[Cloud/GCP] gcpdiag

안녕하세요

gcpdiag라는게 새로 나와서 좀 테스트를 해 봤는데요.

결론부터 얘기하자면…뭐랄까…aquasecurity/kube-bench 도구 같은 느낌이네요?

그리고 좀 단점부터 보이는게…컨테이너 베이스로 동작을 시키다 보니…

linux 커널이 필요한거 같다? 이고…근데 왜 이걸 오픈소스화 했을까…?

어짜피 GCP에서만 쓸꺼 같은 이름인데…

어쨌든 GCP를 쓰고, 내부적으로 audit을 전체적으로 하고 싶다 하면 써볼만 할꺼 같기도 합니다.

실행 내용과 결과는 다음과 같습니다.
[gcpdiag run]

[hj@cs-491314827780-default ~ (☸️ |hj-gke:default)]$ gcpdiag
Unable to find image 'us-docker.pkg.dev/gcpdiag-dist/release/gcpdiag:0.55' locally
0.55: Pulling from gcpdiag-dist/release/gcpdiag
1fe172e4850f: Pull complete
caf521ccaac6: Pull complete
3ead6fa29328: Pull complete
5c2a1cbceb83: Pull complete
a8d5f1318db7: Pull complete
358ca086baa3: Pull complete
30197a52639a: Pull complete
65545a04dace: Pull complete
80cf14a4e373: Pull complete
00f344331f1a: Pull complete
5fbfd71d5b4d: Pull complete
Digest: sha256:0b5fcc0fd3e2f1b822cec492b0128f7e1df5173c19990570ee072c80cf6164c4
Status: Downloaded newer image for us-docker.pkg.dev/gcpdiag-dist/release/gcpdiag:0.55
gcpdiag 🩺 - Diagnostics for Google Cloud Platform

Usage:
        gcpdiag COMMAND [OPTIONS]

Commands:
        help     Print this help text.
        lint     Run diagnostics on GCP projects.
        version  Print gcpdiag version.

See: gcpdiag COMMAND --help for command-specific usage.

실행 결과 실패한 내용을 rough하게 살펴보면…

1. serial port가 enable 안되어 있어서 실패…(👎️)
2. date ops가 설치 안되서 실패…? (👎️)
3. uniform access(단일 접근) 아니라서 실패…(👍️)
4. GKE가 리저널이 아니라서 실패…(👌)
5. GKE가 기본 서비스 어카운트를 써서 실패(👍️)

뭐 이정도면…한번쯤 cloud shell에서 돌려봐도 무방할꺼 같기도 하네요.

[gcpdiag --project]

[hj@cs-491314827780-default ~ (☸️ |hj-gke:default)]$ gcpdiag lint --project=hj-int-20200908
gcpdiag

Starting lint inspection (project: hj-int-20200908)...

🔎  gce/BP/2021_001: Serial port logging is enabled.
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [FAIL]
   - hj-int-20200908/windows                                              [FAIL]
   - hj-int-20200908/zipkin-vm                                            [FAIL]

   Serial port output can be often useful for troubleshooting, and enabling
   serial logging makes sure that you don't lose the information when the VM is
   restarted. Additionally, serial port logs are timestamped, which is useful to
   determine when a particular serial output line was printed.

   https://gcpdiag.dev/rules/gce/BP/2021_001

🔎  gce/BP/2021_002: GCE nodes have an up to date ops agent installed.
   - hj-int-20200908/gui-pipe                                             [FAIL] not installed
   - hj-int-20200908/windows                                              [FAIL] not installed
   - hj-int-20200908/zipkin-vm                                            [FAIL] not installed

   Verify that the ops agent is used by the GCE instances and that the agent is
   recent enough. If the monitoring agent is found it is recommended to upgrade
   to the ops agent.  see:
   https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent

   https://gcpdiag.dev/rules/gce/BP/2021_002

🔎  gce/ERR/2021_003: Google APIs service agent has the Editor role.
   - hj-int-20200908                                                      [ OK ]

🔎  gce/ERR/2021_004: Serial logs don't contain Secure Boot error messages
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/ERR/2021_005: Serial logs don't contain mount error messages
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_001: GCE instance service account permissions for logging.
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_003: GCE instance service account permissions for monitoring.
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_004: Serial logs don't contain disk full messages
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_005: Serial logs don't contain out-of-memory messages
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_006: Serial logs don't contain "Kernel panic" messages
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2021_007: Serial logs don't contain "BSOD" messages
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2022_001: GCE connectivity: IAP service can connect to SSH/RDP port on instances.
   - hj-int-20200908/gke-gke-nodes-634bd260-gefn                          [ OK ]
   - hj-int-20200908/gui-pipe                                             [ OK ]
   - hj-int-20200908/windows                                              [ OK ]
   - hj-int-20200908/zipkin-vm                                            [ OK ]

🔎  gce/WARN/2022_002: Instance groups named ports are using unique names.
   - hj-int-20200908/gke-gke-nodes-634bd260-grp                           [ OK ]

🔎  gce/WARN/2022_003: GCE VM instances quota is not near the limit.
   - projects/hj-int-20200908/regions/asia-northeast3                     [ OK ]
   - projects/hj-int-20200908/regions/us-central1                         [ OK ]

🔎  gcs/BP/2022_001: Buckets are using uniform access
   - hj-int-20200908/artifacts.hj-int-20200908.appspot.com                [FAIL]
     it is recommend to use uniform access on your bucket
   - hj-int-20200908/hoon                                                 [FAIL]
     it is recommend to use uniform access on your bucket

   Google recommends using uniform access for a Cloud Storage bucket IAM policy
   https://cloud.google.com/storage/docs/access-
   control#choose_between_uniform_and_fine-grained_access

   https://gcpdiag.dev/rules/gcs/BP/2022_001

🔎  gke/BP/2021_001: GKE system logging and monitoring enabled.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/BP/2022_001: GKE clusters are regional.
   - hj-int-20200908/us-central1-c/gke                                    [FAIL]
      is not regional

   The availability of regional clusters (both control plane and nodes) is
   higher for regional clusters as they are replicated across zones in the
   region. It is recommended to use regional clusters for the production
   workload.

   https://gcpdiag.dev/rules/gke/BP/2022_001

🔎  gke/BP/2022_002: GKE clusters are using unique subnets.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_001: GKE nodes service account permissions for logging.
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/ERR/2021_002: GKE nodes service account permissions for monitoring.
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/ERR/2021_004: GKE nodes aren't reporting connection issues to apiserver.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_005: GKE nodes aren't reporting connection issues to storage.google.com.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_006: GKE Autoscaler isn't reporting scaleup failures.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_007: GKE service account permissions.
   - hj-int-20200908                                                      [ OK ]

🔎  gke/ERR/2021_008: Google APIs service agent has Editor role.
   - hj-int-20200908                                                      [ OK ]

🔎  gke/ERR/2021_009: Version skew between cluster and node pool.
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/ERR/2021_010: Check internal peering forwarding limits which affect GKE.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_011: ip-masq-agent not reporting errors
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_012: Node pool service account exists and not is disabled.
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/ERR/2021_013: GKE cluster firewall rules are configured.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2021_015: GKE connectivity: node to pod communication.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2022_001: GKE connectivity: pod to pod communication.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/ERR/2022_002: GKE nodes of private clusters can access Google APIs and services.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/SEC/2021_001: GKE nodes don't use the GCE default service account.
   - hj-int-20200908/us-central1-c/gke/nodes                              [FAIL]
     node pool uses the GCE default service account

   The GCE default service account has more permissions than are required to run
   your Kubernetes Engine cluster. You should either use GKE Workload Identity
   or create and use a minimally privileged service account.

   https://gcpdiag.dev/rules/gke/SEC/2021_001

🔎  gke/WARN/2021_003: GKE cluster size close to maximum allowed by pod range
   - hj-int-20200908/us-central1-c/gke                                    [ OK ] 1/1024 nodes used.

🔎  gke/WARN/2021_004: GKE system workloads are running stable.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2021_005: GKE nodes have good disk performance.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2021_006: GKE nodes aren't reporting conntrack issues.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2021_007: GKE nodes have enough free space on the boot disk.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2021_009: GKE nodes use a containerd image.
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/WARN/2022_001: GKE clusters with workload identity are regional.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2022_002: GKE metadata concealment is not in use
   - hj-int-20200908/us-central1-c/gke/nodes                              [ OK ]

🔎  gke/WARN/2022_003: GKE service account permissions to manage project VPC firewall rules.
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  gke/WARN/2022_004: Cloud Logging API enabled when GKE logging is enabled
   - hj-int-20200908/us-central1-c/gke                                    [ OK ]

🔎  iam/SEC/2021_001: No service accounts have the Owner role
   - hj-int-20200908                                                      [ OK ]

Rules summary: 24 skipped, 40 ok, 5 failed

[Continue reading...]

[클라우드] 3사 Cloud Shell 비교 for 약간 쿠버네티스

안녕하세요 

쿠버 만지는 조훈 입니다. 

요즘 GCP console로 쿠버를 좀 만지다 보니 (안토스 포함), 생각보다 console에 재미있는게 많이 들어가 있더라고요? 안 그래도 이런 저런 쿠버 툴(i.e. kubetail)등을 보고 있었는데...그래서 

신기해서 좀 정리합니다. 인스턴스에 따라서 /usr/bin에 있는 것들도 꽤 되서...좀 애매한거 같기도 하긴 하지만...나중에 기회가 되면 세션으로 해도 될 것 같아요. 

!중요참고! 이것은 개인 의견으로 ..어쩌구 저쩌구...

0. 이름 

GCP: Cloud Shell

Azure: Cloud Shell

AWS: Cloud9 (for Lambda)

1. 기본 설치 숫자 (/usr/local/bin을 기준) 

GCP  | Azure | AWS
107    |   67    |  47    (어? 다 7로 떨어짐!!) 

2. 설치 패키지 

2.1. GCP 

1 apt-get

2 autopep8

3 bundle

4 bundler

5 chardetect

6 cloudshell

7 console

8 django-admin

9 django-admin.py

10 dmypy

11 docker-compose

12 docker-credential-gcr

13 docker-machine

14 easy_install

15 easy_install-2.7

16 easy_install-3.7

17 epylint

18 erb

19 f2py

20 f2py2

21 f2py2.7

22 f2py3

23 f2py3.7

24 fixup_language_v1beta2_keywords.py

25 fixup_language_v1_keywords.py

26 fixup_translate_v3beta1_keywords.py

27 fixup_translate_v3_keywords.py

28 fixup_vision_v1_keywords.py

29 fixup_vision_v1p1beta1_keywords.py

30 fixup_vision_v1p2beta1_keywords.py

31 fixup_vision_v1p3beta1_keywords.py

32 fixup_vision_v1p4beta1_keywords.py

33 flake8

34 flask

35 freeze_graph

36 futurize

37 gem

38 helm

39 iptest

40 iptest2

41 iptest3

42 ipython

43 ipython2

44 ipython3

45 irb

46 isort

47 jsonschema

48 kubectx

49 kubens

50 markdown_py

51 maruku

52 marutex

53 mssql-cli

54 mssql-cli.bat

55 mypy

56 mypyc

57 nokogiri

58 nomos

59 pack

60 pasteurize

61 __pycache__

62 pycodestyle

63 pydocstyle

64 pyflakes

65 pygmentize

66 pylint

67 pyls

68 pyreverse

69 pyrsa-decrypt

70 pyrsa-encrypt

71 pyrsa-keygen

72 pyrsa-priv2pub

73 pyrsa-sign

74 pyrsa-verify

75 python

76 rackup

77 rails

78 rake

79 rdoc

80 reverse_markdown

81 ri

82 rubocop

83 ruby

84 ruby-parse

85 ruby-rewrite

86 saved_model_cli

87 solargraph

88 sprockets

89 sqlformat

90 stubgen

91 stubtest

92 symilar

93 tensorboard

94 terraform

95 tflite_convert

96 tf_upgrade_v2

97 thor

98 tilt

99 toco

100 toco_from_protos

101 virtualenv

102 vmtouch

103 wheel

104 yapf

105 yard

106 yardoc

107 yri

2.2 Azure 

1 activate-global-python-argcomplete

2 ansible

3 ansible-config

4 ansible-connection

5 ansible-console

6 ansible-doc

7 ansible-galaxy

8 ansible-inventory

9 ansible-playbook

10 ansible-pull

11 ansible-vault

12 azcopy

13 blobxfer

14 bolt

15 bundle

16 bundler

17 chardetect

18 dcos

19 docker-machine

20 draft

21 erb

22 futurize

23 gem

24 helm

25 htmldiff

26 irb

27 jp.py

28 jx

29 kubectl

30 kubelogin

31 ldiff

32 m365

33 m365_comp

34 microsoft365

35 mssql-scripter

36 mssql-scripter.bat

37 node

38 nodejs

39 npm

40 npx

41 packer

42 pasteurize

43 pip

44 pip2

45 pip2.7

46 pip3

47 pip3.5

48 portal

49 __pycache__

50 pygmentize

51 pyjwt

52 python-argcomplete-check-easy-install-script

53 python-argcomplete-tcsh

54 rake

55 rdoc

56 register-python-argcomplete

57 ri

58 rspec

59 ruby

60 sfctl

61 shipyard

62 tabulate

63 terraform

64 update_rubygems

65 virtualenv

66 yo

67 yo-complete

2.3 AWS 

1 aws

2 aws_bash_completer

3 aws.cmd

4 aws_completer

5 aws_zsh_completer.sh

6 codeintel

7 django-admin

8 django-admin.py

9 django-admin.pyc

10 easy_install

11 easy_install-2.7

12 epylint

13 git-remote-codecommit

14 iptest

15 iptest3

16 ipython

17 ipython3

18 isort

19 jp.py

20 pbr

21 __pycache__

22 pygmentize

23 pylint

24 pyreverse

25 pyrsa-decrypt

26 pyrsa-encrypt

27 pyrsa-keygen

28 pyrsa-priv2pub

29 pyrsa-sign

30 pyrsa-verify

31 rst2html4.py

32 rst2html5.py

33 rst2html.py

34 rst2latex.py

35 rst2man.py

36 rst2odt_prepstyles.py

37 rst2odt.py

38 rst2pseudoxml.py

39 rst2s5.py

40 rst2xetex.py

41 rst2xml.py

42 rstpep2html.py

43 symilar

44 virtualenv

45 virtualenv-clone

46 virtualenvwrapper_lazy.sh

47 virtualenvwrapper.sh

3. 공통적인 패키지 

GCP -AZure

bundle

bundler

chardetect

docker-machine

erb

futurize

gem

helm

irb

pasteurize

__pycache__

pygmentize

rake

rdoc

ri

ruby

terraform

virtualenv

Azure - AWS

jp.py

__pycache__

pygmentize

virtualenv

AWS - GCP 

django-admin

django-admin.py

easy_install

easy_install-2.7

epylint

iptest

iptest3

ipython

ipython3

isort

__pycache__

pygmentize

pylint

pyreverse

pyrsa-decrypt

pyrsa-encrypt

pyrsa-keygen

pyrsa-priv2pub

pyrsa-sign

pyrsa-verify

symilar

virtualenv

4. 특화패키지 중에 내가 봤을때 유명한거 (또는 쿠버에 유용한거?)

GCP

 - kubectx

 - kubens

 - mysql-cli 

특이한 점: kubectl은 /usr/bin 에 있음 

Azure 

 - ansible 

AWS

 - 애초에 cloud9이 람다 개발용이니 같은 동급 비교가 어려움. 

5. TL; DR 

3사 모두 파이썬은 모두 가지고 있음 pip3이나 그런 것도 모두 됨. 하물며 루비도 됨 신기함 

그리고 git도 있고 왠만한건 다 자체 개발 머신으로 사용하는데 무리는 없어 보임 

그런데 쿠버네티스을 바라보는 관점은 조금 차이가 있어 보임. 

GCP에 context가 많아질 경우 유용한 kubectx나 kubens 가 기본 내장인 것은 매우 의외임. 

그거 이외에도 pack이나 thor나 신기한게 많이 포함됨. 

그리고 중요한 kubectl bash_completion의 경우는 GCP에는 기본 설정은 되어 있음 

(물론...이해는 안되는데 alias k=kubectl ; complete -F __start_kubectl k 이게 안 들어가 있음 kubectl 다 치라는건가 -_-) 

Azure의 경우 다른건 불편한 편인데 tmux! 컭님이 사랑하는 tmux가 기본 설치되어 있음 

신기한 일임...

AWS 경우 kubectl도 물론 없고 지인에게 물어봤는데 그런 관점이 아닌듯 하다는 분위기?

그래서 클라우드 네이티브 콘솔을 만들때 GCP Cloud Console 참고로 포팅하는 것도 매우 좋을 것 같음. 

끝!

[Continue reading...]

[클라우드] 클라우드 대시 보드 정리 (5사)

안녕하세요 

필요하실때 보기 위해서 

3사의 상태를 체크할 수 있는 대시보드를 정리합니다. 

1. AWS

- Feed : https://status.aws.amazon.com/rss/all.rss 

- Link: https://status.aws.amazon.com/

2. Alibaba cloud: 

- Link: https://cloudharmony.com/status-for-alibaba

- 공식을 못찾음

3. Tencent cloud 

- Link: https://cloudharmony.com/status-for-tencent 

- 공식을 못 찾음  

4. Azure

 - Feed: https://azurestatuscdn.azureedge.net/ko-kr/status/feed/

 - Link: https://status.azure.com/ko-kr/status

5. GCP

 - Feed: https://status.cloud.google.com/feed.atom

 - Link: https://status.cloud.google.com/

[Continue reading...]