好记性不如烂笔头，记录学习点滴！https://www.kops.cc。技术交流加微 haxicc
Question 1Task weight: 1%You have access to multiple clusters from your main terminal through kubectl contexts. Write all context names into /opt/course/1/contexts, one per line.From the kubeconfig extract the certificate of user restricted@infra-prod and write it decoded to /opt/course/1/cert.题目解析：考点kubectl解题➜ kubectl config get-contexts -o name > /opt/course/1/contexts # 从 .kube/config 文件中找到 - name: restricted@infra-prod user client-certificate-data: LS0tLS1CRUdJ... ➜ echo LS0tLS1CRUdJ... | base64 -d > /opt/course/1/certQuestion 2Task weight: 4%Use context: kubectl config use-context workload-prodFalco is installed with default configuration on node cluster1-worker1. Connect using ssh cluster1-worker1. Use it to:Find a Pod running image nginx which creates unwanted package management processes inside its container.Find a Pod running image httpd which modifies /etc/passwd.Save the Falco logs for case 1 under /opt/course/2/falco.log in format time,container-id,container-name,user-name. No other information should be in any line. Collect the logs for at least 30 seconds.Afterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.题目解析：考点falco解题通过 falco 日志查找容器 ID, PODFind a Pod running image nginx which creates unwanted package management processes inside its container➜ ssh cluster1-worker1 ➜ root@cluster1-worker1:~# grep nginx /var/log/syslog | grep -i "package management process" Dec 13 01:03:23 cluster1-worker1 falco[28640]: 01:03:23.564163340: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=3ed4079e7f61 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine) ... ➜ root@cluster1-worker1:~# crictl ps -id 3ed4079e7f61 CONTAINER ID IMAGE NAME ... POD ID 3ed4079e7f61 6f715d38cfe0e nginx ... 7a864406b9794 ➜ root@cluster1-worker1:~# crictl pods -id 7a864406b9794 POD ID ... NAME NAMESPACE ... 7a864406b9794 ... webapi-6cfddcd6f4-ftxg4 team-blue ... # 可得这个容器属于 deployment team-blue/webapiFind a Pod running image httpd which modifies /etc/passwd➜ root@cluster1-worker1:~# grep httpd /var/log/syslog | grep -i "/etc/passwd" Dec 13 06:13:20 cluster1-worker1 falco: 06:13:20.562402988: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sed -i $d /etc/passwd parent=sh pcmdline=sh -c echo hacker >> /etc/passwd; sed -i '$d' /etc/passwd; true file=/etc/passwdIKpDGh program=sed gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=5c781a912497 image=docker.io/library/httpd) ➜ root@cluster1-worker1:~# crictl ps -id 5c781a912497 CONTAINER ID IMAGE NAME ... POD ID 5c781a912497 f6b40f9f8ad71 httpd ... 595af943c3245 ➜ root@cluster1-worker1:~# crictl pods -id 595af943c3245 POD ID ... NAME NAMESPACE ... 595af943c3245 ... rating-service-68cbdf7b7-v2p6g team-purple ... # 可得这个容器属于 deployment team-purple/rating-serviceSave the Falco logs for case 1 under /opt/course/2/falco.log in format time,container-id,container-name,user-name...通过上面我们查看到的信息，打出的日志格式是这样的：(user=root user_loginuid=-1 command=apk container_id=3ed4079e7f61 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)而题目要求格式是 time,container-id,container-name,user-name，因此我们需要修改 falco 的配置文件➜ root@cluster1-worker1:~# vim /etc/falco/falco_rules.yaml ... - rule: Launch Package Management Process in Container desc: Package management process ran inside container condition: > spawned_process and container and user.name != "_apt" and package_mgmt_procs and not package_mgmt_ancestor_procs and not user_known_package_manager_in_container output: > # Package management process launched in container (user=%user.name user_loginuid=%user.loginuid # command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag) Package management process launched in container %evt.time,%container.id,%container.name,%user.name # 删除上两行，新增这行 priority: ERROR tags: [process, mitre_persistence] ...重启 falco 服务，查看日志输出➜ root@cluster1-worker1:~# systemctl restart falco ➜ root@cluster1-worker1:~# tail -f /var/log/syslog | grep "Package management process launched in container" Dec 13 06:07:57 cluster1-worker1 falco[220990]: 06:07:57.756206761: Error Package management process launched in container 06:07:57.756206761,3ed4079e7f61,nginx,root Dec 13 06:07:57 cluster1-worker1 falco: 06:07:57.756206761: Error Package management process launched in container 06:07:57.756206761,3ed4079e7f61,nginx,root Dec 13 06:08:02 cluster1-worker1 falco[220990]: 06:08:02.754047763: Error Package management process launched in container 06:08:02.754047763,3ed4079e7f61,nginx,root Dec 13 06:08:02 cluster1-worker1 falco: 06:08:02.754047763: Error Package management process launched in container 06:08:02.754047763,3ed4079e7f61,nginx,root Dec 13 06:08:07 cluster1-worker1 falco[220990]: 06:08:07.755037087: Error Package management process launched in container 06:08:07.755037087,3ed4079e7f61,nginx,root Dec 13 06:08:07 cluster1-worker1 falco: 06:08:07.755037087: Error Package management process launched in container 06:08:07.755037087,3ed4079e7f61,nginx,root Dec 13 06:08:12 cluster1-worker1 falco[220990]: 06:08:12.751331220: Error Package management process launched in container 06:08:12.751331220,3ed4079e7f61,nginx,root Dec 13 06:08:12 cluster1-worker1 falco: 06:08:12.751331220: Error Package management process launched in container 06:08:12.751331220,3ed4079e7f61,nginx,root Dec 13 06:08:17 cluster1-worker1 falco[220990]: 06:08:17.763051479: Error Package management process launched in container 06:08:17.763051479,3ed4079e7f61,nginx,root Dec 13 06:08:17 cluster1-worker1 falco: 06:08:17.763051479: Error Package management process launched in container 06:08:17.763051479,3ed4079e7f61,nginx,root Dec 13 06:08:22 cluster1-worker1 falco[220990]: 06:08:22.744677645: Error Package management process launched in container 06:08:22.744677645,3ed4079e7f61,nginx,root将结果写入 /opt/course/2/falco.log06:07:57.756206761,3ed4079e7f61,nginx,root 06:08:02.754047763,3ed4079e7f61,nginx,root 06:08:07.755037087,3ed4079e7f61,nginx,root 06:08:12.751331220,3ed4079e7f61,nginx,root 06:08:17.763051479,3ed4079e7f61,nginx,root 06:08:22.744677645,3ed4079e7f61,nginx,rootAfterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0通过上面查到的 Pod 归属，将副本数改为 0➜ kubectl -n team-blue scale deployment webapi --replicas=0 ➜ kubectl -n team-purple scale deployment rating-service --replicas=0Question 3Task weight: 3%Use context: kubectl config use-context workload-prodYou received a list from the DevSecOps team which performed a security investigation of the k8s cluster1 (workload-prod). The list states the following about the apiserver setup:Accessible through a NodePort ServiceChange the apiserver setup so that:Only accessible through a ClusterIP Service题目解析：考点service解题删除 apiserver 启动参数➜ ssh cluster1-master1 ➜ root@cluster1-master1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml \ - --kubernetes-service-node-port=31000 # 删除修改 kubernetes service，将 NodePort 类型改为 ClusterIP➜ kubectl edit svc kubernetes ... ports: - name: https port: 443 protocol: TCP targetPort: 6443 # nodePort: 31000 # 删除 sessionAffinity: None type: ClusterIP # 修改 ...Question 4Task weight: 8%Use context: kubectl config use-context workload-prodThere is Deployment container-host-hacker in Namespace team-red which mounts /run/containerd as a hostPath volume on the Node where its running. This means that the Pod can access various data about other containers running on the same Node.You're asked to forbid this behavior by:Enabling Admission Plugin PodSecurityPolicy in the apiserverCreating a PodSecurityPolicy named psp-mount which allows hostPath volumes only for directory /tmpCreating a ClusterRole named psp-mount which allows to use the new PSPCreating a RoleBinding named psp-mount in Namespace team-red which binds the new ClusterRole to all ServiceAccounts in the Namespace team-redRestart the Pod of Deployment container-host-hacker afterwards to verify new creation is prevented.NOTE: PSPs can affect the whole cluster. Should you encounter issues you can always disable the Admission Plugin again.题目解析：考点PodSecurityPolicy Pod 安全策略 | Kubernetes解题启用准入插件➜ ssh cluster1-master1 ➜ root@cluster1-master1:~# ssh cluster1-master1 ➜ root@cluster1-master1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy # 修改，在末尾加入,PodSecurityPolicy创建 PodSecurityPolicy，从官网示例拷过来修改apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp-mount namespace: team-red spec: privileged: false # Don't allow privileged pods! # The rest fills in some required fields. seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny allowedHostPaths: - pathPrefix: /tmp volumes: - 'hostPath'创建 ClusterRole，从官网示例拷过来修改apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: psp-mount rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - team-red:psp-mount创建 RoleBinding➜ kubectl -n team-red create rolebinding psp-mount --clusterrole=psp-mount --group=system:serviceaccountsQuestion 5Task weight: 3%Use context: kubectl config use-context infra-prodYou're ask to evaluate specific settings of cluster2 against the CIS Benchmark recommendations. Use the tool kube-bench which is already installed on the nodes.Connect using ssh cluster2-master1 and ssh cluster2-worker1.On the master node ensure (correct if necessary) that the CIS recommendations are set for:The --profiling argument of the kube-controller-managerThe ownership of directory /var/lib/etcdOn the worker node ensure (correct if necessary) that the CIS recommendations are set for:The permissions of the kubelet configuration /var/lib/kubelet/config.yamlThe --client-ca-file argument of the kubelet题目解析：考点kube-bench解题使用 kube-bench 检查 cluster2-master1➜ ssh cluster2-master1 ➜ root@cluster2-master1:~# kube-bench master ... == Summary == 41 checks PASS 13 checks FAIL 11 checks WARN 0 checks INFO解：The --profiling argument of the kube-controller-manager# 从 kube-bench master 命令的输出中，找 kube-controller-manager --profiling 有关的信息 # 可以看到有 1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the below parameter. --profiling=false # 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml ... - --service-cluster-ip-range=10.96.0.0/12 - --use-service-account-credentials=true - --profiling=false # 新增参数 ...解：The ownership of directory /var/lib/etcd# 从 kube-bench master 命令的输出中，找 etcd 有关的信息 # 可以看到有 1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, chown etcd:etcd /var/lib/etcd # 执行命令修改 etcd 数据目录属主属组 ➜ root@cluster2-master1:~# chown etcd:etcd /var/lib/etcd使用 kube-bench 检查 cluster2-worker1➜ ssh cluster2-worker1 ➜ root@cluster2-worker1:~# kube-bench node ... == Summary == 13 checks PASS 10 checks FAIL 2 checks WARN 0 checks INFO解：The permissions of the kubelet configuration /var/lib/kubelet/config.yaml# 从 kube-bench node 命令中找 kubelet 配置文件相关的信息 # 可以找到 2.2.10 Run the following command (using the config file location identified in the Audit step) chmod 644 /var/lib/kubelet/config.yaml # 执行命令修改文件权限 ➜ root@cluster2-worker1:~# chmod 644 /var/lib/kubelet/config.yaml解：The --client-ca-file argument of the kubelet# 从 kube-bench node 命令中找 client-ca-file 相关的信息 # 可以找到 [PASS] 2.1.4 Ensure that the --client-ca-file argument is set as appropriate (Scored) 2.2.7 Run the following command to modify the file permissions of the --client-ca-file 2.2.8 Run the following command to modify the ownership of the --client-ca-file . # 检查通过的，不用处理Question 6Task weight: 2%(can be solved in any kubectl context)There are four Kubernetes server binaries located at /opt/course/6/binaries. You're provided with the following verified sha512 values for these:kube-apiserver f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032ckube-controller-manager 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60kube-proxy 52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6kubelet 4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26cDelete those binaries that don't match with the sha512 values above.题目解析：考点sha512命令对比校验码解题这题好解，通过对比 sha512 的值，发现值不同，则删除二进制程序➜ sha512sum /opt/course/6/binaries/kube-apiserver | grep f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c ➜ sha512sum /opt/course/6/binaries/kube-controller-manager | grep 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 ➜ sha512sum /opt/course/6/binaries/kube-proxy | grep 52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6 ➜ sha512sum /opt/course/6/binaries/kubelet | grep 4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26c没有输出的说明校验码不匹配，删除文件➜ rm /opt/course/6/binaries/kube-controller-manager /opt/course/6/binaries/kubeletQuestion 7Task weight: 6%Use context: kubectl config use-context infra-prodThe Open Policy Agent and Gatekeeper have been installed to, among other things, enforce blacklisting of certain image registries. Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com.Test it by creating a single Pod using image very-bad-registry.com/image in Namespace default, it shouldn't work.You can also verify your changes by looking at the existing Deployment untrusted in Namespace default, it uses an image from the new untrusted source. The OPA contraint should throw violation messages for this one.题目解析：考点OPA OPA Gatekeeper：Kubernetes 的策略和管理 | Kubernetes解题查看 OPA➜ kubectl get constraints NAME AGE requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 87d NAME AGE blacklistimages.constraints.gatekeeper.sh/pod-trusted-images 87d ➜ kubectl get constrainttemplates NAME AGE blacklistimages 87d requiredlabels 87d解：Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com➜ kubectl edit constrainttemplates blacklistimages ... images { image := input.review.object.spec.containers[_].image not startswith(image, "docker-fake.io/") not startswith(image, "google-gcr-fake.com/") not startswith(image, "very-bad-registry.com/") # 新增 } ...验证➜ kubectl run opa-test --image=very-bad-registry.com/image Error from server ([pod-trusted-images] not trusted image!): admission webhook "validation.gatekeeper.sh" denied the request: [pod-trusted-images] not trusted image! ➜ kubectl describe blacklistimages pod-trusted-images ... Total Violations: 1 Violations: Enforcement Action: deny Kind: Pod Message: not trusted image! Name: untrusted-68c4944d48-d2mzc Namespace: default Events: <none>Question 8Task weight: 3%Use context: kubectl config use-context workload-prodThe Kubernetes Dashboard is installed in Namespace kubernetes-dashboard and is configured to:Allow users to "skip login"Allow insecure access (HTTP without authentication)Allow basic authenticationAllow access from outside the clusterYou are asked to make it more secure by:Deny users to "skip login"Deny insecure access, enforce HTTPS (self signed certificates are ok for now)Add the --auto-generate-certificates argumentEnforce authentication using a token (with possibility to use RBAC)Allow only cluster internal access题目解析：解题修改 kubernetes-dashboard 启动参数➜ kubectl -n kubernetes-dashboard edit deploy kubernetes-dashboard ... template: spec: containers: - args: - --namespace=kubernetes-dashboard - --authentication-mode=token # 修改 - --auto-generate-certificates # 新增 #- --enable-skip-login=true # 删除 #- --enable-insecure-login # 删除 image: kubernetesui/dashboard:v2.0.3 imagePullPolicy: Always name: kubernetes-dashboard修改 service➜ kubectl -n kubernetes-dashboard edit svc ... spec: clusterIP: 10.107.176.19 externalTrafficPolicy: Cluster ports: - name: http nodePort: 32513 # 删除 port: 9090 protocol: TCP targetPort: 9090 - name: https nodePort: 32441 # 删除 port: 443 protocol: TCP targetPort: 8443 selector: k8s-app: kubernetes-dashboard sessionAffinity: None type: ClusterIP # 修改 status: loadBalancer: {}Question 9Task weight: 3%Use context: kubectl config use-context workload-prodSome containers need to run more secure and restricted. There is an existing AppArmor profile located at /opt/course/9/profile for this.Install the AppArmor profile on Node cluster1-worker1. Connect using ssh cluster1-worker1.Add label security=apparmor to the NodeCreate a Deployment named apparmor in Namespace default with:One replica of image nginx:1.19.2NodeSelector for security=apparmorSingle container named c1 with the AppArmor profile enabledThe Pod might not run properly with the profile enabled. Write the logs of the Pod into /opt/course/9/logs so another team can work on getting the application running.题目解析：考点Apparmor 使用 AppArmor 限制容器对资源的访问 | Kubernetes解题Install the AppArmor profile on Node cluster1-worker1. Connect using ssh cluster1-worker1➜ scp /opt/course/9/profile cluster1-worker1:/tmp ➜ ssh cluster1-worker1 ➜ root@cluster1-worker1:~# apparmor_parser /tmp/profileAdd label security=apparmor to the Node➜ kubectl label node cluster1-worker1 security=apparmorCreate a Deployment named apparmor in Namespace default with...apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: apparmor name: apparmor spec: replicas: 1 selector: matchLabels: app: apparmor strategy: {} template: metadata: creationTimestamp: null labels: app: apparmor annotations: container.apparmor.security.beta.kubernetes.io/c1: localhost/very-secure spec: containers: - image: nginx:1.19.2 name: c1 resources: {} nodeSelector: security: apparmor status: {}Write the logs of the Pod into /opt/course/9/logs➜ kubectl logs apparmor-85c65645dc-ctwtv > /opt/course/9/logsQuestion 10Task weight: 4%Use context: kubectl config use-context workload-prodTeam purple wants to run some of their workloads more secure. Worker node cluster1-worker2 has container engine containerd already installed and its configured to support the runsc/gvisor runtime.Create a RuntimeClass named gvisor with handler runsc.Create a Pod that uses the RuntimeClass. The Pod should be in Namespace team-purple, named gvisor-test and of image nginx:1.19.2. Make sure the Pod runs on cluster1-worker2.Write the dmesg output of the successfully started Pod into /opt/course/10/gvisor-test-dmesg.题目解析：考点RuntimeClass 容器运行时类（Runtime Class） | Kubernetes解题创建 RuntimeClass，在官网实例拷过来修改apiVersion: node.k8s.io/v1 # RuntimeClass 定义于 node.k8s.io API 组 kind: RuntimeClass metadata: name: gvisor # 用来引用 RuntimeClass 的名字 # RuntimeClass 是一个集群层面的资源 handler: runsc # 对应的 CRI 配置的名称创建Pod指定runtimeClassName 为 gvisor指定到 cluster1-worker2 上运行apiVersion: v1 kind: Pod metadata: name: gvisor-test namespace: team-purple spec: runtimeClassName: gvisor # 新增 nodeName: cluster1-worker2 # 新增 containers: - name: gvisor-test image: nginx:1.19.2在 Pod 里运行 dmesg 写入 /opt/course/10/gvisor-test-dmesg➜ kubectl -n team-purple exec gvisor-test > /opt/course/10/gvisor-test-dmesg -- dmesgQuestion 11Task weight: 7%Use context: kubectl config use-context workload-prodThere is an existing Secret called database-access in Namespace team-green.Read the complete Secret content directly from ETCD (using etcdctl) and store it into /opt/course/11/etcd-secret-content. Write the plain and decoded Secret's value of key "pass" into /opt/course/11/database-password.题目解析：考点etcd, secret 静态加密 Secret 数据 | Kubernetes解题Read the complete Secret content directly from ETCD (using etcdctl) ...➜ ssh cluster1-master1 ➜ root@cluster1-master1:~# ETCDCTL_API=3 etcdctl \ --cert /etc/kubernetes/pki/apiserver-etcd-client.crt \ --key /etc/kubernetes/pki/apiserver-etcd-client.key \ --cacert /etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/team-green/database-access > etcd-secret-content ➜ scp cluster1-master1:/root/etcd-secret-content /opt/course/11/etcd-secret-contentWrite the plain and decoded Secret's value of key "pass" into /opt/course/11/database-password➜ echo Y29uZmlkZW50aWFs | base64 -d > /opt/course/11/database-passwordQuestion 12Task weight: 8%Use context: kubectl config use-context restricted@infra-prodYou're asked to investigate a possible permission escape in Namespace restricted. The context authenticates as user restricted which has only limited permissions and shouldn't be able to read Secret values.Try to find the password-key values of the Secrets secret1, secret2 and secret3 in Namespace restricted. Write the decoded plaintext values into files /opt/course/12/secret1, /opt/course/12/secret2 and /opt/course/12/secret3.题目解析：考点用户认证 | Kubernetes, API 概述 | Kubernetes解题尝试➜ kubectl -n restricted get secrets Error from server (Forbidden): secrets is forbidden: User "restricted" cannot list resource "secrets" in API group "" in the namespace "restricted" ➜ kubectl -n restricted get secrets -o yaml apiVersion: v1 items: [] kind: List metadata: resourceVersion: "" selfLink: "" Error from server (Forbidden): secrets is forbidden: User "restricted" cannot list resource "secrets" in API group "" in the namespace "restricted" ➜ kubectl -n restricted get all NAME READY STATUS RESTARTS AGE pod1-6dcc97847d-jfrxj 1/1 Running 0 88d pod2-5bb9f58f9-zxnrj 1/1 Running 0 88d pod3-59967cd4c7-pxmsk 1/1 Running 0 88d Error from server (Forbidden): replicationcontrollers is forbidden: User "restricted" cannot list resource "replicationcontrollers" in API group "" in the namespace "restricted" Error from server (Forbidden): services is forbidden: User "restricted" cannot list resource "services" in API group "" in the namespace "restricted" Error from server (Forbidden): daemonsets.apps is forbidden: User "restricted" cannot list resource "daemonsets" in API group "apps" in the namespace "restricted" Error from server (Forbidden): deployments.apps is forbidden: User "restricted" cannot list resource "deployments" in API group "apps" in the namespace "restricted" Error from server (Forbidden): replicasets.apps is forbidden: User "restricted" cannot list resource "replicasets" in API group "apps" in the namespace "restricted" Error from server (Forbidden): statefulsets.apps is forbidden: User "restricted" cannot list resource "statefulsets" in API group "apps" in the namespace "restricted" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "restricted" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "restricted" Error from server (Forbidden): cronjobs.batch is forbidden: User "restricted" cannot list resource "cronjobs" in API group "batch" in the namespace "restricted" Error from server (Forbidden): jobs.batch is forbidden: User "restricted" cannot list resource "jobs" in API group "batch" in the namespace "restricted" # 看来是没有权限直接查看，但是有查看 Pod 的权限，尝试通过 Pod 查找漏洞通过 Pod 查找漏洞➜ kubectl -n restricted describe pod pod1-6dcc97847d-jfrxj ... Environment: <none> Mounts: /etc/secret-volume from secret-volume (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: secret-volume: Type: Secret (a volume populated by a Secret) SecretName: secret1 Optional: false ... # 咦，看到 pod1 挂载了secret卷 ➜ k8s@terminal:~$ kubectl -n restricted exec -it pod1-6dcc97847d-jfrxj -- /bin/sh ➜ / # cat /etc/secret-volume/password you-are # 这个就是 secret1 的明文数据了 # 继续看 pod2 ➜ kubectl -n restricted describe pod pod2-5bb9f58f9-zxnrj ... Environment: PASSWORD: <set to the key 'password' in secret 'secret2'> Optional: false Mounts: <none> ... # 看到有一个环境变量PASSWORD，引用的是 secret2 的 password ➜ kubectl -n restricted exec -it pod2-5bb9f58f9-zxnrj -- /bin/sh ➜ / # echo $PASSWORD an-amazing # 这个就是 secret2 的明文数据了 # 继续看 pod3 ➜ kubectl -n restricted describe pod pod3-59967cd4c7-pxmsk ... Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7ggd9 (ro) ... Volumes: kube-api-access-7ggd9: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ... # 这个 pod 挂了 ca 证书，那么可以通过这个 ca 证书访问 apiserver 获取 secret ➜ kubectl -n restricted exec -it pod3-59967cd4c7-pxmsk -- /bin/sh ➜ / # ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token # 有 ca 证书和 token，那就好办了 ➜ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) ➜ / # curl -k https://kubernetes/api/v1/namespaces/restricted/secrets -H "Authorization: Bearer $TOKEN" { "kind": "SecretList", "apiVersion": "v1", "metadata": { "resourceVersion": "45186" }, "items": [ ... { "metadata": { "name": "secret1", "namespace": "restricted", ... }, "data": { "password": "eW91LWFyZQo=" }, "type": "Opaque" }, { "metadata": { "name": "secret2", "namespace": "restricted", ... }, "data": { "password": "YW4tYW1hemluZwo=" }, "type": "Opaque" }, { "metadata": { "name": "secret3", "namespace": "restricted", ... }, "data": { "password": "cEVuRXRSYVRpT24tdEVzVGVSCg==" }, "type": "Opaque" } ] } # 一锅端，全出来了 echo -n cEVuRXRSYVRpT24tdEVzVGVSCg== | base64 -d # 这个就是 secret3 的明文数据了Question 13Task weight: 7%Use context: kubectl config use-context infra-prodThere is a metadata service available at http://192.168.100.21:32000 on which Nodes can reach sensitive data, like cloud credentials for initialisation. By default, all Pods in the cluster also have access to this endpoint. The DevSecOps team has asked you to restrict access to this metadata server.In Namespace metadata-access:Create a NetworkPolicy named metadata-deny which prevents egress to 192.168.100.21 for all Pods but still allows access to everything elseCreate a NetworkPolicy named metadata-allow which allows Pods having label role: metadata-accessor to access endpoint 192.168.100.21There are existing Pods in the target Namespace with which you can test your policies, but don't change their labels.题目解析：考点网络策略 | Kubernetes解题创建 NetworkPolicy，从官网示例中拷过来修改apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: metadata-deny namespace: metadata-access spec: podSelector: matchExpressions: - key: app operator: Exists policyTypes: - Egress egress: - to: - ipBlock: cidr: 0.0.0.0/0 except: - 192.168.100.21/32 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: metadata-allow namespace: metadata-access spec: podSelector: matchLabels: role: metadata-accessor policyTypes: - Egress egress: - to: - ipBlock: cidr: 192.168.100.21/32Question 14Task weight: 4%Use context: kubectl config use-context workload-prodThere are Pods in Namespace team-yellow. A security investigation noticed that some processes running in these Pods are using the Syscall kill, which is forbidden by a Team Yellow internal policy.Find the offending Pod(s) and remove these by reducing the replicas of the parent Deployment to 0.题目解析：考点syscall解题查看有哪些 Pod 在运行➜ kubectl -n team-yellow get all -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/collector1-7585cc58cb-6mzn9 1/1 Running 0 88d 10.44.0.11 cluster1-worker1 <none> <none> pod/collector1-7585cc58cb-9gmsq 1/1 Running 0 88d 10.44.0.13 cluster1-worker1 <none> <none> pod/collector2-8556679d96-72ctk 1/1 Running 0 88d 10.44.0.12 cluster1-worker1 <none> <none> pod/collector3-8b58fdc88-q5ggz 1/1 Running 0 88d 10.44.0.14 cluster1-worker1 <none> <none> pod/collector3-8b58fdc88-qp2b9 1/1 Running 0 88d 10.44.0.15 cluster1-worker1 <none> <none>登录节点，strace 查看系统调用➜ ssh cluster1-worker1 ➜ root@cluster1-worker1:~# ps aux | grep collector root 40367 0.0 0.0 702208 308 ? Ssl 02:13 0:05 ./collector1-process root 40471 0.0 0.0 702208 308 ? Ssl 02:13 0:05 ./collector1-process root 40530 0.0 0.0 702472 888 ? Ssl 02:13 0:05 ./collector3-process root 40694 0.0 0.0 702472 992 ? Ssl 02:13 0:05 ./collector3-process root 40846 0.0 0.0 702216 304 ? Ssl 02:13 0:05 ./collector2-process root 301954 0.0 0.0 9032 672 pts/2 S+ 12:37 0:00 grep --color=auto collector ➜ root@cluster1-worker1:~# strace -p 40367 strace: Process 40367 attached epoll_pwait(3, [], 128, 194, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 kill(666, SIGTERM) = -1 ESRCH (No such process) epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, [], 128, 999, NULL, 1) = 0 epoll_pwait(3, ^Cstrace: Process 40367 detached <detached ...> # 依次使用 strace 命令查看进程的系统调用，这里省略了，最终发现 collector1-process 进程有 kill 系统调用将副本数改为 0➜ kubectl -n team-yellow scale deploy collector1 --replicas=0Question 15Task weight: 4%Use context: kubectl config use-context workload-prodIn Namespace team-pink there is an existing Nginx Ingress resources named secure which accepts two paths /app and /api which point to different ClusterIP Services.From your main terminal you can connect to it using for example:HTTP: curl -v http://secure-ingress.test:31080/appHTTPS: curl -kv https://secure-ingress.test:31443/appRight now it uses a default generated TLS certificate by the Nginx Ingress Controller.You're asked to instead use the key and certificate provided at /opt/course/15/tls.key and /opt/course/15/tls.crt. As it's a self-signed certificate you need to use curl -k when connecting to it.题目解析：考点Ingress | Kubernetes解题查看现有 ingress➜ kubectl -n team-pink get ingress NAME CLASS HOSTS ADDRESS PORTS AGE secure nginx secure-ingress.test 192.168.100.12 80, 443 88d创建 tls 证书 secret➜ kubectl -n team-pink create secret tls secure-ingress --key=/opt/course/15/tls.key --cert=/opt/course/15/tls.crt修改 ingress，使用自定义证书➜ kubectl -n team-pink edit ingress secure apiVersion: networking.k8s.io/v1 kind: Ingress ... spec: ingressClassName: nginx rules: - host: secure-ingress.test http: paths: - backend: service: name: secure-app port: number: 80 path: /app pathType: Prefix - backend: service: name: secure-api port: number: 80 path: /api pathType: Prefix tls: # 新增 - hosts: # 新增 - secure-ingress.test # 新增 secretName: secure-ingress # 新增 status: loadBalancer: ingress: - ip: 192.168.100.12验证➜ curl -kv https://secure-ingress.test:31443/app ... * Server certificate: * subject: CN=secure-ingress.test; O=secure-ingress.test * start date: Sep 25 18:22:10 2020 GMT * expire date: Sep 20 18:22:10 2040 GMT * issuer: CN=secure-ingress.test; O=secure-ingress.test * SSL certificate verify result: self signed certificate (18), continuing anyway. ...Question 16Task weight: 7%Use context: kubectl config use-context workload-prodThere is a Deployment image-verify in Namespace team-blue which runs image registry.killer.sh:5000/image-verify:v1. DevSecOps has asked you to improve this image by:Changing the base image to alpine:3.12Not installing curlUpdating nginx to use the version constraint >=1.18.0Running the main process as user myuserDo not add any new lines to the Dockerfile, just edit existing ones. The file is located at /opt/course/16/image/Dockerfile.Tag your version as v2. You can build, tag and push using:cd /opt/course/16/image podman build -t registry.killer.sh:5000/image-verify:v2 . podman run registry.killer.sh:5000/image-verify:v2 # to test your changes podman push registry.killer.sh:5000/image-verify:v2Make the Deployment use your updated image tag v2.题目解析：考点制作镜像解题修改DockerfileFROM alpine:3.12 # 修改 RUN apk update && apk add vim nginx=1.18.0-r3 # 修改, 去掉curl，nginx去掉版本号，或者起一个alpine容器搜索nginx版本 RUN addgroup -S myuser && adduser -S myuser -G myuser COPY ./run.sh run.sh RUN ["chmod", "+x", "./run.sh"] USER myuser # 修改 ENTRYPOINT ["/bin/sh", "./run.sh"]打镜像podman build -t registry.killer.sh:5000/image-verify:v2 . podman push registry.killer.sh:5000/image-verify:v2更新镜像版本➜ kubectl -n team-blue set image deployment image-verify *=registry.killer.sh:5000/image-verify:v2 # 或者 kubectl -n team-blue edit deployment image-verify 进行修改Question 17Task weight: 7%Use context: kubectl config use-context infra-prodAudit Logging has been enabled in the cluster with an Audit Policy located at /etc/kubernetes/audit/policy.yaml on cluster2-master1.Change the configuration so that only one backup of the logs is stored.Alter the Policy in a way that it only stores logs:From Secret resources, level MetadataFrom "system:nodes" userGroups, level RequestResponseAfter you altered the Policy make sure to empty the log file so it only contains entries according to your changes, like using truncate -s 0 /etc/kubernetes/audit/logs/audit.log.NOTE: You can use jq to render json more readable. cat data.json | jq题目解析：考点审计 | Kubernetes解题修改 apiserver 启动参数➜ ssh cluster2-master1 ➜ root@cluster2-master1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml - --audit-log-maxbackup=1 # 修改修改审计策略➜ root@cluster2-master1:~# vim /etc/kubernetes/audit/policy.yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: # log Secret resources audits, level Metadata - level: Metadata resources: - group: "" resources: ["secrets"] # log node related audits, level RequestResponse - level: RequestResponse userGroups: ["system:nodes"] # for everything else don't log anything - level: None重启 apiserver➜ root@cluster2-master1:~# cd /etc/kubernetes/manifests/ ➜ root@cluster2-master1:/etc/kubernetes/manifests# mv kube-apiserver.yaml .. # 等 apiserver 停止后 ➜ root@cluster2-master1:/etc/kubernetes/manifests# mv ../kube-apiserver.yaml .验证# shows Secret entries cat /etc/kubernetes/audit/logs/audit.log | grep '"resource":"secrets"' | wc -l # confirms Secret entries are only of level Metadata cat /etc/kubernetes/audit/logs/audit.log | grep '"resource":"secrets"' | grep -v '"level":"Metadata"' | wc -l # shows RequestResponse level entries cat /etc/kubernetes/audit/logs/audit.log | grep -v '"level":"RequestResponse"' | wc -l # shows RequestResponse level entries are only for system:nodes cat /etc/kubernetes/audit/logs/audit.log | grep '"level":"RequestResponse"' | grep -v "system:nodes" | wc -lQuestion 18Task weight: 4%Use context: kubectl config use-context infra-prodNamespace security contains five Secrets of type Opaque which can be considered highly confidential. The latest Incident-Prevention-Investigation revealed that ServiceAccount p.auster had too broad access to the cluster for some time. This SA should've never had access to any Secrets in that Namespace.Find out which Secrets in Namespace security this SA did access by looking at the Audit Logs under /opt/course/18/audit.log.Change the password to any new string of only those Secrets that were accessed by this SA.NOTE: You can use jq to render json more readable. cat data.json | jq题目解析：考点审计日志分析解题粗略查看一下审计日志 /opt/course/18/audit.log{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"824df58e-4986-4c60-8eb1-6d1eb89516f0","stage":"RequestReceived","requestURI":"/readyz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["192.168.102.11"],"userAgent":"kube-probe/1.19","requestReceivedTimestamp":"2020-09-24T21:25:56.713207Z","stageTimestamp":"2020-09-24T21:25:56.713207Z"}{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"824df58e-4986-4c60-8eb1-6d1eb89516f0","stage":"ResponseComplete","requestURI":"/readyz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["192.168.102.11"],"userAgent":"kube-probe/1.19","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-09-24T21:25:56.713207Z","stageTimestamp":"2020-09-24T21:25:56.714900Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:public-info-viewer\" of ClusterRole \"system:public-info-viewer\" to Group \"system:unauthenticated\""}}{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"f9e355c9-6221-4214-abc4-ba313bb07df2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s","verb":"get","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["192.168.102.11"],"userAgent":"kube-controller-manager/v1.19.0 (linux/amd64) kubernetes/e199641/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-controller-manager","apiVersion":"v1"},"requestReceivedTimestamp":"2020-09-24T21:25:56.880803Z","stageTimestamp":"2020-09-24T21:25:56.880803Z"} ...可以发现都是 json 格式的日志，而且文件有几千行，因此需要做一定的筛选➜ cat /opt/course/18/audit.log | wc -l # 4448 行 ➜ cat /opt/course/18/audit.log | grep p.auster | wc -l # 28 行，还是有点多，还得继续筛选 ➜ cat /opt/course/18/audit.log | grep p.auster | grep Secret | wc -l # 2 行 ➜ cat /opt/course/18/audit.log | grep p.auster | grep Secret | jq { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "RequestResponse", "auditID": "74fd9e03-abea-4df1-b3d0-9cfeff9ad97a", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/security/secrets/vault-token", "verb": "get", "user": { "username": "system:serviceaccount:security:p.auster", "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c", "groups": [ "system:serviceaccounts", "system:serviceaccounts:security", "system:authenticated" ] }, "sourceIPs": [ "192.168.102.21" ], "userAgent": "curl/7.64.0", "objectRef": { "resource": "secrets", "namespace": "security", "name": "vault-token", "apiVersion": "v1" }, "responseObject": { "kind": "Secret", "apiVersion": "v1", ... "data": { "password": "bERhR0hhcERKcnVuCg==" }, "type": "Opaque" }, ... } { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "RequestResponse", "auditID": "aed6caf9-5af0-4872-8f09-ad55974bb5e0", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/security/secrets/mysql-admin", "verb": "get", "user": { "username": "system:serviceaccount:security:p.auster", "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c", "groups": [ "system:serviceaccounts", "system:serviceaccounts:security", "system:authenticated" ] }, "sourceIPs": [ "192.168.102.21" ], "userAgent": "curl/7.64.0", "objectRef": { "resource": "secrets", "namespace": "security", "name": "mysql-admin", "apiVersion": "v1" } "responseObject": { "kind": "Secret", "apiVersion": "v1", ... "data": { "password": "bWdFVlBSdEpEWHBFCg==" }, "type": "Opaque" }, ... }通过以上过滤查到的审计记录，可以发现请求了 vault-token, mysql-admin 两个 secrets修改 secret➜ echo new-pass | base64 bmV3LXBhc3MK ➜ kubectl -n security edit secret vault-token apiVersion: v1 data: password: bmV3LXBhc3MK # 修改 kind: Secret metadata: ... name: vault-token namespace: security type: Opaque ➜ kubectl -n security edit secret mysql-admin apiVersion: v1 data: password: bmV3LXBhc3MK # 修改 kind: Secret metadata: ... name: mysql-admin namespace: security type: OpaqueQuestion 19Task weight: 2%Use context: kubectl config use-context workload-prodThe Deployment immutable-deployment in Namespace team-purple should run immutable, it's created from file /opt/course/19/immutable-deployment.yaml. Even after a successful break-in, it shouldn't be possible for an attacker to modify the filesystem of the running container.Modify the Deployment in a way that no processes inside the container can modify the local filesystem, only /tmp directory should be writeable. Don't modify the Docker image.Save the updated YAML under /opt/course/19/immutable-deployment-new.yaml and update the running Deployment.题目解析：考点为 Pod 或容器配置安全性上下文 | Kubernetes解题修改 yaml➜ cp /opt/course/19/immutable-deployment.yaml /opt/course/19/immutable-deployment-new.yaml ➜ vim /opt/course/19/immutable-deployment-new.yaml apiVersion: apps/v1 kind: Deployment metadata: namespace: team-purple name: immutable-deployment labels: app: immutable-deployment spec: replicas: 1 selector: matchLabels: app: immutable-deployment template: metadata: labels: app: immutable-deployment spec: containers: - image: busybox:1.32.0 command: ['sh', '-c', 'tail -f /dev/null'] imagePullPolicy: IfNotPresent name: busybox securityContext: # 新增 readOnlyRootFilesystem: true # 新增 volumeMounts: # 新增 - name: tmp # 新增 mountPath: /tmp # 新增 restartPolicy: Always volumes: # 新增 - name: tmp # 新增 emptyDir: {} # 新增验证➜ kubectl -n team-purple exec -it immutable-deployment-7cf85b8f74-z8zb9 -- /bin/sh / # echo 1 > /tmp/1 / # echo 1 > /1 /bin/sh: can't create /1: Read-only file systemQuestion 20Task weight: 8%Use context: kubectl config use-context workload-stageThe cluster is running Kubernetes 1.21.4. Update it to 1.22.1 available via apt package manager.Use ssh cluster3-master1 and ssh cluster3-worker1 to connect to the instances.题目解析：考点升级 kubeadm 集群 | Kubernetes解题查看节点➜ kubectl get node NAME STATUS ROLES AGE VERSION cluster3-master1 Ready control-plane,master 88d v1.21.4 cluster3-worker1 Ready <none> 88d v1.21.4查看 master 节点 kubeadm, kubectl, kubelet 版本，升级计划➜ ssh cluster3-master1 ➜ root@cluster3-master1:~# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:44:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} ➜ root@cluster3-master1:~# kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:16:05Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} ➜ root@cluster3-master1:~# kubelet --version Kubernetes v1.21.4 # kubeadm 已经是 1.22.1 版本了，不用升级 kubeadm # 执行升级计划，kubeadm upgrade plan 或 kubeadm upgrade plan 1.22.1 ➜ root@cluster3-master1:~# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.21.4 [upgrade/versions] kubeadm version: v1.22.1 I1214 01:16:17.575626 228047 version.go:255] remote version is much newer: v1.23.0; falling back to: stable-1.22 [upgrade/versions] Target version: v1.22.4 [upgrade/versions] Latest version in the v1.21 series: v1.21.7 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 2 x v1.21.4 v1.21.7 Upgrade to the latest version in the v1.21 series: COMPONENT CURRENT TARGET kube-apiserver v1.21.4 v1.21.7 kube-controller-manager v1.21.4 v1.21.7 kube-scheduler v1.21.4 v1.21.7 kube-proxy v1.21.4 v1.21.7 CoreDNS v1.8.0 v1.8.4 etcd 3.4.13-0 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.21.7 _____________________________________________________________________ Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 2 x v1.21.4 v1.22.4 Upgrade to the latest stable version: COMPONENT CURRENT TARGET kube-apiserver v1.21.4 v1.22.4 kube-controller-manager v1.21.4 v1.22.4 kube-scheduler v1.21.4 v1.22.4 kube-proxy v1.21.4 v1.22.4 CoreDNS v1.8.0 v1.8.4 etcd 3.4.13-0 3.5.0-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.22.4 Note: Before you can perform this upgrade, you have to update kubeadm to v1.22.4. _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________升级 master➜ root@cluster3-master1:~# kubeadm upgrade apply v1.22.1 ... [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.22.1". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. # 驱逐节点运行的 Pod ➜ root@cluster3-master1:~# kubectl drain cluster3-master1 --ignore-daemonsets # 升级 kubelet, kubectl ➜ root@cluster3-master1:~# apt-get install kubelet=1.22.1-00 kubectl=1.22.1-00 # 重启 kubelet ➜ root@cluster3-master1:~# systemctl daemon-reload ➜ root@cluster3-master1:~# systemctl restart kubelet # 解除保护 ➜ root@cluster3-master1:~# kubectl uncordon cluster3-master1升级 workerssh cluster3-worker1 # 检查版本，这里略过 # 升级 ➜ root@cluster3-worker1:~# kubeadm upgrade node # 驱逐节点运行的 Pod ➜ root@cluster3-worker1:~# kubectl drain cluster3-worker1 --ignore-daemonsets # 升级 kubelet, kubectl ➜ root@cluster3-worker1:~# apt-get install kubelet=1.22.1-00 kubectl=1.22.1-00 # 重启 kubelet ➜ root@cluster3-worker1:~# systemctl daemon-reload ➜ root@cluster3-worker1:~# systemctl restart kubelet # 解除保护 ➜ root@cluster3-worker1:~# kubectl uncordon cluster3-worker1验证➜ kubectl get node NAME STATUS ROLES AGE VERSION cluster3-master1 Ready control-plane,master 88d v1.22.1 cluster3-worker1 Ready <none> 88d v1.22.1Question 21Task weight: 2%(can be solved in any kubectl context)The Vulnerability Scanner trivy is installed on your main terminal. Use it to scan the following images for known CVEs:nginx:1.16.1-alpinek8s.gcr.io/kube-apiserver:v1.18.0k8s.gcr.io/kube-controller-manager:v1.18.0docker.io/weaveworks/weave-kube:2.7.0Write all images that don't contain the vulnerabilities CVE-2020-10878 or CVE-2020-1967 into /opt/course/21/good-images.题目解析：考点trivy解题使用 trivy 工具扫描镜像➜ trivy image nginx:1.16.1-alpine | egrep "CVE-2020-10878|CVE-2020-1967" | libcrypto1.1 | CVE-2020-1967 | HIGH | 1.1.1d-r2 | 1.1.1g-r0 | openssl: Segmentation fault in | | libssl1.1 | CVE-2020-1967 | | 1.1.1d-r2 | 1.1.1g-r0 | openssl: Segmentation fault in | ➜ trivy image k8s.gcr.io/kube-apiserver:v1.18.0 | egrep "CVE-2020-10878|CVE-2020-1967" | | CVE-2020-10878 | | | | perl: corruption of | ➜ trivy k8s.gcr.io/kube-controller-manager:v1.18.0 | egrep "CVE-2020-10878|CVE-2020-1967" | | CVE-2020-10878 | | | | perl: corruption of | ➜ trivy docker.io/weaveworks/weave-kube:2.7.0 | egrep "CVE-2020-10878|CVE-2020-1967" # 只有 docker.io/weaveworks/weave-kube:2.7.0 没有扫出漏洞写入答案➜ echo docker.io/weaveworks/weave-kube:2.7.0 > /opt/course/21/good-imagesQuestion 22Task weight: 3%(can be solved in any kubectl context)The Release Engineering Team has shared some YAML manifests and Dockerfiles with you to review. The files are located under /opt/course/22/files.As a container security expert, you are asked to perform a manual static analysis and find out possible security issues with respect to unwanted credential exposure. Running processes as root is of no concern in this task.Write the filenames which have issues into /opt/course/22/security-issues.NOTE: In the Dockerfile and YAML manifests, assume that the referred files, folders, secrets and volume mounts are present. Disregard syntax or logic errors.题目解析：考点找安全隐患解题查看都有哪些文件➜ ll /opt/course/22/files total 48 drwxr-xr-x 2 k8s k8s 4096 Dec 14 02:15 ./ drwxr-xr-x 3 k8s k8s 4096 Sep 16 08:37 ../ -rw-r--r-- 1 k8s k8s 341 Sep 16 08:37 deployment-nginx.yaml -rw-r--r-- 1 k8s k8s 723 Sep 16 08:37 deployment-redis.yaml -rw-r--r-- 1 k8s k8s 384 Sep 16 08:37 Dockerfile-go
Question 1Task weight: 1%You have access to multiple clusters from your main terminal through kubectl contexts. Write all those context names into /opt/course/1/contexts.Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh, the command should use kubectl.Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh, but without the use of kubectl.题目解析：考点kubectl, contexts kubectl 概述 | Kubernetes, kubectl-commands解题根据题意：Write all those context names into /opt/course/1/contexts➜ kubectl config get-contexts -o name > /opt/course/1/contexts根据题意：Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh, the command should use kubectl➜ vim /opt/course/1/context_default_kubectl.sh kubectl config current-context根据题意：Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh, but without the use of kubectl➜ vim /opt/course/1/context_default_no_kubectl.sh grep "current-context: " ~/.kube/config | awk '{print $2}'Question 2Task weight: 3%Use context: kubectl config use-context k8s-c1-HCreate a single Pod of image httpd:2.4.41-alpine in Namespace default. The Pod should be named pod1 and the container should be named pod1-container. This Pod should only be scheduled on a master node, do not add new labels any nodes.Shortly write the reason on why Pods are by default not scheduled on master nodes into /opt/course/2/master_schedule_reason .题目解析：考点Pod Pods | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H Switched to context "k8s-c1-H".快速生成创建 Pod 的 yaml 模板➜ kubectl run pod1 --image=httpd:2.4.41-alpine --dry-run=client -o yaml > 2.yaml根据题意，修改模板➜ vim 2.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 spec: containers: - image: httpd:2.4.41-alpine name: pod1-container # 修改, the container should be named `pod1-container` resources: {} dnsPolicy: ClusterFirst restartPolicy: Always tolerations: # 新增, This Pod should only be scheduled on a master node，所以要容忍master的污点 - effect: NoSchedule # 新增 key: node-role.kubernetes.io/master # 新增 nodeSelector: # 新增，指定nodeSelector，标签通过kubectl get node --show-labels查看 node-role.kubernetes.io/master: "" # 新增 status: {}创建资源kubectl apply -f 2.yaml根据题意：Shortly write the reason on why Pods are by default not scheduled on master nodes into /opt/course/2/master_schedule_reason➜ vim /opt/course/2/master_schedule_reason master nodes usually have a taint definedQuestion 3Task weight: 1%Use context: kubectl config use-context k8s-c1-HThere are two Pods named o3db-* in Namespace project-c13. C13 management asked you to scale the Pods down to one replica to save resources. Record the action.题目解析：考点Pod伸缩解题切换上下文kubectl config use-context k8s-c1-H根据题意，注意两点将副本数设为 1记录操作➜ kubectl -n project-c13 scale sts o3db --replicas 1 --recordQuestion 4Task weight: 4%Use context: kubectl config use-context k8s-c1-HDo the following in Namespace default. Create a single Pod named ready-if-service-ready of image nginx:1.16.1-alpine. Configure a LivenessProbe which simply runs true. Also configure a ReadinessProbe which does check if the url http://service-am-i-ready:80 is reachable, you can use wget -T2 -O- http://service-am-i-ready:80 for this. Start the Pod and confirm it isn't ready because of the ReadinessProbe.Create a second Pod named am-i-ready of image nginx:1.16.1-alpine with label id: cross-server-ready. The already existing Service service-am-i-ready should now have that second Pod as endpoint.Now the first Pod should be in ready state, confirm that.题目解析：考点LivenessProbe, ReadinessProbe, Service 配置存活、就绪和启动探测器 | Kubernetes, 服务 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H快速生成创建 Pod 的 yaml 模版➜ kubectl run ready-if-service-ready --image=nginx:1.16.1-alpine --dry-run=client -o yaml > 4.yaml根据题意：Create a single Pod named ready-if-service-ready...，修改模板➜ vim 4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: ready-if-service-ready name: ready-if-service-ready spec: containers: - image: nginx:1.16.1-alpine name: ready-if-service-ready resources: {} livenessProbe: # 新增 httpGet: # 新增 path: / # 新增 port: 80 # 新增 readinessProbe: # 新增 exec: # 新增 command: # 新增 - sh # 新增 - -c # 新增 - wget -T2 -O- http://service-am-i-ready:80 # 新增 dnsPolicy: ClusterFirst restartPolicy: Always status: {}根据题意：Create a second Pod named am-i-ready of image nginx:1.16.1-alpine with label id: cross-server-ready...# 快速生成创建 Pod 的 yaml 模板 ➜ kubectl run am-i-ready --image=nginx:1.16.1-alpine --dry-run=client -o yaml > 4-2.yaml # 修改模板 ➜ vim 4-2.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: id: cross-server-ready # 修改，with label id: cross-server-ready name: am-i-ready spec: containers: - image: nginx:1.16.1-alpine name: am-i-ready resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {}创建资源➜ kubectl apply -f 4.yaml -f 4-2.yamlQuestion 5Task weight: 1%Use context: kubectl config use-context k8s-c1-HThere are various Pods in all namespaces. Write a command into /opt/course/5/find_pods.sh which lists all Pods sorted by their AGE (metadata.creationTimestamp).Write a second command into /opt/course/5/find_pods_uid.sh which lists all Pods sorted by field metadata.uid. Use kubectl sorting for both commands.题目解析：考点kubectl kubectl 概述 | Kubernetes, kubectl-commands解题列出所有 Pod 并以创建时间排序➜ vim /opt/course/5/find_pods.sh kubectl get pod --all-namespaces --sort-by=metadata.creationTimestamp列出所有 Pod 并以创建 uid 排序➜ vim /opt/course/5/find_pods_uid.sh kubectl get pod --all-namespaces --sort-by=metadata.uidQuestion 6Task weight: 8%Use context: kubectl config use-context k8s-c1-HCreate a new PersistentVolume named safari-pv. It should have a capacity of 2Gi, accessMode ReadWriteOnce, hostPath /Volumes/Data and no storageClassName defined.Next create a new PersistentVolumeClaim in Namespace project-tiger named safari-pvc . It should request 2Gi storage, accessMode ReadWriteOnce and should not define a storageClassName. The PVC should bound to the PV correctly.Finally create a new Deployment safari in Namespace project-tiger which mounts that volume at /tmp/safari-data. The Pods of that Deployment should be of image httpd:2.4.41-alpine.题目解析：考点pv, pvc, deployment 持久卷 | Kubernetes, Deployments | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-Hpv, pvc 的 yaml 要自己写，可以从官网拷贝示例来修改➜ vim 6.yaml apiVersion: v1 kind: PersistentVolume metadata: name: safari-pv spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/Volumes/Data" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: safari-pvc namespace: project-tiger spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi volumeName: safari-pv ---快速生成创建 deployment 的 yaml 模板➜ kubectl -n project-tiger create deployment safari --image=httpd:2.4.41-alpine --dry-run -o yaml >> 6.yaml根据题意，修改模板➜ vim 6.yaml ... --- apiVersion: apps/v1 kind: Deployment metadata: name: safari namespace: project-tiger labels: app: safari spec: replicas: 1 selector: matchLabels: app: safari template: metadata: labels: app: safari spec: containers: - name: httpd image: httpd:2.4.41-alpine volumeMounts: # 新增 - name: data # 新增 mountPath: /tmp/safari-data # 新增 volumes: # 新增 - name: data # 新增 persistentVolumeClaim: # 新增 claimName: safari-pvc # 新增创建资源➜ kubectl apply -f 6.yamlQuestion 7Task weight: 1%Use context: kubectl config use-context k8s-c1-HThe metrics-server hasn't been installed yet in the cluster, but it's something that should be done soon. Your college would already like to know the kubectl commands to:show node resource usageshow Pod and their containers resource usagePlease write the commands into /opt/course/7/node.sh and /opt/course/7/pod.sh.题目解析：考点metric-server解题➜ vim /opt/course/7/node.sh kubectl top node ➜ vim /opt/course/7/pod.sh kubectl top pod --containers=trueQuestion 8Task weight: 2%Use context: kubectl config use-context k8s-c1-HSsh into the master node with ssh cluster1-master1. Check how the master components kubelet, kube-apiserver, kube-scheduler, kube-controller-manager and etcd are started/installed on the master node. Also find out the name of the DNS application and how it's started/installed on the master node.Write your findings into file /opt/course/8/master-components.txt. The file should be structured like:# /opt/course/8/master-components.txt kubelet: [TYPE] kube-apiserver: [TYPE] kube-scheduler: [TYPE] kube-controller-manager: [TYPE] etcd: [TYPE] dns: [TYPE] [NAME]Choices of [TYPE] are: not-installed, process, static-pod, pod题目解析：考点系统组件, 部署解题切换上下文➜ kubectl config use-context k8s-c1-H通过 kubectl -n kube-system get pod 可以看出 kube-apiserver, kube-scheduler, kube-controller-manager, etcd 是静态 pod. dns 是 coredns, 由 deployment 管理的，因此是 pod 类型➜ vim /opt/course/8/master-components.txt kubelet: process kube-apiserver: static-pod kube-scheduler: static-pod kube-controller-manager: static-pod etcd: static-pod dns: pod corednsQuestion 9Use context: kubectl config use-context k8s-c2-ACSsh into the master node with ssh cluster2-master1. Temporarily stop the kube-scheduler, this means in a way that you can start it again afterwards.Create a single Pod named manual-schedule of image httpd:2.4-alpine, confirm its started but not scheduled on any node.Now you're the scheduler and have all its power, manually schedule that Pod on node cluster2-master1. Make sure it's running.Start the kube-scheduler again and confirm its running correctly by creating a second Pod named manual-schedule2 of image httpd:2.4-alpine and check if it's running on cluster2-worker1.题目解析：考点调度 Kubernetes 调度器 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c2-AC停止 kube-scheduler➜ ssh cluster2-master1 ➜ root@cluster2-master1:~# cd /etc/kubernetes/manifests/ ➜ root@cluster2-master1:~# mv kube-scheduler.yaml ..创建 Pod：Create a single Pod named manual-schedule of image httpd:2.4-alpine➜ kubectl run manual-schedule --image=httpd:2.4-alpine查看状态➜ kubectl get pod manual-schedule -o wide NAME READY STATUS ... NODE NOMINATED NODE manual-schedule 0/1 Pending ... <none> <none> 手动调度➜ kubectl edit pod manual-schedule apiVersion: v1 kind: Pod metadata: ... spec: nodeName: cluster2-master1 # 新增 containers: - image: httpd:2.4-alpine imagePullPolicy: IfNotPresent name: manual-schedule ...再次查看状态➜ kubectl get pod manual-schedule -o wide NAME READY STATUS ... NODE manual-schedule 1/1 Running ... cluster2-master1启动 kube-scheduler➜ ssh cluster2-master1 ➜ root@cluster2-master1:~# cd /etc/kubernetes/ ➜ root@cluster2-master1:~# mv kube-scheduler.yaml manifests/创建 Pod：➜ kubectl run manual-schedule2 --image=httpd:2.4-alpine查看状态➜ kubectl get pod -o wide | grep schedule manual-schedule 1/1 Running ... cluster2-master1 manual-schedule2 1/1 Running ... cluster2-worker1Question 10Task weight: 6%Use context: kubectl config use-context k8s-c1-HCreate a new ServiceAccount processor in Namespace project-hamster. Create a Role and RoleBinding, both named processor as well. These should allow the new SA to only create Secrets and ConfigMaps in that Namespace.题目解析：考点鉴权概述 | Kubernetes, 使用 RBAC 鉴权 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H创建 ServiceAccount➜ kubectl -n project-hamster create sa processor创建 Role➜ kubectl -n project-hamster create role processor --verb=create --resource=secrets,configmap创建 RoleBanding➜ kubectl -n project-hamster create rolebinding processor --role=processor --serviceaccount=project-hamster:processor验证➜ kubectl -n project-hamster auth can-i create secret --as system:serviceaccount:project-hamster:processor yes ➜ kubectl -n project-hamster auth can-i create configmap --as system:serviceaccount:project-hamster:processor yes ➜ kubectl -n project-hamster auth can-i create pod --as system:serviceaccount:project-hamster:processor no ➜ kubectl -n project-hamster auth can-i delete secret --as system:serviceaccount:project-hamster:processor no ➜ kubectl -n project-hamster auth can-i get configmap --as system:serviceaccount:project-hamster:processor noQuestion 11Task weight: 4%Use context: kubectl config use-context k8s-c1-HUse Namespace project-tiger for the following. Create a DaemonSet named ds-important with image httpd:2.4-alpine and labels id=ds-important and uuid=18426a0b-5f59-4e10-923f-c0e078e82462. The Pods it creates should request 10 millicore cpu and 10 megabytes memory. The Pods of that DaemonSet should run on all nodes, master and worker.题目解析：考点为容器管理资源 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H在官网找一份 DaemonSet 示例修改➜ vim 11.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: ds-important namespace: project-tiger labels: k8s-app: ds-important spec: selector: matchLabels: id: ds-important uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 template: metadata: labels: id: ds-important uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 spec: tolerations: # this toleration is to have the daemonset runnable on master nodes # remove it if your masters can't run pods - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule containers: - name: httpd image: httpd:2.4-alpine resources: requests: cpu: 10m memory: 10Mi创建资源➜ kubectl apply -f 11.yamlQuestion 12Task weight: 6%Use context: kubectl config use-context k8s-c1-HUse Namespace project-tiger for the following. Create a Deployment named deploy-important with label id=very-important (the Pods should also have this label) and 3 replicas. It should contain two containers, the first named container1 with image nginx:1.17.6-alpine and the second one named container2 with image kubernetes/pause.There should be only ever one Pod of that Deployment running on one worker node. We have two worker nodes: cluster1-worker1 and cluster1-worker2. Because the Deployment has three replicas the result should be that on both nodes one Pod is running. The third Pod won't be scheduled, unless a new worker node will be added.In a way we kind of simulate the behaviour of a DaemonSet here, but using a Deployment and a fixed number of replicas.题目解析：考点将 Pod 分配给节点 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H快速生成部署 deployment 的 yaml 文件➜ kubectl -n project-tiger create deployment --image=nginx:1.17.6-alpine deploy-important --dry-run=client -o yaml > 12.yaml根据题意修改模板➜ vim 12.yaml apiVersion: apps/v1 kind: Deployment metadata: name: deploy-important namespace: project-tiger labels: id: very-important # 修改 spec: replicas: 3 # 修改 selector: matchLabels: id: very-important # 修改 template: metadata: labels: id: very-important # 修改 spec: containers: - name: container1 # 修改 image: nginx:1.17.6-alpine - name: container2 # 新增 image: kubernetes/pause # 新增 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: # 新增 - topologyKey: kubernetes.io/hostname # 新增 labelSelector: # 新增 #matchExpressions: # 新增 #- key: id # 新增 # operator: In # 新增 # values: # 新增 # - very-important # 新增 matchLabels: # 新增 id: very-important # 新增Question 13Task weight: 4%Use context: kubectl config use-context k8s-c1-HCreate a Pod named multi-container-playground in Namespace default with three containers, named c1, c2 and c3. There should be a volume attached to that Pod and mounted into every container, but the volume shouldn't be persisted or shared with other Pods.Container c1 should be of image nginx:1.17.6-alpine and have the name of the node where its Pod is running on value available as environment variable MY_NODE_NAME.Container c2 should be of image busybox:1.31.1 and write the output of the date command every second in the shared volume into file date.log. You can use while true; do date >> /your/vol/path/date.log; sleep 1; done for this.Container c3 should be of image busybox:1.31.1 and constantly write the content of file date.log from the shared volume to stdout. You can use tail -f /your/vol/path/date.log for this.Check the logs of container c3 to confirm correct setup.题目解析：考点临时卷 | Kubernetes, 通过环境变量将 Pod 信息呈现给容器 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H快速生成部署 pod 的 yaml 模板➜ kubectl run multi-container-playground --image=nginx:1.17.6-alpine > 13.yaml根据题意，修改模板➜ vim 13.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: multi-container-playground name: multi-container-playground spec: containers: - image: nginx:1.17.6-alpine name: c1 # 修改 volumeMounts: # 新增 - name: data # 新增 mountPath: /data # 新增 env: # 新增 - name: MY_NODE_NAME # 新增 valueFrom: # 新增 fieldRef: # 新增 fieldPath: spec.nodeName # 新增 - image: busybox:1.31.1 # 新增 name: c2 # 新增 command: # 新增 - sh # 新增 - -c # 新增 - while true; do date >> /data/date.log; sleep 1; done # 新增 volumeMounts: # 新增 - name: data # 新增 mountPath: /data # 新增 - image: busybox:1.31.1 # 新增 name: c3 # 新增 command: # 新增 - sh # 新增 - -c # 新增 - tail -f /data/date.log # 新增 volumeMounts: # 新增 - name: data # 新增 mountPath: /data # 新增 volumes: # 新增 - name: data # 新增 emptyDir: {} # 新增Question 14Task weight: 2%Use context: kubectl config use-context k8s-c1-HYou're ask to find out following information about the cluster k8s-c1-H:How many master nodes are available?How many worker nodes are available?What is the Service CIDR?Which Networking (or CNI Plugin) is configured and where is its config file?Which suffix will static pods have that run on cluster1-worker1?Write your answers into file /opt/course/14/cluster-info, structured like this:# /opt/course/14/cluster-info 1: [ANSWER] 2: [ANSWER] 3: [ANSWER] 4: [ANSWER] 5: [ANSWER]题目解析：考点集群信息解题切换上下文➜ kubectl config use-context k8s-c1-H通过查看节点，1,2 可解➜ kubectl get node查看配置文件，3,4 可解➜ ssh cluster1-master1 ➜ root@cluster1-master1:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep range - --service-cluster-ip-range=10.96.0.0/12 ➜ root@cluster1-master1:~# ls /etc/cni/net.d/ 10-weave.conflist可得答案1: 1 2: 2 3: 10.96.0.0/12 4: Weave, /etc/cni/net.d/10-weave.conflist 5: -cluster1-worker1Question 15Task weight: 3%Use context: kubectl config use-context k8s-c2-ACWrite a command into /opt/course/15/cluster_events.sh which shows the latest events in the whole cluster, ordered by time. Use kubectl for it.Now kill the kube-proxy Pod running on node cluster2-worker1 and write the events this caused into /opt/course/15/pod_kill.log.Finally kill the containerd container of the kube-proxy Pod on node cluster2-worker1 and write the events into /opt/course/15/container_kill.log.Do you notice differences in the events both actions caused?题目解析：考点event解题切换上下文➜ kubectl config use-context k8s-c2-AC获取集群events，以时间排序kubectl get events -A --sort-by=.metadata.creationTimestamp删除 cluster2-worker1 上的 kube-proxy pod➜ kubectl -n kube-system get pod ➜ kubectl -n kube-system delete pod kube-proxy-xxxx获取 events➜ sh /opt/course/15/cluster_events.sh # 将最新的 events 写入 /opt/course/15/pod_kill.log删除 cluster2-worker1 上的 kube-proxy 容器➜ ssh cluster2-worker1 ➜ root@cluster2-worker1:~# crictl ps ➜ root@cluster2-worker1:~# crictl rm CONTAINER_ID获取 events➜ sh /opt/course/15/cluster_events.sh # 将最新的 events 写入 /opt/course/15/container_kill.logQuestion 16Task weight: 2%Use context: kubectl config use-context k8s-c1-HCreate a new Namespace called cka-master.Write the names of all namespaced Kubernetes resources (like Pod, Secret, ConfigMap...) into /opt/course/16/resources.txt.Find the project-* Namespace with the highest number of Roles defined in it and write its name and amount of Roles into /opt/course/16/crowded-namespace.txt.题目解析：考点kubectl解题切换上下文➜ kubectl config use-context k8s-c1-HCreate a new Namespace called cka-master➜ kubectl create ns cka-masterWrite the names of all namespaced Kubernetes resources (like Pod, Secret, ConfigMap...) into /opt/course/16/resources.txt➜ kubectl api-resources --namespaced=true -o name > /opt/course/16/resources.txtFind the project-* Namespace with the highest number of Roles defined in it...➜ kubectl get role -A | grep ^project- | awk '{print $1}' | sort | uniq -c # 将结果写入/opt/course/16/crowded-namespace.txtQuestion 17Task weight: 3%Use context: kubectl config use-context k8s-c1-HIn Namespace project-tiger create a Pod named tigers-reunite of image httpd:2.4.41-alpine with labels pod=container and container=pod. Find out on which node the Pod is scheduled. Ssh into that node and find the containerd container belonging to that Pod.Using command crictl:Write the ID of the container and the info.runtimeType into /opt/course/17/pod-container.txtWrite the logs of the container into /opt/course/17/pod-container.log题目解析：考点containerd, crictl解题切换上下文➜ kubectl config use-context k8s-c1-H部署 Pod➜ kubectl -n project-tiger run tigers-reunite --image=httpd:2.4.41-alpine --labels=pod=container,container=pod查看 info.runtimeType# 先查看 Pod 调度到了哪个节点 ➜ kubectl -n project-tiger get pod -o wide ➜ ssh cluster1-worker2 crictl ps | grep tigers-reunite b01edbe6f89ed 54b0995a63052 5 seconds ago Running tigers-reunite ... ➜ ssh cluster1-worker2 crictl inspect b01edbe6f89ed | grep runtimeType "runtimeType": "io.containerd.runc.v2", # 将container id 和 runtimeType 写入 /opt/course/17/pod-container.txt b01edbe6f89ed io.containerd.runc.v2获取容器日志➜ kubectl -n project-tiger logs tigers-reunite > /opt/course/17/pod-container.logQuestion 18Task weight: 8%Use context: kubectl config use-context k8s-c3-CCCThere seems to be an issue with the kubelet not running on cluster3-worker1. Fix it and confirm that cluster3 has node cluster3-worker1 available in Ready state afterwards. You should be able to schedule a Pod on cluster3-worker1 afterwards.Write the reason of the issue into /opt/course/18/reason.txt.题目解析：考点故障排查解题切换上下文➜ kubectl config use-context k8s-c3-CCC检查 cluster3-worker1 节点 kubelet➜ ssh cluster3-worker1 ➜ root@cluster3-worker1:~# ps aux | grep kubelet root 29294 0.0 0.2 14856 1016 pts/0 S+ 11:30 0:00 grep --color=auto kubelet ➜ root@cluster3-worker1:~# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: inactive (dead) since Sun 2019-12-08 11:30:06 UTC; 50min 52s ago ➜ root@cluster3-worker1:~# systemctl start kubelet ➜ root@cluster3-worker1:~# systemctl status kubelet ➜ root@cluster3-worker1:~# journalctl -xe # 通过查看 `systemctl status kubelet` 和 journalctl -xe，发现报错信息：/usr/local/bin/kubelet: No such file or directory # 查看配置文件：/etc/systemd/system/kubelet.service.d/10-kubeadm.conf # 修改 /usr/local/bin/kubelet 为 /usr/bin/kubelet ➜ root@cluster3-worker1:~# systemctl start kubelet故障原因➜ vim /opt/course/18/reason.txt wrong path to kubelet binary specified in service configQuestion 19Task weight: 3%this task can only be solved if questions 18 or 20 have been successfully implemented and the k8s-c3-CCC cluster has a functioning worker nodeUse context: kubectl config use-context k8s-c3-CCCDo the following in a new Namespace secret. Create a Pod named secret-pod of image busybox:1.31.1 which should keep running for some time, it should be able to run on master nodes as well.There is an existing Secret located at /opt/course/19/secret1.yaml, create it in the secret Namespace and mount it readonly into the Pod at /tmp/secret1.Create a new Secret in Namespace secret called secret2 which should contain user=user1 and pass=1234. These entries should be available inside the Pod's container as environment variables APP_USER and APP_PASS.Confirm everything is working.题目解析：考点Secret | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c3-CCC创建命名空间 secret➜ kubectl create ns secret快速生成部署 Pod 的 yaml 模板➜ kubectl -n secret run secret-pod --image=busybox:1.31.1 --dry-run=client -o yaml > 19.yaml修改模板➜ vim 19.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: secret-pod name: secret-pod namespace: secret spec: containers: - image: busybox:1.31.1 name: secret-pod resources: {} command: # 新增 - sh # 新增 - -c # 新增 - sleep 1d # 新增 volumeMounts: # 新增，mount it readonly into the *Pod* at `/tmp/secret1` - name: secret1 # 新增 mountPath: /tmp/secret1 # 新增 readOnly: true # 新增 env: # 新增，environment variables APP_USER and APP_PASS. - name: APP_USER # 新增 valueFrom: # 新增 secretKeyRef: # 新增 name: secret2 # 新增 key: pass # 新增 - name: APP_PASS # 新增 valueFrom: # 新增 secretKeyRef: # 新增 name: secret2 # 新增 key: pass # 新增 dnsPolicy: ClusterFirst restartPolicy: Always tolerations: # 新增，it should be able to run on master nodes as well. - key: "node-role.kubernetes.io/master" # 新增 operator: "Exists" # 新增 effect: "NoSchedule" # 新增 volumes: # 新增 - name: secret1 # 新增 secret: # 新增 secretName: secret1 # 新增 status: {}创建 secret1➜ cp /opt/course/19/secret1.yaml 19_secret1.yaml ➜ vim 19_secret1.yaml apiVersion: v1 data: halt: IyEgL2Jpbi9zaAo... kind: Secret metadata: creationTimestamp: null name: secret1 namespace: secret # 修改 ➜ kubectl apply -f 19_secret1.yaml创建secret2➜ kubectl -n secret create secret generic secret2 --from-literal=user=user1 --from-literal=pass=1234创建 Podkubectl apply -f 19.yamlQuestion 20Task weight: 10%Use context: kubectl config use-context k8s-c3-CCCYour coworker said node cluster3-worker2 is running an older Kubernetes version and is not even part of the cluster. Update kubectl and kubeadm to the version that's running on cluster3-master1. Then add this node to the cluster, you can use kubeadm for this.题目解析：考点升级 kubeadm 集群 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c3-CCC查看版本➜ kubectl get node NAME STATUS ROLES AGE VERSION cluster3-master1 Ready control-plane,master 116m v1.22.1 cluster3-worker1 NotReady <none> 112m v1.22.1查看 cluster3-worker2 节点 kubelet, kubectl, kubeadm 版本➜ ssh cluster3-worker2 ➜ root@cluster3-worker2:~# kubeadm version ubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:44:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} ➜ root@cluster3-worker2:~# kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:16:05Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port? ➜ root@cluster3-worker2:~# kubelet --version Kubernetes v1.21.4尝试一下 kubeadm 升级➜ root@cluster3-worker2:~# kubeadm upgrade node couldn't create a Kubernetes client from file "/etc/kubernetes/kubelet.conf": failed to load admin kubeconfig: open /etc/kubernetes/kubelet.conf: no such file or directory To see the stack trace of this error execute with --v=5 or higher # 看到这个报错，说明这个节点没有做过初始化，即从没有加入过集群 # 这种情况，就升级 kubectl, kubeadm, kubelet 到指定版本，然后 kubeadm join 加入集群升级 kubectl, kubeadm, kubelet➜ root@cluster3-worker2:~# apt-get update ➜ root@cluster3-worker2:~# apt-cache show kubelet | grep 1.22 Version: 1.22.1-00 Filename: pool/kubectl_1.22.1-00_amd64_2a00cd912bfa610fe4932bc0a557b2dd7b95b2c8bff9d001dc6b3d34323edf7d.deb Version: 1.22.0-00 Filename: pool/kubectl_1.22.0-00_amd64_052395d9ddf0364665cf7533aa66f96b310ec8a2b796d21c42f386684ad1fc56.deb Filename: pool/kubectl_1.17.1-00_amd64_0dc19318c9114db2931552bb8bf650a14227a9603cb73fe0917ac7868ec7fcf0.deb SHA256: 0dc19318c9114db2931552bb8bf650a14227a9603cb73fe0917ac7868ec7fcf0 ... ➜ root@cluster3-worker2:~# apt install kubectl=1.22.1-00 kubeadm=1.22.1-00 kubelet=1.22.1-00获取 kubeadm join 命令➜ ssh cluster3-master1 ➜ root@cluster3-master1:~# kubeadm token create --print-join-command kubeadm join 192.168.100.31:6443 --token leqq1l.1hlg4rw8mu7brv73 --discovery-token-ca-cert-hash sha256:2e2c3407a256fc768f0d8e70974a8e24d7b9976149a79bd08858c4d7aa2ff79acluster3-worker2 节点加入集群➜ ssh cluster3-worker2 ➜ root@cluster3-worker2:~# kubeadm join 192.168.100.31:6443 --token leqq1l.1hlg4rw8mu7brv73 --discovery-token-ca-cert-hash sha256:2e2c3407a256fc768f0d8e70974a8e24d7b9976149a79bd08858c4d7aa2ff79aQuestion 21Task weight: 2%Use context: kubectl config use-context k8s-c3-CCCCreate a Static Pod named my-static-pod in Namespace default on cluster3-master1. It should be of image nginx:1.16-alpine and have resource requests for 10m CPU and 20Mi memory.Then create a NodePort Service named static-pod-service which exposes that static Pod on port 80 and check if it has Endpoints and if its reachable through the cluster3-master1 internal IP address. You can connect to the internal node IPs from your main terminal.题目解析：考点静态 Pod解题切换上下文➜ kubectl config use-context k8s-c3-CCC创建静态Pod➜ ssh cluster3-master1 ➜ root@cluster1-master1:~# cd /etc/kubernetes/manifests/ ➜ root@cluster1-master1:~# kubectl run my-static-pod --image=nginx:1.16-alpine --requests "cpu=10m,memory=20Mi" --dry-run=client -o yaml > my-static-pod.yaml创建Service➜ kubectl expose pod my-static-pod-cluster3-master1 --name static-pod-service --type=NodePort --port 80Questin 22Task weight: 2%Use context: kubectl config use-context k8s-c2-ACCheck how long the kube-apiserver server certificate is valid on cluster2-master1. Do this with openssl or cfssl. Write the exipiration date into /opt/course/22/expiration.Also run the correct kubeadm command to list the expiration dates and confirm both methods show the same date.Write the correct kubeadm command that would renew the apiserver server certificate into /opt/course/22/kubeadm-renew-certs.sh.题目解析：考点证书 证书 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c2-AC查看证书有效期➜ ssh cluster2-master1 ➜ root@cluster2-master1:~# openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt ... Validity Not Before: Jan 14 18:18:15 2021 GMT Not After : Jan 14 18:49:40 2022 GMT ... # 将结果写入 /opt/course/22/expiration Jan 14 18:49:40 2022 GMT证书续期命令➜ vim /opt/course/22/kubeadm-renew-certs.sh kubeadm certs renew apiserverQuestion 23Task weight: 2%Use context: kubectl config use-context k8s-c2-ACNode cluster2-worker1 has been added to the cluster using kubeadm and TLS bootstrapping.Find the "Issuer" and "Extended Key Usage" values of the cluster2-worker1:kubelet client certificate, the one used for outgoing connections to the kube-apiserver.kubelet server certificate, the one used for incoming connections from the kube-apiserver.Write the information into file /opt/course/23/certificate-info.txt.Compare the "Issuer" and "Extended Key Usage" fields of both certificates and make sense of these.题目解析：考点证书 证书 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c2-AC查看证书信息➜ ssh cluster2-worker1 ➜ root@cluster2-worker1:~# openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet-client-current.pem | grep Issuer Issuer: CN = kubernetes ➜ root@cluster2-worker1:~# openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet-client-current.pem | grep "Extended Key Usage" -A1 X509v3 Extended Key Usage: TLS Web Client Authentication ➜ root@cluster2-worker1:~# openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep Issuer Issuer: CN = cluster2-worker1-ca@1588186506 ➜ root@cluster2-worker1:~# openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep "Extended Key Usage" -A1 X509v3 Extended Key Usage: TLS Web Server Authentication # 将证书信息写入 /opt/course/23/certificate-info.txtQuestion 24Task weight: 9%Use context: kubectl config use-context k8s-c1-HThere was a security incident where an intruder was able to access the whole cluster from a single hacked backend Pod.To prevent this create a NetworkPolicy called np-backend in Namespace project-snake. It should allow the backend-* Pods only to:connect to db1-* Pods on port 1111connect to db2-* Pods on port 2222Use the app label of Pods in your policy.After implementation, connections from backend-* Pods to vault-* Pods on port 3333 should for example no longer work.题目解析：考点网络策略 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H从官网找一份示例 yaml 来修改➜ vim 24.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: np-backend namespace: project-snake spec: podSelector: matchLabels: app: backend policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: db1 ports: - protocol: TCP port: 1111 - to: - podSelector: matchLabels: app: db2 ports: - protocol: TCP port: 2222Question 25Task weight: 8%Use context: kubectl config use-context k8s-c3-CCCMake a backup of etcd running on cluster3-master1 and save it on the master node at /tmp/etcd-backup.db.Then create a Pod of your kind in the cluster.Finally restore the backup, confirm the cluster is still working and that the created Pod is no longer with us.题目解析：考点etcd备份 为 Kubernetes 运行 etcd 集群 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c3-CCC备份 etcd➜ cluster3-master1 ➜ root@cluster3-master1:~# ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key创建 Pod➜ kubectl run test --image=nginx恢复etcdroot@cluster3-master1:~# cd /etc/kubernetes/manifests/ root@cluster3-master1:/etc/kubernetes/manifests# mv * .. # 等系统组件静态 Pod 停止运行 ➜ root@cluster3-master1:~# ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db \ --data-dir /var/lib/etcd-backup \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key ➜ root@cluster3-master1:~# vim /etc/kubernetes/etcd.yaml ... - hostPath: path: /var/lib/etcd-backup # 修改 type: DirectoryOrCreate ... ➜ root@cluster3-master1:/etc/kubernetes/manifests# mv ../*.yaml . # 等系统组件静态 Pod 启动Extra Question 1Use context: kubectl config use-context k8s-c1-HCheck all available Pods in the Namespace project-c13 and find the names of those that would probably be terminated first if the Nodes run out of resources (cpu or memory) to schedule all Pods. Write the Pod names into /opt/course/e1/pods-not-stable.txt.题目解析：考点配置 Pod 的服务质量 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H查看 Pod 的 QoS 级别➜ kubectl get pods -n project-c13 -o jsonpath="{range .items[*]}{.metadata.name} {.status.qosClass}{'\n'}" c13-2x3-api-86784557bd-cgs8g Burstable c13-2x3-api-86784557bd-lnxvj Burstable c13-2x3-api-86784557bd-mnp77 Burstable c13-2x3-web-769c989898-6hbgt Burstable c13-2x3-web-769c989898-g57nq Burstable c13-2x3-web-769c989898-hfd5v Burstable c13-2x3-web-769c989898-jfx64 Burstable c13-2x3-web-769c989898-r89mg Burstable c13-2x3-web-769c989898-wtgxl Burstable c13-3cc-runner-98c8b5469-dzqhr Burstable c13-3cc-runner-98c8b5469-hbtdv Burstable c13-3cc-runner-98c8b5469-n9lsw Burstable c13-3cc-runner-heavy-65588d7d6-djtv9 BestEffort c13-3cc-runner-heavy-65588d7d6-v8kf5 BestEffort c13-3cc-runner-heavy-65588d7d6-wwpb4 BestEffort c13-3cc-web-675456bcd-glpq6 Burstable c13-3cc-web-675456bcd-knlpx Burstable c13-3cc-web-675456bcd-nfhp9 Burstable c13-3cc-web-675456bcd-twn7m Burstable o3db-0 BestEffort o3db-1 BestEffort # BestEffort 为最低级别，会优先被 kill写入答案➜ vim /opt/course/e1/pods-not-stable.txt c13-3cc-runner-heavy-65588d7d6-djtv9 c13-3cc-runner-heavy-65588d7d6-v8kf5 c13-3cc-runner-heavy-65588d7d6-wwpb4 o3db-0 o3db-1Extra Question 2Use context: kubectl config use-context k8s-c1-HThere is an existing ServiceAccount secret-reader in Namespace project-hamster. Create a Pod of image curlimages/curl:7.65.3 named tmp-api-contact which uses this ServiceAccount. Make sure the container keeps running.Exec into the Pod and use curl to access the Kubernetes Api of that cluster manually, listing all available secrets. You can ignore insecure https connection. Write the command(s) for this into file /opt/course/e4/list-secrets.sh.题目解析：考点用户认证 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c1-H创建 PodapiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: tmp-api-contact name: tmp-api-contact namespace: project-hamster spec: serviceAccountName: secret-reader containers: - command: - sh - -c - sleep 1d image: curlimages/curl:7.65.3 name: tmp-api-contact resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {}命令➜ vim /opt/course/e4/list-secrets.sh TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) curl -k https://kubernetes.default/api/v1/secrets -H "Authorization: Bearer ${TOKEN}" # 或者 ➜ vim /opt/course/e4/list-secrets.sh TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt curl --cacert ${CACERT} https://kubernetes.default/api/v1/secrets -H "Authorization: Bearer ${TOKEN}"Preview Question 1Use context: kubectl config use-context k8s-c2-ACThe cluster admin asked you to find out the following information about etcd running on cluster2-master1:Server private key locationServer certificate expiration dateIs client certificate authentication enabledWrite these information into /opt/course/p1/etcd-info.txtFinally you're asked to save an etcd snapshot at /etc/etcd-snapshot.db on cluster2-master1 and display its status.题目解析：考点证书, etcd备份 证书 | Kubernetes, 为 Kubernetes 运行 etcd 集群 | Kubernetes解题切换上下文➜ kubectl config use-context k8s-c2-AC证书信息# 查看 /etc/kubernetes/manifests/etcd.yaml 可得证书路径和鉴权是否开启 # openssl x509 -noout -text -in /etc/kubernetes/pki/etcd/server.crt 可得证书有效期 ➜ vim /opt/course/p1/etcd-info.txt Server private key location: /etc/kubernetes/pki/etcd/server.key Server certificate expiration date: Sep 13 13:01:31 2022 GMT Is client certificate authentication enabled: yes备份etcd➜ root@cluster2-master1:~# ETCDCTL_API=3 etcdctl snapshot save /etc/etcd-snapshot.db \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key # 查看备份状态 ➜ root@cluster2-master1:~# ETCDCTL_API=3 etcdctl snapshot status /etc/etcd-snapshot.db 4d4e953, 7213, 1291, 2.7 MBPreview Question 2Use context: kubectl config use-context k8s-c1-HYou're asked to confirm that kube-proxy is running correctly on all nodes. For this perform the following in Namespace project-hamster:Create a new Pod named p2-pod with two containers, one of image nginx:1.21.3-alpine and one of image busybox:1.31. Make sure the busybox container keeps running for some time.Create a new Service named p2-service which exposes that Pod internally in the cluster on port 3000->80.Find the kube-proxy container on all nodes cluster1-master1, cluster1-worker1 and cluster1-worker2 and make sure that it's using iptables. Use command crictl for this.Write the iptables rules of all nodes belonging the created Service p2-service into file /opt/course/p2/iptables.txt.Finally delete the Service and confirm that the iptables rules are gone from all nodes.题目解析：考点iptables解题切换上下文➜ kubectl config use-context k8s-c1-H部署清单apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: p2-pod name: p2-pod namespace: project-hamster spec: containers: - image: nginx:1.21.3-alpine name: c1 resources: {} - image: busybox:1.31 name: c2 resources: {} command: ["sh", "-c", "sleep 1d"] dnsPolicy: ClusterFirst restartPolicy: Always status: {} --- apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: p2-service name: p2-service namespace: project-hamster spec: ports: - name: 3000-80 port: 3000 protocol: TCP targetPort: 80 selector: run: p2-pod type: ClusterIP status: loadBalancer: {}iptables规则➜ ssh cluster1-master1 iptables-save | grep p2-service >> /opt/course/p2/iptables.txt ➜ ssh cluster1-worker1 iptables-save | grep p2-service >> /opt/course/p2/iptables.txt ➜ ssh cluster1-worker2 iptables-save | grep p2-service >> /opt/course/p2/iptables.txtPreview Question 3Use context: kubectl config use-context k8s-c2-ACCreate a Pod named check-ip in Namespace default using image httpd:2.4.41-alpine. Expose it on port 80 as a ClusterIP Service named check-ip-service. Remember/output the IP of that Service.Change the Service CIDR to 11.96.0.0/12 for the cluster.Then create a second Service named check-ip-service2 pointing to the same Pod to check if your settings did take effect. Finally check if the IP of the first Service has changed.题目解析：考点iptables解题切换上下文➜ kubectl config use-context k8s-c2-AC部署清单apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: check-ip name: check-ip spec: containers: - image: httpd:2.4.41-alpine name: check-ip resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} --- apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: check-ip-service name: check-ip-service spec: ports: - name: 80-80 port: 80 protocol: TCP targetPort: 80 selector: run: check-ip type: ClusterIP status: loadBalancer: {}修改网段➜ root@cluster2-master1:~# vim /etc/kubernetes/manifests/kube-controller-manager.yaml - --service-cluster-ip-range=11.96.0.0/12 # 修改创建另一个 service➜ kubectl expose pod check-ip --name check-ip-service2 --port 80