Simble的小站

K8s Python SDK

2020-12-23T06:40:21.000Z

用户及授权

创建用户

script_sc_rolebinding.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: script-admin
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: script-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: script-admin
  namespace: kube-system

1	$ kubectl apply -f script_sc_rolebinding.yml

获取token

$ kubectl get secret -n kube-system | grep script-admin
script-admin-token-9rkpl                         kubernetes.io/service-account-token   3      4m23s
$ kubectl describe secret -n kube-system script-admin-token-9rkpl
Name:         script-admin-token-9rkpl
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name: script-admin
              kubernetes.io/service-account.uid: 9071a506-ae74-4b52-b3d7-25381349fd8b

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1070 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt4SWlabUpwY3U0NWQ4eHY0UWdGNmE0Rm5IRnBtZUMxMjRIV0YzM1kwUFUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzY3JpcHQtYWRtaW4tdG9rZW4tOXJrcGwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic2NyaXB0LWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTA3MWE1MDYtYWU3NC00YjUyLWIzZDctMjUzODEzNDlmZDhiIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOnNjcmlwdC1hZG1pbiJ9.gnSMNViPwf53aG66qn_vUGkMMLKcSIsug2uZPqxH-Lwq7dgAHK7Xen01xWGA9l51xq2wH88xylLP7j6BeZVhtX40DIs3hquxCvX4U7lEollG3AUybxvgeo3Et8G290-eurt6YLS2F1AbSd5qB5SsTfoshVP2UzOqe-gQY1JhPSUK-2CgDH2jMnxLY2qHWbaHB0VY0E8A8keBrHxctetRzyIXHnpbFxCHTLQwEl5rIfoHdZyNn1Q3ALefvpgk_XYijPrbcAFObNtM2HG8Ethrrw0vQuyb1EBHu07RXN_qNsn-QGjEwj_B3k9fbkNA3AxoYN26szPIkpCzhuVeulk5vw

使用python sdk

安装python sdk

参考https://github.com/kubernetes-client/python

1	$ pip install kubernetes

Demo

import requests
requests.packages.urllib3.disable_warnings()
from kubernetes import client, config
from kubernetes.client.rest import ApiException

token = "eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt4SWlabUpwY3U0NWQ4eHY0UWdGNmE0Rm5IRnBtZUMxMjRIV0YzM1kwUFUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzY3JpcHQtYWRtaW4tdG9rZW4tOXJrcGwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic2NyaXB0LWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTA3MWE1MDYtYWU3NC00YjUyLWIzZDctMjUzODEzNDlmZDhiIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOnNjcmlwdC1hZG1pbiJ9.gnSMNViPwf53aG66qn_vUGkMMLKcSIsug2uZPqxH-Lwq7dgAHK7Xen01xWGA9l51xq2wH88xylLP7j6BeZVhtX40DIs3hquxCvX4U7lEollG3AUybxvgeo3Et8G290-eurt6YLS2F1AbSd5qB5SsTfoshVP2UzOqe-gQY1JhPSUK-2CgDH2jMnxLY2qHWbaHB0VY0E8A8keBrHxctetRzyIXHnpbFxCHTLQwEl5rIfoHdZyNn1Q3ALefvpgk_XYijPrbcAFObNtM2HG8Ethrrw0vQuyb1EBHu07RXN_qNsn-QGjEwj_B3k9fbkNA3AxoYN26szPIkpCzhuVeulk5vw"
configuration = client.Configuration()
configuration.api_key_prefix['authorization'] = 'Bearer'
configuration.api_key['authorization'] = token
configuration.verify_ssl = False
configuration.host = "https://10.160.12.184:6443"

api_client = client.ApiClient(configuration)
api_instance = client.CoreV1Api(api_client)
# create a namespace
ns_name = "script-test"
body = {"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": ns_name}}
try:
    res = api_instance.create_namespace(body)
    if res.status.phase == "Active":
        print("Namespace '{}' create success.".format(ns_name))
except ApiException as e:
    print("Namespace '{}' create failed: {}".format(ns_name, e))

# get namespace list
try:
    res = api_instance.list_namespace()
    for ns in res.items:
        print(ns.metadata.name)
except ApiException as e:
    print("Get namespace list failed: {}".format(e))

# delete namespace
try:
    api_instance.delete_namespace(ns_name)
    print("Namespace '{}' delete success.".format(ns_name))
except ApiException as e:
    print("Namespace '{}' delete failed: {}".format(ns_name, e))

Output

Namespace 'script-test' create success.
default
kube-public
kube-system
script-test
Namespace 'script-test' delete success.

后记

看了https://github.com/kubernetes-client/python/blob/master/kubernetes/README.md中的api列表，可以看到有一些方法是用CoreV1Api，有些是BatchApi。

平日里大部分时间使用yaml文件来部署的，包括v1,AppsV1Api,BatchV1Api等，也是同yaml文件中。也就是说，不同的资源需要使用不同的api instance来进行操作。

K8s学习笔记——Job与CronJob

2020-08-25T05:55:16.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：22 | 撬动离线业务：Job与CronJob

Job

创建Job

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - name: hello
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["echo", " hello, Job"]
      restartPolicy: Never

1
2
3

$ kubectl get jobs -o wide
NAME     COMPLETIONS   DURATION   AGE    CONTAINERS   IMAGES    SELECTOR
my-job   1/1           3s         102s   hello        busybox   controller-uid=c85adfa3-f52f-40f7-a772-cab087df71cf

1
2
3

$ kubectl get pod
NAME                READY   STATUS      RESTARTS   AGE
my-job-d4rnl        0/1     Completed   0          2m4s

$ kubectl describe pod my-job-d4rnl
Name:         my-job-d4rnl
Namespace:    default
Priority:     0
Node:         k8s-node4/10.160.18.184
Start Time:   Mon, 03 Aug 2020 14:05:42 +0800
Labels:       controller-uid=c85adfa3-f52f-40f7-a772-cab087df71cf
              job-name=my-job
Annotations:  
Status:       Succeeded
IP:           172.1.3.12
IPs:
  IP:           172.1.3.12
Controlled By:  Job/my-job
...
Events:
  Type    Reason     Age    From                Message
  ----    ------     ----   ----                -------
  Normal  Scheduled  2m43s  default-scheduler   Successfully assigned default/my-job-d4rnl to k8s-node4
  Normal  Pulled     2m42s  kubelet, k8s-node4  Container image "busybox" already present on machine
  Normal  Created    2m41s  kubelet, k8s-node4  Created container hello
  Normal  Started    2m41s  kubelet, k8s-node4  Started container hello
  
$ kubectl describe jobs my-job
Name:           my-job
Namespace:      default
Selector:       controller-uid=c85adfa3-f52f-40f7-a772-cab087df71cf
Labels:         controller-uid=c85adfa3-f52f-40f7-a772-cab087df71cf
                job-name=my-job
Annotations:    Parallelism:  1
Completions:    1
Start Time:     Mon, 03 Aug 2020 14:05:41 +0800
Completed At:   Mon, 03 Aug 2020 14:05:44 +0800
Duration:       3s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=c85adfa3-f52f-40f7-a772-cab087df71cf
           job-name=my-job
  Containers:
   hello:
    Image:      busybox
    Port:       
    Host Port:  
    Command:
      echo
       hello, Job
    Environment:  
    Mounts:       
  Volumes:        
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  3m56s  job-controller  Created pod: my-job-d4rnl
  Normal  Completed         3m54s  job-controller  Job completed

$ kubectl logs my-job-d4rnl
 hello, Job

分析

Job创建后，pod模版中添加了一个controller-uid为一个随机字符串的label，而job则拥有一个同样id的selector

而job完成后，进入了completed的状态

删除对应的pod

1 2	$ kubectl delete pod my-job-d4rnl pod "my-job-d4rnl" deleted

删除对应的pod后，会发现没有再创建新的pod

错误的job

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - name: hello
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["exit 1"]
      restartPolicy: Never

$ kubectl get jobs -o wide
NAME     COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES    SELECTOR
my-job   0/1           54s        54s   hello        busybox   controller-uid=9ac8a40c-1548-4683-bbd6-870a236199bf
$ kubectl describe job my-job
Name:           my-job
Namespace:      default
Selector:       controller-uid=9ac8a40c-1548-4683-bbd6-870a236199bf
Labels:         controller-uid=9ac8a40c-1548-4683-bbd6-870a236199bf
                job-name=my-job
Annotations:    Parallelism:  1
Completions:    1
Start Time:     Mon, 03 Aug 2020 14:21:59 +0800
Pods Statuses:  1 Running / 0 Succeeded / 4 Failed
Pod Template:
  Labels:  controller-uid=9ac8a40c-1548-4683-bbd6-870a236199bf
           job-name=my-job
  Containers:
   hello:
    Image:      busybox
    Port:       
    Host Port:  
    Command:
      exit 1
    Environment:  
    Mounts:       
  Volumes:        
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  77s   job-controller  Created pod: my-job-n8qf2
  Normal  SuccessfulCreate  75s   job-controller  Created pod: my-job-j484w
  Normal  SuccessfulCreate  65s   job-controller  Created pod: my-job-576nv
  Normal  SuccessfulCreate  45s   job-controller  Created pod: my-job-xdk29
  Normal  SuccessfulCreate  5s    job-controller  Created pod: my-job-ghfcv

$ kubectl get pods
NAME               READY   STATUS               RESTARTS   AGE
my-job-576nv       0/1     ContainerCannotRun   0          109s
my-job-ghfcv       0/1     ContainerCannotRun   0          49s
my-job-j484w       0/1     ContainerCannotRun   0          119s
my-job-n8qf2       0/1     ContainerCannotRun   0          2m1s
my-job-xdk29       0/1     ContainerCannotRun   0          89s

由于restartPolicy=Never，可以看到，对应的pod的RESTARTS始终为0，job-controller会不停的创建新的pod。

现在，讲restartPolicy改为OnFailure

$ kubectl describe job my-job
Name:           my-job
Namespace:      default
Selector:       controller-uid=d404af28-5d83-4943-85cf-301c2910d967
Labels:         controller-uid=d404af28-5d83-4943-85cf-301c2910d967
                job-name=my-job
Annotations:    Parallelism:  1
Completions:    1
Start Time:     Mon, 03 Aug 2020 14:28:45 +0800
Pods Statuses:  1 Running / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=d404af28-5d83-4943-85cf-301c2910d967
           job-name=my-job
  Containers:
   hello:
    Image:      busybox
    Port:       
    Host Port:  
    Command:
      exit 1
    Environment:  
    Mounts:       
  Volumes:        
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  2m50s  job-controller  Created pod: my-job-2ndzd
$ kubectl get pod
NAME             READY   STATUS             RESTARTS   AGE
my-job-2ndzd     0/1     CrashLoopBackOff   3          96s

可以看到，对应的pod在不断的重启

并行作业

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  parallelism: 2
  completions: 4
  template:
    metadata:
      labels:
        name: my-job
    spec:
      containers:
      - name: my-job
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["echo", "hello Job"]
      restartPolicy: Never

$ kubectl get pods -l name=my-job
NAME           READY   STATUS      RESTARTS   AGE
my-job-2x4lc   0/1     Completed   0          10s
my-job-lnhxx   0/1     Completed   0          7s
my-job-md4nr   0/1     Completed   0          10s
my-job-s5znf   0/1     Completed   0          7s

1
2
3

$ kubectl get jobs
NAME     COMPLETIONS   DURATION   AGE
my-job   4/4           5s         16s

可以根据pod的时间看出，my-job-2x4lc和my-job-md4nr是同一时间创建的，而另外两个pod则在其之后3s创建的。

CronJob

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            args:
            - /bin/sh
            - -c
            - date; echo Hello from Kubernetes Cluster
          restartPolicy: OnFailure

$ kubectl get cronjobs.batch
NAME    SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
hello   */1 * * * *   False     0        46s             2m39s
$ kubectl get jobs
NAME               COMPLETIONS   DURATION   AGE
hello-1596438780   1/1           2s         2m38s
hello-1596438840   1/1           1s         98s
hello-1596438900   1/1           2s         38s
$ kubectl get pods
NAME                     READY   STATUS      RESTARTS   AGE
hello-1596438720-2j5t4   0/1     Completed   0          3s
hello-1596438780-lbwnz   0/1     Completed   0          12s

$ kubectl describe pod hello-1596438960-4c5ft
Name:         hello-1596438960-4c5ft
Namespace:    default
Priority:     0
Node:         k8s-node4/10.160.18.184
Start Time:   Mon, 03 Aug 2020 15:16:03 +0800
Labels:       controller-uid=e421bffd-df9f-4342-8c3f-a852076deb6e
              job-name=hello-1596438960
Annotations:  
Status:       Succeeded
IP:           172.1.3.42
IPs:
  IP:           172.1.3.42
Controlled By:  Job/hello-1596438960

$ kubectl describe job hello-1596438960
Name:           hello-1596438960
Namespace:      default
Selector:       controller-uid=e421bffd-df9f-4342-8c3f-a852076deb6e
Labels:         controller-uid=e421bffd-df9f-4342-8c3f-a852076deb6e
                job-name=hello-1596438960
Annotations:    
Controlled By:  CronJob/hello

分析

从yaml文件中可以看到，jobTemplate的描述同Job，所以，CronJob实际上是一个job的调度器。

CronJob创建了以hello-+随机串的方式，创建了随后的多个Job，再由Job控制Pod运行

而从pod一层一层往上追溯：

Pod测controller为Job
Job的controller为CronJob

小结

本章节主要了解了一下Job和CronJob。需要理解的是：

Job控制pod，且Job仅运行一次
CronJob通过定时创建Job来运行

K8s学习笔记——DaemonSet

2020-08-25T05:51:28.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：21 | 容器化守护进程的意义：DaemonSet

nodeAffinity

原文中的nodeSelector和nodeAffinity的设置的yaml，apply之后一直处于pending状态。机智的我查看了一下flannel的pod设置，发现原文中使用了matchExpressions，而flannel的则使用了matchFields

apiVersion: v1
kind: Pod
metadata:
  name: node-affinity-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - node2
  containers:
  - name: busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true

这样，我指定了调度到node2上

1
2
3

$ kubectl get pod node-affinity-pod -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
node-affinity-pod   1/1     Running   0          23s   172.1.1.100   node2

当然，这样只是指定了node来调度，但并不惟一。比如修改上面yaml文件中的name后，再创建一个pod，同样可以创建成功

$ kubectl get pod node-affinity-pod -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
node-affinity-pod   1/1     Running   0          23s   172.1.1.100   node2              
node-affinity-pod2  1/1     Running   0          14s   172.1.1.101   node2

但如果我将node设置为node1（node1是我的环境的master节点）

apiVersion: v1
kind: Pod
metadata:
  name: node-affinity-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - node1
  containers:
  - name: busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true

1
2
3

$ kubectl get pod node-affinity-pod -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
node-affinity-pod   0/1     Pending   0          96s

$ kubectl describe pod node-affinity-pod
Name:         node-affinity-pod
Namespace:    default
Priority:     0
Node:         
Labels:       
Annotations:  Status:  Pending
...
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age               From               Message
  ----     ------            ----              ----               -------
  Warning  FailedScheduling  36s (x3 over 2m)  default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) didn't match node selector.

1 node(s) had taint {node-role.kubernetes.io/master: }，由于master节点不允许普通pod调度上去，所以，pod处于pending状态。

污点

apiVersion: v1
kind: Pod
metadata:
  name: toleration-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - node1
  tolerations:
  - key: node-role.kubernetes.io/master
    effect: NoSchedule
  containers:
  - name: busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true

改造了上面的pod，增加了对node-role.kubernetes.io/master的容忍

1
2
3

$ kubectl get pod toleration-pod -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
toleration-pod   1/1     Running   0          1s    172.1.0.89   node1

现在，pod已经被调度到了node1上

这样，就解决了master node上不能被调度的问题。同样，课程中提到了unschedulable的污点容忍。

DaemonSet 自动地给被管理的 Pod 加上了这个特殊的 Toleration，就使得这些 Pod 可以忽略这个限制，继而保证每个节点上都会被调度一个 Pod

DaemonSet

创建ds

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test-ds
spec:
  selector:
    matchLabels:
      name: my-test
  template:
    metadata:
      labels:
        name: my-test
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: my-test-busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        stdin: true
        tty: true

创建了一个test-ds的DaemonSet，在污点部分，容忍了master的污点。使其可以被调度在master节点上

查看结果

1
2
3

kubectl get ds
NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
test-ds     4         4         4       4            4                     3m7s

$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-5nxj9   1/1     Running   0          4m33s   172.1.1.103   node2                      
test-ds-bc9jx   1/1     Running   0          4m33s   172.1.2.54    bqi-k8s-node3              
test-ds-kgxm5   1/1     Running   0          4m33s   172.1.3.9     k8s-node4                  
test-ds-wvhm2   1/1     Running   0          4m33s   172.1.0.90    node1

$ kubectl describe pod test-ds-kgxm5
Name:         test-ds-kgxm5
Namespace:    default
Priority:     0
Node:         k8s-node4/10.160.18.184
Start Time:   Fri, 31 Jul 2020 11:50:29 +0800
Labels:       controller-revision-hash=7cdb9f7c5c
              name=my-test
              pod-template-generation=1
Annotations:  
Status:       Running
IP:           172.1.3.9
IPs:
  IP:           172.1.3.9
Controlled By:  DaemonSet/test-ds
...
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age    From                Message
  ----    ------     ----   ----                -------
  Normal  Scheduled  4m55s  default-scheduler   Successfully assigned default/test-ds-kgxm5 to k8s-node4
  Normal  Pulling    4m54s  kubelet, k8s-node4  Pulling image "busybox"
  Normal  Pulled     4m53s  kubelet, k8s-node4  Successfully pulled image "busybox"
  Normal  Created    4m53s  kubelet, k8s-node4  Created container my-test-busybox
  Normal  Started    4m52s  kubelet, k8s-node4  Started container my-test-busybox

可以看到，每个node上都创建了一个pod，并且Tolerations字段中，除了node-role.kubernetes.io/master:NoSchedule，还自动增加了很多污点

kill一个pod

$ kubectl delete pod test-ds-kgxm5
pod "test-ds-kgxm5" deleted
$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS    RESTARTS   AGE    IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-5nxj9   1/1     Running   0          9m9s   172.1.1.103   node2                      
test-ds-bc9jx   1/1     Running   0          9m9s   172.1.2.54    bqi-k8s-node3              
test-ds-dckg2   1/1     Running   0          5s     172.1.3.10    k8s-node4                  
test-ds-wvhm2   1/1     Running   0          9m9s   172.1.0.90    node1

更新

尝试更新镜像版本为一个错误的镜像

$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS              RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-5nxj9   1/1     Running             0          5h27m   172.1.1.103   node2                      
test-ds-dckg2   1/1     Running             0          5h18m   172.1.3.10    k8s-node4                  
test-ds-sbsbq   0/1     ContainerCreating   0          23s             bqi-k8s-node3              
test-ds-wvhm2   1/1     Running             0          5h27m   172.1.0.90    node1                      
$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS             RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-5nxj9   1/1     Running            0          5h28m   172.1.1.103   node2                      
test-ds-dckg2   1/1     Running            0          5h19m   172.1.3.10    k8s-node4                  
test-ds-sbsbq   0/1     ImagePullBackOff   0          42s     172.1.2.55    bqi-k8s-node3              
test-ds-wvhm2   1/1     Running            0          5h28m   172.1.0.90    node1

可以看到，DaemonSet的控制器会选择一个pod进行更新，当遇到更新失败时，将停止更新

$ kubectl describe pod test-ds-5nxj9
Name:         test-ds-5nxj9
Namespace:    default
Priority:     0
Node:         node2/10.160.18.181
Start Time:   Fri, 31 Jul 2020 11:50:29 +0800
Labels:       controller-revision-hash=7cdb9f7c5c
              name=my-test
              pod-template-generation=1
...
$ kubectl describe pod test-ds-dckg2
Name:         test-ds-dckg2
Namespace:    default
Priority:     0
Node:         k8s-node4/10.160.18.184
Start Time:   Fri, 31 Jul 2020 11:59:33 +0800
Labels:       controller-revision-hash=7cdb9f7c5c
              name=my-test
              pod-template-generation=1
...
$ kubectl describe pod test-ds-sbsbq
Name:         test-ds-sbsbq
Namespace:    default
Priority:     0
Node:         bqi-k8s-node3/10.160.18.183
Start Time:   Fri, 31 Jul 2020 17:18:02 +0800
Labels:       controller-revision-hash=6755d9c956
              name=my-test
              pod-template-generation=2

也可以看到，labels中：

controller-revision-hash更新为一个新的
pod-template-generation更新为2

现在，修改镜像为一个可用的镜像

$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS        RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-dckg2   1/1     Terminating   0          5h25m   172.1.3.10    k8s-node4                  
test-ds-jd99w   1/1     Running       0          47s     172.1.2.56    bqi-k8s-node3              
test-ds-nw5lk   1/1     Running       0          9s      172.1.1.104   node2                      
test-ds-wvhm2   1/1     Running       0          5h34m   172.1.0.90    node1                      
$ kubectl get pods -l name=my-test -o wide
NAME            READY   STATUS        RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
test-ds-72z5v   1/1     Running       0          40s     172.1.3.11    k8s-node4                  
test-ds-jd99w   1/1     Running       0          118s    172.1.2.56    bqi-k8s-node3              
test-ds-nw5lk   1/1     Running       0          80s     172.1.1.104   node2                      
test-ds-wvhm2   1/1     Terminating   0          5h35m   172.1.0.90    node1

可以看到，当更新成功后，对应的pod被逐个替换

$ kubectl describe pod test-ds-72z5v
Name:         test-ds-72z5v
Namespace:    default
Priority:     0
Node:         k8s-node4/10.160.18.184
Start Time:   Fri, 31 Jul 2020 17:25:23 +0800
Labels:       controller-revision-hash=86b8bf4df4
              name=my-test
              pod-template-generation=3

更新后的pod：

controller-revision-hash被更新为一个新的
pod-template-generation也增加到了3

小结

DaemonSet分别采用了遍历node来创建pod以及toleration等措施，保证了DaemonSet对应的pod在每一个node上被创建。

通过 nodeAffinity 和 Toleration 这两个调度器的小功能，保证了每个节点上有且只有一个 Pod

同时，通过controller-revision进行版本管理

K8s学习笔记——StatefulSet实践

2020-08-04T02:06:04.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：20 | 深入理解StatefulSet（三）：有状态应用实践

MySQL集群

描述

是一个“主从复制“的MySQL集群
有一个主节点和多个从节点
从节点可水平扩展
所有写操作都只能在主节点上完成
读操作可以在所有节点上完成

创建my.conf的configMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql
  labels:
    app: mysql
data:
  master.cnf: |
    [mysqld]
    log-bin
  slave.cnf: |
    [mysqld]
    super-read-only

1 2	$ kubectl get configmap mysql 2 19s

创建两个Service

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  clusterIP: None
  selector:
    app: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  selector:
    app: mysql

$ kubectl get service
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
mysql                         ClusterIP      None                     3306/TCP                        4s
mysql-read                    ClusterIP      10.105.117.44            3306/TCP                        4s

创建StatefulSet

这个yaml文件是如此的长，在不太懂的情况下手抄作业

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 3
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:5.7
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate mysql server-id from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          echo [mysqld] > /mnt/conf.d/server-id.cnf
          # Add an offset to avoid reserved server-id=0 value.
          echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
          # Copy appropriate conf.d files from config-map to emptyDir.
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/master.cnf /mnt/conf.d/
          else
            cp /mnt/config-map/slave.cnf /mnt/conf.d/
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map
      - name: clone-mysql
        image: 10.160.15.5/google_containers/xtrabackup:1.0
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Skip the clone if data already exists.
          [[ -d /var/lib/mysql/mysql ]] && exit 0
          # Skip the clone on master (ordinal index 0).
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          [[ $ordinal -eq 0 ]] && exit 0
          # Clone data from previous peer.
          ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
          # Prepare the backup.
          xtrabackup --prepare --target-dir=/var/lib/mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      containers:
      - name: mysql
        image: mysql:5.7
        env:
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "1"
        ports:
        - name: mysql
          containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
        livenessProbe:
          exec:
            command: ["mysqladmin", "ping"]
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            # Check we can execute queries over TCP (skip-networking is off).
            command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
      - name: xtrabackup
        image: 10.160.15.5/google_containers/xtrabackup:1.0
        ports:
        - name: xtrabackup
          containerPort: 3307
        command:
        - bash
        - "-c"
        - |
          set -ex
          cd /var/lib/mysql
          # Determine binlog position of cloned data, if any.
          if [[ -f xtrabackup_slave_info ]]; then
            # XtraBackup already generated a partial "CHANGE MASTER TO" query
            # because we're cloning from an existing slave.
            mv xtrabackup_slave_info change_master_to.sql.in
            # Ignore xtrabackup_binlog_info in this case (it's useless).
            rm -f xtrabackup_binlog_info
          elif [[ -f xtrabackup_binlog_info ]]; then
            # We're cloning directly from master. Parse binlog position.
            [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
            rm xtrabackup_binlog_info
            echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                  MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
          fi
          # Check if we need to complete a clone by starting replication.
          if [[ -f change_master_to.sql.in ]]; then
            echo "Waiting for mysqld to be ready (accepting connections)"
            until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
            echo "Initializing replication from clone position"
            # In case of container restart, attempt this at-most-once.
            mv change_master_to.sql.in change_master_to.sql.orig
            mysql -h 127.0.0.1 <
          $(
            MASTER_HOST='mysql-0.mysql',
            MASTER_USER='root',
            MASTER_PASSWORD='',
            MASTER_CONNECT_RETRY=10;
          START SLAVE;
          EOF
          fi
          # Start a server to send backups when requested by peers.
          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
      volumes:
      - name: conf
        emptyDir: {}
      - name: config-map
        configMap:
          name: mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "nfs-client"
      resources:
        requests:
          storage: 10Gi

检查

PV及PVC

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                 STORAGECLASS   REASON   AGE
pvc-ba5e0a68-59f6-4812-b3cc-919898c72f21   10Gi       RWO            Delete           Bound    default/data-mysql-2                                  nfs-client              116s
pvc-cd69aa2b-6e92-48da-998b-363fb3c796e9   10Gi       RWO            Delete           Bound    default/data-mysql-0                                  nfs-client              106m
pvc-f89ba84c-f87c-498a-ac19-99eca1645f4f   10Gi       RWO            Delete           Bound    default/data-mysql-1                                  nfs-client              105m

$ kubectl get pvc
NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mysql-0                                  Bound    pvc-cd69aa2b-6e92-48da-998b-363fb3c796e9   10Gi       RWO            nfs-client     106m
data-mysql-1                                  Bound    pvc-f89ba84c-f87c-498a-ac19-99eca1645f4f   10Gi       RWO            nfs-client     105m
data-mysql-2                                  Bound    pvc-ba5e0a68-59f6-4812-b3cc-919898c72f21   10Gi       RWO            nfs-client     110s

$ kubectl get pods -l app=mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          104m
mysql-1   2/2     Running   1          3m27s
mysql-2   2/2     Running   1          3m7s

$ kubectl describe pod mysql-2
...
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  3m35s                  default-scheduler  running "VolumeBinding" filter plugin for pod "mysql-2": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  3m35s                  default-scheduler  persistentvolumeclaim "data-mysql-2" not found
  Normal   Scheduled         3m32s                  default-scheduler  Successfully assigned default/mysql-2 to node1
  Normal   Pulled            3m32s                  kubelet, node1     Container image "mysql:5.7" already present on machine
  Normal   Created           3m32s                  kubelet, node1     Created container init-mysql
  Normal   Started           3m31s                  kubelet, node1     Started container init-mysql
  Normal   Started           3m30s                  kubelet, node1     Started container clone-mysql
  Normal   Pulled            3m30s                  kubelet, node1     Container image "10.160.15.5/google_containers/xtrabackup:1.0" already present on machine
  Normal   Created           3m30s                  kubelet, node1     Created container clone-mysql
  Normal   Pulled            2m55s                  kubelet, node1     Container image "10.160.15.5/google_containers/xtrabackup:1.0" already present on machine
  Normal   Created           2m55s                  kubelet, node1     Created container xtrabackup
  Normal   Started           2m55s                  kubelet, node1     Started container xtrabackup
  Normal   Pulled            2m54s (x2 over 2m55s)  kubelet, node1     Container image "mysql:5.7" already present on machine
  Normal   Created           2m54s (x2 over 2m55s)  kubelet, node1     Created container mysql
  Normal   Started           2m54s (x2 over 2m55s)  kubelet, node1     Started container mysql

service

$ kubectl get service
NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
mysql                                   ClusterIP      None                     3306/TCP                        21h
mysql-read                              ClusterIP      10.105.117.44            3306/TCP                        21h

1
2
3

$ kubectl get statefulset
NAME                   READY   AGE
mysql                  3/3     107m

验证集群

创建测试数据

$ kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\
  mysql -h mysql-0.mysql <
CREATE DATABASE test;
CREATE TABLE test.messages (message VARCHAR(250));
INSERT INTO test.messages VALUES ('hello');
EOF

If you don't see a command prompt, try pressing enter.
pod "mysql-client" deleted

读取测试数据

$ kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read -e "SELECT * FROM test.messages"
If you don't see a command prompt, try pressing enter.
+---------+
| message |
+---------+
| hello   |
+---------+
pod "mysql-client" deleted

扩容mysql集群

修改replicas为5

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 5

pods

$ kubectl get pods -l app=mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          4h51m
mysql-1   2/2     Running   1          3h10m
mysql-2   2/2     Running   1          3h10m
mysql-3   2/2     Running   1          119s
mysql-4   2/2     Running   0          71s

获取数据

$ kubectl run mysql-client --image=mysql:5.7 -it --rm --restart=Never -- mysql -h mysql-4.mysql -e "SELECT * FROM test.messages"
+---------+
| message |
+---------+
| hello   |
+---------+
pod "mysql-client" deleted

尝试写入

$ kubectl run mysql-client --image=mysql:5.7 -it --rm --restart=Never -- mysql -h mysql-4.mysql -e "INSERT INTO test.messages VALUES ('test')"
ERROR 1290 (HY000) at line 1: The MySQL server is running with the --super-read-only option so it cannot execute this statement
pod "mysql-client" deleted
pod default/mysql-client terminated (Error)

分析

实验伴随着解决各种问题完成了，现在再回头来总结一下我理解的YAML文件的内容

init_containers中定义了两个container，分别是mysql和xtrabackup
1. mysql的container根据hostname进行判断，是初始化为master还是slave
2. xtrabackup的container，如果不为master，则根据数据是否存在决定是否要拷贝数据
containers中也同样定义了两个container，也同样是mysql和xtrabackup
1. mysql的container直接使用init_container中创建的数据，启动mysql服务
2. xtrabackup的container则开启3307用来为其他的slave提供拷贝

更新

修改镜像版本为10.160.15.5/pub/mysql:5.7

由于新镜像实际上和原来的镜像是同一个md5，不会出现其他问题。

$ kubectl get pods -l app=mysql
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   2/2     Running       0          26h
mysql-1   2/2     Running       1          24h
mysql-2   2/2     Running       1          24h
mysql-3   2/2     Running       1          21h
mysql-4   2/2     Terminating   0          46s
$ kubectl get pods -l app=mysql
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   2/2     Running       0          26h
mysql-1   2/2     Running       1          24h
mysql-2   2/2     Running       1          24h
mysql-3   2/2     Terminating   1          21h
mysql-4   2/2     Running       0          46s
$ kubectl get pods -l app=mysql
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   2/2     Running       0          26h
mysql-1   2/2     Terminating   1          25h
mysql-2   2/2     Running       0          17s
mysql-3   2/2     Running       0          68s
mysql-4   2/2     Running       0          2m19s
$ kubectl get pods -l app=mysql
NAME      READY   STATUS     RESTARTS   AGE
mysql-0   0/2     Init:0/2   0          1s
mysql-1   2/2     Running    0          47s
mysql-2   2/2     Running    0          90s
mysql-3   2/2     Running    0          2m21s
mysql-4   2/2     Running    0          3m32s

可以看到，StatefulSet更新后，会先将最后一个pod进行更新，成功后，会更新mysql-3，依次类推，直到更新到mysql-0

小结

StatefulSet 其实是一种特殊的 Deployment，只不过这个“Deployment”的每个 Pod 实例的名字里，都携带了一个唯一并且固定的编号。这个编号的顺序，固定了 Pod 的拓扑关系；这个编号对应的 DNS 记录，固定了 Pod 的访问方式；这个编号对应的 PV，绑定了 Pod 与持久化存储的关系。所以，当 Pod 被删除重建时，这些“状态”都会保持不变。

可以看到，由于pod名称固定，PV以及PVC固定，即使删掉一个pod，重建后，仍然会创建一个与其名字相同的pod并使用原有的PV及PVC。从而对应的pod启动后，内容一致

然而，由于将master和slave写在一个StatefulSet，感觉实现的非常复杂。

K8s学习笔记——StatefulSet之存储状态

2020-08-04T01:51:32.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：19 | 深入理解StatefulSet（二）：存储状态

注：本节实验中用到的StorageClassName依赖于另一篇博文中的使用NFS

创建StatefulSet

创建带volumeClaim的StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ss-ss
spec:
  serviceName: "ss-ss-nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      storageClassName: nfs-client
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

$ kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
ss-ss-0                                                  1/1     Running   0          4s
ss-ss-1                                                  0/1     Pending   0          0s
$ kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
ss-ss-0                                                  1/1     Running   0          23s
ss-ss-1                                                  1/1     Running   0          19s

查看PV及PVC

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-1a215f93-37df-4278-9723-51d365c4cf96   1Gi        RWO            Delete           Bound    default/www-ss-ss-0   nfs-client              56s
pvc-f07c6d74-6eee-4930-9ad6-20fdb4faa5d5   1Gi        RWO            Delete           Bound    default/www-ss-ss-1   nfs-client              52s
$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-ss-ss-0   Bound    pvc-1a215f93-37df-4278-9723-51d365c4cf96   1Gi        RWO            nfs-client     59s
www-ss-ss-1   Bound    pvc-f07c6d74-6eee-4930-9ad6-20fdb4faa5d5   1Gi        RWO            nfs-client     55s

查看nfs server：

1
2
3

$ ls
default-www-ss-ss-0-pvc-1a215f93-37df-4278-9723-51d365c4cf96
default-www-ss-ss-1-pvc-f07c6d74-6eee-4930-9ad6-20fdb4faa5d5

在各个pod上创建文件

$ for i in 0 1; do kubectl exec ss-ss-$i -- sh -c 'echo hello $(hostname) > /usr/share/nginx/html/index.html'; done
$ for i in 0 1; do kubectl exec -it ss-ss-$i -- curl localhost; done
hello ss-ss-0
hello ss-ss-1

删除对应的pod

$ kubectl get pods -l app=nginx
NAME      READY   STATUS    RESTARTS   AGE
ss-ss-0   1/1     Running   0          10m
ss-ss-1   1/1     Running   0          10m
$ kubectl delete pod -l app=nginx
pod "ss-ss-0" deleted
pod "ss-ss-1" deleted

查看pvc

$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-ss-ss-0   Bound    pvc-1a215f93-37df-4278-9723-51d365c4cf96   1Gi        RWO            nfs-client     13m
www-ss-ss-1   Bound    pvc-f07c6d74-6eee-4930-9ad6-20fdb4faa5d5   1Gi        RWO            nfs-client     13m

可以看到，pod删除，但pvc还存在

$ kubectl get pod -l app=nginx
NAME      READY   STATUS    RESTARTS   AGE
ss-ss-0   1/1     Running   0          4m21s
ss-ss-1   1/1     Running   0          4m19s

随后pod被重建，再来curl看看

1
2
3

$ for i in 0 1; do kubectl exec -it ss-ss-$i -- curl localhost; done
hello ss-ss-0
hello ss-ss-1

由此：

当删除一个pod时，这个 Pod 对应的 PVC 和 PV，并不会被删除，而这个 Volume 里已经写入的数据，也会保存在远程存储服务里
控制器就会重新创建一个新的、名字还是叫作 ss-ss-0 的 Pod 来，“纠正”这个不一致的情况
在新的 ss-ss-0 Pod 被创建出来之后，Kubernetes 为它查找名叫 web-ss-ss-0 的 PVC 时，就会直接找到旧 Pod 遗留下来的同名的 PVC，进而找到跟这个 PVC 绑定在一起的 PV

那么，如果删除了PVC呢？

$ kubectl delete pvc www-ss-ss-0
persistentvolumeclaim "www-ss-ss-0" deleted
$ kubectl delete pvc www-ss-ss-1
persistentvolumeclaim "www-ss-ss-1" deleted
$ kubectl get pvc
No resources found in default namespace.
$ kubectl get pv
No resources found in default namespace.

可以看到，删除PVC后，对应的PV也同样被删除了

而nfs服务器端：

$ ls
archived-default-www-ss-ss-0-pvc-1a215f93-37df-4278-9723-51d365c4cf96   archived-default-www-ss-ss-1-pvc-f07c6d74-6eee-4930-9ad6-20fdb4faa5d5
$ ls archived-default-www-ss-ss-0-pvc-1a215f93-37df-4278-9723-51d365c4cf96/
index.html
$ cat archived-default-www-ss-ss-0-pvc-1a215f93-37df-4278-9723-51d365c4cf96/index.html
hello ss-ss-0

服务器端的目录变成了archived开头的目录，但内容依然存在，以备后续查看。

小结

首先，StatefulSet 的控制器直接管理的是 Pod
其次，Kubernetes 通过 Headless Service，为这些有编号的 Pod，在 DNS 服务器中生成带有同样编号的 DNS 记录
最后，StatefulSet 还为每一个 Pod 分配并创建一个同样编号的 PVC

StatefulSet 其实就是一种特殊的 Deployment，而其独特之处在于，它的每个 Pod 都被编号了

K8s学习笔记——StatefulSet之拓扑状态

2020-07-29T09:02:29.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：18 | 深入理解StatefulSet（一）：拓扑状态

Headless Service

创建service

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx

1
2
3

$ kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
nginx        ClusterIP   None                 80/TCP    129m

创建StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: web

$ kubectl apply -f statefulset.yaml
statefulset.apps/web created
$ kubectl get pods -w -l app=nginx
NAME    READY   STATUS              RESTARTS   AGE
web-0   1/1     Running             0          2s
web-1   0/1     ContainerCreating   0          1s
web-1   1/1     Running             0          2s
$ kubectl get statefulset -o wide
NAME   READY   AGE   CONTAINERS   IMAGES
web    2/2     52s   nginx        nginx
$ kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          85s   172.1.1.199   node2                      
web-1   1/1     Running   0          81s   172.1.2.180   bqi-k8s-node3

可以看出，statefulset创建出来的pod，名字是固定的，而不再像deployment中的，pod名字会变化

验证

$ kubectl exec web-0 -- sh -c 'hostname'
web-0
$ kubectl exec web-1 -- sh -c 'hostname'
web-1

$ kubectl run -i --tty --image busybox:1.28.4 web-test --restart=Never --rm -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 172.1.1.199 web-0.nginx.default.svc.cluster.local
/ # nslookup web-1.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-1.nginx
Address 1: 172.1.2.180 web-1.nginx.default.svc.cluster.local
/ #

重建pod

$ kubectl delete pod -l app=nginx
pod "web-0" deleted
pod "web-1" deleted
# 快速执行
$ kubectl get pod -w -l app=nginx
NAME    READY   STATUS              RESTARTS   AGE
web-0   0/1     ContainerCreating   0          0s
web-0   1/1     Running             0          1s
web-1   0/1     Pending             0          0s
web-1   0/1     Pending             0          0s
web-1   0/1     ContainerCreating   0          0s
web-1   1/1     Running             0          3s
$  kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          85s   172.1.1.201   node2                      
web-1   1/1     Running   0          81s   172.1.2.181   bqi-k8s-node3

可以看到，当删除了相关的pod后，kubernetes会以同样的名字重建pod

$ kubectl run -i --tty --image busybox:1.28.4 connect-test --restart=Never --rm -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 172.1.1.201 web-0.nginx.default.svc.cluster.local
/ # nslookup web-1.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-1.nginx
Address 1: 172.1.2.181 web-1.nginx.default.svc.cluster.local
/ #

重建后，依然可以访问web-0.nginx，但IP地址发生了变化

小结

本节课程主要学习了StatefulSet这个控制器，K8s在按照pod模版进行创建时：

对pod进行编号并逐一完成创建工作
当重建发生时，也会按照编号对pod逐一进行操作
通过headless service，StatefulSet 为每个 Pod 创建了一个固定并且稳定的 DNS 记录

K8s学习笔记——作业副本与水平扩展

2020-07-29T08:51:05.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：17 | 经典PaaS的记忆：作业副本与水平扩展

ReplicaSet

书接上文中，创建了一个nginx-dp的deployment

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-67f857c57f-4wptz   1/1     Running   0          7m34s
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          25m

先来查看下ReplicaSet

1
2
3

$ kubectl get replicasets
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-67f857c57f   2         2         2       39m

由此可见，实际上当我们创建一个deployment的时候，k8s为我们创建了一个名为nginx-dp-67f857c57f的ReplicaSet。是不是觉得这个名字中的ID这么熟悉。正是Pod的labels字段中的那个ID

创建一个ReplicaSet

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: test-rs
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Never

$ kubectl apply -f rs-test.yaml
replicaset.apps/rs-test created
$ kubectl get pods
NAME                        READY   STATUS        RESTARTS   AGE
nginx-dp-67f857c57f-4wptz   1/1     Running       0          33m
nginx-dp-67f857c57f-qx7l7   1/1     Running       0          51m
rs-test-8zd6q               0/1     Terminating   0          4s
rs-test-f5vvr               0/1     Terminating   0          4s
rs-test-rkh7r               0/1     Terminating   0          4s
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-67f857c57f   2         2         2       51m
rs-test               0         0         0       14s

WHAT?!

居然创建失败，所谓在作死的道路上不停试探。谁让我用了和上节课中的Deployment一样的labels呢

再来一遍：

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: test-rs
  labels:
    app: nginx-new
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-new
  template:
    metadata:
      labels:
        app: nginx-new
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Never

$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-67f857c57f   2         2         2       54m
test-rs               3         3         3       7m30s
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-67f857c57f-4wptz   1/1     Running   0          37m
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          54m
test-rs-9wfx8               1/1     Running   0          7m33s
test-rs-qbckm               1/1     Running   0          7m33s
test-rs-rxjbv               1/1     Running   0          7m33s

现在看起来，创建出了一个名为test-rs的ReplicaSet，而test-rs的rs又创建出了三个pod

水平收缩

修改test-rs的RS：replicas: 3为replicas: 4

$ kubectl apply -f rs-test.yaml
replicaset.apps/test-rs configured
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
test-rs-9wfx8               1/1     Running   0          37m
test-rs-dwm9j               1/1     Running   0          4s
test-rs-qbckm               1/1     Running   0          37m
test-rs-rxjbv               1/1     Running   0          37m

修改为4个后，自动创建了一个新的test-rs-dwm9j的pod

再次修改为replicas: 2

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
test-rs-9wfx8               1/1     Running   0          41m
test-rs-rxjbv               1/1     Running   0          41m
$ kubectl describe rs test-rs
Name:         test-rs
Namespace:    default
Selector:     app=nginx-new
Labels:       app=nginx-new
Annotations:  Replicas:  2 current / 2 desired
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
...
Events:
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  41m    replicaset-controller  Created pod: test-rs-qbckm
  Normal  SuccessfulCreate  41m    replicaset-controller  Created pod: test-rs-9wfx8
  Normal  SuccessfulCreate  41m    replicaset-controller  Created pod: test-rs-rxjbv
  Normal  SuccessfulCreate  4m41s  replicaset-controller  Created pod: test-rs-dwm9j
  Normal  SuccessfulDelete  16s    replicaset-controller  Deleted pod: test-rs-dwm9j
  Normal  SuccessfulDelete  16s    replicaset-controller  Deleted pod: test-rs-qbckm

查看ReplicaSet的Events可以看出来：

修改为4的时候，创建了一个新的pod
修改为2的时候，两个pod被删除

滚动更新

创建Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dp
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

$ kubectl apply -f nginx-dp.yaml --record
deployment.apps/nginx-dp created
# 注：执行完立即执行rollout才能看到
$ kubectl rollout status deployment/nginx-dp
Waiting for deployment "nginx-dp" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "nginx-dp" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "nginx-dp" rollout to finish: 2 of 3 updated replicas are available...
deployment "nginx-dp" successfully rolled out
$ kubectl get rs
NAME                 DESIRED   CURRENT   READY   AGE
nginx-dp-d46f5678b   3         3         3       3m41s
$ kubectl get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-dp   3/3     3            3           3m45s

可以看到:

ReplicaSet的状态有：Desired, Current, Ready
Deployment的有： Ready, Available, UP-to-date

注：与课程中的不一样

修改image版本

$ kubectl edit deployment/nginx-dp
....
    spec:
      containers:
      - image: nginx:stable
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP

$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-7fb9ff5685   1         1         0       41s
nginx-dp-d46f5678b    3         3         3       8m2s
$ kubectl rollout status deployment/nginx-dp
Waiting for deployment "nginx-dp" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-dp" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-dp" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-dp" successfully rolled out
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-7fb9ff5685   3         3         3       10m
nginx-dp-d46f5678b    0         0         0       17m
$ kubectl describe deploy nginx-dp
...
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
...
OldReplicaSets:  
NewReplicaSet:   nginx-dp-7fb9ff5685 (3/3 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  18m    deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 3
  Normal  ScalingReplicaSet  11m    deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 1
  Normal  ScalingReplicaSet  7m22s  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 2
  Normal  ScalingReplicaSet  7m22s  deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 2
  Normal  ScalingReplicaSet  3m34s  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 1
  Normal  ScalingReplicaSet  3m34s  deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 3
  Normal  ScalingReplicaSet  3m17s  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 0

从上面的各种结果可以看出，当修改了deployment中的image后：

k8s创建了一个新的ReplicaSet: nginx-dp-7fb9ff5685
逐个增加新的ReplicaSet，然后逐个减少旧的ReplicaSet
当滚动更新完毕后，旧的RS nginx-dp-d46f5678b数量变为0，但仍旧保留，没有自动删除
Deployment的NewReplicaSet字段变成了nginx-dp-7fb9ff5685

修改为一个错误的image

$ kubectl set image deployment/nginx-dp nginx=nginx:1.91
deployment.apps/nginx-dp image updated
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   1         1         0       6s
nginx-dp-7fb9ff5685   3         3         3       24m
nginx-dp-d46f5678b    0         0         0       31m
$ kubectl get pods
NAME                        READY   STATUS         RESTARTS   AGE
nginx-dp-6d6678fb55-4xxpv   0/1     ErrImagePull   0          72s
nginx-dp-7fb9ff5685-25xlv   1/1     Running        0          25m
nginx-dp-7fb9ff5685-jxvt8   1/1     Running        0          17m
nginx-dp-7fb9ff5685-ld64t   1/1     Running        0          21m
$ kubectl describe deployment nginx-dp
...
OldReplicaSets:  nginx-dp-7fb9ff5685 (3/3 replicas created)
NewReplicaSet:   nginx-dp-6d6678fb55 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  33m   deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 3
  Normal  ScalingReplicaSet  25m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 1
  Normal  ScalingReplicaSet  21m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 2
  Normal  ScalingReplicaSet  21m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 2
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 1
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 3
  Normal  ScalingReplicaSet  17m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 0
  Normal  ScalingReplicaSet  98s   deployment-controller  Scaled up replica set nginx-dp-6d6678fb55 to 1

可以看出：

再次创建了一个新的RS: nginx-dp-6d6678fb55
创建了一个名为nginx-dp-6d6678fb55-4xxpv的pod，但状态为ErrImagePull
Deployment的OldReplicaSets为原来的nginx-dp-7fb9ff5685，NewReplicaSet为新的nginx-dp-6d6678fb55

Rollout

$ kubectl rollout undo deployment/nginx-dp
deployment.apps/nginx-dp rolled back
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       6m49s
nginx-dp-7fb9ff5685   3         3         3       30m
nginx-dp-d46f5678b    0         0         0       38m
$ kubectl describe deployment/nginx-dp
...
OldReplicaSets:  
NewReplicaSet:   nginx-dp-7fb9ff5685 (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  38m   deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 3
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 1
  Normal  ScalingReplicaSet  27m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 2
  Normal  ScalingReplicaSet  27m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 2
  Normal  ScalingReplicaSet  23m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 1
  Normal  ScalingReplicaSet  23m   deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 3
  Normal  ScalingReplicaSet  23m   deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 0
  Normal  ScalingReplicaSet  7m2s  deployment-controller  Scaled up replica set nginx-dp-6d6678fb55 to 1
  Normal  ScalingReplicaSet  19s   deployment-controller  Scaled down replica set nginx-dp-6d6678fb55 to 0
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-7fb9ff5685-25xlv   1/1     Running   0          40m
nginx-dp-7fb9ff5685-jxvt8   1/1     Running   0          33m
nginx-dp-7fb9ff5685-ld64t   1/1     Running   0          36m

可以看出：

nginx-dp-6d6678fb55的RS的期望值变为了0
对应的pod被自动删除

history

$ kubectl rollout history deployment/nginx-dp
deployment.apps/nginx-dp
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-dp.yaml --record=true
3         kubectl apply --filename=nginx-dp.yaml --record=true
4         kubectl apply --filename=nginx-dp.yaml --record=true

$ kubectl rollout history deployment/nginx-dp --revision=4
deployment.apps/nginx-dp with revision #4
Pod Template:
  Labels:   app=nginx
    pod-template-hash=7fb9ff5685
  Annotations:  kubernetes.io/change-cause: kubectl apply --filename=nginx-dp.yaml --record=true
  Containers:
   nginx:
    Image:  nginx:stable
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    
    Mounts: 
  Volumes:

注：新版本的命令输出已经和课程中的不一样了

现在可以使用revision字段回滚到镜像为nginx:latest

$ kubectl rollout history deployment/nginx-dp --revision=1
deployment.apps/nginx-dp with revision #1
Pod Template:
  Labels:   app=nginx
    pod-template-hash=d46f5678b
  Annotations:  kubernetes.io/change-cause: kubectl apply --filename=nginx-dp.yaml --record=true
  Containers:
   nginx:
    Image:  nginx
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    
    Mounts: 
  Volumes:  
$ kubectl rollout undo deployment/nginx-dp --to-revision=1
deployment.apps/nginx-dp rolled back
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       29m
nginx-dp-7fb9ff5685   3         3         3       53m
nginx-dp-d46f5678b    1         1         0       60m
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       29m
nginx-dp-7fb9ff5685   1         1         1       53m
nginx-dp-d46f5678b    3         3         2       60m
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       29m
nginx-dp-7fb9ff5685   0         0         0       53m
nginx-dp-d46f5678b    3         3         3       60m

$ kubectl rollout history deployment/nginx-dp
deployment.apps/nginx-dp
REVISION  CHANGE-CAUSE
3         kubectl apply --filename=nginx-dp.yaml --record=true
4         kubectl apply --filename=nginx-dp.yaml --record=true
5         kubectl apply --filename=nginx-dp.yaml --record=true

$ kubectl rollout history deployment/nginx-dp --revision=5
deployment.apps/nginx-dp with revision #5
Pod Template:
  Labels:   app=nginx
    pod-template-hash=d46f5678b
  Annotations:  kubernetes.io/change-cause: kubectl apply --filename=nginx-dp.yaml --record=true
  Containers:
   nginx:
    Image:  nginx
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    
    Mounts: 
  Volumes:

可以看到，回滚之后：

原来的revision 1已经不存在了
出现了一个revision = 5
revision 5和原来的revision 1的内容一致

所以，虽然是回滚到某个版本，但revision的号依然是增加了的。是通过滚动升级的方式实现了回滚的操作

暂停与恢复

现在，先暂停

1 2	$ kubectl rollout pause deployment/nginx-dp deployment.apps/nginx-dp paused

修改image版本

$ kubectl set image deployment/nginx-dp nginx=nginx:1.18
deployment.apps/nginx-dp image updated
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       37m
nginx-dp-7fb9ff5685   0         0         0       61m
nginx-dp-d46f5678b    3         3         3       68m
$ kubectl describe deployment/nginx-dp
...
OldReplicaSets:  nginx-dp-d46f5678b (3/3 replicas created)
NewReplicaSet:   
Events:
  Type    Reason             Age                  From                   Message
  ----    ------             ----                 ----                   -------
  Normal  ScalingReplicaSet  57m                  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 2
  Normal  ScalingReplicaSet  57m                  deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 2
  Normal  ScalingReplicaSet  54m                  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 1
  Normal  ScalingReplicaSet  54m                  deployment-controller  Scaled up replica set nginx-dp-7fb9ff5685 to 3
  Normal  ScalingReplicaSet  53m                  deployment-controller  Scaled down replica set nginx-dp-d46f5678b to 0
  Normal  ScalingReplicaSet  37m                  deployment-controller  Scaled up replica set nginx-dp-6d6678fb55 to 1
  Normal  ScalingReplicaSet  30m                  deployment-controller  Scaled down replica set nginx-dp-6d6678fb55 to 0
  Normal  ScalingReplicaSet  8m32s                deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 1
  Normal  ScalingReplicaSet  8m26s                deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 2
  Normal  ScalingReplicaSet  8m26s                deployment-controller  Scaled down replica set nginx-dp-7fb9ff5685 to 2
  Normal  ScalingReplicaSet  8m22s (x2 over 69m)  deployment-controller  Scaled up replica set nginx-dp-d46f5678b to 3
  Normal  ScalingReplicaSet  8m22s                deployment-controller  Scaled down replica set nginx-dp-7fb9ff5685 to 1
  Normal  ScalingReplicaSet  8m17s                deployment-controller  Scaled down replica set nginx-dp-7fb9ff5685 to 0

可以看到：

修改了image为nginx:1.18后，没有创建出新的RS
但：Deployment的OldReplicaSet变成了nginx-dp-d46f5678b，而NewReplicaSet变成了，这与最早之前看到的不一样

如果在暂停状态下将image再设置回nginx:latest会怎样呢

$ kubectl set image deployment/nginx-dp nginx=nginx
deployment.apps/nginx-dp image updated
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       44m
nginx-dp-7fb9ff5685   0         0         0       68m
nginx-dp-d46f5678b    3         3         3       76m
# 取消暂停
$ kubectl rollout resume deployment/nginx-dp
deployment.apps/nginx-dp resumed
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       46m
nginx-dp-7fb9ff5685   0         0         0       70m
nginx-dp-d46f5678b    3         3         3       77m
$ kubectl rollout history deployment/nginx-dp
deployment.apps/nginx-dp
REVISION  CHANGE-CAUSE
3         kubectl apply --filename=nginx-dp.yaml --record=true
4         kubectl apply --filename=nginx-dp.yaml --record=true
5         kubectl apply --filename=nginx-dp.yaml --record=true
$ kubectl describe deployment/nginx-dp
...
OldReplicaSets:  
NewReplicaSet:   nginx-dp-d46f5678b (3/3 replicas created)
...

可以看到，修改回暂停前的镜像后，没有生成新的revision

继续暂停的实验

# 暂停
$ kubectl rollout pause deployment/nginx-dp
deployment.apps/nginx-dp paused
# 修改
$ kubectl set image deployment/nginx-dp nginx=nginx:1.18
deployment.apps/nginx-dp image updated
# 恢复
$ kubectl rollout resume deployment nginx-dp
deployment.apps/nginx-dp resumed
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       53m
nginx-dp-7f6cd547bd   1         1         0       3s
nginx-dp-7fb9ff5685   0         0         0       77m
nginx-dp-d46f5678b    3         3         3       85m
$ kubectl rollout history deployment/nginx-dp
deployment.apps/nginx-dp
REVISION  CHANGE-CAUSE
3         kubectl apply --filename=nginx-dp.yaml --record=true
4         kubectl apply --filename=nginx-dp.yaml --record=true
5         kubectl apply --filename=nginx-dp.yaml --record=true
6         kubectl apply --filename=nginx-dp.yaml --record=true
$ kubectl get rs
NAME                  DESIRED   CURRENT   READY   AGE
nginx-dp-6d6678fb55   0         0         0       55m
nginx-dp-7f6cd547bd   3         3         3       90s
nginx-dp-7fb9ff5685   0         0         0       79m
nginx-dp-d46f5678b    0         0         0       86m
$ kubectl rollout history deployment/nginx-dp --revision=6
deployment.apps/nginx-dp with revision #6
Pod Template:
  Labels:   app=nginx
    pod-template-hash=7f6cd547bd
  Annotations:  kubernetes.io/change-cause: kubectl apply --filename=nginx-dp.yaml --record=true
  Containers:
   nginx:
    Image:  nginx:1.18
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    
    Mounts: 
  Volumes:

可以看出，当取消暂停后，对应的动作按照滚动更新的方式开始执行。

小结

本节课程主要学习了：

水平扩展时，实际上是修改了ReplicaSet的期望值，并按照期望值去逐步拉起pod
而滚动更新时，会创建新的ReplicaSet，并通过逐个增加新的，而逐个减少旧的的方式来逐步替换pod

最后，一个对原文的引用：

Deployment 实际上是一个两层控制器。首先，它通过 ReplicaSet 的个数来描述应用的版本；然后，它再通过 ReplicaSet 的属性（比如 replicas 的值），来保证 Pod 的副本数量。

K8s使用NFS跨节点数据持久化

2020-07-21T07:35:34.000Z

《深入剖析Kubernetes》中使用了Rook来搭建存储，而K8s本身也支持直接使用NFS。本次实验主要为后续学习做铺垫，使用现有NFS存储做为跨节点数据持久化，并简单的学习其实现原理。

安装NFS客户端

分别在各个node上安装

1	$ apt install nfs-common

测试

1	$ mount -t nfs 10.160.12.7:/data/share /mnt

NFS的PV及PVC测试

新建PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs-client
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/share/bqi
    server: 10.160.12.7

1
2
3

$ kubectl get pv
NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
nfs-pv   100Gi      RWO            Recycle          Bound    default/pvc-nfs-test   nfs-client              28m

新建PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: nfs-client
  resources:
    requests:
      storage: 1Gi

注：storageClassName同PV的storageClassName

1
2
3

$ kubectl get pvc
NAME           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-nfs-test   Bound    nfs-pv   100Gi      RWO            nfs-client     7m41s

测试PVC

测试思路：

创建一个pod，使用pvc
在挂载目录下创建文件
再次创建一个pod，挂载pvc，检查其中文件
删除两个pod，重新创建一个使用pvc的pod，然后检查数据
同步检查跨节点时的状况

1. 创建一个pod，使用pvc

apiVersion: v1
kind: Pod
metadata:
  name: pvc-test-pod1
spec:
  containers:
  - name: pvc-busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true
    volumeMounts:
    - name: pvc
      mountPath: /mnt
  volumes:
  - name: pvc
    persistentVolumeClaim:
      claimName: pvc-nfs-test

1
2
3

$ kubectl get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
pvc-test-pod1   1/1     Running   0          83s   172.1.1.229   node2

2. 在挂载目录下创建文件

1 2	$ kubectl exec -it pvc-test-pod1 -- /bin/sh / # echo "hello nfs pvc" > /mnt/test.txt

3. 创建第二个pod

apiVersion: v1
kind: Pod
metadata:
  name: pvc-test-pod2
spec:
  containers:
  - name: pvc-busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true
    volumeMounts:
    - name: pvc
      mountPath: /mnt
  volumes:
  - name: pvc
    persistentVolumeClaim:
      claimName: pvc-nfs-test

$ kubectl get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
pvc-test-pod1   1/1     Running   0          5m26s   172.1.1.229   node2                      
pvc-test-pod2   1/1     Running   0          2s      172.1.1.231   node2

$ kubectl exec -it pvc-test-pod2 -- ls /mnt/
test.txt
$ kubectl exec -it pvc-test-pod2 -- cat /mnt/test.txt
hello nfs pvc

4. 删除原来的pod并重建新的pod

$ kubectl delete pod pvc-test-pod1
pod "pvc-test-pod1" deleted
$ kubectl delete pod pvc-test-pod2
pod "pvc-test-pod2" deleted

apiVersion: v1
kind: Pod
metadata:
  name: pvc-test-pod3
spec:
  containers:
  - name: pvc-busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true
    volumeMounts:
    - name: pvc
      mountPath: /mnt
  volumes:
  - name: pvc
    persistentVolumeClaim:
      claimName: pvc-nfs-test

1
2
3

$ kubectl get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
pvc-test-pod3   1/1     Running   0          7s    172.1.1.232   node2

$ kubectl exec -it pvc-test-pod3 -- ls /mnt
test.txt
$ kubectl exec -it pvc-test-pod3 -- cat /mnt/test.txt
hello nfs pvc

5. 测试跨节点

由于总是调度在node2上，不得已，使用deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-pvc-dp
  labels:
    app: busybox
spec:
  replicas: 5
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        stdin: true
        tty: true
        volumeMounts:
        - name: pvc
          mountPath: /mnt
      volumes:
      - name: pvc
        persistentVolumeClaim:
          claimName: pvc-nfs-test

$ kubectl get pods -o wide
NAME                          READY   STATUS    RESTARTS   AGE    IP            NODE            NOMINATED NODE   READINESS GATES
nfs-pvc-dp-559d75db66-dg6th   1/1     Running   0          119s   172.1.1.233   node2                      
nfs-pvc-dp-559d75db66-kwrnk   1/1     Running   0          119s   172.1.2.210   bqi-k8s-node3              
nfs-pvc-dp-559d75db66-qk4pr   1/1     Running   0          119s   172.1.2.209   bqi-k8s-node3              
nfs-pvc-dp-559d75db66-z224z   1/1     Running   0          119s   172.1.1.234   node2                      
nfs-pvc-dp-559d75db66-z4xs5   1/1     Running   0          119s   172.1.2.211   bqi-k8s-node3

$ kubectl exec -it nfs-pvc-dp-559d75db66-dg6th -- cat /mnt/test.txt
hello nfs pvc
$ kubectl exec -it nfs-pvc-dp-559d75db66-kwrnk -- cat /mnt/test.txt
hello nfs pvc

看看底层

可以使用Linux的mount命令查看两个worker节点上的mount状况

# node2
$ mount | grep 10.160.12.7
10.160.12.7:/data/share/bqi on /var/lib/kubelet/pods/d2375620-7164-468e-a212-0f73042983d7/volumes/kubernetes.io~nfs/nfs-pv type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.160.18.181,local_lock=none,addr=10.160.12.7)
10.160.12.7:/data/share/bqi on /var/lib/kubelet/pods/015ee988-c22e-4a8e-aa9d-7a85d0dd6045/volumes/kubernetes.io~nfs/nfs-pv type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.160.18.181,local_lock=none,addr=10.160.12.7)

$ mount | grep 10.160.12.7
10.160.12.7:/data/share/bqi on /var/lib/kubelet/pods/2ff007b3-6568-4b90-9c76-2c994c18f71c/volumes/kubernetes.io~nfs/nfs-pv type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.160.18.183,local_lock=none,addr=10.160.12.7)
10.160.12.7:/data/share/bqi on /var/lib/kubelet/pods/aa545d93-0178-4adc-87b5-df27ac070a2e/volumes/kubernetes.io~nfs/nfs-pv type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.160.18.183,local_lock=none,addr=10.160.12.7)
10.160.12.7:/data/share/bqi on /var/lib/kubelet/pods/2098028e-8be5-44fd-b5d8-b358baf40791/volumes/kubernetes.io~nfs/nfs-pv type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.160.18.183,local_lock=none,addr=10.160.12.7)

可以看出：10.160.12.7:/data/share/bqi被分别挂载到了对应的pod的目录下

自动创建PV及PVC

当我以为这样就完了的时候，用着用着，我发现，一个PV只能被一个PVC绑定。所以，实际上之前的操作，都局限在将一个nfs的目录变成了一个PV，然后绑定到了PVC，最后挂载到了某个pod上。

而后续实验中发现，存储状态的实验需要不同的pod用不同的PVC。那我不可能总是手工去创建一个PV。于是，需要用到nfs-client-provisioner

nfs-client-provisioner

nfs-client-provisioner可以利用NFS Server给Kubernetes作为持久存储的后端，并且动态提供PV。可以参考https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client

安装方式也有多种，这里采用helm方式安装，非常简单

1	$ helm install --set nfs.server=10.160.12.7 --set nfs.path=/data/share/bqi stable/nfs-client-provisioner

参数说明：

nfs.server - nfs服务器的地址
nfs.path - nfs共享出来的目录

1
2
3

$ helm list
NAME              REVISION    UPDATED                     STATUS      CHART                           APP VERSION    NAMESPACE
crusty-penguin    1           Wed Jul 22 19:07:19 2020    DEPLOYED    nfs-client-provisioner-1.2.8    3.1.0          default

$ helm status crusty-penguin
LAST DEPLOYED: Wed Jul 22 19:07:19 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                          CREATED AT
crusty-penguin-nfs-client-provisioner-runner  2020-07-22T11:07:19Z

==> v1/ClusterRoleBinding
NAME                                       ROLE                                                      AGE
run-crusty-penguin-nfs-client-provisioner  ClusterRole/crusty-penguin-nfs-client-provisioner-runner  15h

==> v1/Deployment
NAME                                   READY  UP-TO-DATE  AVAILABLE  AGE
crusty-penguin-nfs-client-provisioner  1/1    1           1          15h

==> v1/Pod(related)
NAME                                                    READY  STATUS   RESTARTS  AGE
crusty-penguin-nfs-client-provisioner-579d48f95f-vncxw  1/1    Running  1         15h

==> v1/Role
NAME                                                  CREATED AT
leader-locking-crusty-penguin-nfs-client-provisioner  2020-07-22T11:07:19Z

==> v1/RoleBinding
NAME                                                  ROLE                                                       AGE
leader-locking-crusty-penguin-nfs-client-provisioner  Role/leader-locking-crusty-penguin-nfs-client-provisioner  15h

==> v1/ServiceAccount
NAME                                   SECRETS  AGE
crusty-penguin-nfs-client-provisioner  1        15h

==> v1/StorageClass
NAME        PROVISIONER                                          RECLAIMPOLICY  VOLUMEBINDINGMODE  ALLOWVOLUMEEXPANSION  AGE
nfs-client  cluster.local/crusty-penguin-nfs-client-provisioner  Delete         Immediate          true                  15h

可以看到，主要部署了：

Role, roleBinding, serviceAccount
Deployment
StorageClass

其中，比较重要的是StorageClass，当前为nfs-client。当创建PVC时，指定了storageClassName为nfs-client后，将会自动创建PV及PVC

测试

创建pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: nfs-client
  resources:
    requests:
      storage: 1Gi

查看pv及pvc

1
2
3

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-nfs-test   Bound    pvc-cd18aa33-8734-461c-81a5-f5cf2aa0ad8f   1Gi        RWO            nfs-client     3s

1
2
3

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-cd18aa33-8734-461c-81a5-f5cf2aa0ad8f   1Gi        RWO            Delete           Bound    default/pvc-nfs-test   nfs-client              10s

1
2
3

$ ls -l
total 0
drwxrwxrwx 2 root root 10 Jul 23 11:01 default-pvc-nfs-test-pvc-cd18aa33-8734-461c-81a5-f5cf2aa0ad8f

可以看到，对应的nfs服务器上的目录中创建了一个目录

小结

本次实验主要是想通过NFS创建PV及PVC，以提供跨节点数据持久化

注：没有演示搭建nfs服务器的方法，网上教程很多

在创建PV时，指定了nfs方式
基于nfs的PV创建PVC，分布于多个节点上的pod可以共享数据
使用nfs-client-provisioner自动创建PV及PVC

K8s学习笔记——控制器模型

2020-07-20T08:36:02.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：16 | 编排其实很简单：谈谈“控制器”模型

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dp
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Never
        ports:
        - containerPort: 80

$ kubectl get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-dp   2/2     2            2           75s
$ kubectl describe deploy nginx-dp
Name:                   nginx-dp
Namespace:              default
CreationTimestamp:      Mon, 06 Jul 2020 15:56:10 +0800
Labels:                 
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
...
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-67f857c57f-kgmsq   1/1     Running   0          100s
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          100s

这个 Deployment 定义的编排动作：确保携带了 app=nginx 标签的 Pod 的个数，永远等于 spec.replicas 指定的个数，即 2 个。

现在，可以手动删除一个pod看看

$ kubectl delete pod nginx-dp-67f857c57f-kgmsq
pod "nginx-dp-67f857c57f-kgmsq" deleted
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-67f857c57f-nmrhh   1/1     Running   0          7s
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          3m8s

可以看到，删除了nginx-dp-67f857c57f-kgmsq的pod，控制器自动拉起一个新的nginx-dp-67f857c57f-nmrhh的pod

扩展一下

如果我手工拉起一个匹配了label的pod会怎样呢

apiVersion: v1
kind: Pod
metadata:
  name: new-nginx-pod
  labels:
    app: nginx
spec:
  containers:
  - name: new-nginx-contrainer
    image: nginx
    imagePullPolicy: Never
    ports:
      - containerPort: 80

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
new-nginx-pod               1/1     Running   0          7m20s
nginx-dp-67f857c57f-nmrhh   1/1     Running   0          14m
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          17m

发现并没有因此而减少一个pod。那么手工删除一个

$ kubectl delete pod nginx-dp-67f857c57f-nmrhh
pod "nginx-dp-67f857c57f-nmrhh" deleted
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
new-nginx-pod               1/1     Running   0          7m50s
nginx-dp-67f857c57f-4wptz   1/1     Running   0          5s
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          17m

依然还是创建了一个新的，为什么呢？

$ kubectl describe pod nginx-dp-67f857c57f-qx7l7
Name:         nginx-dp-67f857c57f-qx7l7
Namespace:    default
Priority:     0
Node:         node2/10.160.18.181
Start Time:   Mon, 06 Jul 2020 15:56:10 +0800
Labels:       app=nginx
              pod-template-hash=67f857c57f

可以看到，Labels中，k8s自动创建了一个pod-template-hash=67f857c57f，来区别于手工创建出来的pod

那如果在自定义的pod中也指定了呢？不妨试试

apiVersion: v1
kind: Pod
metadata:
  name: new-nginx-pod
  labels:
    app: nginx
    pod-template-hash: 67f857c57f
spec:
  containers:
  - name: new-nginx-contrainer
    image: nginx
    imagePullPolicy: Never
    ports:
      - containerPort: 80

$ kubectl apply -f new-pod.yaml
pod/new-nginx-pod created
# 以极快的速度查看pod
$ kubectl get pods
NAME                        READY   STATUS        RESTARTS   AGE
new-nginx-pod               0/1     Terminating   0          1s
nginx-dp-67f857c57f-4wptz   1/1     Running       0          6m25s
nginx-dp-67f857c57f-qx7l7   1/1     Running       0          24m
# 稍等一下再试
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dp-67f857c57f-4wptz   1/1     Running   0          7m34s
nginx-dp-67f857c57f-qx7l7   1/1     Running   0          25m

可以看到，这个new-nginx-pod一创建出来，就进入了Terminating的状态，再次查看时，已经不存在了。

小结

本章节主要学习了Deployment的基础用法。可以看到，Deployment这种控制器，对携带指定标签的Pod，会维持一个指定的份数。

而且也注意到，为了有别于非deployment创建出来的pod，会为pod添加一个pod-template-hash的标签。

K8s网络相关问题及解决

2020-07-20T07:55:15.000Z

随着学习日渐深入，遇到了不少问题。有问题，度娘+谷哥。学习阶段，我只是各种文章的搬运工。先解决用的问题

本文主要解决以下几个问题：

pod无法访问ClusterIP
busybox做dns查询
pod间互访及访问外网

问题1: pod无法访问ClusterIP

这个问题困扰了我好些天，最后，配置了IPVS就OK了

step0 - kube-proxy日志

从kube-proxy的日志中看到Unknown proxy mode "", assuming iptables proxy

$ kubectl logs -n kube-system kube-proxy-5n29r | more
W0720 03:22:47.942827       1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
I0720 03:22:48.245820       1 node.go:136] Successfully retrieved node IP: 10.160.18.183
I0720 03:22:48.245876       1 server_others.go:186] Using iptables Proxier.
I0720 03:22:48.246253       1 server.go:583] Version: v1.18.3
I0720 03:22:48.395170       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0720 03:22:48.395210       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0720 03:22:48.395578       1 conntrack.go:83] Setting conntrack hashsize to 32768
I0720 03:22:48.414004       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0720 03:22:48.414067       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0720 03:22:48.533638       1 config.go:315] Starting service config controller
I0720 03:22:48.533673       1 shared_informer.go:223] Waiting for caches to sync for service config
I0720 03:22:48.533997       1 config.go:133] Starting endpoints config controller
I0720 03:22:48.534016       1 shared_informer.go:223] Waiting for caches to sync for endpoints config

step1 - 安装相关包

1 2	# 在所有节点上 $ apt install ipset ipvsadm

step2 - 加载module

$ modprobe -- ip_vs
$ modprobe -- ip_vs_rr
$ modprobe -- ip_vs_wrr
$ modprobe -- ip_vs_sh
$ modprobe -- nf_conntrack
# 注：很多帖子上都写的是nf_conntrack_ipv4，但在Ubuntu20.04上，这个已经变成了nf_conntrack
# 检查配置
$ lsmod | grep -e ipvs -e nf_conntrack
nf_conntrack_netlink    45056  0
nfnetlink              16384  3 nf_conntrack_netlink,ip_set
nf_conntrack          139264  5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs
# 再注：Ubuntu 20.04上这些module可能默认已经加载了

注：如果ipvs默认没有加载的话，需要写一个脚本，系统重启时也需要加载

step3 - 修改kube-proxy配置文件

修改kube-proxy的configmap中的mode字段为ipvs

$ kubectl edit configmap kube-proxy -n kube-system
...
     kind: KubeProxyConfiguration
     metricsBindAddress: ""
     mode: "ipvs"
     nodePortAddresses: null
     ...

step4 - 重启kube-proxy

可以逐个删除kube-proxy的pod，由k8s自动重启，也可以批量删除

1	$ kubectl get pod -n kube-system \| grep kube-proxy \|awk '{system("kubectl delete pod "$1" -n kube-system")}'

查看kube-proxy的日志

$ kubectl logs -n kube-system kube-proxy-44zw5
I0720 05:37:30.026304       1 node.go:136] Successfully retrieved node IP: 10.160.18.181
I0720 05:37:30.026349       1 server_others.go:259] Using ipvs Proxier.
W0720 05:37:30.026600       1 proxier.go:429] IPVS scheduler not specified, use rr by default
I0720 05:37:30.026814       1 server.go:583] Version: v1.18.3
I0720 05:37:30.027200       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0720 05:37:30.027452       1 config.go:133] Starting endpoints config controller
I0720 05:37:30.027474       1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0720 05:37:30.027507       1 config.go:315] Starting service config controller
I0720 05:37:30.027529       1 shared_informer.go:223] Waiting for caches to sync for service config
I0720 05:37:30.127736       1 shared_informer.go:230] Caches are synced for endpoints config
I0720 05:37:30.127790       1 shared_informer.go:230] Caches are synced for service config

可以看到Using ipvs Proxier.，说明IPVS已经启用了

现在，可以启动一个busybox的container来ping一下coredns的clusterIP了

问题2: busybox做dns查询失败

step0 - 问题现象

在解决了pod无法访问dns clusterIP的问题之后，发现busybox还是无法解析到某个service的IP

kubectl run -i --tty --image busybox dns-test --restart=Never --rm /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
;; connection timed out; no servers could be reached

上网查完后发现：busybox的版本高于1.28.4都存在这个问题

step1 - 解决方法

使用1.28.4的busybox镜像执行dns查询

$ kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 172.1.2.175 web-0.nginx.default.svc.cluster.local
/ # ping web-0.nginx
PING web-0.nginx (172.1.2.175): 56 data bytes
64 bytes from 172.1.2.175: seq=0 ttl=62 time=1.050 ms
64 bytes from 172.1.2.175: seq=1 ttl=62 time=0.432 ms

问题3: pod互访及访问外网不通

问题原因：iptables

解决方法：

分别在每个节点上执行

$ iptables -P INPUT ACCEPT
$ iptables -P FORWARD ACCEPT
$ iptables -F
$ iptables -L -n
$ iptables -t nat -I POSTROUTING -s 172.1.2.0/24 -j MASQUERADE
# 注：172.1.2.0/24是每个节点的pod-network-cidr

测试

$ kubectl run -i --tty --image busybox:1.28.4 connect-test --restart=Never --rm -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # ping 10.96.0.10 -c 1
PING 10.96.0.10 (10.96.0.10): 56 data bytes
64 bytes from 10.96.0.10: seq=0 ttl=64 time=0.082 ms

--- 10.96.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.082/0.082/0.082 ms
/ # ping 10.96.0.1 -c 1
PING 10.96.0.1 (10.96.0.1): 56 data bytes
64 bytes from 10.96.0.1: seq=0 ttl=64 time=0.072 ms

--- 10.96.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.072/0.072/0.072 ms
/ # ping 172.1.1.193 -c 1
PING 172.1.1.193 (172.1.1.193): 56 data bytes
64 bytes from 172.1.1.193: seq=0 ttl=64 time=0.108 ms

--- 172.1.1.193 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.108/0.108/0.108 ms
/ # ping 223.5.5.5 -c 1
PING 223.5.5.5 (223.5.5.5): 56 data bytes
64 bytes from 223.5.5.5: seq=0 ttl=114 time=5.659 ms

--- 223.5.5.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 5.659/5.659/5.659 ms
/ #
# 注: 10.96.0.1为kube-apiserver的ClusterIP
#     10.96.0.10为coredns的ClusterIP
#     172.1.1.193为一个pod的IP

使iptables规则重启生效

分别在每个节点上执行：

1
2
3

$ iptables-save > /etc/iptables.up.rules
$ echo -e '#!/bin/bash\n/sbin/iptables-restore < /etc/iptables.up.rules' > /etc/network/if-pre-up.d/iptables
$ chmod +x /etc/network/if-pre-up.d/iptables

K8s学习笔记——Pod使用进阶

2020-07-09T03:01:36.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：15 | 深入解析Pod对象（二）：使用进阶

Project Volume

Project Volume（投射数据卷）是为Pod提供事先定义好的一些数据，包括4种：

Secret
ConfigMap
Downward API
ServiceAccountToken

Secret

以文件形式创建secret

# 先创建两个文件，username.txt和password.txt
$ cat username.txt
secuser
$ cat password.txt
secPassword
$ kubectl create secret generic user --from-file=./username.txt
secret/user created
$ kubectl create secret generic pass --from-file=./password.txt
secret/pass created
$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-nlz8h   kubernetes.io/service-account-token   3      36d
pass                  Opaque                                1      6s
user                  Opaque                                1      38s
# 可以看到default-token-nlz8h，实际上是k8s的默认token

apiVersion: v1
kind: Pod
metadata:
  name: test-pv-secret
spec:
  containers:
  - name: test-secret
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: sec-test
      mountPath: "/pv"
      readOnly: true
  volumes:
  - name: sec-test
    projected:
      sources:
      - secret:
          name: user
      - secret:
          name: pass

$ kubectl describe pod test-pv-secret
Name:         test-pv-secret
...
Volumes:
  sec-test:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          user
    SecretOptionalName:  
    SecretName:          pass
    SecretOptionalName:  
  default-token-nlz8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nlz8h
    Optional:    false
...

可以看到，pod的Volumes中有两个：

sec-test，即为yaml文件中的volume
default-token-nlz8h，k8s默认的token

容器层面

$ docker inspect a77ad18a7a2a
[
    {
        "Id": "a77ad18a7a2aad64fba15bf1dcef8b3f2fa803aa3da3cab5def23b40a0aa5b21",
        ...
        HostConfig": {
            "Binds": [
                "/var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test:/pv:ro",
                "/var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~secret/default-token-nlz8h:/var/run/secrets/kubernetes.io/serviceaccount:ro",
                ...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test",
                "Destination": "/pv",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            ...

从上面的信息可以看出，/var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test目录以只读方式挂载到了容器上

$ ls /var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test/
password.txt  username.txt
$ cat /var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test/username.txt
secuser
$ cat /var/lib/kubelet/pods/cba17e39-f12d-416b-a9d1-6390b590d754/volumes/kubernetes.io~projected/sec-test/password.txt
secPassword

查看结果

$ kubectl exec test-pv-secret -- ls /pv
password.txt
username.txt
$ kubectl exec test-pv-secret -- cat /pv/username.txt
secuser
$ kubectl exec test-pv-secret -- cat /pv/password.txt
secPassword

secret对象

创建secret对象

Secret 对象要求这些数据必须是经过 Base64 转码的，以免出现明文密码的安全隐患

$ echo -n 'secuser' | base64
c2VjdXNlcg==
$ echo -n 'secPassword' | base64
c2VjUGFzc3dvcmQ=

apiVersion: v1
kind: Secret
metadata:
  name: sec-obj-test
type: Opaque
data:
  username: c2VjdXNlcg==
  password: c2VjUGFzc3dvcmQ=

$ kubectl apply -f secret-obj.yaml
secret/sec-obj-test created
$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-nlz8h   kubernetes.io/service-account-token   3      36d
sec-obj-test          Opaque                                2      45s

创建pod

apiVersion: v1
kind: Pod
metadata:
  name: sec-obj-test
spec:
  containers:
  - name: test-secobj
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: secobj-test
      mountPath: "/pv"
      readOnly: true
  volumes:
  - name: sec-test
    projected:
      sources:
      - secret:
          name: sec-obj-test

$ kubectl describe pod sec-obj-test
Name:         sec-obj-test
...
    Mounts:
      /pv from secobj-test (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
...
Volumes:
  secobj-test:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          sec-obj-test
    SecretOptionalName:  
...

检查容器上的内容

$ kubectl exec sec-obj-test -- ls /pv
password
username
$ kubectl exec sec-obj-test -- cat /pv/username
secuser
# 注：这里没有换行
$ kubectl exec sec-obj-test -- cat /pv/password
secPassword
# 注：这里没有换行

ConfigMap

创建configMap

以如下配置文件为例

1 2	APP_NAME = "demo" APP_PORT = 80

$ kubectl create configmap flask-config --from-file=config.py
$ kubectl describe configmaps
Name:         flask-config
Namespace:    default
Labels:       
Annotations:  

Data
====
config.py:
----
APP_NAME = "demo"
APP_PORT = 80

Events:

创建pod

apiVersion: v1
kind: Pod
metadata:
  name: configmap-test
spec:
  containers:
  - name: configmap-test-container
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: flask-config
      mountPath: "/flask_config"
      readOnly: true
  volumes:
  - name: flask-config
    projected:
      sources:
      - configMap:
          name: flask-config

查看容器内的结果

$ kubectl exec configmap-test -- ls /flask_config
config.py
$ kubectl exec configmap-test -- cat /flask_config/config.py
APP_NAME = "demo"
APP_PORT = 80

DownwardAPI

创建pod

apiVersion: v1
kind: Pod
metadata:
  name: downward-api-test
  labels:
    arch: amd64
spec:
  containers:
  - name: downward-api-container
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: pod-label
      mountPath: /pod_label
      readOnly: true
  volumes:
  - name: pod-label
    downwardAPI:
      items:
        - path: "labels"
          fieldRef:
            fieldPath: metadata.labels

说明：在volumes中声明了downwardAPI，将metadata.labels挂载到容器中的/pod_label里

注：我当前的k8s环境中，downwardAPI可以不用定义在projected之下，当然也可以定义在之下

查看结果

$ kubectl exec downward-api-test -- ls /pod_label
labels
$ kubectl exec downward-api-test -- cat /pod_label/labels
arch="amd64"

尝试一个其他filed

如将metadata.labels换成metadata.namespace

1 2	$ kubectl exec downward-api-test -- cat /pod_label/labels default

ServiceAccountToken

其实之前查看pod信息的时候已经看到了相关内容

$ kubectl describe pod xxxxxx
...
    Mounts:
      /pod_label from pod-label (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
...
Volumes:
  default-token-nlz8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nlz8h
    Optional:    false
    ...

$ kubectl exec downward-api-test -- ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token
$ kubectl exec downward-api-test -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IlhTSnlXMUhXTlNnUmd4MlVMTzdtbm14YVdiSzNUdjk4UnVoZ3RRbUFXZGsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tbmx6OGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjU1MDAxNDE1LWU5N2MtNDVmNS04ZTNlLThkMzZhNGE5ZmYxMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.wPr9cO6ySUGV65jjA4WxheIk5mpDkim9jYNSN9hmDOOitOkKrC1qj5fg5eF_GnETUZbS7xaf7sitJgIWmNZfA9YJDMn_nY7a6TLGMMRlhRx3ZXhYg6XFheFCQkCwHLJt2FGNlBtf6WpRGBFVZK_bNbjrcxkBAE-tzz7wYFuA4L1zK3UQfZoie11F8NeEWVOliuMXlVT31GZeMG5MCsuSYk_c-S2c2eTT4ru1k74uH7W1nUi9P5O2CzQPjKHTIRcP3HSBuA1CQYxuBk8Av0eE6vQB41tYMIlpIavzljpd-_jWngPkZAUnQbislaXWZhzzKkGkGqsxsFGb0P57sX4Dyg

查看帮助

$ kubectl explain pod.spec.volumes.projected.sources
KIND:     Pod
VERSION:  v1

RESOURCE: sources <[]Object>

DESCRIPTION:
     list of volume projections

     Projection that may be projected along with other supported volume types

FIELDS:
   configMap    
     information about the configMap data to project

   downwardAPI  
     information about the downwardAPI data to project

   secret   
     information about the secret data to project

   serviceAccountToken  
     information about the serviceAccountToken data to project

容器健康检查及恢复

创建pod

apiVersion: v1
kind: Pod
metadata:
  name: liveness-test
  labels:
    test: liveness
spec:
  containers:
  - name: liveness
    image: busybox
    imagePullPolicy: Never
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

1
2
3

$ kubectl get pod
NAME            READY   STATUS    RESTARTS   AGE
liveness-test   1/1     Running   0          21s

可以看到，在21s的时候，pod的运行状态正常，RESTARTS字段为0

给他一会儿时间

查看pod详情

$ ubectl describe pod liveness-test
Name:         liveness-test
Namespace:    default
Priority:     0
Node:         node2/10.160.18.181
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  75s                default-scheduler  Successfully assigned default/liveness-test to node2
  Normal   Pulled     74s                kubelet, node2     Container image "busybox" already present on machine
  Normal   Created    74s                kubelet, node2     Created container liveness
  Normal   Started    74s                kubelet, node2     Started container liveness
  Warning  Unhealthy  32s (x3 over 42s)  kubelet, node2     Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    32s                kubelet, node2     Container liveness failed liveness probe, will be restarted
  Warning  Unhealthy  30s (x3 over 40s)  kubelet, node2     Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    30s                kubelet, node2     Container liveness failed liveness probe, will be restarted
  Normal   Started    2s (x2 over 74s)   kubelet, node2     Started container liveness
  Normal   Pulled     2s (x2 over 74s)   kubelet, node2     Container image "busybox" already present on machine
  Normal   Created    2s (x2 over 74s)   kubelet, node2     Created container liveness

当pod运行了一会儿后，/tmp/healthy被删除，随后，Container liveness failed liveness probe, will be restarted

查看pod重启状况

1
2
3

$ kubectl get pod
NAME            READY   STATUS    RESTARTS   AGE
liveness-test   1/1     Running   5          6m19s

podPresets是一个需要开启的选项，在我当前环境中没有开启，暂不学习

小结

Pod使用进阶主要涉及了：

Project Volume的4种类型
健康状况检查和恢复

K8s学习笔记——Pod的基本概念

2020-07-08T05:25:35.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则。动手实践并记录结果
对应章节：14 | 深入解析Pod对象（一）：基本概念

`hostAliases`测试

Pod配置

apiVersion: v1
kind: Pod
metadata:
  name: my-first-test
spec:
  hostAliases:
  - ip: 10.9.8.7
    hostnames:
    - "host.test.site"
    - "host.aliases.test.site"
  containers:
  - name: my-test-container1
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
  - name: my-test-container2
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true

查看容器内的hosts信息

$ kubectl exec my-first-test -c my-test-container1 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
172.172.1.41    my-first-test

# Entries added by HostAliases.
10.9.8.7    host.test.site  host.aliases.test.site
$ kubectl exec my-first-test -c my-test-container2 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
172.172.1.41    my-first-test

# Entries added by HostAliases.
10.9.8.7    host.test.site  host.aliases.test.site

`shareProcessNamespace`

不配置`shareProcessNamespace`

apiVersion: v1
kind: Pod
metadata:
  name: no-share-process-ns-test
spec:
  containers:
  - name: shell
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
  - name: web
    image: nginx:latest
    imagePullPolicy: Never

通过shell的container查看进程

$ kubectl exec no-share-process-ns-test -c shell -- ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
    6 root      0:00 ps
# shell的容器仅能看到自己容器内的进程

container层面信息

$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED              STATUS              PORTS               NAMES
117497ca101c        2622e6cca7eb                                        "/docker-entrypoint.…"   About a minute ago   Up About a minute                       k8s_web_no-share-process-ns-test_default_94884606-ea15-4038-9fa0-0e3b11ca4f31_0
ca8b6e6a1796        1c35c4412082                                        "sh"                     About a minute ago   Up About a minute                       k8s_shell_no-share-process-ns-test_default_94884606-ea15-4038-9fa0-0e3b11ca4f31_0
eb0fd0f91677        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 About a minute ago   Up About a minute                       k8s_POD_no-share-process-ns-test_default_94884606-ea15-4038-9fa0-0e3b11ca4f31_0
$ docker inspect ca8b6e6a1796 | grep \"Pid\"
            "Pid": 1561218,
$ docker inspect 117497ca101c | grep \"Pid\"
            "Pid": 1561252,
$ ls -l /proc/1561218/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 17 14:43 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 17 14:41 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 17 14:41 mnt -> 'mnt:[4026532718]'
lrwxrwxrwx 1 root root 0 Jun 17 14:41 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 17 14:41 pid -> 'pid:[4026532723]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 pid_for_children -> 'pid:[4026532723]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 17 14:41 uts -> 'uts:[4026532721]'
$ ls -l /proc/1561252/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 17 14:43 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 mnt -> 'mnt:[4026532724]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 pid -> 'pid:[4026532726]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 pid_for_children -> 'pid:[4026532726]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 17 14:43 uts -> 'uts:[4026532725]'

两个container的pid和pid_for_children不同

开启shareProcessNamespace

apiVersion: v1
kind: Pod
metadata:
  name: share-process-ns-test
spec:
  shareProcessNamespace: true
  containers:
  - name: shell
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
  - name: web
    image: nginx:latest
    imagePullPolicy: Never

通过shell的container查看进程

$ kubectl exec share-process-ns-test -c shell -- ps
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    6 root      0:00 sh
   12 root      0:00 nginx: master process nginx -g daemon off;
   39 101       0:00 nginx: worker process
   40 root      0:00 ps

可以看到，名为shell的container中也能看到nginx的进程

container层面

$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
955d3e3b9cba        2622e6cca7eb                                        "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes                            k8s_web_share-process-ns-test_default_d1b8348e-5399-4d30-9d53-de91ffdb9746_0
d065b6907db0        1c35c4412082                                        "sh"                     2 minutes ago       Up 2 minutes                            k8s_shell_share-process-ns-test_default_d1b8348e-5399-4d30-9d53-de91ffdb9746_0
79a59e6e245c        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_share-process-ns-test_default_d1b8348e-5399-4d30-9d53-de91ffdb9746_0
$ docker inspect 955d3e3b9cba | grep \"Pid\"
            "Pid": 1564637,
$ docker inspect d065b6907db0 | grep \"Pid\"
            "Pid": 1564603,
$ ls -l /proc/1564637/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 17 14:51 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 mnt -> 'mnt:[4026532724]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 pid -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 pid_for_children -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 uts -> 'uts:[4026532725]'
$ ls -l /proc/1564603/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 17 14:51 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 17 14:48 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 17 14:48 mnt -> 'mnt:[4026532721]'
lrwxrwxrwx 1 root root 0 Jun 17 14:48 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 17 14:48 pid -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 pid_for_children -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 17 14:51 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 17 14:48 uts -> 'uts:[4026532723]'

shell的container和web的container的pid和pid_for_children都是pid:[4026532626]的namespace

`hostNetwork`练习

apiVersion: v1
kind: Pod
metadata:
  name: host-net-test
spec:
  hostNetwork: true
  containers:
  - name: shell
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
  - name: web
    image: nginx:latest
    imagePullPolicy: Never

$ kubectl exec host-net-test -c shell -- ifconfig
cni0      Link encap:Ethernet  HWaddr 16:80:F0:F9:A0:B4
          inet addr:172.172.1.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::1480:f0ff:fef9:a0b4/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:31 errors:0 dropped:0 overruns:0 frame:0
          TX packets:600 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:868 (868.0 B)  TX bytes:43824 (42.7 KiB)

docker0   Link encap:Ethernet  HWaddr 02:42:BB:9B:62:76
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::42:bbff:fe9b:6276/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:1666 (1.6 KiB)

...

container中看到的网卡信息与host相同

`lifecycle`练习

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: nginx
    imagePullPolicy: Never
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/usr/sbin/nginx","-s","quit"]

1 2	$ kubectl exec lifecycle-demo -- cat /usr/share/message Hello from the postStart handler

可以看到container启动后就打印了

小结

这章节主要是理解哪些操作是属于pod层面，而哪些操作是属于容器层面。比如hostAliases和shareProcessNamespace，都是属于整个pod层面，而imagePullPolicy和lifecycle则属于container层面。

再次引用原文中的总结：

**凡是 Pod 中的容器要共享宿主机的 Namespace，也一定是 Pod 级别的定义

K8s学习笔记——为什么需要Pod

2020-07-06T05:34:23.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则.
对应章节：13 | 为什么我们需要Pod？

共享net的container

先来起两个共享net和volume的容器

$ docker run -it -d --name test1 busybox
15dafb04a3fae857c0db7ce09c02d6ebc565de71a49df2511eded48457e0945e
$ docker run -it -d --network container:test1 --volumes-from test1 --name test2 busybox
6e15995e63a4f2cc0b293be848a448c2e34a1502477645c434e529245ffaa1f3

查看两个container的信息

$ docker inspect test1
[
    {
        "Id": "15dafb04a3fae857c0db7ce09c02d6ebc565de71a49df2511eded48457e0945e",
            ...
            "Pid": 2282,
            ...
        },
        ...
            "Networks": {
                "bridge": {
                    ...
                    "NetworkID": "5bdb44786ae497e27f9626360b59ff35eead5c02f6f6da4392306132e8910ad0",
                    "EndpointID": "8a9d1594a9949509e42eee8af6145d30e1162cc4c8384c0f7d7e8d7bece5cfdd",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                }
        ...

$ docker inspect test2
[
    {
        "Id": "6e15995e63a4f2cc0b293be848a448c2e34a1502477645c434e529245ffaa1f3",
        ...
            "Pid": 3555,
        ...
            "NetworkMode": "container:15dafb04a3fae857c0db7ce09c02d6ebc565de71a49df2511eded48457e0945e",
        ...

查看进程

$ ls -l /proc/2282/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 12 06:45 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 12 06:31 ipc -> 'ipc:[4026532571]'
lrwxrwxrwx 1 root root 0 Jun 12 06:31 mnt -> 'mnt:[4026532569]'
lrwxrwxrwx 1 root root 0 Jun 12 06:16 net -> 'net:[4026532574]'
lrwxrwxrwx 1 root root 0 Jun 12 06:31 pid -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 12 06:45 pid_for_children -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 12 06:45 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 12 06:31 uts -> 'uts:[4026532570]'
$ ls -l /proc/3555/ns/
total 0
lrwxrwxrwx 1 root root 0 Jun 12 06:45 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 12 06:42 ipc -> 'ipc:[4026532630]'
lrwxrwxrwx 1 root root 0 Jun 12 06:42 mnt -> 'mnt:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 12 06:42 net -> 'net:[4026532574]'
lrwxrwxrwx 1 root root 0 Jun 12 06:42 pid -> 'pid:[4026532631]'
lrwxrwxrwx 1 root root 0 Jun 12 06:45 pid_for_children -> 'pid:[4026532631]'
lrwxrwxrwx 1 root root 0 Jun 12 06:45 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 12 06:42 uts -> 'uts:[4026532629]'

可以看到，cgroup、net和user是相同的

实际上，在不指定共享net的container的时候，cgroup和user也都是相同的

$ docker run -it -d ubuntu
25b637a2facf0ba713403c14f300644ee21cd107b45fb57d47f7cb4be0fcc3ad
$ docker inspect 25b637a2f | grep Pid
      "Pid": 10895,
      "PidMode": "",
         "PidsLimit": null,
$ ls /proc/10895/ns/ -l
total 0
lrwxrwxrwx 1 root root 0 Jun 15 03:10 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 ipc -> 'ipc:[4026532706]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 mnt -> 'mnt:[4026532704]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 net -> 'net:[4026532709]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 pid -> 'pid:[4026532707]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 pid_for_children -> 'pid:[4026532707]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 15 03:10 uts -> 'uts:[4026532705]'

查看网络

$ docker exec -it test1 ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1522 (1.4 KiB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$ docker exec -it test2 ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1522 (1.4 KiB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

两个container的eth0的IP，MAC等都相同

Pod

先起一个pod看看

apiVersion: v1
kind: Pod
metadata:
  name: two-containers
spec:
  containers:
  - name: my-test-container1
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
  - name: my-test-container2
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true

注：因为已经pull了image，所以这里设置了imagePullPolicy: Never

1
2
3

$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
two-containers   2/2     Running   0          3s

$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED              STATUS              PORTS               NAMES
14aea3a7ec58        1c35c4412082                                        "sh"                     About a minute ago   Up About a minute                       k8s_my-test-container2_two-containers_default_dd628b40-a070-4d9d-9837-7d3031092e86_0
fb9a5a4e5ee2        1c35c4412082                                        "sh"                     About a minute ago   Up About a minute                       k8s_my-test-container1_two-containers_default_dd628b40-a070-4d9d-9837-7d3031092e86_0
bac4514284b2        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 About a minute ago   Up About a minute                       k8s_POD_two-containers_default_dd628b40-a070-4d9d-9837-7d3031092e86_0

可以看到，分别起了3个container：

k8s_my-test-container1_two-containers_default
k8s_my-test-container2_two-containers_default
k8s_POD_two-containers_default

其中，k8s_POD_two-containers_default的image是pause

先看看namespace

$ docker inspect 14aea3a7ec58 | grep \"Pid\"
            "Pid": 359647,
$ docker inspect fb9a5a4e5ee2 | grep \"Pid\"
            "Pid": 359614,
$ docker inspect bac4514284b2 | grep \"Pid\"
            "Pid": 359530,
$ ls -l /proc/359647/ns
total 0
lrwxrwxrwx 1 root root 0 Jun 15 14:41 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 mnt -> 'mnt:[4026532715]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid -> 'pid:[4026532717]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid_for_children -> 'pid:[4026532717]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 uts -> 'uts:[4026532716]'
$ ls -l /proc/359614/ns
total 0
lrwxrwxrwx 1 root root 0 Jun 15 14:41 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 mnt -> 'mnt:[4026532710]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid -> 'pid:[4026532714]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid_for_children -> 'pid:[4026532714]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 uts -> 'uts:[4026532713]'
$ ls -l /proc/359530/ns
total 0
lrwxrwxrwx 1 root root 0 Jun 15 14:41 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 15 14:38 ipc -> 'ipc:[4026532625]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 mnt -> 'mnt:[4026532623]'
lrwxrwxrwx 1 root root 0 Jun 15 14:38 net -> 'net:[4026532628]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 pid_for_children -> 'pid:[4026532626]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 15 14:41 uts -> 'uts:[4026532624]'

看看IP

Pod层面

$ kubectl describe pod two-containers
Name:         two-containers
Namespace:    default
Priority:     0
Node:         node2/10.160.18.181
Start Time:   Mon, 15 Jun 2020 14:38:10 +0800
Labels:       
Annotations:  Status:  Running
IP:           172.172.1.35
IPs:
  IP:  172.172.1.35
Containers:
  my-test-container1:
    Container ID:   docker://fb9a5a4e5ee2eef15dddc159bdb507b435ebbbc189d3ceefbaca7d7d4dac994d
    Image:          busybox
    ...
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
  my-test-container2:
    Container ID:   docker://14aea3a7ec58e4f4220d6eea1266799a55fbdd945956d4845d4e062fce6542d7
    Image:          busybox
    ...
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
...
Volumes:
  default-token-nlz8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nlz8h
    Optional:    false

Container层面

$ docker exec -it 14aea3a7ec58 ifconfig
eth0      Link encap:Ethernet  HWaddr 26:64:FF:FA:7D:BE
          inet addr:172.172.1.35  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1198 (1.1 KiB)  TX bytes:42 (42.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$ docker exec -it fb9a5a4e5ee2 ifconfig
eth0      Link encap:Ethernet  HWaddr 26:64:FF:FA:7D:BE
          inet addr:172.172.1.35  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1198 (1.1 KiB)  TX bytes:42 (42.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Pod共享volume

Pod配置

apiVersion: v1
kind: Pod
metadata:
  name: two-containers-volume
spec:
  volumes:
  - name: shared-data
    hostPath:
      path: /data
  containers:
  - name: c1
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: shared-data
      mountPath: /data
  - name: c2
    image: busybox
    imagePullPolicy: Never
    stdin: true
    tty: true
    volumeMounts:
    - name: shared-data
      mountPath: /data

查看pod信息

1
2
3

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
two-containers-volume   2/2     Running   0          5s

$ kubectl describe pod two-containers-volume
Name:         two-containers-volume
Namespace:    default
Priority:     0
Node:         node2/10.160.18.181
Start Time:   Mon, 15 Jun 2020 14:53:43 +0800
Labels:       
Annotations:  Status:  Running
IP:           172.172.1.36
IPs:
  IP:  172.172.1.36
Containers:
  c1:
    Container ID:   docker://b5b285305aae0d9155fca8415a1002697b0a446d222ce8f8bb27456c51680bc0
    Image:          busybox
    ...
    Mounts:
      /data from shared-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
  c2:
    Container ID:   docker://52767f5bfa9e246e79547b11bfca0eb5948f3458850beaf173fbd921e43aa949
    Image:          busybox
    ...
    Mounts:
      /data from shared-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlz8h (ro)
...
Volumes:
  shared-data:
    Type:          HostPath (bare host directory volume)
    Path:          /data
    HostPathType:
  default-token-nlz8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nlz8h
    Optional:    false
...

对比之前的，可以看到，volumes中增加了一个shared-data的volume，每个container的Mounts中多了一个/data from shared-data (rw)

查看container层面的信息

$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
52767f5bfa9e        1c35c4412082                                        "sh"                     6 minutes ago       Up 6 minutes                            k8s_c2_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0
b5b285305aae        1c35c4412082                                        "sh"                     6 minutes ago       Up 6 minutes                            k8s_c1_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0
09b081c04b1b        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0

$ docker inspect 52767f5bfa9e
[
    {
        ...
        "Name": "/k8s_c2_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0",
        ...
        "HostConfig": {
            ...
            "NetworkMode": "container:09b081c04b1ba2b973d7fb31aadd7d796d8f6aef03113764f4a868022b6332ee",
            ...
        },
        ...
        "Mounts": [
            ...
            {
                "Type": "bind",
                "Source": "/data",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            ...
        ],
        ...
    }
]
$ docker inspect b5b285305aae
[
    {
        ...
        "Name": "/k8s_c1_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0",
        ...
        "HostConfig": {
            ...
            "NetworkMode": "container:09b081c04b1ba2b973d7fb31aadd7d796d8f6aef03113764f4a868022b6332ee",
            ...
        },
        ...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/data",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            ...
        ],
        ...
    }
]
$ docker inspect 09b081c04b1b
[
    {
        ...
        "Name": "/k8s_POD_two-containers-volume_default_a66c7066-35c2-4be6-b58a-0df6ede9d95d_0",
        ...
        "HostConfig": {
            ...
            "NetworkMode": "none",
            ...
        },
        ...
        "Mounts": [],
        ...
        "NetworkSettings": {
            ...
            "SandboxKey": "/var/run/docker/netns/cb1ed85e8f94",
            ...
            "Networks": {
                "none": {
                    ...
                    "NetworkID": "d5b75ce86af4c81134a382c1b77ed2de74e8ec9d484a68a5002f52ed3905b239",
                    "EndpointID": "f6ea091a8d1445dae3a4b961c6cfbfa05c671f604dab8c9284eca959d02e7867",
                    ...
                }
            }
        }
    }
]

说明：
k8s_POD_two-containers-volume_default这个container的NetworkSettings字段声明有网络相关内容，但Mounts为空
k8s_c1和k8s_c2的container的NetworkSettings为空，但Mounts中则挂载了/data目录

Init container

apiVersion: v1
kind: Pod
metadata:
  name: init-container-test
spec:
  volumes:
  - name: app-volume
    emptyDir: {}
  initContainers:
  - image: busybox
    imagePullPolicy: Never
    name: data-container
    stdin: true
    tty: true
    command: ["touch", "/data/test.txt"]
    volumeMounts:
    - mountPath: "/data"
      name: app-volume
  containers:
  - image: busybox
    imagePullPolicy: Never
    name: app-container
    stdin: true
    tty: true
    volumeMounts:
    - mountPath: "/data"
      name: app-volume

查看pod状况

1
2
3

$ kubectl get pod
NAME                  READY   STATUS    RESTARTS   AGE
init-container-test   1/1     Running   0          5s

查看container

$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
ce67fd98bc68        1c35c4412082                                        "sh"                     30 seconds ago      Up 29 seconds                           k8s_app-container_init-container-test_default_5960cae6-00b3-48ae-9991-741d73077dc4_0
e75657cb2079        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 31 seconds ago      Up 30 seconds                           k8s_POD_init-container-test_default_5960cae6-00b3-48ae-9991-741d73077dc4_0

会发现这里只有一个k8s_app-container和k8s_POD，而没有init-container
是因为
在 Pod 中，所有 Init Container 定义的容器，都会比 spec.containers 定义的用户容器先启动。并且，Init Container 容器会按顺序逐一启动，而直到它们都启动并且退出了，用户容器才会启动。
但k8s并没有在这个container结束运行后重启它

$ docker ps -a
CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS                      PORTS               NAMES
ce67fd98bc68        1c35c4412082                                         "sh"                     36 seconds ago      Up 36 seconds                                   k8s_app-container_init-container-test_default_5960cae6-00b3-48ae-9991-741d73077dc4_0
bbc45d10357c        1c35c4412082                                         "touch /data/test.txt"   37 seconds ago      Exited (0) 37 seconds ago                       k8s_data-container_init-container-test_default_5960cae6-00b3-48ae-9991-741d73077dc4_0
e75657cb2079        registry.aliyuncs.com/google_containers/pause:3.2    "/pause"                 37 seconds ago      Up 37 seconds                                   k8s_POD_init-container-test_default_5960cae6-00b3-48ae-9991-741d73077dc4_0

从这里可以看到， data-container已经Exited (0) 37 seconds ago

1 2	$ kubectl exec init-container-test -- ls /data test.txt

通过容器可以看到test.txt已经存在在init-container-test的/data目录下了

总结引用

一个运行在虚拟机里的应用，哪怕再简单，也是被管理在 systemd 或者 supervisord 之下的一组进程，而不是一个进程。
对于容器来说，一个容器永远只能管理一个进程
Pod，实际上是在扮演传统基础设施里“虚拟机”的角色；而容器，则是这个虚拟机里运行的用户程序。

K8s学习笔记——重新认识Docker容器

2020-06-23T04:48:13.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则.
对应章节：08 | 白话容器基础（四）：重新认识Docker容器

build一个镜像

创建Flask相关文件

app.py

from flask import Flask
import socket
import os

app = Flask(__name__)

@app.route('/')
def hello():
    html = "Hello {name}!
" \
           "Hostname: {hostname}
"           
    return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname())
    
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

requirements.txt

Flask

创建Dockerfile

FROM python:2.7-slim

WORKDIR /app

# app.py和requirements.txt都放在当前文件夹的app目录下
ADD ./app /app

# 使用pip命令安装这个应用所需要的依赖
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# 允许外界访问容器的80端口
EXPOSE 80

# 设置环境变量
ENV NAME World

# 设置容器进程为：python app.py，即：这个Python应用的启动命令
CMD ["python", "app.py"]

当前目录结构

$ tree .
.
├── app
│   ├── app.py
│   └── requirements.txt
└── Dockerfile

1 directory, 3 files

build docker镜像

1	$ docker build -t flaskapp .

当build镜像时：

docker每执行一行，会以上一层为基础拉起一个容器
然后在这个容器里执行对应的命令
完成后，将这一层提交成一个image

查看镜像的build history

$ docker history 06e1a19665ce
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
06e1a19665ce        5 weeks ago         /bin/sh -c apk update && apk add nginx          2.98MB
f70734b6a266        2 months ago        /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
           2 months ago        /bin/sh -c #(nop) ADD file:b91adb67b670d3a6f…   5.61MB

docker exec的本质

进入一个namespace

set_ns.c

#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 

#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE);} while (0)

int main(int argc, char *argv[]) {
    int fd;
    
    fd = open(argv[1], O_RDONLY);
    if (setns(fd, 0) == -1) {
        errExit("setns");
    }
    execvp(argv[2], &argv[2]); 
    errExit("execvp");
}

1	$ gcc -o set_ns set_ns.c

启动一个container

$ docker run -it -d --rm ubuntu
59ed1b0423ac42b5659e9c3d1759000e934c8383f605875d86db42b6ae7bf098
$ docker inspect 59ed1b0423a | grep \"Pid\"
            "Pid": 14123,

查看进程相关ns

$ ls /proc/14123/ns/ -l
total 0
lrwxrwxrwx 1 root root 0 Jun 16 07:34 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 ipc -> 'ipc:[4026532571]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 mnt -> 'mnt:[4026532569]'
lrwxrwxrwx 1 root root 0 Jun 16 07:32 net -> 'net:[4026532574]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 pid -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 pid_for_children -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 16 07:34 uts -> 'uts:[4026532570]'

以net的namespace运行ifconfig

$ ./set_ns /proc/14123/ns/net /bin/bash
$ ifconfig
eth0: flags=4163  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 13  bytes 1046 (1.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

分别用set_ns和docker exec查看

$ ps -aux | grep /bin/bash
root     14123  0.0  0.3   4112  3288 pts/0    Ss+  07:32   0:00 /bin/bash
root     14682  0.0  5.8 706028 58772 pts/3    Sl+  07:47   0:00 docker exec -it 59ed1b0423ac /bin/bash
root     14698  0.0  0.3   4112  3376 pts/1    Ss+  07:47   0:00 /bin/bash
root     14722  0.0  0.3  20416  3792 pts/4    S+   07:52   0:00 /bin/bash

$ ls -l /proc/14123/ns/net
lrwxrwxrwx 1 root root 0 Jun 16 07:32 /proc/14123/ns/net -> 'net:[4026532574]'
$ ls -l /proc/14698/ns/net
lrwxrwxrwx 1 root root 0 Jun 16 07:49 /proc/14698/ns/net -> 'net:[4026532574]'
$ ls -l /proc/14722/ns/net
lrwxrwxrwx 1 root root 0 Jun 16 07:52 /proc/14722/ns/net -> 'net:[4026532574]'

可以看出，最终都指向了同一个net namespace

Linux的ip命令也支持创建一个network namespace，如：

$ ip netns add ns_test
$ ip netns exec ns_test /bin/bash
$ ifconfig
# 由于并没有为这个namespace设定接口，所以，这里显示为空

$ ps -aux | grep /bin/bash
root     14956  0.0  0.4  20416  4056 pts/0    S+   08:33   0:00 /bin/bash
$ ls -l /proc/14956/ns/net
lrwxrwxrwx 1 root root 0 Jun 16 08:33 /proc/14956/ns/net -> 'net:[4026532629]'

当然，同样可以使用前面的set_ns的工具进行查看

1
2
3

$ ./set_ns /proc/14956/ns/net /bin/bash
$ ifconfig
# 这里同样没有内容输出

$ ps -aux | grep /bin/bash
root     14956  0.0  0.4  20416  4056 pts/0    S+   08:33   0:00 /bin/bash
root     14992  0.0  0.4  20416  4084 pts/4    S+   08:42   0:00 /bin/bash
$ ls -l /proc/14992/ns/net
lrwxrwxrwx 1 root root 0 Jun 16 08:45 /proc/14992/ns/net -> 'net:[4026532629]'

可以看到，使用ip命令进入namespace和set_ns进入namespace后的的/bin/bash的ns

Volume的本质

启动一个挂载volume的容器

$ docker run --rm -it -d -v /test ubuntu
65f6facb7c3c8e2f239bed07481e70ca7e0d09fc61617d0ef07e00a066ae7f96
$ docker volume ls
DRIVER              VOLUME NAME
local               ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614

$ docker inspect 65f6fa
[
    {
        "Id": "65f6facb7c3c8e2f239bed07481e70ca7e0d09fc61617d0ef07e00a066ae7f96",
        ...
        "Mounts": [
            {
                "Type": "volume",
                "Name": "ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614",
                "Source": "/var/lib/docker/volumes/ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614/_data",
                "Destination": "/test",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        ...
]

可以看到，在不指定本地目录的时候，docker会自动创建一个volume，且在/var/lib/docker/volumes下创建一个目录

启动一个挂载本地目录的容器

1 2	$ docker run --rm -it -d -v /test:/test ubuntu ddbbb888a84a3b700a321db79e0743576d894a2c6c6b9be58f73142d921b60aa

$ docker inspect ddbbb8
[
    {
        "Id": "ddbbb888a84a3b700a321db79e0743576d894a2c6c6b9be58f73142d921b60aa",
        ...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/test",
                "Destination": "/test",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        ...
]

Volume挂载的真相

这一段话有必要引述

Docker 又是如何做到把一个宿主机上的目录或者文件，挂载到容器里面去呢？难道又是 Mount Namespace 的黑科技吗？
实际上，并不需要这么麻烦。在《白话容器基础（三）：深入理解容器镜像》的分享中，我已经介绍过，当容器进程被创建之后，尽管开启了 Mount Namespace，但是在它执行 chroot（或者 pivot_root）之前，容器进程一直可以看到宿主机上的整个文件系统。而宿主机上的文件系统，也自然包括了我们要使用的容器镜像。这个镜像的各个层，保存在 /var/lib/docker/aufs/diff 目录下，在容器进程启动后，它们会被联合挂载在 /var/lib/docker/aufs/mnt/ 目录中，这样容器所需的 rootfs 就准备好了。
所以，我们只需要在 rootfs 准备好之后，在执行 chroot 之前，把 Volume 指定的宿主机目录（比如 /home 目录），挂载到指定的容器目录（比如 /test 目录）在宿主机上对应的目录（即 /var/lib/docker/aufs/mnt/[可读写层 ID]/test）上，这个 Volume 的挂载工作就完成了

由此可见，如果要将一个目录挂载到一个容器里，其操作是：

进入mount namespace
将需要挂载的目录挂载到容器的目录上
chroot切换到对应的文件系统

示例

1
2
3

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
65f6facb7c3c        ubuntu              "/bin/bash"         14 minutes ago      Up 14 minutes                           recursing_clarke

$ docker inspect 65f6facb7c3c
[
    {
        "Id": "65f6facb7c3c8e2f239bed07481e70ca7e0d09fc61617d0ef07e00a066ae7f96",
        ...
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/9f0bd8e84ddffd8663f01c959a0ced115a4d598b096896be15b3c680039d7754-init/diff:/var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff:/var/lib/docker/overlay2/823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff:/var/lib/docker/overlay2/37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff:/var/lib/docker/overlay2/40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff",
                "MergedDir": "/var/lib/docker/overlay2/9f0bd8e84ddffd8663f01c959a0ced115a4d598b096896be15b3c680039d7754/merged",
                "UpperDir": "/var/lib/docker/overlay2/9f0bd8e84ddffd8663f01c959a0ced115a4d598b096896be15b3c680039d7754/diff",
                "WorkDir": "/var/lib/docker/overlay2/9f0bd8e84ddffd8663f01c959a0ced115a4d598b096896be15b3c680039d7754/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614",
                "Source": "/var/lib/docker/volumes/ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614/_data",
                "Destination": "/test",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        ...
]

可以看到Mounts中的/var/lib/docker/volumes/ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614/_data

UpperDir中已经可以看到test目录了

1 2	$ ls /var/lib/docker/overlay2/9f0bd8e84ddffd8663f01c959a0ced115a4d598b096896be15b3c680039d7754/diff/ test

现在，在docker中创建一个文件

1
2
3

$ docker exec -it 65f6facb7c3c touch /test/test.txt
$ ls /var/lib/docker/volumes/ca1683d1e9cc06657c7857d8b8c7196176a59e409d74fc72ff30f0a76f98d614/_data/
test.txt

绑定挂载机制

$ ls /test
# 没有挂载前，目录为空
# 挂载tmp目录
$ mount --bind /tmp /test
$ ls /test
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-resolved.service-A5ml7i   vmware-root_550-2991137472
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-timesyncd.service-qu9LFm
# /test目录与/tmp目录内容一致
$ ls /tmp
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-resolved.service-A5ml7i   vmware-root_550-2991137472
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-timesyncd.service-qu9LFm
# umount
$ umount /test
$ ls /test
# 目录为空

但是，启动容器前host上的目录没有挂载内容，容器启动后，host上挂载目录，查看容器中的内容

$ ls /test
# 无内容
$ docker run -it -d --rm -v /test:/test ubuntu
5f2aaad59abb7546ecd4b7a47ede09fb6a1541c4bda27b79de893bd27350b93c
$ docker exec -it 5f2 ls /test
# 无内容
$ mount --bind /tmp /test
$ ls /test
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-resolved.service-A5ml7i   vmware-root_550-2991137472
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-timesyncd.service-qu9LFm
$ docker exec -it 5f2 ls /test
# 依旧没有内容
$ docker exec -it 5f2 touch /test/test.txt
$ ls /test
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-resolved.service-A5ml7i   vmware-root_550-2991137472
systemd-private-0ea54a48e403454ba91e8c8d816d2cbd-systemd-timesyncd.service-qu9LFm
# 咦，test.txt去哪儿了呢
$ umount /test
$ ls /test
test.txt
# 这样就有了

如果/test目录已经mount了呢？

会不会将本地挂载的目录提交到image里面呢？

不会
容器的镜像操作，比如 docker commit，都是发生在宿主机空间的。而由于 Mount Namespace 的隔离作用，宿主机并不知道这个绑定挂载的存在。所以，在宿主机看来，容器中可读写层的 /test 目录（/var/lib/docker/aufs/mnt/[可读写层 ID]/test），始终是空的。

小结

本节主要学习了DockerFile编写、镜像的build方法。以及docker exec和volume的底层实现原理。

通过所有前面几节的实验，不难发现，docker就是通过linux namespace进行隔离，cgroup对资源进行限制，rootfs作为容器的文件系统。无论是docker镜像还是docker容器，以及网络和volume，都是在linux的这些基础功能的基础上实现起来的。

K8s学习笔记——深入理解容器镜像

2020-06-23T04:25:55.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则.
对应章节：07 | 白话容器基础（三）：深入理解容器镜像

以系统调用方式创建namespace实验

ns.c

#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#define STACK_SIZE (1024 * 1024)
static char container_stack[STACK_SIZE];
char* const container_args[] = {
  "/bin/bash",
  NULL
};

int container_main(void* arg)
{
  printf("Container - inside the container!\n");
  execv(container_args[0], container_args);
  printf("Something's wrong!\n");
  return 1;
}

int main()
{
  printf("Parent - start a container!\n");
  int container_pid = clone(container_main, container_stack+STACK_SIZE, CLONE_NEWNS | SIGCHLD , NULL);
  waitpid(container_pid, NULL, 0);
  printf("Parent - container stopped!\n");
  return 0;
}

build ns并进入ns

$ gcc -o ns ns.c
$ ./ns
Parent - start a container!
Container - inside the container!
$ ls /tmp
systemd-private-8bac4934482d484d879054051ff48730-systemd-resolved.service-54yGbQ   vmware-root_563-4281712267
systemd-private-8bac4934482d484d879054051ff48730-systemd-timesyncd.service-ti2Lns
$ exit
exit
Parent - container stopped!

查看进程(在另外一个窗口中执行)

$ ps -aux
...
root     12217  0.0  0.0   5532   720 pts/3    S    06:41   0:00 ./ns
root     12218  0.0  0.3  20312  3980 pts/3    S+   06:41   0:00 /bin/bash
...
$ pstree -g
systemd(1)─┬─VGAuthService(562)
           ├─sshd(1144)─┬─sshd(2307)───bash(2394)
           │            ├─sshd(2448)───bash(2529)
           │            ├─sshd(11698)───bash(11784)───pstree(12265)
           │            └─sshd(11845)───bash(11927)───ns(12217)───bash(12218)
           ...

由上面的操作可见，即使开启了namespace，容器进程看到的文件系统与宿主机一样

重新挂载目录实验

修改代码

修改ns.c的container_main函数，新创建文件ns_new.c

注：因为是在虚拟机上实验，根目录类型默认是shared，所以，需要先重新挂载根目录

int container_main(void* arg)
{
  printf("Container - inside the container!\n");
  // 如果你的机器的根目录的挂载类型是shared，那必须先重新挂载根目录
  mount("", "/", NULL, MS_PRIVATE, "");
  mount("none", "/tmp", "tmpfs", 0, "");
  execv(container_args[0], container_args);
  printf("Something's wrong!\n");
  return 1;
}

编译及测试效果

在容器内

$ gcc -o ns_new ns_new.c
$ ./ns_new
Parent - start a container!
Container - inside the container!
$ ls /tmp
$ mount -l | grep tmpfs
udev on /dev type devtmpfs (rw,nosuid,relatime,size=473204k,nr_inodes=118301,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=100928k,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=100924k,mode=700)
none on /tmp type tmpfs (rw,relatime)
$ exit
exit
Parent - container stopped!

在宿主机上

$ mount -l | grep tmpfs
udev on /dev type devtmpfs (rw,nosuid,relatime,size=473204k,nr_inodes=118301,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=100928k,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=100924k,mode=700)

chroot实验

准备

$ mkdir test
$ mkdir -p test/{bin,lib64,lib}
# 拷贝bash和ls命令
$ cp -v /bin/{bash,ls} test/bin/
'/bin/bash' -> 'test/bin/bash'
'/bin/ls' -> 'test/bin/ls'
# 拷贝lib
$ list="$(ldd /bin/ls | egrep -o '/lib.*\.[0-9]')"
$ mkdir test/lib/x86_64-linux-gnu
$ for i in $list; do cp -v "$i" "test${i}"; done
'/lib/x86_64-linux-gnu/libselinux.so.1' -> 'test/lib/x86_64-linux-gnu/libselinux.so.1'
'/lib/x86_64-linux-gnu/libc.so.6' -> 'test/lib/x86_64-linux-gnu/libc.so.6'
'/lib/x86_64-linux-gnu/libpcre.so.3' -> 'test/lib/x86_64-linux-gnu/libpcre.so.3'
'/lib/x86_64-linux-gnu/libdl.so.2' -> 'test/lib/x86_64-linux-gnu/libdl.so.2'
'/lib64/ld-linux-x86-64.so.2' -> 'test/lib64/ld-linux-x86-64.so.2'
'/lib/x86_64-linux-gnu/libpthread.so.0' -> 'test/lib/x86_64-linux-gnu/libpthread.so.0'
$ cp /lib/x86_64-linux-gnu/libtinfo.so.5 test/lib/x86_64-linux-gnu/

chroot

1
2
3

$ chroot test /bin/bash
bash-4.4# ls
bin  lib  lib64

以busybox的镜像为例，同样可以chroot

1
2
3

$ chroot busybox /bin/sh
/ # ls
bin   dev   etc   home  root  tmp   usr   var

UnionFS实验

$ mkdir A
$ mkdir B
$ mkdi^C
$ touch A/t1.txt
$ touch A/t2.txt
$ touch B/t2.txt
$ touch B/t3.txt
$ mkdir C
$ mount -t aufs -o dirs=./A:./B none ./C
$ ls C
t1.txt  t2.txt  t3.txt
$ mount -l | grep aufs
none on /root/bqi/C type aufs (rw,relatime,si=f9e234d74656f278)

此时，可以尝试修改A/t1.txt, A/t2.txt, B/t2.txt, B/t3.txt

$ echo 'This is a test' > A/t1.txt
$ cat C/t1.txt
This is a test
$ echo 'This is a test2' > A/t2.txt
$ cat C/t2.txt
This is a test2
$ cat B/t2.txt
$ echo 'This is test2 for B' > B/t2.txt
$ cat C/t2.txt
This is a test2

docker image解析

$ docker inspect ubuntu
...
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff:/var/lib/docker/overlay2/37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff:/var/lib/docker/overlay2/40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff",
                "MergedDir": "/var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/merged",
                "UpperDir": "/var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff",
                "WorkDir": "/var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:7789f1a3d4e9258fbe5469a8d657deb6aba168d86967063e9b80ac3e1154333f",
                "sha256:9e53fd4895597d04f8871a68caea4c686011e1fbd0be32e57e89ada2ea5c24c4",
                "sha256:2a19bd70fcd4ce7fd73b37b1b2c710f8065817a9db821ff839fe0b4b4560e643",
                "sha256:8891751e0a1733c5c214d17ad2b0040deccbdea0acebb963679735964d516ac2"
            ]
        },
...

你会看到，Ubuntu镜像，在我的环境里面是4层

overlay挂载方式

先启动一个container

$ docker run -it -d ubuntu
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
a274e38c218a        ubuntu              "/bin/bash"         17 minutes ago      Up 17 minutes                           musing_knuth
$ docker inspect a274e38c218a
...
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2-init/diff:/var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff:/var/lib/docker/overlay2/823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff:/var/lib/docker/overlay2/37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff:/var/lib/docker/overlay2/40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff",
                "MergedDir": "/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/merged",
                "UpperDir": "/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/diff",
                "WorkDir": "/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/work"
            },
            "Name": "overlay2"
        },
...

查看系统挂载表

1
2

$ cat /proc/mounts | grep overlay
overlay /var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/7ZDIR6KYXXF6RMDW3JCBEQUGMH:/var/lib/docker/overlay2/l/TH3OYMS4POUF3S22QNB7UJPORG:/var/lib/docker/overlay2/l/BJILDL5W6H6U7LGVSUUS2QUTGO:/var/lib/docker/overlay2/l/6F6BVIETKMGL5QLJIOCP6CONB3:/var/lib/docker/overlay2/l/P5BANYVEKZJYYPT4M6IFNLAR7Z,upperdir=/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/diff,workdir=/var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/work 0 0

可以看到，lowerdir由5个目录共同挂载而成，分别是

7ZDIR6KYXXF6RMDW3JCBEQUGMH
TH3OYMS4POUF3S22QNB7UJPORG
BJILDL5W6H6U7LGVSUUS2QUTGO
6F6BVIETKMGL5QLJIOCP6CONB3
P5BANYVEKZJYYPT4M6IFNLAR7Z

查看overlay2目录下的文件

$ ls /var/lib/docker/overlay2/l/ -l
total 56
lrwxrwxrwx 1 root root 72 May 14 07:09 2L7W765NSNAZAJUW324PTRY6AF -> ../6bd794b03d6772755f61a55ff28f0f20caf1541192c57030b1c0d92e4d3134fa/diff
lrwxrwxrwx 1 root root 72 May 13 07:09 2VHUX6G37XVXLON33KHDZBOVBH -> ../2787c91d4cd57511162a5b17a1ad9cca5204e57b541127223dd11b8c084710bb/diff
lrwxrwxrwx 1 root root 72 Jun 11 05:51 6F6BVIETKMGL5QLJIOCP6CONB3 -> ../37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff
lrwxrwxrwx 1 root root 72 May 13 07:24 6P3MYX3ANRVWUUGIBBTJYPGRLP -> ../6d18d5f8aed3820e7500e1f70b3c5d896b90c109977a1097e957667a6b0f48f3/diff
lrwxrwxrwx 1 root root 77 Jun 11 07:53 7ZDIR6KYXXF6RMDW3JCBEQUGMH -> ../0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2-init/diff
lrwxrwxrwx 1 root root 72 Jun 11 05:51 BJILDL5W6H6U7LGVSUUS2QUTGO -> ../823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff
lrwxrwxrwx 1 root root 72 May 13 07:24 D26SOVMOFLZLRVVBCVXJVRYAE2 -> ../a74c293f4eb20bef383865bbba97f84a51fd0d894ced280dc1cfe6021be3ae77/diff
lrwxrwxrwx 1 root root 72 Jun 11 07:53 EY46SF3NEYTFL4QOXUZT5YHMA3 -> ../0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2/diff
lrwxrwxrwx 1 root root 72 May 13 07:30 IXP5XYXVUASXI2FOX5UKMYW2JN -> ../c58e315ff14dda2b6ec7f75a3a0a8099dfe269604a0acecf6ecf026c6b56de63/diff
lrwxrwxrwx 1 root root 72 May 14 07:18 N7NRQKX4E62F2JKRQ5JCI3BSSH -> ../d6fed7e45abcf9e0055b9de876a81a5347cdcf364736b0f50053630f8f189e30/diff
lrwxrwxrwx 1 root root 72 May 13 06:16 NJCD6YK72MQFTUSJUMGOEJL4WM -> ../bf1fb537d794b4460c81ae39fc45c3230c22b47e4509a35c282ca15727fe81ac/diff
lrwxrwxrwx 1 root root 72 Jun 11 05:51 P5BANYVEKZJYYPT4M6IFNLAR7Z -> ../40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff
lrwxrwxrwx 1 root root 72 Jun 11 05:51 TH3OYMS4POUF3S22QNB7UJPORG -> ../17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff
lrwxrwxrwx 1 root root 72 May 13 07:24 YQB24YJNNQSDHU2IDBA2L4LCX4 -> ../c3bee923bb1d5cd56503c976bc8353a6a579698186536f6023524b84373a6834/diff

做个对比

容器ID为a274e38，对应的DIR的ID是0c3ff90
lowerdir=/var/lib/docker/overlay2/l/7ZDIR6KYXXF6RMDW3JCBEQUGMH，实际上指向了0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2-init/diff
1
2
$ ls /var/lib/docker/overlay2/0c3ff90aba26b4197b2293789f75d4e3db7a9213601b8d222b1f0c413c7115b2-init/diff/
dev etc
/var/lib/docker/overlay2/l/TH3OYMS4POUF3S22QNB7UJPORG 实际上指向了17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff
1
2
$ ls /var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff/
run
而17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff实际上是ubuntu镜像的UpperDir
1
2
$ ls /var/lib/docker/overlay2/17bd5da0cda20e8ecd1d4955d25f49609ff0d7aa72fe45a0388a357fcc5b625f/diff/
run
/var/lib/docker/overlay2/l/BJILDL5W6H6U7LGVSUUS2QUTGO实际上是823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff

而823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff实际上是ubuntu镜像的LowerDir

1 2	$ ls /var/lib/docker/overlay2/823e415d4256d05fb0101af4dcc42a4389d44cf6467972d654e93e0cc575cd9b/diff/ etc usr var

/var/lib/docker/overlay2/l/6F6BVIETKMGL5QLJIOCP6CONB3实际上是37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff
1
2
$ ls /var/lib/docker/overlay2/37d3e588905fae55c8a0481e9cda7be36177af874631abb15724c893887e260b/diff/
var

/var/lib/docker/overlay2/l/P5BANYVEKZJYYPT4M6IFNLAR7Z实际上是40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff

1
2

$ ls /var/lib/docker/overlay2/40d198d6f624e455800254766eb6a7190ce02442fc48f02f6f16f72105cefd0d/diff/
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

小结

容器的镜像即rootfs是按照一层一层的组合起来的。
启动容器进程时，将多个增量的rootfs联合挂载成一个完整的rootfs
启动容器时，会只读模式挂载一个init层，以及一个可写的层

K8s学习笔记——限制与隔离

2020-06-17T08:08:05.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则.
对应章节：06 | 白话容器基础（二）：隔离与限制

隔离与限制

cgroup的mount点

$ mount -t cgroup
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

查看CPU相关信息

$ cd /sys/fs/cgroup
$ ls cpu
cgroup.clone_children  cpuacct.stat       cpuacct.usage_percpu       cpuacct.usage_sys   cpu.cfs_quota_us  docker             system.slice
cgroup.procs           cpuacct.usage      cpuacct.usage_percpu_sys   cpuacct.usage_user  cpu.shares        notify_on_release  tasks
cgroup.sane_behavior   cpuacct.usage_all  cpuacct.usage_percpu_user  cpu.cfs_period_us   cpu.stat          release_agent      user.slice
$ cat cpu/cpu.cfs_period_us
100000
$ cat cpu/cpu.cfs_quota_us
-1

在CPU目录下创建一个container目录

$ mkdir container
$ ls container/
cgroup.clone_children  cpuacct.usage_all          cpuacct.usage_sys   cpu.shares
cgroup.procs           cpuacct.usage_percpu       cpuacct.usage_user  cpu.stat
cpuacct.stat           cpuacct.usage_percpu_sys   cpu.cfs_period_us   notify_on_release
cpuacct.usage          cpuacct.usage_percpu_user  cpu.cfs_quota_us    tasks

会看到创建目录后，会自动创建一堆文件

用一个while循环检测CPU使用状况

$ while : ; do : ; done &
[1] 16508
$ top
top - 09:03:06 up  3:32,  4 users,  load average: 0.28, 0.07, 0.02
Tasks: 105 total,   2 running,  61 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1009256 total,   208232 free,   220168 used,   580856 buff/cache
KiB Swap:  2018300 total,  2008256 free,    10044 used.   647692 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
16508 root      20   0   21640   3180   1216 R 99.7  0.3   0:17.88 bash

在有的虚拟机上，你会看到CPU使用率不是99%，可能是50%，25%等，你可以思考一下这是为什么

使用container的cgroup对while循环进行资源限制

$ echo 20000 > container/cpu.cfs_quota_us
$ echo 16508 > container/tasks
$ top
top - 09:06:24 up  3:35,  4 users,  load average: 0.97, 0.52, 0.21
Tasks: 105 total,   2 running,  61 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.0 us,  0.0 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  1009256 total,   208636 free,   219736 used,   580884 buff/cache
KiB Swap:  2018300 total,  2008256 free,    10044 used.   648120 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
16508 root      20   0   21640   3180   1216 R 20.6  0.3   3:31.08 bash

起一个容器看看

$ docker run -it --cpu-period=100000 --cpu-quota=20000 ubuntu /bin/bash
root@26cbffdcb5bf:/# cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
root@26cbffdcb5bf:/# cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
20000
# 在宿主机上
$ cat /sys/fs/cgroup/cpu/docker/26cbffdcb5bf2f9d9ecdc9207f4211c8f5b3cfbc39d83c77ed4666db3ca0bac3/cpu.cfs_period_us
100000
$ cat /sys/fs/cgroup/cpu/docker/26cbffdcb5bf2f9d9ecdc9207f4211c8f5b3cfbc39d83c77ed4666db3ca0bac3/cpu.cfs_quota_us
20000

top对比

# 容器中
root@26cbffdcb5bf:/# while : ; do : ; done &
[1] 11
root@26cbffdcb5bf:/# top
top - 09:10:34 up  3:39,  0 users,  load average: 0.01, 0.21, 0.15
Tasks:   3 total,   2 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.3 us,  0.0 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    985.6 total,    163.9 free,    251.5 used,    570.2 buff/cache
MiB Swap:   1971.0 total,   1961.2 free,      9.8 used.    595.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   11 root      20   0    4224    560      0 R  20.3   0.1   0:01.21 bash
    1 root      20   0    4224   3504   2944 S   0.0   0.3   0:00.19 bash
   12 root      20   0    6080   3244   2740 R   0.0   0.3   0:00.00 top
   
# 宿主机上
$ top
top - 09:11:36 up  3:40,  5 users,  load average: 0.10, 0.20, 0.15
Tasks: 111 total,   2 running,  66 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.0 us,  0.3 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1009256 total,   167472 free,   257600 used,   584184 buff/cache
KiB Swap:  2018300 total,  2008256 free,    10044 used.   610176 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
16755 root      20   0    4224    560      0 R 19.9  0.1   0:13.62 bash
$ pstree -g
systemd(1)─┬─VGAuthService(562)
           ├─accounts-daemon(866)─┬─{accounts-daemon}(866)
           │                      └─{accounts-daemon}(866)
           ├─atd(928)
           ├─containerd(1055)─├─containerd-shim(16557)─┬─bash(16585)───bash(16755)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        ├─{containerd-shim}(16557)
           │                  │                        └─{containerd-shim}(16557)

可以看到容器内和容器外部看到的是一样的

总结

主要使用了cgroup对一个进程的CPU使用率进行了限制，通过了解进程的CPU使用率限制机制，了解docker通过cgroup对相关资源使用的限制

K8s学习笔记——container之于进程

2020-06-17T05:35:07.000Z

学习极客时间上的《深入剖析Kubernetes》
秉持眼过千遍不如手过一遍的原则.
对应章节：05 | 白话容器基础（一）：从进程说开去

操作

start一个container

1	$ docker run -it -d busybox

查看进程

$ ps -aux
...
root      2817  0.0  0.2 107700  2296 ?        Sl   05:42   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.run
root      2851  0.0  0.0   1308     4 pts/0    Ss+  05:42   0:00 sh
...

查看进程树

$ pstree  -g
systemd(1)─┬─VGAuthService(562)
           ├─accounts-daemon(866)─┬─{accounts-daemon}(866)
           │                      └─{accounts-daemon}(866)
           ├─atd(928)
           ├─containerd(1055)─┬─containerd-shim(2817)─┬─sh(2851)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       ├─{containerd-shim}(2817)
           │                  │                       └─{containerd-shim}(2817)
 ....

查看容器内进程

$ ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
    6 root      0:00 ps

分别查看两个进程的namespace

/proc/2817/ns# ls -l
total 0
lrwxrwxrwx 1 root root 0 Jun 11 05:43 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 net -> 'net:[4026531993]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Jun 11 06:16 pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 uts -> 'uts:[4026531838]
/proc/2851/ns# ls -l
total 0
lrwxrwxrwx 1 root root 0 Jun 11 05:43 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 ipc -> 'ipc:[4026532571]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 mnt -> 'mnt:[4026532569]'
lrwxrwxrwx 1 root root 0 Jun 11 05:42 net -> 'net:[4026532574]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 pid -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 11 06:16 pid_for_children -> 'pid:[4026532572]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Jun 11 05:43 uts -> 'uts:[4026532570]'

总结

启动一个docker容器后，会看到启动了一个2817的进程，这个进程是1055的子进程
而因为busybox容器启动后，启动了sh，其实际上是2817的子进程2851
而在容器中，能看到1号进程是sh
通过/proc下可以看到，进程2817和进程2851的ns下，cgroup是都是4026531835
而很明显，每个container都会创建ipc, mnt, net, pid, pid_for_children, user, uts这些namespace

国内源安装k8s——Ubuntu

2020-05-19T09:16:31.000Z

阿里云源
Ubuntu20.04
master + slave

安装docker

分别在两个node上安装docker-ce

$ apt-get update 
$ apt-get -y install apt-transport-https ca-certificates curl software-properties-common
$ curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
$ add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
$ apt update
$ apt-get -y install docker-ce

安装kubeadm，kubectl，kubelet

分别在两个node上安装

$ apt-get update && apt-get install -y apt-transport-https
$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
$ cat </etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl

初始化master节点

1	$ kubeadm init --pod-network-cidr=172.172.0.0/16 --image-repository registry.aliyuncs.com/google_containers

安装成功后，会有如下信息：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.160.18.180:6443 --token 5xxosi.3du1z15pevcvnyyx \
    --discovery-token-ca-cert-hash sha256:4cc4977482e04ac0ca845bf3520a6a5fa8a0cf6ac8233e734a47e0250c259f73

根据提示，执行

1
2
3

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装flannel

参考： https://github.com/coreos/flannel

1	$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

安装dashboard

参考：https://github.com/kubernetes/dashboard

1	$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

修改dashboard配置

修改spec中的type为NodePort

$ kubectl -n kubernetes-dashboard edit service kubernetes-dashboard
.......
spec:
  clusterIP: 10.101.212.193
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 32609
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort

修改成功后，查看port信息

1
2
3

$ kubectl -n kubernetes-dashboard get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.101.212.193           443:32609/TCP   27m

现在，可以通过https://:（这里的port是32609）来访问dashboard了

虽然页面已经展示出来了，但需要使用token或Kubeconfig才能访问

创建sample-user

创建服务账号

新建dashboard-adminuser.yaml并写入：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

执行：

1	$ kubectl apply -f dashboard-adminuser.yaml

创建ClusterRoleBinding

新建cluster-role-binding.yaml并写入：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

执行：

1	$ kubetcl apply -f cluster-role-binding.yaml

获取token

$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
Name:         admin-user-token-jmggp
Namespace:    kubernetes-dashboard
Labels:       
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: 58210c16-0fac-438c-8867-d0a3e7b950b9

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlhTSnlXMUhXTlNnUmd4MlVMTzdtbm14YVdiSzNUdjk4UnVoZ3RRbUFXZGsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWptZ2dwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1ODIxMGMxNi0wZmFjLTQzOGMtODg2Ny1kMGEzZTdiOTUwYjkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.F4TKNO_6Guu-vcLUtELUOhRI2dGMcZ3V1et2evono_a6f-TvCR9c4pbyYCnRdCG6_MumTmyE5W1g3zHioVnb5TgnGwfmAfIWLltwwLEOxOdLfO7oqM8zrYfzZnIH16SoOZQYMU7xIk5MhE5WN265n8Q2kpDMraf0L06_nqNy1pq8h9eaX0QIntosl4fmf9KVew0geLCKbknEwpnzGGfSCcKLLgE7a45ACWwStJiL29t69gcKJ6ze33MXpA5_irk2nKkavXbKEk7ejapgYK66nOxJnDKgbNVDcBP47xHrPjGeeupB6bw6uUMWxA6z4kJUTVRepk6yTMGVDPzB9Muicw

现在，可以使用token登录Dashboard了

slave节点加入集群

1 2	$ kubeadm join 10.160.18.180:6443 --token 5xxosi.3du1z15pevcvnyyx \ --discovery-token-ca-cert-hash sha256:4cc4977482e04ac0ca845bf3520a6a5fa8a0cf6ac8233e734a47e0250c259f73

问题及解决方案

docker cgroup driver问题

问题日志

1	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

解决方法

在/etc/docker/下创建daemon.json

cat > /etc/docker/daemon.json <
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

重启docker进程

1 2	$ systemctl restart docker $ systemctl status docker

swap问题

问题日志

1	[ERROR Swap]: running with swap on is not supported. Please disable swap

解决方法

1 2	$ swapoff -a # 在所有node上执行

但这只是暂时关闭了swap，重启node后，就会再次打开。需要修改/etc/fstab，在swap那行加上#

1	#UUID=4eeb5155-41f9-4478-a420-2beb4290a721 none swap sw 0 0

Node处于NotReady状态

node处于NotReady状态的原因有很多。可以一步一步处理

$ kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
node1   Ready    master   4h56m   v1.18.2
node2   NotReady       4h6m    v1.18.2

先查看错误原因：

$ kubectl get pod -n kube-system
NAME                            READY   STATUS                  RESTARTS   AGE
coredns-7ff77c879f-2k7rw        1/1     Running                 1          4h47m
coredns-7ff77c879f-q76jr        1/1     Running                 1          4h47m
etcd-node1                      1/1     Running                 2          4h47m
kube-apiserver-node1            1/1     Running                 2          4h47m
kube-controller-manager-node1   1/1     Running                 2          4h47m
kube-flannel-ds-amd64-2jn8n     0/1     Init:ImagePullBackOff   0          3h49m
kube-flannel-ds-amd64-ftpxl     1/1     Running                 1          3h49m
kube-proxy-5q8wp                1/1     Running                 2          4h47m
kube-proxy-wfcjq                0/1     ContainerCreating       0          5m46s
kube-scheduler-node1            1/1     Running                 2          4h47m

k8s有些服务会在各个节点上启动，比如这里的proxy，flannel。

$ kubectl describe pod -n kube-system kube-flannel-ds-amd64-2jn8n
.....
Events:
  Type     Reason                  Age                    From            Message
  ----     ------                  ----                   ----            -------
  Normal   Pulling                 6m51s                  kubelet, node2  Pulling image "quay.io/coreos/flannel:v0.12.0-amd64"
  Warning  Failed                  5m48s                  kubelet, node2  Failed to pull image "quay.io/coreos/flannel:v0.12.0-amd64": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/coreos/flannel/manifests/v0.12.0-amd64: received unexpected HTTP status: 500 Internal Server Error
  Warning  Failed                  5m48s                  kubelet, node2  Error: ErrImagePull
  Normal   BackOff                 5m47s                  kubelet, node2  Back-off pulling image "quay.io/coreos/flannel:v0.12.0-amd64"
  Warning  Failed                  5m47s                  kubelet, node2  Error: ImagePullBackOff
  Normal   Pulling                 5m36s (x2 over 5m51s)  kubelet, node2  Pulling image "quay.io/coreos/flannel:v0.12.0-amd64"

最常见的是ImagePull失败。比如master node上镜像拉取正常，而其他节点拉取失败。

解决方法

1. 在slave节点上手工拉取镜像

1	$ docker pull quay.io/coreos/flannel:v0.12.0-amd64

2. 将master节点上的镜像导入slave节点

查看master节点上的镜像

(master)$ docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
kubernetesui/dashboard                                            v2.0.0              8b32422733b3        3 weeks ago         222MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.18.2             0d40868643c6        4 weeks ago         117MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.18.2             a3099161e137        4 weeks ago         95.3MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.18.2             6ed75ad404bd        4 weeks ago         173MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.18.2             ace0a8c17ba9        4 weeks ago         162MB
kubernetesui/metrics-scraper                                      v1.0.4              86262685d9ab        7 weeks ago         36.9MB
quay.io/coreos/flannel                                            v0.12.0-amd64       4e9f801d2217        2 months ago        52.8MB
registry.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        3 months ago        683kB
registry.aliyuncs.com/google_containers/coredns                   1.6.7               67da37a9a360        3 months ago        43.8MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        6 months ago        288MB

将镜像导出为文件

1	(master)$ docker save quay.io/coreos/flannel > flannel.tar

将文件传输到slave节点上
slave节点上导入镜像

(slave)$ docker load < flannel.tar
256a7af3acb1: Loading layer [==================================================>]  5.844MB/5.844MB
d572e5d9d39b: Loading layer [==================================================>]  10.37MB/10.37MB
57c10be5852f: Loading layer [==================================================>]  2.249MB/2.249MB
7412f8eefb77: Loading layer [==================================================>]  35.26MB/35.26MB
05116c9ff7bf: Loading layer [==================================================>]   5.12kB/5.12kB
Loaded image: quay.io/coreos/flannel:v0.12.0-amd64

Now， all is OK

Ubuntu安装IPSAN

2020-04-17T09:00:27.000Z

参考文章

IPSAN的定义就不讲了，主要讲一下搭建环境。IPSAN主要包括两种node：

initiator：可以理解为客户端
target：可以理解为服务器端

实际上也不存在客户端服务器端的理解。target端主要是把磁盘share出来给其他设备用的。initiator相当于用来映射target端的磁盘到本地的。

安装及配置target端

服务器准备(ip: 192.168.0.100)

服务器需要有两块磁盘，如果使用虚拟机环境，则特别简单

第一块硬盘用来安装系统，大小40G就够了
第二块硬盘用来通过IPSAN来share，按需来创建，最小2G

安装软件包

在Ubuntu上安装target端很简单，主要是安装tgt软件包

# 升级软件包
$ apt update -y
$ apt upgrade -y
# 安装tgt
$ apt install tgt -y

默认状况下，安装完毕后，tgt服务就应该已经正常运行了。当然，可以确认是否已经运行正常

1	$ systemctl status tgt

配置

安装完成后，一般情况是没有这个文件的，需要自己创建一个。创建配置文件/etc/tgt/conf.d/iscsi.conf

<target abc.id123.testsite.xyz:lun1>
     backing-store /dev/sdb
     initiator-address 192.168.0.0/16
     incominguser test password
     outgoinguser test password
target>

配置说明

target后面为节点名称，随便写
backing-store，即将要share出去的磁盘
initiator-address，即可以允许连接的IP地址或地址范围
incominguser，入的用户名密码（我的理解是写权限）
outgoinguser，出的用户名密码（我的理解是读权限）

检查配置

1	$ systemctl restart tgt

查看状态

$ tgtadm --mode target --op show
Target abc.id123.testsite.xyz:lun1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 2146 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/sdb
            Backing store flags: 
    Account information:
        test
        test (outgoing)
    ACL information:
        192.168.0.0/16

安装及配置initiator

如果你只是想创建一个IPSAN共享存储出去，那么不需要安装initiator。本例则同样通过另外一台Ubuntu服务器来连接target端

安装及配置initiator(192.168.0.200)

安装open-iscsi软件包

1	$ apt-get install open-iscsi -y

探测target端

1 2	$ iscsiadm -m discovery -t st -p 192.168.0.100 192.168.0.100:3260,1 abc.id123.testsite.xyz:lun1

修改配置文件

探测完成后，将会在/etc/iscsi/nodes/目录和/etc/iscsi/send_targets/目录下看到对应的target

$ ls -l /etc/iscsi/nodes/abc.id123.testsite.xyz\:lun1/192.168.0.100\,3260\,1/ 
total 4
-rw------- 1 root root 1840 Nov  8 13:17 default

$ ls -l /etc/iscsi/send_targets/192.168.0.100,3260/
total 8
lrwxrwxrwx 1 root root  66 Nov  8 13:17 abc.id123.testsite.xyz:lun1,192.168.0.100,3260,1,default -> /etc/iscsi/nodes/abc.id123.testsite.xyz:lun1/192.168.0.100,3260,1
-rw------- 1 root root 547 Nov  8 13:17 st_config

修改default文件，增加CHAP的配置（认证相关）

vim /etc/iscsi/nodes/abc.id123.testsite.xyz\:lun1/192.168.0.100\,3260\,1/default，新增如下内容

node.session.auth.authmethod = CHAP
node.session.auth.username = test
node.session.auth.password = password
node.session.auth.username_in = test
node.session.auth.password_in = password
node.startup = automatic

验证

重启服务

1	$ systemctl restart open-iscsi

检查是否映射完成

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   40G  0 disk
├─sda1   8:1    0  512M  0 part /boot/efi
└─sda2   8:2    0 39.5G  0 part /
sdb      8:16   0  120G  0 disk
└─sdb1   8:17   0  120G  0 part
sr0     11:0    1 1024M  0 rom

其中的sdb即为通过IPSAN连接的磁盘。可以以本地磁盘的方式进行访问

Openstack学习 —— 跨主机虚拟机访问2

2019-11-27T06:47:24.000Z

上篇文章梳理了分别在两个node上创建了VM后，底层的Linux系统上的namespace、linux bridge以及ovs中发生的事情。

本文将来着重关注两个node上的VM相互访问的流量通路。特别是令人头疼的ovs流表以及两个node是如何通过VXLAN网络将两台虚拟机连在了一起的。

br-tun

在前面收集信息时，你或许已经关注到了linux bridge和ovs中还有其他变化，那么，来看看 br-tun

(node0)$ ovs-vsctl show
...
    Bridge br-tun
        ...
        Port "vxlan-ac0a000b"
            Interface "vxlan-ac0a000b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.10.0.10", out_key=flow, remote_ip="172.10.0.11"}
        ...
        
(node1)$ ovs-vsctl show
...
    Bridge br-tun
        ...
        Port "vxlan-ac0a000a"
            Interface "vxlan-ac0a000a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.10.0.11", out_key=flow, remote_ip="172.10.0.10"}
        ...

当环境搭建完毕后，并不会立即创建这个接口，而是当创建了虚拟机后，会创建VXLAN的VTEP接口。

流表

node0

br-int

$ ovs-ofctl dump-flows br-int
 table=0, ..., priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
 table=0, ..., priority=10,icmp6,in_port="qvo02407769-97",icmp_type=136 actions=resubmit(,24)
 table=0, ..., priority=10,arp,in_port="qvo02407769-97" actions=resubmit(,24)
 table=0, ..., priority=2,in_port="int-br-provider" actions=drop
 table=0, ..., priority=2,in_port="int-br-ext" actions=drop
 table=0, ..., priority=9,in_port="qvo02407769-97" actions=resubmit(,25)
 table=0, ..., priority=0 actions=resubmit(,60)
 table=23, ..., priority=0 actions=drop
 table=24, ..., priority=2,icmp6,in_port="qvo02407769-97",icmp_type=136,nd_target=fe80::f816:3eff:fead:669e actions=resubmit(,60)
 table=24, ..., priority=2,arp,in_port="qvo02407769-97",arp_spa=200.0.0.219 actions=resubmit(,25)
 table=24, ..., priority=0 actions=drop
 table=25, ..., priority=2,in_port="qvo02407769-97",dl_src=fa:16:3e:ad:66:9e actions=resubmit(,60)
 table=60, ..., priority=3 actions=NORMAL

table 0

从qvo02407769-97送入的icmp，arp报文，送往table 24
从qvo02407769-97送入其他报文送往table 25
其他报文送table 60

table 24

从qvo02407769-97送入的报文，送往table 60。fe80::f816:3eff:fead:669e是vm-1的IPv6地址
从qvo02407769-97送入的arp报文，arp源地址为200.0.0.219即vm-1来的报文，送往table 25

table 25

从qvo02407769-97送入的报文，源mac为fa:16:3e:ad:66:9e即vm-1来的报文，送往table 60

table 60

正常转发

综上，从vm-1来的报文以及其他报文，都将正常在br-int上转发

br-tun

$ ovs-ofctl dump-flows br-tun
 table=0, ..., priority=1,in_port="patch-int" actions=resubmit(,2)
 table=0, ..., priority=1,in_port="vxlan-ac0a000b" actions=resubmit(,4)
 table=0, ..., priority=0 actions=drop
 table=2, ..., priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 table=2, ..., priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 table=3, ..., priority=0 actions=drop
 table=4, ..., priority=1,tun_id=0xf actions=mod_vlan_vid:1,resubmit(,10)
 table=4, ..., priority=0 actions=drop
 table=6, ..., priority=0 actions=drop
 table=10, ..., priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x9c915752f14d2544,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int"
 table=20, ..., priority=2,dl_vlan=1,dl_dst=fa:16:3e:f5:ca:f5 actions=strip_vlan,load:0xf->NXM_NX_TUN_ID[],output:"vxlan-ac0a000b"
 table=20, ..., hard_timeout=300, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:f5:ca:f5 actions=load:0->NXM_OF_VLAN_TCI[],load:0xf->NXM_NX_TUN_ID[],output:"vxlan-ac0a000b"
 table=20, ..., priority=0 actions=resubmit(,22)
 table=22, ..., priority=1,dl_vlan=1 actions=strip_vlan,load:0xf->NXM_NX_TUN_ID[],output:"vxlan-ac0a000b"
 table=22, ..., priority=0 actions=drop

table 0

从br-int来的报文（即需要从本节点送出的报文），送到table 2
从vxlan-ac0a000b送来的报文（即从其他节点送到node 0的），送到table 4

table 2：将要送出本节点的报文

送往table 20
送往table 22

table 4：从其他节点送到本节点的报文

tun_id为15（0xf）的报文，打上vlan标签1，送往table 10

即从vxlan的tunnel出来的报文，如果tunnel id是15，则报文设置vlan为1
而qvo02407769-97（与vm-1相连）和tap17a89323-fd（DHCP的tap接口）的vlan tag正是1

table 10

learn(table=20,hard_timeout=300,priority=1,cookie=0x9c915752f14d2544,NXM_OF_VLAN_TCI[0…11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:“patch-int”

这个action是如此的复杂，但不看learn()中的内容，报文最终被送往了br-int

learn则是在table20中增加对于回程报文的转发规则

table 20

送往table 22

table 22

VLAN tag为1的报文，去掉VLAN，设置tun_id为15（0xf）后，送往vxlan-ac0a000b

综上：
从vm-1送来的报文，携带VLAN tag为1，去掉VLAN tag，设置tun_id为15，从vxlan接口送出
从其他节点送来的报文，如果tun_id为15，设置VLAN tag为1，送往br-int

总结

完整的分析了node0上的流表，node1上的流表内容基本相似，就不再展开。

至此，跨节点的虚拟机相互访问的实验及分析正式完结。你会发现

每一个port在连接到虚拟机的时候，都创建了一个网桥
每个虚拟机连在br-int上的接口，都按照subnet分配了一个VLAN tag，且每个节点的并不相同
当虚拟机的报文要送出/进入当前节点时，会有VLAN tag和VXLAN的tun_id相互转换

现在，思考一下：

node1上的vm-1是如何通过DHCP获取IP地址的？
为什么虚拟机不直接连在br-int上，而是要通过一个linux bridge连接到br-int上呢？