《基于 Kubernetes 的云原生 DevOps》第12章部署 Kubernetes 应用程序

> I lay on my back, surprised at how calm and focused I felt, strapped to four and a half million pounds of explosives.
> -- Ron Garan, astronaut

## 12.1 使用 Helm 构建清单

**Helm Chart 包含什么？**

每个 *Chart* 都有一个标准的结构。

首先 *Chart* 包含在同名目录中（这里的例子中为 *demo* ）。

```powershell
demo
  │  Chart.yaml
  │  production-values.yaml
  │  staging-values.yaml
  │  values.yaml
  │
  └─templates
          deployment.yaml
          service.yaml
```

其次，包含一个名为 *Chart.yaml* 的文，其中指定了 *Chart* 的名称及版本：

```yaml
name: demo
sources:
- https://github.com/cloudnativedevops/cloudnatived
version: 1.0.1
```

**Chart.yaml 文件**

*Chart.yaml* 提供很多可选字段，包括指向项目源代码的链接，但必须的信息只有名称和版本。

**values.yaml**

*values.yaml* 文件包含 Chart 作者公开的、可供用户修改的设置：

```yaml
environment: development
container:
  name: demo
  port: 8888
  image: cloudnatived/demo
  tag: hello
replicas: 2
```

*values.yaml* 完全是自由格式的 YAML ，没有预定义的结构，可以随意选择定义哪些变量、变量的名称和值。

**Helm 模板**

模板文件保存在 *templates* 目录下，这些文件正是前面示例中的清单文件，不同之处在于它包含了很多占位符，而 Helm 可以利用 *values.yaml* 中的实际值替换这些占位符。

*deployment.yaml*

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.container.name }}-{{ .Values.environment }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.container.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.container.name }}
        environment: {{ .Values.environment }}
    spec:
      containers:
        - name: {{ .Values.container.name }}
          image: {{ .Values.container.image }}:{{ .Values.container.tag }}
          ports:
            - containerPort: {{ .Values.container.port }}
          env:
            - name: environment
              value: {{ .Values.environment }}
```

> 大括号指明了 Helm 应替换成变量值的位置，但实际上它们是 Go 模板语法的一部分。

**插值变量**

例如

```yaml
metadata:
  name: {{ .Values.container.name }}-{{ .Values.environment }}
```

会被替换为

```yaml
metadata:
  name: demo-development
```

使用 Helm 变量可以避免重复。除了简单的变量替换之外，还支持循环、表达式、条件分支，甚至函数调用。

**引用模板中的值**

使用 Helm 中的 `quote` 函数给模板中的值自动添加引号：

```yaml
  name: {{ .Values.MyName | quote }}
```

只有字符串应该加引号，切勿对端口号等数字值使用 `quote` 函数。

**指定依赖项**

*Chart* 可以依赖于其它 *Chart* 。如你的程序适应 *Redis* , 就可以指定 *redis* Chart 作为依赖。

可以在 *requirements.yaml* 文件中指定依赖：

```yaml
dependencies:
  - name: redis
    version: 1.2.3
  - name: nginx
    version: 3.2.1
```

接下来运行 `helm dependency update` 命令， Helm 就会下载这些 Chart ，然后与应用程序一起安装。

## 12.2 部署 Helm Chart

**设置变量**

可以通过 `helm install` 命令在命令行中指定包含其它值的文件，而这些值会覆盖 *values.yaml* 中的默认值。

创建一个新的 YAML 文件 *staging-values.yaml* ：

```yaml
environment: staging
```

**在 Helm Release 中指定值**

通过 `--values` 标志指定额外的值的文件。

```powershell
helm install --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\
```

命令行的书写格式和书中不太一样，估计是因为 *Helm* 的版本不一样。

> **运行结果：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm install --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\
> NAME: demo-staging
> LAST DEPLOYED: Wed May 18 17:27:56 2022
> NAMESPACE: default
> STATUS: deployed
> REVISION: 1
> TEST SUITE: None
> ```
>
> **查看部署：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl describe deploy/demo
> Name:                   demo
> Namespace:              default
> CreationTimestamp:      Wed, 18 May 2022 17:27:56 +0800
> Labels:                 app.kubernetes.io/managed-by=Helm
> Annotations:            deployment.kubernetes.io/revision: 1
>                         meta.helm.sh/release-name: demo-staging
>                         meta.helm.sh/release-namespace: default
> Selector:               app=demo
> Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
> StrategyType:           RollingUpdate
> MinReadySeconds:        0
> RollingUpdateStrategy:  25% max unavailable, 25% max surge
> Pod Template:
>   Labels:  app=demo
>            environment=staging
>   Containers:
>    demo:
>     Image:      cloudnatived/demo:hello
>     Port:       8888/TCP
>     Host Port:  0/TCP
>     Environment:
>       environment:  staging
>     Mounts:         <none>
>   Volumes:          <none>
> Conditions:
>   Type           Status  Reason
>   ----           ------  ------
>   Available      True    MinimumReplicasAvailable
>   Progressing    True    NewReplicaSetAvailable
> OldReplicaSets:  <none>
> NewReplicaSet:   demo-5b8b9fd8d9 (2/2 replicas created)
> Events:
>   Type    Reason             Age   From                   Message
>   ----    ------             ----  ----                   -------
>   Normal  ScalingReplicaSet  11m   deployment-controller  Scaled up replica set demo-5b8b9fd8d9 to 2
> ```
>
> 可以看到环境变量 *environment* 的值为 *staging* 。
>
> **查看机密：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl get secret
> NAME                                 TYPE                                  DATA   AGE
> default-token-hxl8b                  kubernetes.io/service-account-token   3      163d
> sh.helm.release.v1.demo-staging.v1   helm.sh/release.v1                    1      14m
> ```
>
> 可以看到 helm release 的名字为 *demo-staging* 。

**使用 Helm 更新应用程序**

```powershell
helm upgrade --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\
```

> **运行结果：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm upgrade --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\
> Release "demo-staging" has been upgraded. Happy Helming!
> NAME: demo-staging
> LAST DEPLOYED: Wed May 18 17:47:37 2022
> NAMESPACE: default
> STATUS: deployed
> REVISION: 2
> TEST SUITE: None
> ```
>
> **查看机密：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl get secret
> NAME                                 TYPE                                  DATA   AGE
> default-token-hxl8b                  kubernetes.io/service-account-token   3      163d
> sh.helm.release.v1.demo-staging.v1   helm.sh/release.v1                    1      19m
> sh.helm.release.v1.demo-staging.v2   helm.sh/release.v1                    1      16s
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl describe secret/sh.helm.release.v1.demo-staging.v2
> Name:         sh.helm.release.v1.demo-staging.v2
> Namespace:    default
> Labels:       modifiedAt=1652867257
>               name=demo-staging
>               owner=helm
>               status=deployed
>               version=2
> Annotations:  <none>
>
> Type:  helm.sh/release.v1
>
> Data
> ====
> release:  1604 bytes
> ```

**回滚到以前的版本**

```powershell
helm rollback demo-staging 1
```

> **运行结果：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm rollback demo-staging 1
> Rollback was a success! Happy Helming!
> ```
>
> **查看机密：**
>
> 生成了一个新的版本 *v3* 。
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> k get secret
> NAME                                 TYPE                                  DATA   AGE
> default-token-hxl8b                  kubernetes.io/service-account-token   3      163d
> sh.helm.release.v1.demo-staging.v1   helm.sh/release.v1                    1      30m
> sh.helm.release.v1.demo-staging.v2   helm.sh/release.v1                    1      10m
> sh.helm.release.v1.demo-staging.v3   helm.sh/release.v1                    1      30s
> ```
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helm3> k describe secret/sh.helm.release.v1.demo-staging.v3
> Name:         sh.helm.release.v1.demo-staging.v3
> Namespace:    default
> Labels:       modifiedAt=1652867847
>               name=demo-staging
>               owner=helm
>               status=deployed
>               version=3
> Annotations:  <none>
>
> Type:  helm.sh/release.v1
>
> Data
> ====
> release:  1600 bytes
> ```

**使用 *helm-monitor* 自动回滚**

*helm-monitor* 插件可以通过任何指标表达式查询 Prometheus 服务器，并在查询成功时触发回滚。 *helm-monitor* 将监视指标五分钟，如果在此期间发现问题，则回滚。有关 *helm-monitor* 的更多信息，请参阅[这篇博客](https://blog.container-solutions.com/automated-rollback-helm-releases-based-logs-metrics "Automated rollback of Helm releases based on logs or metrics")。

**创建 Helm Chart 库**

通常应用程序的 Helm Chart 会存储在应用程序自己的代码库中。但是如果有需要，维护自己的 Helm Chart 库也非常简单。

你需要通过 HTTP 提供 Chart，实现方式有两种：

1. 将它们存储到云存储桶中；
2. 使用 GitHub Pages 或利用现有的 Web 服务器。

将所有的 Chart 汇集到一个目录中后，直接在目录中运行 `helm repo index` 即可创建 Chart 库元数据的 *index.yaml* 文件。

有关 Chart 库的更多信息，请参见 Helm 文档（ [The Chart Repository Guide](https://v2.helm.sh/docs/developing_charts/#the-chart-repository-guide "The Chart Repository Guide") ）。

如果想安装自己库中的 Chart ，首先需要将你的 Chart 库添加到 Helm 的列表中：

```powershell
helm repo add myrepo http://myrepo.example.com
helm install myrepo/myapp
```

卸载 *demo-staging* :

```powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm uninstall demo-staging
release "demo-staging" uninstalled
```

**使用 Sops 管理 Helm Chart 的机密数据**

*demo* 代码库的 *hello-sops* 目录下提供了一个示例。

```powershell
PS C:\projects\github\cloudnativedevops\demo> cd .\hello-sops\
PS C:\projects\github\cloudnativedevops\demo\hello-sops> tree /f
卷 system 的文件夹 PATH 列表
卷序列号为 0000022B 1ABC:16A8
C:.
│  temp.yaml
│
└─k8s
    └─demo
        │  Chart.yaml
        │  production-secrets.yaml
        │  production-values.yaml
        │  staging-secrets.yaml
        │  staging-values.yaml
        │  values.yaml
        │
        └─templates
                deployment.yaml
                secrets.yaml
```

模板文件里有个机密数据清单文件 *k8s\demo\templates\secrets.yaml* ：

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.container.name }}-secrets
type: Opaque
data:
  {{ $environment := .Values.environment }}
  secrets.yaml: {{ .Files.Get (nospace (cat $environment "-secrets.yaml")) | b64enc }}
```

真正的机密数据则保存在 *k8s\demo\production-secrets.yaml* 和 *k8s\demo\staging-secrets.yaml* 文件中。

最后一行的末尾添加了 `| b64enc` 。这种写法是一种快捷方式，可以使用 Helm 的 Go 模板自动将机密数据从纯文本转化为 *base64* ，在默认情况下， Kubernetes 希望机密数据保存成 *base64* 的形式。

*staging-secrets.yaml*

```yaml
secret_one: ENC[AES256_GCM,data:hJ/QcnvOxgfjMhY=,iv:7AC+hW/ouAeIF9yTACttPQ7sq/QHSSvHgI+DSuZDOnY=,tag:fRqJ2QiORl04uxa1FpZZKA==,type:str]
secret_two: ENC[AES256_GCM,data:swFK66/mAEX3Ohg=,iv:fBe4aGLomnYldLfEwUZ2r+CzcPGmCP3VKfgfioys7uQ=,tag:Bj6/E/+JIQrCXqgzH8rgGg==,type:str]
secret_three: ENC[AES256_GCM,data:xsf1dK4KCzFru2TOKQ==,iv:hf0f8K2ctbZXbn/s0aYhYUoUyUBEYZOTj6Eay4Q/Id0=,tag:ozi+Fy9Fe3StAi6P7tdCsw==,type:str]
list_one:
-   one: ENC[AES256_GCM,data:EraI,iv:zMGncu9m4nj4TxK2NB4nnmHQHHMtbQTA9xWkWg+5kh8=,tag:4EGWtlOCGu2J0RoOFoK1WQ==,type:str]
-   two: ENC[AES256_GCM,data:inVA,iv:zqQQGbJAT8xrzYGqybFN5UwMOPyVdaJRt8oluLXfFDc=,tag:OZbxXMnngqQGvkcqQGv6Dw==,type:str]
-   three: ENC[AES256_GCM,data:OWXA,iv:nH2vZXRg/Aoz9IHPuDyKuYfxMkE+iY4B0f1k9MUMKRM=,tag:PUk7aQOdhBfBsYRqbHXFRQ==,type:str]
list_two:
-   four: ENC[AES256_GCM,data:Oij/,iv:tsFYBmZbJKGXlH4m2hl5TfM27lIevx2gxClB2xWmiGU=,tag:kamdz1aNt7XQ643L+NjhmQ==,type:str]
-   five: ENC[AES256_GCM,data:0Gw5,iv:SVD7FeDMmByq6KUlsi0AkHtXu/UxK4jEaVBUKTgu+00=,tag:r29v6wKqCRw6Oq2PYvkYKg==,type:str]
-   six: ENC[AES256_GCM,data:vw0N,iv:UOLRoZb0nL375E+tpbt+lN/6QLZyN8n7Kh0BT6G3gfQ=,tag:YKWxZZgNNtdco0QCKVmWgQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    lastmodified: '2018-07-27T13:48:34Z'
    mac: ENC[AES256_GCM,data:3td8f4cPGAqOJT7lZyCCYch6R5kfsiWqoJlryhBq1WAJDse7LG1NJ2eKKyuI4KXThEwvOLdjMG8+uPa6JNzZsvRTzRRzU9r6dYBnmGAlJhoMe4nJpFLwBd2qSmDeix8D8GtChXrwKAK43QFycEJpTdErVxoTeZ17TEF/vK9BuUU=,iv:UQlx16FKsixgwzVB29Nj21d6AET0TSgwIMXGVZcVuR0=,tag:zmnAibqXOwuB75vYTquejA==,type:str]
    pgp:
    -   created_at: '2018-07-27T13:48:34Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----

wcFMA3GCSQ3MKyJgARAAHLgRHT2CgtV7FhoI4im2YMfYOZeOPWLnsHgzhdMZU97Q
            Pq4lZ8lOjLMzSDo/IGNkF/Lb3h2efvZ0899O6Bm1HyeN+W6wJV5dqOvERIbD5vAT
            ztXEHQqkJZN0+M0msflx/dCbcOdS7fXeQrdM2dO768yW31cHWr0zgWj3RymYjNNr
            rRj0a9VZ79xB4Aw7MziUdHE/7HcE0K9kb3qLEbXC9pWJrjr5kQTZrDpDAsAuOseI
            BXfHT1yoEZ02A5+QjZvB7pLM9qBXErOwELrLbTqtQbctFOi7BHyDi7xKpDgHnqz7
            J6H1bjQbINHEo2nRhYzT6OyCmNIi0XpUuzThRNCCbR7HeHwH4Jaqo9B4rVZ9KOKG
            KdwMn2nt1R+Ib89Yro0w/07PkV6Pzm2rcdCumr0lUsL0qcsOltjCe7ekAZuccqHY
            GzF1lYAwMnlQu9G22ut1d2CGTGTVNJA08s3Yi7I0bey9vuoGNJb4T5cQ1htAT4HC
            7J1H/Z8x3JLG8G+elh5KytHQponWOMKNYk6dj/lh3FvDfoe9607yfcpERqddY2yF
            H4IpiYlQT/k9ux6Y027RBIF8X1C4s6F6CZ+30t9kirkTyROOZILzmDR8kfgGxhKZ
            VgUiEKSMrntuFf5lo5w9YEbJnmi5MIyn1PsaPjY2zCyz1nz2do15KXbnn0jo2XPS
            4AHkacYw0WMzioA/XmaTFKylBeHBLeBe4NThq9DgQ+LFzD2g4I/lhWS4QG/uI6Pu
            YJiL34k3uEEO3CzCxTYSOIWhlT4TxPLg6OTie+M5cFALzcZ7StggItWg4hSOxM7h
            qAgA
            =pCRc
            -----END PGP MESSAGE-----
        fp: 8252CFC3A36E12F7214687F44E66A6921AAADBBF
    unencrypted_suffix: _unencrypted
    version: 3.0.5
```

这是加密后的机密文件，需要解密后才可使用：

```bash
sops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml && helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo && rm temp-staging-secrets.yaml
```

这里将解密后的文件保存为临时文件，并且使用后立即删除。由于在同一条命令中执行，不会留下明文的机密数据。

> **JiaJia：**
>
> 在 PowerShell 中需要将 `&&` 修改为 `;` 。
>
> ```powershell
> sops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml ; helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo ; rm temp-staging-secrets.yaml
> ```
>
> 尝试执行时报错了，应该是缺少密钥文件导致的，*demo* 仓库里貌似没提供测试用的密钥。
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-sops> sops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml ; helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo ; rm temp-staging-secrets.yaml
> Failed to get the data key required to decrypt the SOPS file.
>
> Group 0: FAILED
>   8252CFC3A36E12F7214687F44E66A6921AAADBBF: FAILED
>     - | could not decrypt data key with PGP key:
>       | github.com/ProtonMail/go-crypto/openpgp error: Could not
>       | load secring: open C:\Users\ljj/.gnupg/secring.gpg: The
>       | system cannot find the file specified.; GPG binary error:
>       | exit status 2
>
> Recovery failed because no master key was able to decrypt the file. In
> order for SOPS to recover the file, at least one key has to be successful,
> but none were.
> Release "staging-demo" does not exist. Installing it now.
> Error: open staging-values.yaml: The system cannot find the file specified.
> ```

## 12.3 使用 Helmfile 管理多个 Chart

虽然 Helm 非常实用，但它一次只能操作一个 Chart 。Helm 可以使用模板和变量部署一个应用程序，而 Helmfile 则可以通过一个命令部署集群应该安装的所有组件。

*helmfile.yaml：*

*demo* 库里的内容和书中的内容不太一样，这里是 *demo* 库里的内容。

```yaml
repositories:
  - name: prometheus-community
    url: https://prometheus-community.github.io/helm-charts

releases:
  - name: demo
    namespace: demo
    chart: ../hello-helm3/k8s/demo
    values:
      - "../hello-helm3/k8s/demo/production-values.yaml"

- name: kube-state-metrics
    namespace: kube-state-metrics
    chart: prometheus-community/kube-state-metrics

- name: prometheus
    namespace: prometheus
    chart: prometheus-community/prometheus
    set:
      - name: rbac.create
        value: true
```

*repositories* 部分定义了我们将要引用的 Helm Chart 库。

接下来定义一组 release ，即一组即将部署到集群的应用程序。每个 *release* 都需要指定以下元数据：

- **name** : 部署的 Helm Chart 名。
- **namespace** : 部署的目标命名空间。
- **chart** : Chart 本身的 URL 或路径。
- **values** : 部署使用的 *values.yaml* 文件的路径。
- **set** : 设置除值文件包含的值意外的所有其它值。

**Chart 元数据**

这里 *Helmfile* 文件中 *demo* Chart 部分使用的是相对路径。因此，你的 Chart 不需要放在 Chart 库中即可共 Helmfile 进行管理。

至于 *prometheus* Chart，*Helmfile* 知道要去仓库中查找这个 Chart 。还可以在 *Helmfile* 的 `set:` 部分指定任何需要在安装应用程序时覆盖的值。

**安装 Scoop**

Windows 下只能通过 [*Scoop*](https://scoop.sh/) 来安装 *Helmfile* 工具。

确认下 *PowerShell* 的版本，需要 *5.1* 或更新的版本。

```powershell
> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      19041  1320
```

执行如下命令安装 *Scoop* ：

```powershell
> Set-ExecutionPolicy RemoteSigned -Scope CurrentUser # Optional: Needed to run a remote script the first time
> Invoke-WebRequest get.scoop.sh | Invoke-Expression
```

> 有些文章使用的是 `iwr get.scoop.sh | iex` 命令，这个其实是 `Invoke-WebRequest get.scoop.sh | Invoke-Expression` 命令的缩写。

本地安装时报了如下错误：

> Invoke-WebRequest : 未能解析此远程名称: 'raw.githubusercontent.com'

参考[这篇博客](https://blog.csdn.net/qq_41061352/article/details/107140887 "Scoop安装遇到 “raw.githubusercontent.com未能解析” 解决方案")修改本地的 *hosts* 文件：

```bash
code C:\Windows\System32\drivers\etc
```

添加如下解析记录：

```host
199.232.68.133 raw.githubusercontent.com
```

如果以管理员身份运行会报如下错误：

> PS C:\k8s> Invoke-WebRequest get.scoop.sh | Invoke-Expression
> Initializing...
> Running the installer as administrator is disabled by default, see https://github.com/ScoopInstaller/Install#for-admin for details.
> Abort.

摘自 https://github.com/ScoopInstaller/Install#for-admin 的说明：

> **For Admin**
>
> Installation under the administrator console has been disabled by default for security considerations. If you know what you are doing and want to install Scoop as administrator. Please download the installer and manually execute it with the -RunAsAdmin parameter in an elevated console. Here is the example:
>
> ```powershell
> iwr get.scoop.sh -outfile 'install.ps1'
> .\install.ps1 -RunAsAdmin [-OtherParameters ...]
> ```

执行结果：

```powershell
PS C:\k8s> iwr get.scoop.sh -outfile 'install.ps1'
PS C:\k8s> .\install.ps1 -RunAsAdmin
Initializing...
Downloading...
Extracting...
Creating shim...
Adding ~\scoop\shims to your path.
Scoop was installed successfully!
Type 'scoop help' for instructions.
```

如果显示 *无法连接到远程服务器* 错误的话，只能多试几次了。

安装成功后可以通过 `scoop help` 命令查看帮助文档：

```powershell
PS C:\k8s> scoop help
Usage: scoop <command> [<args>]

Some useful commands are:

alias       Manage scoop aliases
bucket      Manage Scoop buckets
cache       Show or clear the download cache
cat         Show content of specified manifest. If available, `bat` will be used to pretty-print the JSON.
checkup     Check for potential problems
cleanup     Cleanup apps by removing old versions
config      Get or set configuration values
create      Create a custom app manifest
depends     List dependencies for an app
download    Download apps in the cache folder and verify hashes
export      Exports (an importable) list of installed apps
help        Show help for a command
hold        Hold an app to disable updates
home        Opens the app homepage
info        Display information about an app
install     Install apps
list        List installed apps
prefix      Returns the path to the specified app
reset       Reset an app to resolve conflicts
search      Search available apps
shim        Manipulate Scoop shims
status      Show status and check for new app versions
unhold      Unhold an app to enable updates
uninstall   Uninstall an app
update      Update apps, or Scoop itself
virustotal  Look for app's hash on virustotal.com
which       Locate a shim/executable (similar to 'which' on Linux)

Type 'scoop help <command>' to get help for a specific command.
```

**安装 Helmfile**

```bash
scoop install helmfile
```

```powershell
PS C:\k8s> scoop install helmfile
Installing 'helmfile' (0.144.0) [64bit]
helmfile_windows_amd64.exe (45.4 MB) [========================================================================================] 100%
Checking hash of helmfile_windows_amd64.exe ... ok.
Linking ~\scoop\apps\helmfile\current => ~\scoop\apps\helmfile\0.144.0
Creating shim for 'helmfile'.
'helmfile' (0.144.0) was installed successfully!
```

**应用 Helmfile**

```bash
helmfile sync
```

```powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile sync
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Building dependency release=demo, chart=..\hello-helm3\k8s\demo
Affected releases are:
  demo (..\hello-helm3\k8s\demo) UPDATED
  kube-state-metrics (prometheus-community/kube-state-metrics) UPDATED
  prometheus (prometheus-community/prometheus) UPDATED

Upgrading release=demo, chart=..\hello-helm3\k8s\demo
Upgrading release=kube-state-metrics, chart=prometheus-community/kube-state-metrics
Upgrading release=prometheus, chart=prometheus-community/prometheus
Release "demo" does not exist. Installing it now.
NAME: demo
LAST DEPLOYED: Thu May 19 17:19:30 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^demo$
demo    demo            1               2022-05-19 17:19:30.6562078 +0800 CST   deployed        demo-1.0.1

Release "kube-state-metrics" does not exist. Installing it now.

Release "prometheus" does not exist. Installing it now.

UPDATED RELEASES:
NAME   CHART                     VERSION
demo   ..\hello-helm3\k8s\demo     1.0.1

FAILED RELEASES:
NAME
kube-state-metrics
prometheus
in ./helmfile.yaml: 2 errors:
err 0: failed processing release kube-state-metrics: command "C:\\k8s\\helm.exe" exited with non-zero status:

...

COMBINED OUTPUT:
  Release "kube-state-metrics" does not exist. Installing it now.
  Error: failed to download "prometheus-community/kube-state-metrics"
err 1: failed processing release prometheus: command "C:\\k8s\\helm.exe" exited with non-zero status:

...

COMBINED OUTPUT:
  Release "prometheus" does not exist. Installing it now.
  Error: failed to download "prometheus-community/prometheus"
```

后面两个 Chart 下载失败了，没有部署成功，重复执行一次后才成功。

Upgrading release=demo, chart=..\hello-helm3\k8s\demo
Upgrading release=prometheus, chart=prometheus-community/prometheus
Upgrading release=kube-state-metrics, chart=prometheus-community/kube-state-metrics
Release "demo" has been upgraded. Happy Helming!
NAME: demo
LAST DEPLOYED: Thu May 19 17:22:18 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 2
TEST SUITE: None

Listing releases matching ^demo$
demo    demo            2               2022-05-19 17:22:18.704396 +0800 CST    deployed        demo-1.0.1

Release "kube-state-metrics" does not exist. Installing it now.
NAME: kube-state-metrics
LAST DEPLOYED: Thu May 19 17:22:35 2022
NAMESPACE: kube-state-metrics
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
The exposed metrics can be found here:
https://github.com/kubernetes/kube-state-metrics/blob/master/docs/README.md#exposed-metrics

The metrics are exported on the HTTP endpoint /metrics on the listening port.
In your case, kube-state-metrics.kube-state-metrics.svc.cluster.local:8080/metrics

They are served either as plaintext or protobuf depending on the Accept header.
They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint.

Listing releases matching ^kube-state-metrics$
kube-state-metrics      kube-state-metrics      1               2022-05-19 17:22:35.3940547 +0800 CST   deployed        kube-state-metrics-4.7.0        2.4.1

Release "prometheus" does not exist. Installing it now.
NAME: prometheus
LAST DEPLOYED: Thu May 19 17:22:36 2022
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local

Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9090

The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.prometheus.svc.cluster.local

Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################

The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.prometheus.svc.cluster.local

Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

Listing releases matching ^prometheus$
prometheus      prometheus      1               2022-05-19 17:22:36.4520302 +0800 CST   deployed        prometheus-15.8.7       2.34.0

UPDATED RELEASES:
NAME                 CHART                                     VERSION
demo                 ..\hello-helm3\k8s\demo                     1.0.1
kube-state-metrics   prometheus-community/kube-state-metrics     4.7.0
prometheus           prometheus-community/prometheus            15.8.7
```

分别查看 3 个命名空间中的 Pod ：

```powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n demo
NAME                    READY   STATUS    RESTARTS   AGE
demo-5475bbbd95-2484z   1/1     Running   0          6m12s
demo-5475bbbd95-q4gb5   1/1     Running   0          6m12s
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n kube-state-metrics
NAME                                  READY   STATUS             RESTARTS   AGE
kube-state-metrics-69db677d7c-k8jfb   0/1     ImagePullBackOff   0          3m41s
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n prometheus
NAME                                             READY   STATUS         RESTARTS   AGE
prometheus-alertmanager-5bb9fdf5b6-f2bpt         2/2     Running        0          3m13s
prometheus-kube-state-metrics-5fd8648d78-kv9hr   0/1     ErrImagePull   0          3m13s
prometheus-node-exporter-5jdgg                   1/1     Running        0          3m13s
prometheus-pushgateway-5846b545ff-6wwf9          1/1     Running        0          3m13s
prometheus-server-8574f9c49b-z8z4s               2/2     Running        0          3m13s
```

其中有几个容器启动失败了，看状态应该镜像没有拉下来。

> **使用唯一的正确标准。**不要一面手动使用 Helm 部署某个 Chart ，一面又通过 Helmfile 以声明式的方式管理整个集群的所有 Chart 。请仅选择一种方式。如果你选择应用 Helmfile ，然后又使用 Helm 来部署或修改应用程序，那么 Helmfile 将不再是集群的唯一正确标准。这势必会引发很多问题，因此，**如果你使用 Helmfile ，则请仅使用 Helmfile 执行所有部署。**

如果你不喜欢 Helmfile ，还可以使用：

- *Landscaper* ：https://github.com/Eneco/landscaper
- *Helmsman* ：https://github.com/Praqma/helmsman

> **JiaJia:**
>
> ***helmfile* 命令说明文档：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile --help
> NAME:
>    helmfile
>
> USAGE:
>    helmfile.exe [global options] command [command options] [arguments...]
>
> VERSION:
>    v0.144.0
>
> COMMANDS:
>    deps          update charts based on their requirements
>    repos         sync repositories from state file (helm repo add && helm repo update)
>    charts        DEPRECATED: sync releases from state file (helm upgrade --install)
>    diff          diff releases from state file against env (helm diff)
>    template      template releases from state file against env (helm template)
>    write-values  write values files for releases. Similar to `helmfile template`, write values files instead of manifests.
>    lint          lint charts from state file (helm lint)
>    fetch         fetch charts from state file
>    sync          sync all resources from state file (repos, releases and chart deps)
>    apply         apply all resources from state file only when there are changes
>    status        retrieve status of releases in state file
>    delete        DEPRECATED: delete releases from state file (helm delete)
>    destroy       deletes and then purges releases
>    test          test releases from state file (helm test)
>    build         output compiled helmfile state(s) as YAML
>    list          list releases defined in state file
>    cache         cache management
>    version       Show the version for Helmfile.
>    help, h       Shows a list of commands or help for one command
>
> GLOBAL OPTIONS:
>    --helm-binary value, -b value           path to helm binary (default: "helm")
>    --file helmfile.yaml, -f helmfile.yaml  load config from file or directory. defaults to helmfile.yaml or `helmfile.d`(means `helmfile.d/*.yaml`) in this preference
>    --environment value, -e value           specify the environment name. defaults to "default"
>    --state-values-set value                set state values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2)
>    --state-values-file value               specify state values in a YAML file
>    --quiet, -q                             Silence output. Equivalent to log-level warn
>    --kube-context value                    Set kubectl context. Uses current context by default
>    --debug                                 Enable verbose output for Helm and set log-level to debug, this disables --quiet/-q effect
>    --no-color                              Output without color
>    --log-level value                       Set log level, default info
>    --namespace value, -n value             Set namespace. Uses the namespace set in the context by default, and is available in templates as {{ .Namespace }}
>    --chart value, -c value                 Set chart. Uses the chart set in release by default, and is available in template as {{ .Chart }}
>    --selector value, -l value              Only run using the releases that match labels. Labels can take the form of foo=bar or foo!=bar.
>                                            A release must match all labels in a group in order to be used. Multiple groups can be specified at once.
>                                            --selector tier=frontend,tier!=proxy --selector tier=backend. Will match all frontend, non-proxy releases AND all backend releases.
>                                            The name of a release can be used as a label. --selector name=myrelease
>    --allow-no-matching-release             Do not exit with an error code if the provided selector has no matching releases.
>    --interactive, -i                       Request confirmation before attempting to modify clusters
>    --help, -h                              show help
>    --version, -v                           print the version
> ```
>
> **清理 helmfile 部署的资源：**
>
> ```powershell
> PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile destroy
> Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
> "prometheus-community" has been added to your repositories
>
> Building dependency release=demo, chart=..\hello-helm3\k8s\demo
> Listing releases matching ^prometheus$
> prometheus      prometheus      1               2022-05-19 17:22:36.4520302 +0800 CST   deployed        prometheus-15.8.7       2.34.0
>
> Listing releases matching ^kube-state-metrics$
> kube-state-metrics      kube-state-metrics      1               2022-05-19 17:22:35.3940547 +0800 CST   deployed        kube-state-metrics-4.7.0        2.4.1
>
> Listing releases matching ^demo$
> demo    demo            2               2022-05-19 17:22:18.704396 +0800 CST    deployed        demo-1.0.1
>
> Deleting prometheus
> Deleting kube-state-metrics
> Deleting demo
> release "demo" uninstalled
>
> release "kube-state-metrics" uninstalled
>
> release "prometheus" uninstalled
>
>
> DELETED RELEASES:
> NAME
> demo
> kube-state-metrics
> prometheus
> ```

## 12.4 高级清单管理工具

**Tanka**

Tanka 允许你使用一种名叫 *Jsonnet* 的语言编写 Kubernetes 清单。 *Jsonnet* 是 JSON 的扩展，它在 JSON 的基础上添加了一些重要的功能：变量、循环、数学运算、条件语句、错误处理等。

**Kapitan**

Kapitan 是另一款基于 *Jsonnet* 的清单工具，专门用于在多个应用程序甚至集群之间共享配置值。 Kapitan 拥有一个按照层级组织的数据库（名叫 *inventory* ），用户保存配置值。有了这个数据库，你就可以根据环境或应用程序的情况，通过给清单的模板插入不同的值来实现重用。

**Kustomize**

Kustomize 使用普通的 YAML 。你可以从基本的 YAML 清单开始，然后通过覆盖给清单打补丁，以供不同的环境或不同的配置使用。 Kustomize 命令行工具可以根据基础文件和覆盖文件来生成最终的清单。

**kompose**

如果你一直在 Docker 容器中运行生产服务，但未使用 Kubernetes ，但你可能非常熟悉 Docker Compose 。

你可以通过 Compose 定义和部署可协同工作的容器集。只需一个 *docker-compose.yml* 文件就可以定义这些容器之间的通信方式。

kompose 是一款能够将 *docker-compose.yml* 文件转换为 Kubernetes 清单的工具，可帮助你完成从 Docker Compose 到 Kubernetes 的迁移，而无需从头开始编写你的 Kubernetes 清单或 Helm Chart 。

**Ansible**

Ansible 是一款流行的基础设置自动化工具。这款工具并非 Kubernetes 专用，但它能够使用扩展模块来管理许多不同种类的资源，就像 Puppet 一样。

除了安装和配置 Kubernetes 集群外， Ansible 还可以使用 k8s 模块直接管理 Kubernetes 资源，例如部署和服务。

与 Helm 类似， Ansible 可以使用它的标准模板语言（ *Jinja* ）来模板化 Kubernetes 清单，而且它还用更成熟的方式来标记变量查找，即使用层次化的系统。

如果你的组织已在使用 Ansible ，则一定要评估一下，是否可以利用它来管理 Kubernetes 资源。如果你的基础设置都在 Kubernetes 上，则对 Ansible 来说，你的需求就是小菜一碟；对于混合基础设施来说，只使用一种工具来管理所有资源也很有益。

**kubeval**

kubeval 的用途是验证清单。例如，检查你是否指定了某个对象的所有必需字段，以及值的类型是否正确。

kubectl 还会在应用清单时进行验证。如果你尝试应用无效的清单，则它会报错。但是，能够预先验证清单也很有用。 kubeval 不需要访问集群，它还可以验证任何版本的 Kubernetes 。

将 kubeval 添加到持续部署流水线的做法非常值得推崇，这样它就能够在清单发生变化时自动进行验证。此外，你还可以使用 kubeval 进行测试，例如，在实际升级之前，测试清单是否需要进行任何调整才能在最新版本的 Kubernetes 上运行。

## 12.5 小结

- Chart 是 Helm 软件包的规范，包括有关软件包的元数据、一些配置值以及引用这些值的模板 Kubernetes 对象。
- 安装 Chart 会创建一个 Helm 发布。每次安装 Chart 的实例都会创建一个新的发布。使用不同的配置值更新发布时， Helm 就会升级发布的版本号。
- 如果想根据自己的需求定制 Helm Chart ，则请创建一个自定义值文件来覆盖你关心的设置，然后将其添加到 `helm install` 或 `helm upgrade` 命令行中。
- 你可以通过一个变量（比如 *environment* ），根据部署环境（预发布、生产等）来选择不同的值或密钥集。
- 你可以通过 Helmfile ，以声明式的方式指定应用到集群的一组 Helm Chart 和值，并使用一个命令来安装或更新所有的 Chart 和值。
- Helm 可以与 Sops 结合使用，处理 Chart 中的机密配置。而且还可以自动对机密数据解析 *base64* 编码（ Kubernetes 希望的格式）。
- Helm 不是唯一的管理 Kubernetes 清单的工具。 Ksonnet 和 Kapitan 使用 Jsonnet （一种模板语言）。 kustomize 采用一种不同的方法，它不会对变量进行插值，但会使用 YAML 覆盖来配置清单。
- kubeval 是一种快速测试和验证清单的方法，它可以检查清单的语法是否有效，还能检查清单中常见的错误。

← 上一篇

目录

关联博客