基于 Kubernetes 的云原生 DevOps 第 12 章 部署 Kubernetes 应用程序
🏷️ Kubernetes 《基于 Kubernetes 的云原生 DevOps》
I lay on my back, surprised at how calm and focused I felt, strapped to four and a half million pounds of explosives.
-- Ron Garan, astronaut
12.1 使用 Helm 构建清单
Helm Chart 包含什么？
每个 Chart 都有一个标准的结构。
首先 Chart 包含在同名目录中（这里的例子中为 demo ）。
powershell
demo
│ Chart.yaml
│ production-values.yaml
│ staging-values.yaml
│ values.yaml
│
└─templates
deployment.yaml
service.yaml其次，包含一个名为 Chart.yaml 的文，其中指定了 Chart 的名称及版本：
yaml
name: demo
sources:
- https://github.com/cloudnativedevops/cloudnatived
version: 1.0.1Chart.yaml 文件
Chart.yaml 提供很多可选字段，包括指向项目源代码的链接，但必须的信息只有名称和版本。
values.yaml
values.yaml 文件包含 Chart 作者公开的、可供用户修改的设置：
yaml
environment: development
container:
name: demo
port: 8888
image: cloudnatived/demo
tag: hello
replicas: 2values.yaml 完全是自由格式的 YAML，没有预定义的结构，可以随意选择定义哪些变量、变量的名称和值。
Helm 模板
模板文件保存在 templates 目录下，这些文件正是前面示例中的清单文件，不同之处在于它包含了很多占位符，而 Helm 可以利用 values.yaml 中的实际值替换这些占位符。
deployment.yaml
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.container.name }}-{{ .Values.environment }}
spec:
replicas: {{ .Values.replicas }}
selector:
matchLabels:
app: {{ .Values.container.name }}
template:
metadata:
labels:
app: {{ .Values.container.name }}
environment: {{ .Values.environment }}
spec:
containers:
- name: {{ .Values.container.name }}
image: {{ .Values.container.image }}:{{ .Values.container.tag }}
ports:
- containerPort: {{ .Values.container.port }}
env:
- name: environment
value: {{ .Values.environment }}大括号指明了 Helm 应替换成变量值的位置，但实际上它们是 Go 模板语法的一部分。
插值变量
例如
yaml
metadata:
name: {{ .Values.container.name }}-{{ .Values.environment }}会被替换为
yaml
metadata:
name: demo-development使用 Helm 变量可以避免重复。除了简单的变量替换之外，还支持循环、表达式、条件分支，甚至函数调用。
引用模板中的值
使用 Helm 中的 quote 函数给模板中的值自动添加引号：
yaml
name: {{ .Values.MyName | quote }}只有字符串应该加引号，切勿对端口号等数字值使用 quote 函数。
指定依赖项
Chart 可以依赖于其它 Chart 。如你的程序适应 Redis , 就可以指定 redis Chart 作为依赖。
可以在 requirements.yaml 文件中指定依赖：
yaml
dependencies:
- name: redis
version: 1.2.3
- name: nginx
version: 3.2.1接下来运行 helm dependency update 命令，Helm 就会下载这些 Chart，然后与应用程序一起安装。
12.2 部署 Helm Chart
设置变量
可以通过 helm install 命令在命令行中指定包含其它值的文件，而这些值会覆盖 values.yaml 中的默认值。
创建一个新的 YAML 文件 staging-values.yaml ：
yaml
environment: staging在 Helm Release 中指定值
通过 --values 标志指定额外的值的文件。
powershell
helm install --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\命令行的书写格式和书中不太一样，估计是因为 Helm 的版本不一样。
运行结果：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm install --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\ NAME: demo-staging LAST DEPLOYED: Wed May 18 17:27:56 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None查看部署：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl describe deploy/demo Name: demo Namespace: default CreationTimestamp: Wed, 18 May 2022 17:27:56 +0800 Labels: app.kubernetes.io/managed-by=Helm Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: demo-staging meta.helm.sh/release-namespace: default Selector: app=demo Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=demo environment=staging Containers: demo: Image: cloudnatived/demo:hello Port: 8888/TCP Host Port: 0/TCP Environment: environment: staging Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: demo-5b8b9fd8d9 (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 11m deployment-controller Scaled up replica set demo-5b8b9fd8d9 to 2可以看到环境变量 environment 的值为 staging 。
查看机密：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl get secret NAME TYPE DATA AGE default-token-hxl8b kubernetes.io/service-account-token 3 163d sh.helm.release.v1.demo-staging.v1 helm.sh/release.v1 1 14m可以看到 helm release 的名字为 demo-staging 。
使用 Helm 更新应用程序
powershell
helm upgrade --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\运行结果：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm upgrade --values=.\k8s\demo\staging-values.yaml demo-staging .\k8s\demo\ Release "demo-staging" has been upgraded. Happy Helming! NAME: demo-staging LAST DEPLOYED: Wed May 18 17:47:37 2022 NAMESPACE: default STATUS: deployed REVISION: 2 TEST SUITE: None查看机密：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl get secret NAME TYPE DATA AGE default-token-hxl8b kubernetes.io/service-account-token 3 163d sh.helm.release.v1.demo-staging.v1 helm.sh/release.v1 1 19m sh.helm.release.v1.demo-staging.v2 helm.sh/release.v1 1 16s PS C:\projects\github\cloudnativedevops\demo\hello-helm3> kubectl describe secret/sh.helm.release.v1.demo-staging.v2 Name: sh.helm.release.v1.demo-staging.v2 Namespace: default Labels: modifiedAt=1652867257 name=demo-staging owner=helm status=deployed version=2 Annotations: <none> Type: helm.sh/release.v1 Data ==== release: 1604 bytes
回滚到以前的版本
powershell
helm rollback demo-staging 1运行结果：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm rollback demo-staging 1 Rollback was a success! Happy Helming!查看机密：
生成了一个新的版本 v3 。
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> k get secret NAME TYPE DATA AGE default-token-hxl8b kubernetes.io/service-account-token 3 163d sh.helm.release.v1.demo-staging.v1 helm.sh/release.v1 1 30m sh.helm.release.v1.demo-staging.v2 helm.sh/release.v1 1 10m sh.helm.release.v1.demo-staging.v3 helm.sh/release.v1 1 30spowershellPS C:\projects\github\cloudnativedevops\demo\hello-helm3> k describe secret/sh.helm.release.v1.demo-staging.v3 Name: sh.helm.release.v1.demo-staging.v3 Namespace: default Labels: modifiedAt=1652867847 name=demo-staging owner=helm status=deployed version=3 Annotations: <none> Type: helm.sh/release.v1 Data ==== release: 1600 bytes
使用 helm-monitor 自动回滚
helm-monitor 插件可以通过任何指标表达式查询 Prometheus 服务器，并在查询成功时触发回滚。 helm-monitor 将监视指标五分钟，如果在此期间发现问题，则回滚。有关 helm-monitor 的更多信息，请参阅这篇博客。
创建 Helm Chart 库
通常应用程序的 Helm Chart 会存储在应用程序自己的代码库中。但是如果有需要，维护自己的 Helm Chart 库也非常简单。
你需要通过 HTTP 提供 Chart，实现方式有两种：
- 将它们存储到云存储桶中；
- 使用 GitHub Pages 或利用现有的 Web 服务器。
将所有的 Chart 汇集到一个目录中后，直接在目录中运行 helm repo index 即可创建 Chart 库元数据的 index.yaml 文件。
有关 Chart 库的更多信息，请参见 Helm 文档（ The Chart Repository Guide ）。
如果想安装自己库中的 Chart，首先需要将你的 Chart 库添加到 Helm 的列表中：
powershell
helm repo add myrepo http://myrepo.example.com
helm install myrepo/myapp卸载 demo-staging :
powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helm3> helm uninstall demo-staging
release "demo-staging" uninstalled使用 Sops 管理 Helm Chart 的机密数据
demo 代码库的 hello-sops 目录下提供了一个示例。
powershell
PS C:\projects\github\cloudnativedevops\demo> cd .\hello-sops\
PS C:\projects\github\cloudnativedevops\demo\hello-sops> tree /f
卷 system 的文件夹 PATH 列表
卷序列号为 0000022B 1ABC:16A8
C:.
│ temp.yaml
│
└─k8s
└─demo
│ Chart.yaml
│ production-secrets.yaml
│ production-values.yaml
│ staging-secrets.yaml
│ staging-values.yaml
│ values.yaml
│
└─templates
deployment.yaml
secrets.yaml模板文件里有个机密数据清单文件 k8s\demo\templates\secrets.yaml ：
yaml
apiVersion: v1
kind: Secret
metadata:
name: {{ .Values.container.name }}-secrets
type: Opaque
data:
{{ $environment := .Values.environment }}
secrets.yaml: {{ .Files.Get (nospace (cat $environment "-secrets.yaml")) | b64enc }}真正的机密数据则保存在 k8s\demo\production-secrets.yaml 和 k8s\demo\staging-secrets.yaml 文件中。
最后一行的末尾添加了 | b64enc 。这种写法是一种快捷方式，可以使用 Helm 的 Go 模板自动将机密数据从纯文本转化为 base64 ，在默认情况下，Kubernetes 希望机密数据保存成 base64 的形式。
staging-secrets.yaml
yaml
secret_one: ENC[AES256_GCM,data:hJ/QcnvOxgfjMhY=,iv:7AC+hW/ouAeIF9yTACttPQ7sq/QHSSvHgI+DSuZDOnY=,tag:fRqJ2QiORl04uxa1FpZZKA==,type:str]
secret_two: ENC[AES256_GCM,data:swFK66/mAEX3Ohg=,iv:fBe4aGLomnYldLfEwUZ2r+CzcPGmCP3VKfgfioys7uQ=,tag:Bj6/E/+JIQrCXqgzH8rgGg==,type:str]
secret_three: ENC[AES256_GCM,data:xsf1dK4KCzFru2TOKQ==,iv:hf0f8K2ctbZXbn/s0aYhYUoUyUBEYZOTj6Eay4Q/Id0=,tag:ozi+Fy9Fe3StAi6P7tdCsw==,type:str]
list_one:
- one: ENC[AES256_GCM,data:EraI,iv:zMGncu9m4nj4TxK2NB4nnmHQHHMtbQTA9xWkWg+5kh8=,tag:4EGWtlOCGu2J0RoOFoK1WQ==,type:str]
- two: ENC[AES256_GCM,data:inVA,iv:zqQQGbJAT8xrzYGqybFN5UwMOPyVdaJRt8oluLXfFDc=,tag:OZbxXMnngqQGvkcqQGv6Dw==,type:str]
- three: ENC[AES256_GCM,data:OWXA,iv:nH2vZXRg/Aoz9IHPuDyKuYfxMkE+iY4B0f1k9MUMKRM=,tag:PUk7aQOdhBfBsYRqbHXFRQ==,type:str]
list_two:
- four: ENC[AES256_GCM,data:Oij/,iv:tsFYBmZbJKGXlH4m2hl5TfM27lIevx2gxClB2xWmiGU=,tag:kamdz1aNt7XQ643L+NjhmQ==,type:str]
- five: ENC[AES256_GCM,data:0Gw5,iv:SVD7FeDMmByq6KUlsi0AkHtXu/UxK4jEaVBUKTgu+00=,tag:r29v6wKqCRw6Oq2PYvkYKg==,type:str]
- six: ENC[AES256_GCM,data:vw0N,iv:UOLRoZb0nL375E+tpbt+lN/6QLZyN8n7Kh0BT6G3gfQ=,tag:YKWxZZgNNtdco0QCKVmWgQ==,type:str]
sops:
kms: []
gcp_kms: []
lastmodified: '2018-07-27T13:48:34Z'
mac: ENC[AES256_GCM,data:3td8f4cPGAqOJT7lZyCCYch6R5kfsiWqoJlryhBq1WAJDse7LG1NJ2eKKyuI4KXThEwvOLdjMG8+uPa6JNzZsvRTzRRzU9r6dYBnmGAlJhoMe4nJpFLwBd2qSmDeix8D8GtChXrwKAK43QFycEJpTdErVxoTeZ17TEF/vK9BuUU=,iv:UQlx16FKsixgwzVB29Nj21d6AET0TSgwIMXGVZcVuR0=,tag:zmnAibqXOwuB75vYTquejA==,type:str]
pgp:
- created_at: '2018-07-27T13:48:34Z'
enc: |-
-----BEGIN PGP MESSAGE-----
wcFMA3GCSQ3MKyJgARAAHLgRHT2CgtV7FhoI4im2YMfYOZeOPWLnsHgzhdMZU97Q
Pq4lZ8lOjLMzSDo/IGNkF/Lb3h2efvZ0899O6Bm1HyeN+W6wJV5dqOvERIbD5vAT
ztXEHQqkJZN0+M0msflx/dCbcOdS7fXeQrdM2dO768yW31cHWr0zgWj3RymYjNNr
rRj0a9VZ79xB4Aw7MziUdHE/7HcE0K9kb3qLEbXC9pWJrjr5kQTZrDpDAsAuOseI
BXfHT1yoEZ02A5+QjZvB7pLM9qBXErOwELrLbTqtQbctFOi7BHyDi7xKpDgHnqz7
J6H1bjQbINHEo2nRhYzT6OyCmNIi0XpUuzThRNCCbR7HeHwH4Jaqo9B4rVZ9KOKG
KdwMn2nt1R+Ib89Yro0w/07PkV6Pzm2rcdCumr0lUsL0qcsOltjCe7ekAZuccqHY
GzF1lYAwMnlQu9G22ut1d2CGTGTVNJA08s3Yi7I0bey9vuoGNJb4T5cQ1htAT4HC
7J1H/Z8x3JLG8G+elh5KytHQponWOMKNYk6dj/lh3FvDfoe9607yfcpERqddY2yF
H4IpiYlQT/k9ux6Y027RBIF8X1C4s6F6CZ+30t9kirkTyROOZILzmDR8kfgGxhKZ
VgUiEKSMrntuFf5lo5w9YEbJnmi5MIyn1PsaPjY2zCyz1nz2do15KXbnn0jo2XPS
4AHkacYw0WMzioA/XmaTFKylBeHBLeBe4NThq9DgQ+LFzD2g4I/lhWS4QG/uI6Pu
YJiL34k3uEEO3CzCxTYSOIWhlT4TxPLg6OTie+M5cFALzcZ7StggItWg4hSOxM7h
qAgA
=pCRc
-----END PGP MESSAGE-----
fp: 8252CFC3A36E12F7214687F44E66A6921AAADBBF
unencrypted_suffix: _unencrypted
version: 3.0.5这是加密后的机密文件，需要解密后才可使用：
bash
sops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml && helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo && rm temp-staging-secrets.yaml这里将解密后的文件保存为临时文件，并且使用后立即删除。由于在同一条命令中执行，不会留下明文的机密数据。
JiaJia：
在 PowerShell 中需要将
&&修改为;。powershellsops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml ; helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo ; rm temp-staging-secrets.yaml尝试执行时报错了，应该是缺少密钥文件导致的，demo 仓库里貌似没提供测试用的密钥。
powershellPS C:\projects\github\cloudnativedevops\demo\hello-sops> sops -d ./k8s/demo/staging-secrets.yaml > temp-staging-secrets.yaml ; helm upgrade --install staging-demo --values ./k8s/demo/staging-values.yaml --values temp-staging-secrets.yaml ./k8s/demo ; rm temp-staging-secrets.yaml Failed to get the data key required to decrypt the SOPS file. Group 0: FAILED 8252CFC3A36E12F7214687F44E66A6921AAADBBF: FAILED - | could not decrypt data key with PGP key: | github.com/ProtonMail/go-crypto/openpgp error: Could not | load secring: open C:\Users\ljj/.gnupg/secring.gpg: The | system cannot find the file specified.; GPG binary error: | exit status 2 Recovery failed because no master key was able to decrypt the file. In order for SOPS to recover the file, at least one key has to be successful, but none were. Release "staging-demo" does not exist. Installing it now. Error: open staging-values.yaml: The system cannot find the file specified.
12.3 使用 Helmfile 管理多个 Chart
虽然 Helm 非常实用，但它一次只能操作一个 Chart。Helm 可以使用模板和变量部署一个应用程序，而 Helmfile 则可以通过一个命令部署集群应该安装的所有组件。
helmfile.yaml：
demo 库里的内容和书中的内容不太一样，这里是 demo 库里的内容。
yaml
repositories:
- name: prometheus-community
url: https://prometheus-community.github.io/helm-charts
releases:
- name: demo
namespace: demo
chart: ../hello-helm3/k8s/demo
values:
- "../hello-helm3/k8s/demo/production-values.yaml"
- name: kube-state-metrics
namespace: kube-state-metrics
chart: prometheus-community/kube-state-metrics
- name: prometheus
namespace: prometheus
chart: prometheus-community/prometheus
set:
- name: rbac.create
value: truerepositories 部分定义了我们将要引用的 Helm Chart 库。
接下来定义一组 release，即一组即将部署到集群的应用程序。每个 release 都需要指定以下元数据：
- name : 部署的 Helm Chart 名。
- namespace : 部署的目标命名空间。
- chart : Chart 本身的 URL 或路径。
- values : 部署使用的 values.yaml 文件的路径。
- set : 设置除值文件包含的值意外的所有其它值。
Chart 元数据
这里 Helmfile 文件中 demo Chart 部分使用的是相对路径。因此，你的 Chart 不需要放在 Chart 库中即可共 Helmfile 进行管理。
至于 prometheus Chart，Helmfile 知道要去仓库中查找这个 Chart。还可以在 Helmfile 的 set: 部分指定任何需要在安装应用程序时覆盖的值。
安装 Scoop
Windows 下只能通过 Scoop 来安装 Helmfile 工具。
确认下 PowerShell 的版本，需要 5.1 或更新的版本。
powershell
> $PSVersionTable.PSVersion
Major Minor Build Revision
----- ----- ----- --------
5 1 19041 1320执行如下命令安装 Scoop ：
powershell
> Set-ExecutionPolicy RemoteSigned -Scope CurrentUser # Optional: Needed to run a remote script the first time
> Invoke-WebRequest get.scoop.sh | Invoke-Expression有些文章使用的是
iwr get.scoop.sh | iex命令，这个其实是Invoke-WebRequest get.scoop.sh | Invoke-Expression命令的缩写。
本地安装时报了如下错误：
Invoke-WebRequest : 未能解析此远程名称: 'raw.githubusercontent.com'
参考这篇博客修改本地的 hosts 文件：
bash
code C:\Windows\System32\drivers\etc添加如下解析记录：
txt
199.232.68.133 raw.githubusercontent.com如果以管理员身份运行会报如下错误：
PS C:\k8s> Invoke-WebRequest get.scoop.sh | Invoke-Expression
Initializing...
Running the installer as administrator is disabled by default, see https://github.com/ScoopInstaller/Install#for-admin for details.
Abort.
摘自 https://github.com/ScoopInstaller/Install#for-admin 的说明：
For Admin
Installation under the administrator console has been disabled by default for security considerations. If you know what you are doing and want to install Scoop as administrator. Please download the installer and manually execute it with the -RunAsAdmin parameter in an elevated console. Here is the example:
powershelliwr get.scoop.sh -outfile 'install.ps1' .\install.ps1 -RunAsAdmin [-OtherParameters ...]
执行结果：
powershell
PS C:\k8s> iwr get.scoop.sh -outfile 'install.ps1'
PS C:\k8s> .\install.ps1 -RunAsAdmin
Initializing...
Downloading...
Extracting...
Creating shim...
Adding ~\scoop\shims to your path.
Scoop was installed successfully!
Type 'scoop help' for instructions.如果显示 无法连接到远程服务器 错误的话，只能多试几次了。
安装成功后可以通过 scoop help 命令查看帮助文档：
powershell
PS C:\k8s> scoop help
Usage: scoop <command> [<args>]
Some useful commands are:
alias Manage scoop aliases
bucket Manage Scoop buckets
cache Show or clear the download cache
cat Show content of specified manifest. If available, `bat` will be used to pretty-print the JSON.
checkup Check for potential problems
cleanup Cleanup apps by removing old versions
config Get or set configuration values
create Create a custom app manifest
depends List dependencies for an app
download Download apps in the cache folder and verify hashes
export Exports (an importable) list of installed apps
help Show help for a command
hold Hold an app to disable updates
home Opens the app homepage
info Display information about an app
install Install apps
list List installed apps
prefix Returns the path to the specified app
reset Reset an app to resolve conflicts
search Search available apps
shim Manipulate Scoop shims
status Show status and check for new app versions
unhold Unhold an app to enable updates
uninstall Uninstall an app
update Update apps, or Scoop itself
virustotal Look for app's hash on virustotal.com
which Locate a shim/executable (similar to 'which' on Linux)
Type 'scoop help <command>' to get help for a specific command.安装 Helmfile
bash
scoop install helmfilepowershell
PS C:\k8s> scoop install helmfile
Installing 'helmfile' (0.144.0) [64bit]
helmfile_windows_amd64.exe (45.4 MB) [========================================================================================] 100%
Checking hash of helmfile_windows_amd64.exe ... ok.
Linking ~\scoop\apps\helmfile\current => ~\scoop\apps\helmfile\0.144.0
Creating shim for 'helmfile'.
'helmfile' (0.144.0) was installed successfully!应用 Helmfile
bash
helmfile syncpowershell
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile sync
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
Building dependency release=demo, chart=..\hello-helm3\k8s\demo
Affected releases are:
demo (..\hello-helm3\k8s\demo) UPDATED
kube-state-metrics (prometheus-community/kube-state-metrics) UPDATED
prometheus (prometheus-community/prometheus) UPDATED
Upgrading release=demo, chart=..\hello-helm3\k8s\demo
Upgrading release=kube-state-metrics, chart=prometheus-community/kube-state-metrics
Upgrading release=prometheus, chart=prometheus-community/prometheus
Release "demo" does not exist. Installing it now.
NAME: demo
LAST DEPLOYED: Thu May 19 17:19:30 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 1
TEST SUITE: None
Listing releases matching ^demo$
demo demo 1 2022-05-19 17:19:30.6562078 +0800 CST deployed demo-1.0.1
Release "kube-state-metrics" does not exist. Installing it now.
Release "prometheus" does not exist. Installing it now.
UPDATED RELEASES:
NAME CHART VERSION
demo ..\hello-helm3\k8s\demo 1.0.1
FAILED RELEASES:
NAME
kube-state-metrics
prometheus
in ./helmfile.yaml: 2 errors:
err 0: failed processing release kube-state-metrics: command "C:\\k8s\\helm.exe" exited with non-zero status:
...
COMBINED OUTPUT:
Release "kube-state-metrics" does not exist. Installing it now.
Error: failed to download "prometheus-community/kube-state-metrics"
err 1: failed processing release prometheus: command "C:\\k8s\\helm.exe" exited with non-zero status:
...
COMBINED OUTPUT:
Release "prometheus" does not exist. Installing it now.
Error: failed to download "prometheus-community/prometheus"后面两个 Chart 下载失败了，没有部署成功，重复执行一次后才成功。
powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile sync
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
Building dependency release=demo, chart=..\hello-helm3\k8s\demo
Affected releases are:
demo (..\hello-helm3\k8s\demo) UPDATED
kube-state-metrics (prometheus-community/kube-state-metrics) UPDATED
prometheus (prometheus-community/prometheus) UPDATED
Upgrading release=demo, chart=..\hello-helm3\k8s\demo
Upgrading release=prometheus, chart=prometheus-community/prometheus
Upgrading release=kube-state-metrics, chart=prometheus-community/kube-state-metrics
Release "demo" has been upgraded. Happy Helming!
NAME: demo
LAST DEPLOYED: Thu May 19 17:22:18 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 2
TEST SUITE: None
Listing releases matching ^demo$
demo demo 2 2022-05-19 17:22:18.704396 +0800 CST deployed demo-1.0.1
Release "kube-state-metrics" does not exist. Installing it now.
NAME: kube-state-metrics
LAST DEPLOYED: Thu May 19 17:22:35 2022
NAMESPACE: kube-state-metrics
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
The exposed metrics can be found here:
https://github.com/kubernetes/kube-state-metrics/blob/master/docs/README.md#exposed-metrics
The metrics are exported on the HTTP endpoint /metrics on the listening port.
In your case, kube-state-metrics.kube-state-metrics.svc.cluster.local:8080/metrics
They are served either as plaintext or protobuf depending on the Accept header.
They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint.
Listing releases matching ^kube-state-metrics$
kube-state-metrics kube-state-metrics 1 2022-05-19 17:22:35.3940547 +0800 CST deployed kube-state-metrics-4.7.0 2.4.1
Release "prometheus" does not exist. Installing it now.
NAME: prometheus
LAST DEPLOYED: Thu May 19 17:22:36 2022
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local
Get the Prometheus server URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward $POD_NAME 9090
The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.prometheus.svc.cluster.local
Get the Alertmanager URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward $POD_NAME 9093
#################################################################################
###### WARNING: Pod Security Policy has been moved to a global property. #####
###### use .Values.podSecurityPolicy.enabled with pod-based #####
###### annotations #####
###### (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################
The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.prometheus.svc.cluster.local
Get the PushGateway URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/
Listing releases matching ^prometheus$
prometheus prometheus 1 2022-05-19 17:22:36.4520302 +0800 CST deployed prometheus-15.8.7 2.34.0
UPDATED RELEASES:
NAME CHART VERSION
demo ..\hello-helm3\k8s\demo 1.0.1
kube-state-metrics prometheus-community/kube-state-metrics 4.7.0
prometheus prometheus-community/prometheus 15.8.7分别查看 3 个命名空间中的 Pod：
powershell
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n demo
NAME READY STATUS RESTARTS AGE
demo-5475bbbd95-2484z 1/1 Running 0 6m12s
demo-5475bbbd95-q4gb5 1/1 Running 0 6m12s
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n kube-state-metrics
NAME READY STATUS RESTARTS AGE
kube-state-metrics-69db677d7c-k8jfb 0/1 ImagePullBackOff 0 3m41s
PS C:\projects\github\cloudnativedevops\demo\hello-helmfile> kubectl get po -n prometheus
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-5bb9fdf5b6-f2bpt 2/2 Running 0 3m13s
prometheus-kube-state-metrics-5fd8648d78-kv9hr 0/1 ErrImagePull 0 3m13s
prometheus-node-exporter-5jdgg 1/1 Running 0 3m13s
prometheus-pushgateway-5846b545ff-6wwf9 1/1 Running 0 3m13s
prometheus-server-8574f9c49b-z8z4s 2/2 Running 0 3m13s其中有几个容器启动失败了，看状态应该镜像没有拉下来。
使用唯一的正确标准。 不要一面手动使用 Helm 部署某个 Chart，一面又通过 Helmfile 以声明式的方式管理整个集群的所有 Chart。请仅选择一种方式。如果你选择应用 Helmfile，然后又使用 Helm 来部署或修改应用程序，那么 Helmfile 将不再是集群的唯一正确标准。这势必会引发很多问题，因此，如果你使用 Helmfile，则请仅使用 Helmfile 执行所有部署。
如果你不喜欢 Helmfile，还可以使用：
- Landscaper ：https://github.com/Eneco/landscaper
- Helmsman ：https://github.com/Praqma/helmsman
JiaJia:
helmfile 命令说明文档：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile --help NAME: helmfile USAGE: helmfile.exe [global options] command [command options] [arguments...] VERSION: v0.144.0 COMMANDS: deps update charts based on their requirements repos sync repositories from state file (helm repo add && helm repo update) charts DEPRECATED: sync releases from state file (helm upgrade --install) diff diff releases from state file against env (helm diff) template template releases from state file against env (helm template) write-values write values files for releases. Similar to `helmfile template`, write values files instead of manifests. lint lint charts from state file (helm lint) fetch fetch charts from state file sync sync all resources from state file (repos, releases and chart deps) apply apply all resources from state file only when there are changes status retrieve status of releases in state file delete DEPRECATED: delete releases from state file (helm delete) destroy deletes and then purges releases test test releases from state file (helm test) build output compiled helmfile state(s) as YAML list list releases defined in state file cache cache management version Show the version for Helmfile. help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --helm-binary value, -b value path to helm binary (default: "helm") --file helmfile.yaml, -f helmfile.yaml load config from file or directory. defaults to helmfile.yaml or `helmfile.d`(means `helmfile.d/*.yaml`) in this preference --environment value, -e value specify the environment name. defaults to "default" --state-values-set value set state values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2) --state-values-file value specify state values in a YAML file --quiet, -q Silence output. Equivalent to log-level warn --kube-context value Set kubectl context. Uses current context by default --debug Enable verbose output for Helm and set log-level to debug, this disables --quiet/-q effect --no-color Output without color --log-level value Set log level, default info --namespace value, -n value Set namespace. Uses the namespace set in the context by default, and is available in templates as {{ .Namespace }} --chart value, -c value Set chart. Uses the chart set in release by default, and is available in template as {{ .Chart }} --selector value, -l value Only run using the releases that match labels. Labels can take the form of foo=bar or foo!=bar. A release must match all labels in a group in order to be used. Multiple groups can be specified at once. --selector tier=frontend,tier!=proxy --selector tier=backend. Will match all frontend, non-proxy releases AND all backend releases. The name of a release can be used as a label. --selector name=myrelease --allow-no-matching-release Do not exit with an error code if the provided selector has no matching releases. --interactive, -i Request confirmation before attempting to modify clusters --help, -h show help --version, -v print the version清理 helmfile 部署的资源：
powershellPS C:\projects\github\cloudnativedevops\demo\hello-helmfile> helmfile destroy Adding repo prometheus-community https://prometheus-community.github.io/helm-charts "prometheus-community" has been added to your repositories Building dependency release=demo, chart=..\hello-helm3\k8s\demo Listing releases matching ^prometheus$ prometheus prometheus 1 2022-05-19 17:22:36.4520302 +0800 CST deployed prometheus-15.8.7 2.34.0 Listing releases matching ^kube-state-metrics$ kube-state-metrics kube-state-metrics 1 2022-05-19 17:22:35.3940547 +0800 CST deployed kube-state-metrics-4.7.0 2.4.1 Listing releases matching ^demo$ demo demo 2 2022-05-19 17:22:18.704396 +0800 CST deployed demo-1.0.1 Deleting prometheus Deleting kube-state-metrics Deleting demo release "demo" uninstalled release "kube-state-metrics" uninstalled release "prometheus" uninstalled DELETED RELEASES: NAME demo kube-state-metrics prometheus
12.4 高级清单管理工具
Tanka
Tanka 允许你使用一种名叫 Jsonnet 的语言编写 Kubernetes 清单。 Jsonnet 是 JSON 的扩展，它在 JSON 的基础上添加了一些重要的功能：变量、循环、数学运算、条件语句、错误处理等。
Kapitan
Kapitan 是另一款基于 Jsonnet 的清单工具，专门用于在多个应用程序甚至集群之间共享配置值。Kapitan 拥有一个按照层级组织的数据库（名叫 inventory ），用户保存配置值。有了这个数据库，你就可以根据环境或应用程序的情况，通过给清单的模板插入不同的值来实现重用。
Kustomize
Kustomize 使用普通的 YAML。你可以从基本的 YAML 清单开始，然后通过覆盖给清单打补丁，以供不同的环境或不同的配置使用。Kustomize 命令行工具可以根据基础文件和覆盖文件来生成最终的清单。
kompose
如果你一直在 Docker 容器中运行生产服务，但未使用 Kubernetes，但你可能非常熟悉 Docker Compose。
你可以通过 Compose 定义和部署可协同工作的容器集。只需一个 docker-compose.yml 文件就可以定义这些容器之间的通信方式。
kompose 是一款能够将 docker-compose.yml 文件转换为 Kubernetes 清单的工具，可帮助你完成从 Docker Compose 到 Kubernetes 的迁移，而无需从头开始编写你的 Kubernetes 清单或 Helm Chart。
Ansible
Ansible 是一款流行的基础设置自动化工具。这款工具并非 Kubernetes 专用，但它能够使用扩展模块来管理许多不同种类的资源，就像 Puppet 一样。
除了安装和配置 Kubernetes 集群外，Ansible 还可以使用 k8s 模块直接管理 Kubernetes 资源，例如部署和服务。
与 Helm 类似，Ansible 可以使用它的标准模板语言（ Jinja ）来模板化 Kubernetes 清单，而且它还用更成熟的方式来标记变量查找，即使用层次化的系统。
如果你的组织已在使用 Ansible，则一定要评估一下，是否可以利用它来管理 Kubernetes 资源。如果你的基础设置都在 Kubernetes 上，则对 Ansible 来说，你的需求就是小菜一碟；对于混合基础设施来说，只使用一种工具来管理所有资源也很有益。
kubeval
kubeval 的用途是验证清单。例如，检查你是否指定了某个对象的所有必需字段，以及值的类型是否正确。
kubectl 还会在应用清单时进行验证。如果你尝试应用无效的清单，则它会报错。但是，能够预先验证清单也很有用。kubeval 不需要访问集群，它还可以验证任何版本的 Kubernetes。
将 kubeval 添加到持续部署流水线的做法非常值得推崇，这样它就能够在清单发生变化时自动进行验证。此外，你还可以使用 kubeval 进行测试，例如，在实际升级之前，测试清单是否需要进行任何调整才能在最新版本的 Kubernetes 上运行。
12.5 小结
- Chart 是 Helm 软件包的规范，包括有关软件包的元数据、一些配置值以及引用这些值的模板 Kubernetes 对象。
- 安装 Chart 会创建一个 Helm 发布。每次安装 Chart 的实例都会创建一个新的发布。使用不同的配置值更新发布时，Helm 就会升级发布的版本号。
- 如果想根据自己的需求定制 Helm Chart，则请创建一个自定义值文件来覆盖你关心的设置，然后将其添加到
helm install或helm upgrade命令行中。 - 你可以通过一个变量（比如 environment ），根据部署环境（预发布、生产等）来选择不同的值或密钥集。
- 你可以通过 Helmfile，以声明式的方式指定应用到集群的一组 Helm Chart 和值，并使用一个命令来安装或更新所有的 Chart 和值。
- Helm 可以与 Sops 结合使用，处理 Chart 中的机密配置。而且还可以自动对机密数据解析 base64 编码（Kubernetes 希望的格式）。
- Helm 不是唯一的管理 Kubernetes 清单的工具。Ksonnet 和 Kapitan 使用 Jsonnet（一种模板语言）。kustomize 采用一种不同的方法，它不会对变量进行插值，但会使用 YAML 覆盖来配置清单。
- kubeval 是一种快速测试和验证清单的方法，它可以检查清单的语法是否有效，还能检查清单中常见的错误。