k8s HPA使用详解 | 爱吃可爱多

kubectl scale 命令可以来实现 Pod 的扩缩容功能，但是这个毕竟是完全手动操作的，要应对线上的各种复杂情况，需要能够做到自动化去感知业务，来自动进行扩缩容，为此Kubernetes 也为我们提供了这样的一个资源对象：Horizontal Pod Autoscaling（Pod 水平自动伸缩），简称HPA，HPA 通过监控分析一些控制器控制的所有 Pod 的负载变化情况来确定是否需要调整 Pod 的副本数量，这是 HPA 最基本的原理： ![hpa1.png](http://www.youqiqi.cn/upload/2020/05/hpa1-9b0ebe020dab4e4abd7f506262c79fb7.png) 我们可以简单的通过 kubectl autoscale 命令来创建一个 HPA 资源对象，HPA Controller默认30s轮询一次（可通过 kube-controller-manager 的--horizontal-pod-autoscaler-sync-period 参数进行设置），查询指定的资源中的 Pod 资源使用率，并且与创建时设定的值和指标做对比，从而实现自动伸缩的功能 # Metrics Server 在 HPA 的第一个版本中，我们需要 Heapster 提供 CPU 和内存指标，在 HPA v2 过后就需要安装 Metrcis Server 了，Metrics Server 可以通过标准的 Kubernetes API 把监控数据暴露出来，有了 Metrics Server 之后，我们就完全可以通过标准的 Kubernetes API 来访问我们想要获取的监控数据了： ```bash https://10.96.0.1/apis/metrics.k8s.io/v1beta1/namespaces/<namespace-name>/pods/<pod-name> ``` 比如当我们访问上面的 API 的时候，我们就可以获取到该 Pod 的资源数据，这些数据其实是来自于 kubelet 的 Summary API 采集而来的，不过需要说明的是我们这里可以通过标准的 API 来获取资源监控数据，并不是因为 Metrics Server 就是 APIServer 的一部分，而是通过 Kubernetes 提供的 Aggregator 汇聚插件来实现的，是独立于 APIServer 之外运行的 ![hpa2.png](http://www.youqiqi.cn/upload/2020/05/hpa2-d28e111984054f5c9a926d75ef2d5dac.png) ## 聚合 API Aggregator 允许开发人员编写一个自己的服务，把这个服务注册到 Kubernetes 的 APIServer 里面去，这样我们就可以像原生的 APIServer 提供的 API 使用自己的 API 了，我们把自己的服务运行在 Kubernetes 集群里面，然后 Kubernetes 的 Aggregator 通过 Service 名称就可以转发到我们自己写的 Service 里面去了。这样这个聚合层就带来了很多好处： 1. 增加了 API 的扩展性，开发人员可以编写自己的 API 服务来暴露他们想要的 API 2. 丰富了 API，核心 kubernetes 团队阻止了很多新的 API 提案，通过允许开发人员将他们的 API 作为单独的服务公开，这样就无须社区繁杂的审查了开发分阶段实验性 API，新的 API 可以在单独的聚合服务中开发，当它稳定之后，在合并会 APIServer 就很容易了 3. 确保新 API 遵循 Kubernetes 约定，如果没有这里提出的机制，社区成员可能会被迫推出自己的东西，这样很可能造成社区成员和社区约定不一致 ## 安装所以现在我们要使用 HPA，就需要在集群中安装 Metrics Server 服务，要安装 Metrics Server 就需要开启 Aggregator，因为 Metrics Server 就是通过该代理进行扩展的，不过我们集群是通过 Kubeadm 搭建的，默认已经开启了，如果是二进制方式安装的集群，需要单独配置 kube-apsierver 添加如下所示的参数： ```bash --requestheader-client-ca-file=<path to aggregator CA cert> --requestheader-allowed-names=aggregator --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=<path to aggregator proxy cert> --proxy-client-key-file=<path to aggregator proxy key> ``` 如果 kube-proxy 没有和 APIServer 运行在同一台主机上，那么需要确保启用了如下 kube-apsierver 的参数 ```bash --enable-aggregator-routing=true ``` 对于这些证书的生成方式，我们可以查看官方文档：https://github.com/kubernetes-sigs/apiserver-builder-alpha/blob/master/docs/concepts/auth.md ## 安装 Metrics Server 见博客Metrics Server安装 [点击跳转](http://www.youqiqi.cn/archives/k8s%E9%83%A8%E7%BD%B2metrics-server) ```bash [root@k8s-master ~]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master 180m 4% 6112Mi 79% k8s-node1 307m 7% 6340Mi 54% k8s-node2 314m 7% 5111Mi 43% ``` 可以通过 kubectl top 命令来获取到资源数据了，证明 Metrics Server 已经安装成功了 # 基于CPU 用 Deployment 来创建一个 Nginx Pod，然后利用 HPA 来进行自动扩缩容，资源清单如下所示：（hpa-demo.yaml） ```bash apiVersion: apps/v1 kind: Deployment metadata: name: hpa-demo namespace: test-env spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 resources: requests: memory: 50Mi cpu: 50m [root@k8s-master test]# kubectl apply -f hpa-demo.yaml deployment.apps/hpa-demo created [root@k8s-master test]# kubectl get -n test-env po NAME READY STATUS RESTARTS AGE hpa-demo-5c9d466b7f-mtqrl 1/1 Running 0 24s ``` 现在我们来创建一个 HPA 资源对象，可以使用kubectl autoscale命令来创建： ```bash [root@k8s-master test]# kubectl autoscale -n test-env deployment hpa-demo --cpu-percent=10 --min=1 --max=10 horizontalpodautoscaler.autoscaling/hpa-demo autoscaled [root@k8s-master test]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo <unknown>/10% 1 10 0 13s ``` 此命令创建了一个关联资源 hpa-demo 的 HPA，最小的 Pod 副本数为1，最大为10。HPA 会根据设定的 cpu 使用率（10%）动态的增加或者减少 Pod 数量当然我们依然还是可以通过创建 YAML 文件的形式来创建 HPA 资源对象，如果我们不知道怎么编写的话，可以查看上面命令行创建的HPA的YAML文件： ```bash [root@k8s-master test]# kubectl get -n test-env hpa hpa-demo -o yaml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: annotations: autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2020-05-25T02:56:54Z","reason":"ScaleDownStabilized","message":"recent recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2020-05-25T02:56:54Z","reason":"ValidMetricFound","message":"the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2020-05-25T02:56:54Z","reason":"DesiredWithinRange","message":"the desired count is within the acceptable range"}]' autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":0,"currentAverageValue":"0"}}]' creationTimestamp: "2020-05-25T02:56:39Z" name: hpa-demo namespace: test-env resourceVersion: "9626632" selfLink: /apis/autoscaling/v1/namespaces/test-env/horizontalpodautoscalers/hpa-demo uid: 3f09896b-9ff1-43f2-9160-781e64bd0ee7 spec: maxReplicas: 10 minReplicas: 1 scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: hpa-demo targetCPUUtilizationPercentage: 10 status: currentCPUUtilizationPercentage: 0 currentReplicas: 1 desiredReplicas: 1 ``` 然后我们可以根据上面的 YAML 文件就可以自己来创建一个基于 YAML 的 HPA 描述文件了，但是我们发现上面信息里面出现了一些 Fail 信息，我们来查看下这个 HPA 对象的信息： ```bash [root@k8s-master test]# kubectl describe -n test-env hpa hpa-demo Name: hpa-demo Namespace: test-env Labels: <none> Annotations: <none> CreationTimestamp: Mon, 25 May 2020 10:56:39 +0800 Reference: Deployment/hpa-demo Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 10% Min replicas: 1 Max replicas: 10 Deployment pods: 1 current / 1 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@k8s-master test]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo 0%/10% 1 10 1 4m40s ``` 可以看到 HPA 资源对象已经正常了，现在我们来增大负载进行测试，我们来创建一个 busybox 的 Pod，并且循环访问上面创建的 Pod： ```bash [root@k8s-master test]# kubectl get -n test-env pods -o wide -l app=nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hpa-demo-5c9d466b7f-mtqrl 1/1 Running 0 13m 10.244.1.88 k8s-node1 <none> <none> [root@k8s-master test]# kubectl run -it --image busybox test-hpa --restart=Never --rm /bin/sh -n test-env / # while true; do wget -q -O- http://10.244.1.88; done ``` 可以看到，HPA 已经开始工作 ```bash [root@k8s-master ~]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo 146%/10% 1 10 10 8m47s [root@k8s-master ~]# kubectl get -n test-env pods -l app=nginx --watch NAME READY STATUS RESTARTS AGE hpa-demo-5c9d466b7f-5p6pw 1/1 Running 0 76s hpa-demo-5c9d466b7f-bx4kf 1/1 Running 0 76s hpa-demo-5c9d466b7f-ddsj7 1/1 Running 0 61s hpa-demo-5c9d466b7f-htn6z 1/1 Running 0 61s hpa-demo-5c9d466b7f-mtqrl 1/1 Running 0 10m hpa-demo-5c9d466b7f-pmhwr 1/1 Running 0 61s hpa-demo-5c9d466b7f-r2dhr 1/1 Running 0 46s hpa-demo-5c9d466b7f-t2gw7 1/1 Running 0 46s hpa-demo-5c9d466b7f-t2sp6 1/1 Running 0 76s hpa-demo-5c9d466b7f-vk8pb 1/1 Running 0 61s ``` 可以看到已经自动拉起了很多新的 Pod，最后定格在了我们上面设置的 10 个 Pod，同时查看资源 hpa-demo 的副本数量，副本数量已经从原来的1变成了10个 ```bash [root@k8s-master ~]# kubectl get -n test-env deployment hpa-demo NAME READY UP-TO-DATE AVAILABLE AGE hpa-demo 10/10 10 10 11m ``` 查看 HPA 资源的对象了解工作过程 ```bash [root@k8s-master ~]# kubectl describe -n test-env hpa hpa-demo Name: hpa-demo Namespace: test-env Labels: <none> Annotations: <none> CreationTimestamp: Mon, 25 May 2020 10:56:39 +0800 Reference: Deployment/hpa-demo Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 27% (13m) / 10% Min replicas: 1 Max replicas: 10 Deployment pods: 10 current / 10 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 2m52s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 2m37s horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 2m22s horizontal-pod-autoscaler New size: 10; reason: cpu resource utilization (percentage of request) above target ``` 同样的这个时候我们来关掉 busybox 来减少负载，然后等待一段时间观察下 HPA 和 Deployment 对象 ```bash [root@k8s-master test]# kubectl get hpa -n test-env NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo 0%/10% 1 10 1 19m [root@k8s-master test]# kubectl get po -n test-env NAME READY STATUS RESTARTS AGE hpa-demo-5c9d466b7f-mtqrl 1/1 Running 0 20m ``` 从 Kubernetes v1.12 版本开始我们可以通过设置 kube-controller-manager 组件的--horizontal-pod-autoscaler-downscale-stabilization 参数来设置一个持续时间，用于指定在当前操作完成后，HPA 必须等待多长时间才能执行另一次缩放操作，默认为5分钟，也就是默认需要等待5分钟后才会开始自动缩放 # 基于内存 HorizontalPodAutoscaler 是 Kubernetes autoscaling API 组的资源，在当前稳定版本 autoscaling/v1 中只支持基于 CPU 指标的缩放。在 Beta 版本 autoscaling/v2beta2，引入了基于内存和自定义指标的缩放。所以我们这里需要使用 Beta 版本的 API ![hpa11.png](http://www.youqiqi.cn/upload/2020/06/hpa11-c4adb295b12d42968a7d26817a9ea871.png) 现在用 Deployment 来创建一个 Nginx Pod，然后利用 HPA 来进行自动扩缩容。资源清单如下所示：（hpa-mem-demo.yaml） ```bash apiVersion: apps/v1 kind: Deployment metadata: name: hpa-mem-demo namespace: test-env spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: volumes: - name: increase-mem-script configMap: name: increase-mem-config containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: increase-mem-script mountPath: /etc/script resources: requests: memory: 50Mi cpu: 50m securityContext: privileged: true ``` 这里和前面普通的应用有一些区别，我们将一个名为 increase-mem-config 的 ConfigMap 资源对象挂载到了容器中，该配置文件是用于后面增加容器内存占用的脚本，配置文件如下所示：（increase-mem-cm.yaml） ```bash apiVersion: v1 kind: ConfigMap metadata: name: increase-mem-config namespace: test-env data: increase-mem.sh: | #!/bin/bash mkdir /tmp/memory mount -t tmpfs -o size=40M tmpfs /tmp/memory dd if=/dev/zero of=/tmp/memory/block sleep 60 rm /tmp/memory/block umount /tmp/memory rmdir /tmp/memory ``` 由于这里增加内存的脚本需要使用到 mount 命令，这需要声明为特权模式，所以我们添加了 securityContext.privileged=true 这个配置。现在我们直接创建上面的资源对象即可 ```bash [root@k8s-master test]# kubectl apply -f increase-mem-cm.yaml [root@k8s-master test]# kubectl apply -f hpa-mem-demo.yaml [root@k8s-master test]# kubectl get -n test-env pods -l app=nginx NAME READY STATUS RESTARTS AGE hpa-mem-demo-66944b79bf-tqrn9 1/1 Running 0 35s ``` 然后需要创建一个基于内存的 HPA 资源对象：（hpa-mem.yaml） ```bash apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa namespcae: test-env spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-mem-demo minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: memory targetAverageUtilization: 60 ``` 要注意这里使用的 apiVersion 是 autoscaling/v2beta1，然后 metrics 属性里面指定的是内存的配置，直接创建上面的资源对象即可 ```bash [root@k8s-master test]# kubectl apply -f hpa-mem.yaml horizontalpodautoscaler.autoscaling/nginx-hpa created [root@k8s-master test]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo 0%/10% 1 10 1 43m nginx-hpa Deployment/hpa-mem-demo 2%/60% 1 5 1 23s ``` 到这里证明 HPA 资源对象已经部署成功了，接下来我们对应用进行压测，将内存压上去，直接执行上面我们挂载到容器中的 increase-mem.sh 脚本即可 ```bash [root@k8s-master test]# kubectl apply -f hpa-mem.yaml horizontalpodautoscaler.autoscaling/nginx-hpa created [root@k8s-master test]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/hpa-demo 0%/10% 1 10 1 43m nginx-hpa Deployment/hpa-mem-demo 2%/60% 1 5 1 23s [root@k8s-master test]# kubectl exec -it -n test-env hpa-mem-demo-6b9bd8f5ff-m7kgd bash root@hpa-mem-demo-6b9bd8f5ff-m7kgd:/# ls /etc/script/ increase-mem.sh root@hpa-mem-demo-6b9bd8f5ff-m7kgd:/# source /etc/script/increase-mem.sh dd: writing to '/tmp/memory/block': No space left on device 81921+0 records in 81920+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.227732 s, 184 MB/s ``` 然后打开另外一个终端观察 HPA 资源对象的变化情况 ```bash [root@k8s-master test]# kubectl get -n test-env hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/hpa-mem-demo 83%/60% 1 5 1 5m3s $ kubectl describe hpa nginx-hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"... CreationTimestamp: Tue, 07 Apr 2020 13:13:59 +0800 Reference: Deployment/hpa-mem-demo Metrics: ( current / target ) resource memory on pods (as a percentage of request): 3% (1740800) / 60% Min replicas: 1 Max replicas: 5 Deployment pods: 2 current / 2 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedGetResourceMetric 5m26s (x3 over 5m58s) horizontal-pod-autoscaler unable to get metrics for resource memory: no metrics returned from resource metrics API Warning FailedComputeMetricsReplicas 5m26s (x3 over 5m58s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API Normal SuccessfulRescale 77s horizontal-pod-autoscaler New size: 2; reason: memory resource utilization (percentage of request) above target $ kubectl top pod hpa-mem-demo-66944b79bf-tqrn9 NAME CPU(cores) MEMORY(bytes) hpa-mem-demo-66944b79bf-tqrn9 0m 41Mi ``` 可以看到内存使用已经超过了我们设定的 60% 这个阈值了，HPA 资源对象也已经触发了自动扩容，变成了两个副本了 ```bash [root@k8s-master test]# kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE hpa-mem-demo-66944b79bf-8m4d9 1/1 Running 0 2m51s hpa-mem-demo-66944b79bf-tqrn9 1/1 Running 0 8m11s ``` # 基于自定义指标除了基于 CPU 和内存来进行自动扩缩容之外，我们还可以根据自定义的监控指标来进行,这个我们就需要使用 Prometheus Adapter，Prometheus 用于监控应用的负载和集群本身的各种指标，Prometheus Adapter 可以帮我们使用 Prometheus 收集的指标并使用它们来制定扩展策略，这些指标都是通过 APIServer 暴露的，而且 HPA 资源对象也可以很轻易的直接使用 ![hpa12.png](http://www.youqiqi.cn/upload/2020/06/hpa12-6a3989e1cb3e4aad86b34953d810e68e.png)

k8s部署Helm

MySQL增量备份与恢复(二)