Kubernetes部署Metrics Server无法访问Apiserver

Kubernetes部署Metrics Server无法访问Apiserver

鲜花的主人 712 2021-01-19

系统环境

  • 部署方式：二进制
  • Docker 版本：19.03.8
  • kubernetes 版本：1.18.4
  • 操作系统版本：CentOS 7.4
  • metrics server 版本：0.4.1

#问题描述
二进制部署完集群后，部署Metrics Server，无法正常查看node及pod的信息，查看日志出现如下错误信息

E1231 10:33:31.978715 1 configmap_cafile_content.go:243] key failed with:
missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
E1231 10:34:22.710836 1 configmap_cafile_content.go:243] kube-system/extension-apiserver-authentication failed with:
missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
E1231 10:34:31.978769 1 configmap_cafile_content.go:243] key failed with:
missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"

根据错误日志信息，可以知道是缺少认证的证书文件，导致不能访问 kube-apiserver 而出现的问题

#问题分析
通过网上查找资料及像大牛们请教，这个错误是因为kube-apiserver 没有开启 API 聚合功能，所以需要配置 kube-apiserver 参数，开启聚合功能即可（kubeadm部署方式默认开启）

什么是API聚合

API 聚合机制 是 Kubernetes 1.7 版本引入的特性，能够将用户扩展的 API 注册到 kube-apiserver 上，仍然通过 API Server 的 HTTP URL 对新的 API 进行访问和操作。为了实现这个机制，Kubernetes 在 kube-apiserver 服务中引入了一个 API 聚合层（API Aggregation Layer），用于将 扩展 API 的访问请求转发到用户服务的功能
为了能够将用户自定义的 API 注册到 Master 的 API Server 中，首先需要在 Master 节点所在服务器，配置 kube-apiserver 应用的启动参数来启用 API 聚合 功能，参数如下：

--runtime-config=api/all=true 
--requestheader-allowed-names=aggregator 
--requestheader-group-headers=X-Remote-Group 
--requestheader-username-headers=X-Remote-User 
--requestheader-extra-headers-prefix=X-Remote-Extra- 
--requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem 
--proxy-client-cert-file=/etc/kubernetes/pki/proxy-client.pem 
--proxy-client-key-file=/etc/kubernetes/pki/proxy-client-key.pem

如果 kube-apiserver 所在的主机上没有运行 kube-proxy，即无法通过服务的 ClusterIP 进行访问，那么还需要设置以下启动参数

--enable-aggregator-routing=true

设置完成重启 kube-apiserver 服务，就启用 API 聚合功能了

问题解决

安装cfssl工具

#下载三个组件
[root@k8s01 ~]# wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl_linux-amd64 -O cfssl
[root@k8s01 ~]# wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssljson_linux-amd64  -O cfssljson
[root@k8s01 ~]# wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl-certinfo_linux-amd64  -O cfssl-certinfo

#复制到 bin 目录下
[root@k8s01 ~]# chmod +x ./cfssl*
[root@k8s01 ~]# mv ./cfssl* /usr/local/bin/

创建 cfssl 配置文件

[root@k8s01 ssl]# cat proxy-client-csr.json 
{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成证书和秘钥

[root@k8s01 ssl]# cfssl gencert \
  -profile=kubernetes \
  -ca=/opt/kubernetes/ssl/ca.pem \
  -ca-key=/opt/kubernetes/ssl/ca-key.pem \
  proxy-client-csr.json

#将上述输出内容输出到文件
[root@k8s01 ssl]# echo -e {"cert":"-----BEGIN CERTIFICATE-----\nMIID4jCCAsqgAwIBAgIUChvFjwuVC2W6aAUG6Pyo48eo42swDQYJKoZIhvcNAQEL\nBQAwZTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaWppbmcxEDAOBgNVBAcTB0Jl\naWppbmcxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwpr\ndWJlcm5ldGVzMB4XDTIxMTEwOTA4MDIwMFoXDTIyMTEwOTA4MDIwMFowcDELMAkG\nA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0JlaUppbmcxFzAV\nBgNVBAoTDnN5c3RlbTptYXN0ZXJzMQ8wDQYDVQQLEwZTeXN0ZW0xEzARBgNVBAMT\nCmFnZ3JlZ2F0b3IwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxdNwm\nD2zo/3S9g0RR+4cXDyor1y9+A3Ou2C9zQ9QH6479O8x3djpXaePYkv3iqmL82qZa\n4mnvnCdMflk248hkVOhAYKPTVXZAE8wn6PIbE5lJJp3coGROoIS9GudNIzv+omcn\nHNW6ON4K+nILxH5r6VhPDr2+M9E777FXBVff5kdu9uCWTJR+iaIavBYLk0rils+9\nYGTQoEqCEXWhymNJxtA2OvPYRbp/p26rSlMbZDZoxVV6XqMX+V/WpIrQP6jLE0yt\n9J7S/oJGG+o2WjxKTdAHQMm0zfeFKEw+nSjGvyjfvhufmQ7LBsztwBiaaIJE/aFm\nBAW+bxgwY9t3paFZAgMBAAGjfzB9MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAU\nBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUN97i\no5AAhba/6NY2boFd0/g9ilwwHwYDVR0jBBgwFoAU5/N2+0KJsCH4/cisAyKMcNQn\n8LowDQYJKoZIhvcNAQELBQADggEBADsIiiEqSDDQ0RJUAOLQzuI4jsHqYC0ELgTJ\nYOADI3+sQ0sQdkxKpMXZplOrrV2h2kAVwciRpuTyEfbEpQaq5pyP55eC+/3bsbHw\nunv9td88o2SukM5dAVP1yiGHrIa9yX8557TgXMvErB5F5nkxhs7+cUw4nQ6ARiOw\nkwklEUpNIUyfX5ADIcqNofnOYKAqvIsQzP36nkeJBL8WpunfC57rlZqPGGzcEoTu\nV+AIFExmgfMxeCs1fPaPYLomgEPIQp+QEZLh07HefLMibiY4ZYjGludMPYgIkfa0\nkIRvf3zlRZhsrhrocQ7ISKxe5VFpPrdvXH/bp8e/1tU5CfzNsB8=\n-----END CERTIFICATE-----\n","csr":"-----BEGIN CERTIFICATE REQUEST-----\nMIICtTCCAZ0CAQAwcDELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAO\nBgNVBAcTB0JlaUppbmcxFzAVBgNVBAoTDnN5c3RlbTptYXN0ZXJzMQ8wDQYDVQQL\nEwZTeXN0ZW0xEzARBgNVBAMTCmFnZ3JlZ2F0b3IwggEiMA0GCSqGSIb3DQEBAQUA\nA4IBDwAwggEKAoIBAQCxdNwmD2zo/3S9g0RR+4cXDyor1y9+A3Ou2C9zQ9QH6479\nO8x3djpXaePYkv3iqmL82qZa4mnvnCdMflk248hkVOhAYKPTVXZAE8wn6PIbE5lJ\nJp3coGROoIS9GudNIzv+omcnHNW6ON4K+nILxH5r6VhPDr2+M9E777FXBVff5kdu\n9uCWTJR+iaIavBYLk0rils+9YGTQoEqCEXWhymNJxtA2OvPYRbp/p26rSlMbZDZo\nxVV6XqMX+V/WpIrQP6jLE0yt9J7S/oJGG+o2WjxKTdAHQMm0zfeFKEw+nSjGvyjf\nvhufmQ7LBsztwBiaaIJE/aFmBAW+bxgwY9t3paFZAgMBAAGgADANBgkqhkiG9w0B\nAQsFAAOCAQEARNxIf1x9T6+/Sei7DGMzR06SYtXnOFlTuNQFJ+ne8iH6b5HRrC9A\nP3ui3hCwiDaLUikeoUTUxJn8+4guSsYnAYoUgWirO+CDHk4Zptym/atVmswEe/Q+\ncvHmZLUPWSTETHLVcc1tgr5wSl/wY2CcWZTWlimgsRR5ey08h8k3sffm0damKzqB\nVSO2kI8+hBSiA9WbMtR+2yCaD1BX+QwF3WIIV2Fk7eN76ADgJszoR+K9t6d2Uzjm\nG1s5UiGwe3ZQrEJygCzdf9DCffL6+rVZ0iaMcBwYLI15e0W32JWtmeeZwqYUueNu\nr7D9VvWgRJtq99t+xhz3cYrYbdH++YPUEg==\n-----END CERTIFICATE REQUEST-----\n","key":"-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAsXTcJg9s6P90vYNEUfuHFw8qK9cvfgNzrtgvc0PUB+uO/TvM\nd3Y6V2nj2JL94qpi/NqmWuJp75wnTH5ZNuPIZFToQGCj01V2QBPMJ+jyGxOZSSad\n3KBkTqCEvRrnTSM7/qJnJxzVujjeCvpyC8R+a+lYTw69vjPRO++xVwVX3+ZHbvbg\nlkyUfomiGrwWC5NK4pbPvWBk0KBKghF1ocpjScbQNjrz2EW6f6duq0pTG2Q2aMVV\nel6jF/lf1qSK0D+oyxNMrfSe0v6CRhvqNlo8Sk3QB0DJtM33hShMPp0oxr8o374b\nn5kOywbM7cAYmmiCRP2hZgQFvm8YMGPbd6WhWQIDAQABAoIBAEsvfwLlMDi720Nc\nNXsivcbIVw7CGk4iukXPNLN6foBojmm+h0/qGKfmpnGhPc5mcJA/N5J0MXwixgfD\n1JrMAaB41PpJt7+XiwV1Hd+e880DDcQPh3ztu/IQUo0os+1by1SEjH3m2qsO4wz1\ntZJEy2d/MzozvbKzk6f4U0PScLvcyPY0SpBoGaiwIC9I4p33ffa0P8T5wPauvj5b\nE1M+hoANzS85vOKdbzzkwGRYLShALtuZGQdnVcTRf47rpU1fFDS1Td2+A495G5Jd\nQb1toXSDlpF+3fEHpTQL17PY5Au4bZRYmUC5tvJI23ymPhKOp8jtWTbK1hxljXvI\nkkTreoECgYEAwqYvfCpM9e4OG+jhNE9hN4mTIvWFBdCPpBaagWCmc8Tl5j8qvsLf\nMClluXxxNkdlkjy29vw2CseLG3kRQxZsBIR7FR6Ddbh8N1OYLljIMFLVx6MO0L9E\n2/2eZmLK8t5B1waYHqzxKzeM1LVnU+5Bz5zWveg8P9DhDjrVoZfkZukCgYEA6WNv\nWPI/dIQOd4zeuizgU4OgEkueSGFvieS8bTQA/I81/8v7HwSUwm0CvENDo+QWxJcT\nz+Xwo8gsESf4hwu4gM9cwTLvFClRk3uL90ng8JLwLnoISWj8p4NcnRdm29rbzvNl\nFy8Ntod7zBTRJp0nRorvEBNunIxf1ziEUIWiwPECgYEApoaOsWdvGCdlorMFmD8h\nKOZmHs5105eui+9al70cKocVOHpqE3GciBOil6HDKXDDkOyoi30Srv4wIHzfK8oO\n935v3o2QYjJG8v5tf5ktC0qS4oy5rBU20A6WaXmzrzw+j7twgS3r1dFchs37bG61\nluzTOv1oLiCBqi7jfa/5E6ECgYBpG6IK8daIF+0YBDZOrGU+11mgw7N+L3t3I6+E\nktYa6DhOgkQPPRXt1gBUXwq4ZcGnOc7vY0QTztOhylmQIEQwprLEQ7cOYmldvVdU\nwE3wqoxGsijHPuQCYRO1n9NwEknKEy2k2kkXuj6Ts5BaDo6go3N0zvuSv1luaQKV\n7QlWUQKBgQC30FGthjGDHqP/6WsZBmcCz/+r6Iz5aXKDuE5WCAXUAtaq5Gh3vTEC\n6S4OQ24HQPXJe5EXu5zKHzB39EU6gb3wnCIvvC4AE+j1RMF5+6PN3ZupthSZi5SX\nf71HusMULSx32iwurgZfhWI+K/bzFUkTAE3Q/u7qNxrdvkI0HNwrtA==\n-----END RSA PRIVATE KEY-----\n"}

查看生成的证书

[root@k8s01 ssl]# ls -l|grep proxy
-rw-r--r-- 1 root root 1017 Dec 31 11:20 proxy-client.csr
-rw-r--r-- 1 root root  236 Dec 31 11:07 proxy-client-csr.json
-rw------- 1 root root 1675 Dec 31 11:20 proxy-client-key.pem
-rw-r--r-- 1 root root 1411 Dec 31 11:20 proxy-client.pem

复制到其它Master节点服务器中

[root@k8s01 ssl]# scp proxy-* root@k8s02:/opt/kubernetes/ssl
[root@k8s01 ssl]# scp proxy-* root@k8s03:/opt/kubernetes/ssl

修改 kube-apiserver参数

修改三个Master节点kube-apiserver配置参数

[root@k8s01 ~]# vim /opt/kubernetes/cfg/kube-apiserver.conf
...
--proxy-client-cert-file=/opt/kubernetes/ssl2/proxy-client.pem \
--proxy-client-key-file=/opt/kubernetes/ssl2/proxy-client-key.pem \
--runtime-config=api/all=true \
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
--requestheader-allowed-names=aggregator \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
...

参数说明：

  • –requestheader-client-ca-file： 客户端CA证书
  • –requestheader-allowed-names： 允许访问的客户端 common names 列表，通过 header 中 –requestheader-username-headers 参数指定的字段获取。客户端 common names 的名称需要在 client-ca-file 中进行设置，将其设置为空值时，表示任意客户端都可访问
  • –requestheader-username-headers： 参数指定的字段获取
  • –requestheader-extra-headers-prefix： 请求头中需要检查的前缀名
  • –requestheader-group-headers 请求头中需要检查的组名
  • –requestheader-username-headers 请求头中需要检查的用户名
  • –proxy-client-cert-file： 在请求期间验证Aggregator的客户端CA证书
  • –proxy-client-key-file： 在请求期间验证Aggregator的客户端私钥
  • –requestheader-allowed-names： 允许访问的客户端 common names 列表，通过 header 中 –requestheader-username-headers 参数指定的字段获取。客户端 common names 的名称需要在 client-ca-file 中进行设置，将其设置为空值时，表示任意客户端都可访问

修改完后重启kube-apiserver服务

验证

查看已有的 metrics server 的 pod

[root@k8s01 ~]# kubectl get -n kube-system po |grep metrics-server
metrics-server-646567c697-456kk              1/1     Running   3          18d

[root@k8s01 ~]# kubectl top node
NAME                   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s01                     132m         1%     1488Mi          9%        
k8s02                     161m         2%     1547Mi          9%        
k8s03                     154m         1%     1571Mi          9%        
k8s04                     1683m        21%    11708Mi         18%       
k8s05                     1550m        19%    5918Mi          9%        
k8s06                     2223m        27%    17364Mi         27%       
k8s07                     2481m        31%    10743Mi         16%       
k8s08                     1674m        20%    11242Mi         17%
Kubernetes