k8s部署SkyWalking8.5.0 | 爱吃可爱多

# 系统环境 - k8s版本：1.20.0 - skywalking版本：8.5.0 - elasticsearch版本：7.6.2 - MySQL版本：8.0.17 由于之前部署过6.1.0的版本，现在skywalking已经更新到8.5，改动的地方有点多，详情可以点击[官网介绍](官网介绍)查看，本文就不在过多描述，官网提供了helm安装，非常简介方便，本文采用yaml文件部署方式 # 架构 ![sky1.png](https://www.youqiqi.cn/upload/2020/12/sky1-0009cd89fae248a69a2749d4a26b5aff.png) 整个架构，分成上、下、左、右四部分： - 上部分 Agent ：负责从应用中，收集链路信息，发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是，SkyWalking Agent 收集 SkyWalking Tracing 数据，传递给服务器 - 下部分 SkyWalking OAP ：负责接收 Agent 发送的 Tracing 数据信息，然后进行分析(Analysis Core) ，存储到外部存储器( Storage )，最终提供查询( Query )功能 - 右部分 Storage ：Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ，主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主 - 左部分 SkyWalking UI ：负责提供控台，查看链路等等 # 部署SkyWalking ## 制作镜像由于时区问题，需要制作oap，ui，agent镜像 ### OAP镜像 ```bash [root@VM-32-194-centos skywalking]# cat oap/Dockerfile FROM apache/skywalking-oap-server:8.5.0-es7 # 时区修改为东八区 RUN apk add --no-cache tzdata ENV TZ=Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone #添加mysql数据驱动包 COPY mysql-connector-java-8.0.17.jar /skywalking/oap-libs [root@VM-32-194-centos skywalking]# docker build -t harbor.xxx.cn/skywalking/oap:8.5 . #上传到私由仓库 [root@VM-32-194-centos skywalking]# docker push harbor.xxx.cn/skywalking/oap:8.5 ``` ### UI镜像 ```bash [root@VM-32-194-centos skywalking]# cat ui/Dockerfile FROM apache/skywalking-ui:8.5.0 # 时区修改为东八区 RUN apk add --no-cache tzdata ENV TZ=Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone [root@VM-32-194-centos skywalking]# docker build -t harbor.xxx.cn/skywalking/ui:8.5 . #上传到私由仓库 [root@VM-32-194-centos skywalking]# docker push harbor.xxx.cn/skywalking/ui:8.5 ``` ### Agent镜像 ```bash [root@VM-32-194-centos skywalking]# cat agent/Dockerfile FROM alpine:3 # 时区修改为东八区 RUN apk add --no-cache tzdata ENV TZ=Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone ENV LANG=C.UTF-8 RUN set -eux && mkdir -p /data ADD agent /data/agent WORKDIR / [root@VM-32-194-centos skywalking]# docker build -t harbor.xxx.cn/skywalking/agent-sidecar:8.5 . #上传到私由仓库 [root@VM-32-194-centos skywalking]# docker push harbor.xxx.cn/skywalking/agent-sidecar ``` # 部署skywalking服务 Skywalking持久化跟踪数据默认使用的是H2，重启后数据就会丢失，本文将演示持久化到ES及MySQL，**生产请根据实际情况选择，一般建议使用ES** ## 部署es ```bash [root@k8s01 skywalking]# cat es-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: es namespace: skywalking spec: serviceName: elasticsearch replicas: 3 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch spec: imagePullSecrets: - name: harborsecret initContainers: - name: increase-vm-max-map image: busybox:latest command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true - name: increase-fd-ulimit image: busybox:latest command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true containers: - name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2 ports: - name: rest containerPort: 9200 - name: inter containerPort: 9300 resources: limits: cpu: 1000m requests: cpu: 1000m volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data env: - name: cluster.name value: k8s-logs - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: cluster.initial_master_nodes value: "es-0,es-1,es-2" - name: discovery.zen.minimum_master_nodes value: "2" - name: discovery.seed_hosts value: "elasticsearch" - name: ES_JAVA_OPTS value: "-Xms512m -Xmx512m" - name: network.host value: "0.0.0.0" volumeClaimTemplates: - metadata: name: data labels: app: elasticsearch spec: accessModes: [ "ReadWriteOnce" ] storageClassName: managed-nfs-storage resources: requests: storage: 100Gi --- kind: Service apiVersion: v1 metadata: name: elasticsearch namespace: skywalking labels: app: elasticsearch spec: selector: app: elasticsearch clusterIP: None ports: - port: 9200 name: rest - port: 9300 name: inter-node --- kind: Service apiVersion: v1 metadata: name: elasticsearch-client namespace: skywalking labels: app: elasticsearch spec: selector: app: elasticsearch ports: - port: 9200 name: rest - port: 9300 name: inter-node [root@k8s01 skywalking]# kubectl apply -f es-statefulset.yaml ``` ## 部署MySQL ```bash [root@k8s01 skywalking]# vim mysql.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: mysql-config namespace: skywalking labels: app: mysql data: my.cnf: |- [client-server] explicit_defaults_for_timestamp=true datadir = /var/lib/mysql [mysqld] port= 3306 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock pid-file=/var/run/mysqld/mysqld.pid log_queries_not_using_indexes = 1 bind-address = 0.0.0.0 skip-name-resolve back_log = 600 max_connections = 1000 max_connect_errors = 6000 lower_case_table_names = 1 open_files_limit = 65535 table_open_cache = 128 max_allowed_packet = 4M binlog_cache_size = 1M max_heap_table_size = 8M tmp_table_size = 16M read_buffer_size = 2M read_rnd_buffer_size = 8M sort_buffer_size = 8M join_buffer_size = 8M thread_cache_size = 8 key_buffer_size = 4M ft_min_word_len = 4 transaction_isolation = REPEATABLE-READ log_bin = mysql-bin binlog_format = mixed performance_schema = 0 explicit_defaults_for_timestamp innodb_file_per_table = 1 innodb_open_files = 500 innodb_buffer_pool_size = 64M innodb_write_io_threads = 4 innodb_read_io_threads = 4 innodb_thread_concurrency = 0 innodb_purge_threads = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 2M innodb_log_file_size = 32M innodb_log_files_in_group = 3 innodb_max_dirty_pages_pct = 90 innodb_lock_wait_timeout = 120 bulk_insert_buffer_size = 8M myisam_sort_buffer_size = 8M myisam_max_sort_file_size = 10G myisam_repair_threads = 1 interactive_timeout = 28800 wait_timeout = 28800 sql_mode="" [mysqldump] quick [myisamchk] key_buffer_size = 8M sort_buffer_size = 8M read_buffer = 4M write_buffer = 4M [client] port = 3306 socket=/var/lib/mysql/mysql.sock [mysqld_safe] log-error=/logs/mysql/mysqld.log innodb_buffer_pool innodb_buffer_pool_instance innodb_data_file_path transaction_isolation innodb_log_buffer_size innodb_log_file_size innodb_log_files_in_group max_connections expire_logs_days slow_query_log long_query_time binlog_format interactive_timeout wait_timeout innodb_flush_method log_queries_not_using_indexes --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-data-pvc namespace: skywalking labels: app: mysql spec: accessModes: - ReadWriteMany resources: requests: storage: 100Gi storageClassName: managed-nfs-storage --- apiVersion: v1 kind: Service metadata: name: mysql namespace: skywalking labels: app: mysql spec: type: NodePort ports: - name: mysql port: 3306 targetPort: 3306 nodePort: 30336 selector: app: mysql --- apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: skywalking labels: app: mysql spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0.17 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD #配置Root用户默认密码 value: "test-pwd" resources: limits: cpu: 2000m memory: 512Mi requests: cpu: 2000m memory: 512Mi livenessProbe: initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 exec: command: ["mysqladmin", "-uroot", "-p${MYSQL_ROOT_PASSWORD}", "ping"] readinessProbe: initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 exec: command: ["mysqladmin", "-uroot", "-p${MYSQL_ROOT_PASSWORD}", "ping"] volumeMounts: - name: data mountPath: /var/lib/mysql - name: config mountPath: /etc/mysql/conf.d/my.cnf subPath: my.cnf - name: localtime readOnly: true mountPath: /etc/localtime volumes: - name: data persistentVolumeClaim: claimName: mysql-data-pvc - name: config configMap: name: mysql-config - name: localtime hostPath: type: File path: /etc/localtime [root@k8s01 skywalking]# kubectl apply -f mysql.yaml configmap/mysql-config created persistentvolumeclaim/mysql-data-pvc created service/mysql created deployment.apps/mysql created ``` 创建skywalking数据库 ``` [root@k8s01 skywalking]# kubectl exec -it -n skywalking mysql-85b49f8f75-vk4mt bash root@mysql-85b49f8f75-vk4mt:/# mysql -ptest-pwd mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 96 Server version: 8.0.17 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create databases skywalking; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'databases skywalking' at line 1 mysql> create database skywalking; Query OK, 1 row affected (0.01 sec) ``` ## 部署oap ```bash [root@k8s01 skywalking]# cat rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: skywalking-oap namespace: skywalking --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: skywalking-oap namespace: skywalking roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: skywalking-oap subjects: - kind: ServiceAccount name: skywalking-oap namespace: skywalking --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: skywalking name: skywalking-oap rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"] [root@k8s01 skywalking]# kubectl apply -f rbac.yaml [root@k8s01 skywalking]# cat oap.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: oap release: oap name: oap namespace: skywalking spec: replicas: 1 selector: matchLabels: app: oap release: oap template: metadata: labels: app: oap release: oap spec: imagePullSecrets: - name: harborsecret containers: - env: - name: JAVA_OPTS value: -Dmode=on-init -Xmx2g -Xms2g - name: SW_CLUSTER value: kubernetes - name: SW_CLUSTER_K8S_NAMESPACE #集群命名空间 value: skywalking - name: SW_CLUSTER_K8S_LABEL value: app=oap,release=oap - name: SKYWALKING_COLLECTOR_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid # - name: SW_STORAGE #存储类型 # value: elasticsearch7 # - name: SW_STORAGE_ES_CLUSTER_NODES #es地址 # value: elasticsearch:9200 # - name: SW_ES_USER # value: xxx # - name: SW_ES_PASSWORD # value: xxx # - name: SW_STORAGE value: mysql - name: SW_JDBC_URL #mysql jdbc地址 value: jdbc:mysql://mysql:3306/skywalking #提前创建好skywalking - name: SW_DATA_SOURCE_USER value: root - name: SW_DATA_SOURCE_PASSWORD value: test-pwd image: harbor.xxx.cn/skywalking/oap:8.5 imagePullPolicy: Always livenessProbe: failureThreshold: 3 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: 12800 timeoutSeconds: 1 name: oap ports: - containerPort: 11800 name: grpc protocol: TCP - containerPort: 12800 name: rest protocol: TCP readinessProbe: failureThreshold: 3 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: 12800 timeoutSeconds: 1 #如果使用mysql存储，以下内容注释 # initContainers: # - command: # - sh # - -c # - for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep # 5; done; exit 1 # image: harbor.xxx.cn/skywalking/busybox:1.30 # imagePullPolicy: Always # name: wait-for-elasticsearch # restartPolicy: Always serviceAccount: skywalking-oap serviceAccountName: skywalking-oap --- apiVersion: v1 kind: Service metadata: labels: app: oap name: oap-svc namespace: skywalking spec: ports: - name: rest port: 12800 protocol: TCP targetPort: 12800 - name: grpc port: 11800 protocol: TCP targetPort: 11800 selector: app: oap sessionAffinity: None type: ClusterIP [root@k8s01 skywalking]# kubectl apply -f oap.yaml ``` **使用MySQL作为存储后，启动oap后查看数据是否创建表（146张表）** ## 部署ui ```bash [root@k8s01 skywalking]# cat ui.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: ui release: ui name: ui namespace: skywalking spec: replicas: 1 selector: matchLabels: app: ui release: ui template: metadata: labels: app: ui release: ui spec: imagePullSecrets: - name: harborsecret containers: - env: - name: SW_OAP_ADDRESS #oap地址 value: oap-svc:12800 - name: security.user.admin.password #账号密码 value: admin image: harbor.xxx.cn/skywalking/ui:8.5 imagePullPolicy: Always name: ui ports: - containerPort: 8080 name: page protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: ui name: ui-svc namespace: skywalking spec: externalTrafficPolicy: Cluster ports: - nodePort: 31234 port: 80 protocol: TCP targetPort: 8080 selector: app: ui sessionAffinity: None type: NodePort [root@k8s01 skywalking]# kubectl apply -f ui.yaml #查看已部署服务 [root@k8s01 skywalking]# kubectl get -n skywalking po NAME READY STATUS RESTARTS AGE es-0 1/1 Running 0 2d23h es-1 1/1 Running 0 2d23h es-2 1/1 Running 0 2d23h oap-7fdfb46778-szvc4 1/1 Running 0 45h ui-6476779bb4-k4xgt 1/1 Running 0 28h ``` ## 访问UI界面 IP:31234或者增加ingress域名访问 ![sk1.png](https://www.youqiqi.cn/upload/2021/06/sk1-6a641e3cec4d4617b404cb6afce87989.png) # 测试部署项目接入agent 测试将agent接入项目，这里简单的接入eureka getway等项目 ## eureka Dockerfile ```bash FROM harbor.xxx.cn/base/centos7:base COPY eureka.jar /home ENV JVM_OPTS="-Xss256k -XX:MaxRAMPercentage=80.0 -Duser.timezone=Asia/Shanghai -Djava.security.egd=file:/dev/./urandom" #jvm调优参数，可根据实际情况填写 ENV JAVA_OPTS="" #空着，给到yaml文件传参 EXPOSE 8081 WORKDIR /home ENTRYPOINT [ "sh", "-c", "java $JVM_OPTS $JAVA_OPTS -jar eureka.jar"] ``` ## 部署eureka ```bash [root@k8s01 ~]# cat eureka-finance5-test.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: eureka-finance5-test namespace: skywalking labels: app: eureka-finance5-test spec: replicas: 1 selector: matchLabels: app: eureka-finance5-test template: metadata: labels: app: eureka-finance5-test spec: imagePullSecrets: - name: harborsecret initContainers: - name: sidecar image: harbor.xxx.cn/skywalking/agent-sidecar:8.5 # 容器镜像，包含静态资源文件 imagePullPolicy: Always command: ["cp", "-r", "/data/agent", "/sidecar"] volumeMounts: - name: sidecar mountPath: /sidecar containers: - name: eureka-finance5-test image: harbor.xxx.cn/itsm/eureka:finance5-test imagePullPolicy: Always env: - name: JAVA_OPTS value: -javaagent:/sidecar/agent/skywalking-agent.jar - name: SW_AGENT_NAME value: eureka-finance5-test - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES #oap地址 value: oap-svc:11800 resources: limits: memory: "1Gi" requests: memory: "1Gi" ports: - name: http containerPort: 8081 protocol: TCP volumeMounts: - name: date mountPath: /etc/localtime - name: sidecar mountPath: /sidecar volumes: - name: date hostPath: path: /etc/localtime - name: sidecar #共享agent文件夹 emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: eureka-finance5-test-svc namespace: skywalking labels: app: eureka-finance5-test spec: sessionAffinity: "ClientIP" ports: - name: http port: 8081 protocol: TCP targetPort: 8081 nodePort: 30835 selector: app: eureka-finance5-test type: NodePort [root@k8s01 ~]# kubectl apply -f eureka-finance5-test.yaml ``` 根据上述步骤一一部署项目 ```bash [root@k8s01 ~]# kubectl get -n skywalking po NAME READY STATUS RESTARTS AGE basics-finance5-test-7665cf7f6d-ndm2n 1/1 Running 0 24h es-0 1/1 Running 0 2d23h es-1 1/1 Running 0 2d23h es-2 1/1 Running 0 2d23h eureka-finance5-test-58db7dbb78-szjrq 1/1 Running 0 24h formflow-finance5-test-57bdfc768-sxzb5 1/1 Running 0 23h getway-finance5-test-6dd6d9f864-lblpb 1/1 Running 0 24h oap-7fdfb46778-szvc4 1/1 Running 0 45h solr-finance5-test-9c9d77fd4-vnp6h 1/1 Running 0 23h ui-6476779bb4-k4xgt 1/1 Running 0 28h ``` ## 访问UI查看 ![sk2.png](https://www.youqiqi.cn/upload/2021/06/sk2-af3a4c5a9d2e4343b4e7c6812191ba92.png) ![sk3.png](https://www.youqiqi.cn/upload/2021/06/sk3-6811718714da4edd8e89b1712818dd59.png)

OpenVPN部署

Kubernetes部署Nacos2.0.1集群