系统环境

  • k8s版本:k8s1.15.4
  • skywalking版本:6.1.0

简介

SkyWalking是什么

分布式系统的应用程序性能监视工具,专为微服务、云原生架构和基于容器(Docker、K8s、Mesos)架构而设计
提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案

功能列表

SkyWalking功能

  • 多种监控手段,可以通过语言探针和 service mesh 获得监控是数据
  • 多个语言自动探针,包括 Java,.NET Core 和 Node.JS
  • 轻量高效,无需大数据平台,和大量的服务器资源
  • 模块化,UI、存储、集群管理都有多种机制可选
  • 支持告警
  • 优秀的可视化解决方案

架构

sky1.png
整个架构,分成上、下、左、右四部分:

  • 上部分 Agent :负责从应用中,收集链路信息,发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是,SkyWalking Agent 收集 SkyWalking Tracing 数据,传递给服务器
  • 下部分 SkyWalking OAP :负责接收 Agent 发送的 Tracing 数据信息,然后进行分析(Analysis Core) ,存储到外部存储器( Storage ),最终提供查询( Query )功能
  • 右部分 Storage :Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ,主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主
  • 左部分 SkyWalking UI :负责提供控台,查看链路等等

部署SkyWalking

制作镜像

镜像分为四个,基础镜像,oap,ui,agent镜像,当然也可以将基础镜像和oap ui写在一起

基础镜像Dockefile

[root@VM-32-194-centos base]# cat Dockerfile
FROM openjdk:8-jre-alpine AS build

RUN apk update \
    apk add --no-cache \
    bash \
    python \
    su-exec

# 时区修改为东八区
RUN apk add --no-cache tzdata
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

ENV SKYWALKING_HOME=/skywalking
RUN mkdir -p "${SKYWALKING_HOME}"
WORKDIR $SKYWALKING_HOME

ENV GPG_KEYS A968F6905E0ACB59E5B24C15D3D9CD50820184C2 B0801BC746F965029A1338072EF5026E70A55777 D360AB2AB20B28403270E2CBE8608938DB25E06B
ENV SKYWALKING_VERSION=6.1.0
ENV SKYWALKING_SHA512 54fe85c984c369fd2731aa10190ee7e82e15e2c7c9528234993f6c3d4db34d19735b5467875e20dc3f8476e8d6d56d4fcd6118b4aff49f7b5a7215ee032918c0

ENV SKYWALKING_TGZ_URLS \
        https://www.apache.org/dyn/closer.cgi?action=download&filename=skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz \
	    https://www-us.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz \
	    https://www.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz \
	    https://archive.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz

ENV SKYWALKING_ASC_URLS \
        https://www.apache.org/dyn/closer.cgi?action=download&filename=skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz.asc \
	    https://www-us.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz.asc \
	    https://www.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz.asc \
	    https://archive.apache.org/dist/skywalking/$SKYWALKING_VERSION/apache-skywalking-apm-$SKYWALKING_VERSION.tar.gz.asc

RUN set -eux; \
	\
	apk add --no-cache --virtual .fetch-deps \
		gnupg \
		\
		ca-certificates \
		openssl \
	; \
	\
	export GNUPGHOME="$(mktemp -d)"; \
	for key in $GPG_KEYS; do \
		gpg --batch --keyserver pgp.surfnet.nl --recv-keys "$key"; \
	done; \
	\
	success=; \
	for url in $SKYWALKING_TGZ_URLS; do \
		if wget -O skywalking.tar.gz "$url"; then \
			success=1; \
			break; \
		fi; \
	done; \
	[ -n "$success" ]; \
	\
	echo "$SKYWALKING_SHA512 *skywalking.tar.gz" | sha512sum -c -; \
	\
	success=; \
	for url in $SKYWALKING_ASC_URLS; do \
		if wget -O skywalking.tar.gz.asc "$url"; then \
			success=1; \
			break; \
		fi; \
	done; \
	[ -n "$success" ]; \
	\
	gpg --batch --verify skywalking.tar.gz.asc skywalking.tar.gz; \
	tar -xvf skywalking.tar.gz --strip-components=1; \
	rm -rf bin/; \
	rm skywalking.tar.gz*; \
	command -v gpgconf && gpgconf --kill all || :; \
	rm -rf "$GNUPGHOME"; \
    apk del .fetch-deps

FROM openjdk:8-jre-alpine

COPY --from=build /skywalking /skywalking

RUN apk add --no-cache \
    bash

#build镜像,这里使用内网harbor地址,根据实际情况修改    
[root@VM-32-194-centos base]# docker build -t 10.133.193.241/skywalking/base:6.1.0 .
Sending build context to Docker daemon  4.608kB
Step 1/17 : FROM openjdk:8-jre-alpine AS build
......
Removing intermediate container bdbe3d303173
 ---> ef1487f03a63
Successfully built ef1487f03a63
Successfully tagged 10.133.193.241/skywalking/base:6.1.0

#上传到私有仓库
[root@VM-32-194-centos base]# docker push 10.133.193.241/skywalking/base:6.1.0

oap镜像Dockefile

文件包扩Dockerfile docker-entrypoint.sh log4j2.xml,保持在同一级目录下

[root@VM-32-194-centos oap]# cat Dockerfile 
FROM 10.133.193.241/skywalking/base:6.1.0
# 时区修改为东八区
RUN apk update \
    apk add tzdata
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

ENV JAVA_OPTS=" -Xms256M " \
    SW_CLUSTER="standalone" \
    SW_STORAGE="h2"

WORKDIR skywalking

RUN set -eux; \
    rm -rf "config/log4j2.xml"; \
    rm -rf "webapp"; rm -rf "agent";

COPY log4j2.xml config/
COPY docker-entrypoint.sh .

EXPOSE 12800 11800 1234

ENTRYPOINT ["bash", "docker-entrypoint.sh"]

[root@VM-32-194-centos oap]# cat log4j2.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!--
       ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  ~
  -->

<Configuration status="INFO">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout charset="UTF-8" pattern="%d - %c -%-4r [%t] %-5p %x - %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <logger name="org.eclipse.jetty" level="INFO"/>
        <logger name="org.apache.zookeeper" level="INFO"/>
        <logger name="org.elasticsearch.common.network.IfConfig" level="INFO"/>
        <logger name="io.grpc.netty" level="INFO"/>
        <Root level="INFO">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>

[root@VM-32-194-centos oap]# cat docker-entrypoint.sh|grep -v "#"


set -e

var_application_file="config/application.yml"

generateClusterStandalone() {
    echo "cluster:" >> ${var_application_file}
    echo "  standalone:" >> ${var_application_file}
}

generateClusterZookeeper() {
    cat <<EOT >> ${var_application_file}
cluster:
  zookeeper:
    nameSpace: \${SW_NAMESPACE:""}
    hostPort: \${SW_CLUSTER_ZK_HOST_PORT:localhost:2181}
EOT
}

generateClusterK8s() {
    cat <<EOT >> ${var_application_file}
cluster:
  kubernetes:
    watchTimeoutSeconds: \${SW_CLUSTER_K8S_WATCH_TIMEOUT:60}
    namespace: \${SW_CLUSTER_K8S_NAMESPACE:default}
    labelSelector: \${SW_CLUSTER_K8S_LABEL:app=collector,release=skywalking}
    uidEnvName: \${SW_CLUSTER_K8S_UID:SKYWALKING_COLLECTOR_UID}
EOT
}

generateClusterConsul() {
    cat <<EOT >> ${var_application_file}
cluster:
  consul:
    serviceName: \${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
     Consul cluster nodes, example: 10.0.0.1:8500,10.0.0.2:8500,10.0.0.3:8500
    hostPort: \${SW_CLUSTER_CONSUL_HOST_PORT:localhost:8500}
EOT
}

generateStorageElastisearch() {
    cat <<EOT >> ${var_application_file}
storage:
  elasticsearch:
    nameSpace: \${SW_NAMESPACE:""}
    clusterNodes: \${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    user: \${SW_ES_USER:""}
    password: \${SW_ES_PASSWORD:""}
    indexShardsNumber: \${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
    indexReplicasNumber: \${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
    metadataQueryMaxSize: \${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
    segmentQueryMaxSize: \${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
EOT
}

generateStorageH2() {
    cat <<EOT >> ${var_application_file}
storage:
  h2:
    driver: \${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
    url: \${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db}
    user: \${SW_STORAGE_H2_USER:sa}
    metadataQueryMaxSize: \${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
EOT
}

generateStorageMySQL() {
    cat <<EOT >> ${var_application_file}
storage:
  mysql:
    metadataQueryMaxSize: \${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
EOT
}

validateVariables() {
    name=$1; value=$2; list=$3
    valid=false
    for c in ${list} ; do
        if [[ "$c" = "$value" ]]; then
            valid=true
        fi
    done

    if ! ${valid}; then
        echo "Error: $name=$value please specify $name = $list"
        exit 1
    fi
}

generateApplicationYaml() {
    [[ -z "$SW_CLUSTER" ]] && [[ -z "$SW_STORAGE" ]] && { echo "Error: please specify \"SW_CLUSTER\" \"SW_STORAGE\""; exit 1; }

    validateVariables "SW_CLUSTER" "$SW_CLUSTER" "standalone zookeeper kubernetes consul"

    validateVariables "SW_STORAGE" "$SW_STORAGE" "elasticsearch h2 mysql"

    case ${SW_CLUSTER} in
    standalone) generateClusterStandalone;;
    zookeeper) generateClusterZookeeper;;
    kubernetes) generateClusterK8s;;
    consul) generateClusterConsul;;
    esac

    cat <<EOT >> ${var_application_file}
core:
  default:
    restHost: \${SW_CORE_REST_HOST:0.0.0.0}
    restPort: \${SW_CORE_REST_PORT:12800}
    restContextPath: \${SW_CORE_REST_CONTEXT_PATH:/}
    gRPCHost: \${SW_CORE_GRPC_HOST:0.0.0.0}
    gRPCPort: \${SW_CORE_GRPC_PORT:11800}
    downsampling:
    - Hour
    - Day
    - Month
EOT

    case ${SW_STORAGE} in
    elasticsearch) generateStorageElastisearch;;
    h2) generateStorageH2;;
    mysql) generateStorageMySQL;;
    esac

    cat <<EOT >> ${var_application_file}
receiver-sharing-server:
  default:
receiver-register:
  default:
receiver-trace:
  default:
    bufferFileCleanWhenRestart: \${SW_RECEIVER_BUFFER_FILE_CLEAN_WHEN_RESTART:false}
receiver-jvm:
  default:
receiver-clr:
  default:
service-mesh:
  default:
    bufferFileCleanWhenRestart: \${SW_SERVICE_MESH_BUFFER_FILE_CLEAN_WHEN_RESTART:false}
istio-telemetry:
  default:
envoy-metric:
  default:
query:
  graphql:
    path: \${SW_QUERY_GRAPHQL_PATH:/graphql}
alarm:
  default:
telemetry:
  prometheus:
    host: \${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
    port: \${SW_TELEMETRY_PROMETHEUS_PORT:1234}
EOT

    if [[ "$SW_RECEIVER_ZIPKIN_ENABLED" = "true" ]]; then
        cat <<EOT >> ${var_application_file}
receiver_zipkin:
  default:
    host: \${SW_RECEIVER_ZIPKIN_HOST:0.0.0.0}
    port: \${SW_RECEIVER_ZIPKIN_PORT:9411}
    contextPath: \${SW_RECEIVER_ZIPKIN_CONTEXT_PATH:/}
EOT
    fi

    if [[ "$SW_RECEIVER_JAEGER_ENABLED" = "true" ]]; then
        cat <<EOT >> ${var_application_file}
receiver_jaeger:
  default:
    gRPCHost: \${SW_RECEIVER_JAEGER_HOST:0.0.0.0}
    gRPCPort: \${SW_RECEIVER_JAEGER_PORT:14250}
EOT
    fi

    if [[ "$SW_EXPORTER_ENABLED" = "true" ]]; then
        cat <<EOT >> \${var_application_file}
exporter:
  grpc:
    targetHost: \${SW_EXPORTER_GRPC_HOST:127.0.0.1}
    targetPort: \${SW_EXPORTER_GRPC_PORT:9870}
EOT
    fi
}

echo "[Entrypoint] Apache SkyWalking Docker Image"

if [[ -z "$SW_L0AD_CONFIG_FILE_FROM_VOLUME" ]] || [[ "$SW_L0AD_CONFIG_FILE_FROM_VOLUME" != "true" ]]; then
    generateApplicationYaml
    echo "Generated application.yml"
    echo "-------------------------"
    cat ${var_application_file}
    echo "-------------------------"
fi

CLASSPATH="config:$CLASSPATH"
for i in oap-libs/*.jar
do
    CLASSPATH="$i:$CLASSPATH"
done

exec java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap \
     ${JAVA_OPTS} -classpath ${CLASSPATH} org.apache.skywalking.oap.server.starter.OAPServerStartUp "$@"
     
#build镜像
 [root@VM-32-194-centos oap]# docker build -t 10.133.193.241/skywalking/oap:6.1.0 .
 #上传到私有仓库
 [root@VM-32-194-centos oap]# docker push 10.133.193.241/skywalking/oap:6.1.0    

ui镜像Dockefile

文件包扩Dockerfile docker-entrypoint.sh logback.xml,保持在同一级目录下

[root@VM-32-194-centos ui]# cat Dockerfile 
FROM 10.133.193.241/skywalking/base:6.1.0

# 时区修改为东八区
RUN apk add --no-cache tzdata
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

ENV JAVA_OPTS=" -Xms256M " \
    SW_OAP_ADDRESS="127.0.0.1:12800" \
    SW_TIMEOUT="20000"

WORKDIR skywalking

RUN set -eux; \
    rm -rf "config"; rm -rf "oap-libs"; \
    rm -rf "agent";

COPY docker-entrypoint.sh .
COPY logback.xml webapp/

EXPOSE 8080

ENTRYPOINT ["bash", "docker-entrypoint.sh"]

[root@VM-32-194-centos ui]# cat docker-entrypoint.sh
#!/bin/bash

set -e

export LOGGING_CONFIG="webapp/logback.xml"

[[ ! -z "$SW_OAP_ADDRESS" ]] && export COLLECTOR_RIBBON_LISTOFSERVERS=${SW_OAP_ADDRESS} && echo "COLLECTOR_RIBBON_LISTOFSERVERS=$COLLECTOR_RIBBON_LISTOFSERVERS"
[[ ! -z "$SW_TIMEOUT" ]] && export COLLECTOR_RIBBON_READTIMEOUT=${SW_TIMEOUT} && echo "COLLECTOR_RIBBON_READTIMEOUT=$COLLECTOR_RIBBON_READTIMEOUT"

exec java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar webapp/skywalking-webapp.jar "$@"

[root@VM-32-194-centos ui]# cat logback.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!--
       ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  ~
  -->
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />
</configuration>

#build镜像
[root@VM-32-194-centos ui]# docker build -t 10.133.193.241/skywalking/ui:6.1.0 .
#上传镜像
[root@VM-32-194-centos ui]# docker push 10.133.193.241/skywalking/ui:6.1.0

agent镜像Dockefile

[root@VM-32-194-centos agent]# cat Dockerfile
FROM alpine:3

# 时区修改为东八区
RUN apk add --no-cache tzdata
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

ENV LANG=C.UTF-8
  
RUN set -eux && mkdir -p /data
  
ADD agent /data/agent
  
WORKDIR /
[root@VM-32-194-centos agent]# docker build -t 10.133.193.241/skywalking/agent-sidecar:6.1.0 .
[root@VM-32-194-centos agent]# docker push 10.133.193.241/skywalking/agent-sidecar:6.1.0

部署skywalking服务

创建namespace

[root@k8s01 ~]# kubectl create ns skywalking
namespace/skywalking created

部署es

[root@k8s01 skywalking]# vim es.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: skywalking
  labels:
    service: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      service: elasticsearch
  template:
    metadata:
      labels:
        service: elasticsearch
    spec:
      terminationGracePeriodSeconds: 300
      initContainers:
      - name: fix-the-volume-permission
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-the-vm-max-map-count
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      - name: increase-the-ulimit
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sh
        - -c
        - ulimit -n 65536
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: tcp
        resources:
          requests:
            memory: 4Gi
          limits:
            memory: 4Gi
        env:
          - name: cluster.name
            value: elasticsearch-cluster
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: elasticsearch:9300
          - name: ES_JAVA_OPTS
            value: -Xms3g -Xmx3g
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: managed-nfs-storage
      resources:
        requests:
          storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: skywalking
  labels:
    service: elasticsearch
spec:
  clusterIP: None
  ports:
  - port: 9200
    name: serving
  - port: 9300
    name: node-to-node
  selector:
    service: elasticsearch

#部署es
[root@k8s01 skywalking]# kubectl apply -f es.yaml
statefulset.apps/elasticsearch created
service/elasticsearch created   
[root@k8s01 skywalking]# kubectl get -n skywalking po
NAME              READY   STATUS    RESTARTS   AGE
elasticsearch-0   1/1     Running   0          4m39s

部署oap

[root@k8s01 oap]# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: skywalking-oap
  namespace: skywalking
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: skywalking-oap
  namespace: skywalking
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: skywalking-oap
subjects:
- kind: ServiceAccount
  name: skywalking-oap
  namespace: skywalking
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: skywalking
  name: skywalking-oap
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
[root@k8s01 oap]# kubectl apply -f rbac.yaml
serviceaccount/skywalking-oap created
rolebinding.rbac.authorization.k8s.io/skywalking-oap created
role.rbac.authorization.k8s.io/skywalking-oap created

[root@k8s01 oap]# cat cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: oap-config
  namespace: skywalking
data:
  application.yml: |-
    cluster:
      # standalone:
      # Please check your ZooKeeper is 3.5+, However, it is also compatible with ZooKeeper 3.4.x. Replace the ZooKeeper 3.5+
      # library the oap-libs folder with your ZooKeeper 3.4.x library.
    #  zookeeper:
    #    nameSpace: ${SW_NAMESPACE:""}
    #    hostPort: ${SW_CLUSTER_ZK_HOST_PORT:localhost:2181}
    #    #Retry Policy
    #    baseSleepTimeMs: ${SW_CLUSTER_ZK_SLEEP_TIME:1000} # initial amount of time to wait between retries
    #    maxRetries: ${SW_CLUSTER_ZK_MAX_RETRIES:3} # max number of times to retry
      kubernetes:
        watchTimeoutSeconds: ${SW_CLUSTER_K8S_WATCH_TIMEOUT:60}
        namespace: ${SW_CLUSTER_K8S_NAMESPACE:skywalking}
        labelSelector: ${SW_CLUSTER_K8S_LABEL:app=oap,release=skywalking}
        uidEnvName: ${SW_CLUSTER_K8S_UID:SKYWALKING_COLLECTOR_UID}
    #  consul:
    #    serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
    #     Consul cluster nodes, example: 10.0.0.1:8500,10.0.0.2:8500,10.0.0.3:8500
    #    hostPort: ${SW_CLUSTER_CONSUL_HOST_PORT:localhost:8500}
    core:
      default:
        # Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
        # Receiver: Receive agent data, Level 1 aggregate
        # Aggregator: Level 2 aggregate
        role: ${SW_CORE_ROLE:Mixed} # Mixed/Receiver/Aggregator
        restHost: ${SW_CORE_REST_HOST:0.0.0.0}
        restPort: ${SW_CORE_REST_PORT:12800}
        restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
        gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
        gRPCPort: ${SW_CORE_GRPC_PORT:11800}
        downsampling:
        - Hour
        - Day
        - Month
        # Set a timeout on metric data. After the timeout has expired, the metric data will automatically be deleted.
        recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:90} # Unit is minute
        minuteMetricsDataTTL: ${SW_CORE_MINUTE_METRIC_DATA_TTL:90} # Unit is minute
        hourMetricsDataTTL: ${SW_CORE_HOUR_METRIC_DATA_TTL:36} # Unit is hour
        dayMetricsDataTTL: ${SW_CORE_DAY_METRIC_DATA_TTL:45} # Unit is day
        monthMetricsDataTTL: ${SW_CORE_MONTH_METRIC_DATA_TTL:18} # Unit is month
    storage:
      elasticsearch:
        nameSpace: ${SW_NAMESPACE:""}
        clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:elasticsearch:9200}
        user: ${SW_ES_USER:""}
        password: ${SW_ES_PASSWORD:""}
        indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
        indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
        # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
        bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:2000} # Execute the bulk every 2000 requests
        bulkSize: ${SW_STORAGE_ES_BULK_SIZE:20} # flush the bulk every 20mb
        flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
        concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
        metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
        segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
      # h2:
      #   driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
      #   url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db}
      #   user: ${SW_STORAGE_H2_USER:sa}
      #   metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
    #  mysql:
    #    metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
    receiver-sharing-server:
      default:
    receiver-register:
      default:
    receiver-trace:
      default:
        bufferPath: ${SW_RECEIVER_BUFFER_PATH:../trace-buffer/}  # Path to trace buffer files, suggest to use absolute path
        bufferOffsetMaxFileSize: ${SW_RECEIVER_BUFFER_OFFSET_MAX_FILE_SIZE:100} # Unit is MB
        bufferDataMaxFileSize: ${SW_RECEIVER_BUFFER_DATA_MAX_FILE_SIZE:500} # Unit is MB
        bufferFileCleanWhenRestart: ${SW_RECEIVER_BUFFER_FILE_CLEAN_WHEN_RESTART:false}
        sampleRate: ${SW_TRACE_SAMPLE_RATE:10000} # The sample rate precision is 1/10000. 10000 means 100% sample in default.
        slowDBAccessThreshold: ${SW_SLOW_DB_THRESHOLD:default:200,mongodb:100} # The slow database access thresholds. Unit ms.
    receiver-jvm:
      default:
    receiver-clr:
      default:
    service-mesh:
      default:
        bufferPath: ${SW_SERVICE_MESH_BUFFER_PATH:../mesh-buffer/}  # Path to trace buffer files, suggest to use absolute path
        bufferOffsetMaxFileSize: ${SW_SERVICE_MESH_OFFSET_MAX_FILE_SIZE:100} # Unit is MB
        bufferDataMaxFileSize: ${SW_SERVICE_MESH_BUFFER_DATA_MAX_FILE_SIZE:500} # Unit is MB
        bufferFileCleanWhenRestart: ${SW_SERVICE_MESH_BUFFER_FILE_CLEAN_WHEN_RESTART:false}
    istio-telemetry:
      default:
    envoy-metric:
      default:
    #receiver_zipkin:
    #  default:
    #    host: ${SW_RECEIVER_ZIPKIN_HOST:0.0.0.0}
    #    port: ${SW_RECEIVER_ZIPKIN_PORT:9411}
    #    contextPath: ${SW_RECEIVER_ZIPKIN_CONTEXT_PATH:/}
    query:
      graphql:
        path: ${SW_QUERY_GRAPHQL_PATH:/graphql}
    alarm:
      default:
    telemetry:
      none:
  log4j2.xml: |-
    <Configuration status="INFO">
        <Appenders>
            <Console name="Console" target="SYSTEM_OUT">
                <PatternLayout charset="UTF-8" pattern="%d - %c -%-4r [%t] %-5p %x - %m%n"/>
            </Console>
        </Appenders>
        <Loggers>
            <logger name="org.eclipse.jetty" level="INFO"/>
            <logger name="org.apache.zookeeper" level="INFO"/>
            <logger name="org.elasticsearch.common.network.IfConfig" level="INFO"/>
            <logger name="io.grpc.netty" level="INFO"/>
            <logger name="org.apache.skywalking.oap.server.receiver.istio.telemetry" level="DEBUG"/>
            <Root level="INFO">
                <AppenderRef ref="Console"/>
            </Root>
        </Loggers>
    </Configuration>
  alarm-settings.yml: |-
    rules:
      service_resp_time_rule:
        indicator-name: service_resp_time
        include-names:
          - dubbox-provider
          - dubbox-consumer
        threshold: 1000
        op: ">"
        period: 10
        count: 1
    webhooks:
  component-libraries.yml: |-
    Tomcat:
      id: 1
      languages: Java
    HttpClient:
      id: 2
      languages: Java,C#,Node.js
    Dubbo:
      id: 3
      languages: Java
    H2:
      id: 4
      languages: Java
    Mysql:
      id: 5
      languages: Java,C#,Node.js
    ORACLE:
      id: 6
      languages: Java
    Redis:
      id: 7
      languages: Java,C#,Node.js
    Motan:
      id: 8
      languages: Java
    MongoDB:
      id: 9
      languages: Java,C#,Node.js
    Resin:
      id: 10
      languages: Java
    Feign:
      id: 11
      languages: Java
    OKHttp:
      id: 12
      languages: Java
    SpringRestTemplate:
      id: 13
      languages: Java
    SpringMVC:
      id: 14
      languages: Java
    Struts2:
      id: 15
      languages: Java
    NutzMVC:
      id: 16
      languages: Java
    NutzHttp:
      id: 17
      languages: Java
    JettyClient:
      id: 18
      languages: Java
    JettyServer:
      id: 19
      languages: Java
    Memcached:
      id: 20
      languages: Java
    ShardingJDBC:
      id: 21
      languages: Java
    PostgreSQL:
      id: 22
      languages: Java,C#,Node.js
    GRPC:
      id: 23
      languages: Java
    ElasticJob:
      id: 24
      languages: Java
    RocketMQ:
      id: 25
      languages: Java
    httpasyncclient:
      id: 26
      languages: Java
    Kafka:
      id: 27
      languages: Java
    ServiceComb:
      id: 28
      languages: Java
    Hystrix:
      id: 29
      languages: Java
    Jedis:
      id: 30
      languages: Java
    SQLite:
      id: 31
      languages: Java,C#
    h2-jdbc-driver:
      id: 32
      languages: Java
    mysql-connector-java:
      id: 33
      languages: Java
    ojdbc:
      id: 34
      languages: Java
    Spymemcached:
      id: 35
      languages: Java
    Xmemcached:
      id: 36
      languages: Java
    postgresql-jdbc-driver:
      id: 37
      languages: Java
    rocketMQ-producer:
      id: 38
      languages: Java
    rocketMQ-consumer:
      id: 39
      languages: Java
    kafka-producer:
      id: 40
      languages: Java
    kafka-consumer:
      id: 41
      languages: Java
    mongodb-driver:
      id: 42
      languages: Java
    SOFARPC:
      id: 43
      languages: Java
    ActiveMQ:
      id: 44
      languages: Java
    activemq-producer:
      id: 45
      languages: Java
    activemq-consumer:
      id: 46
      languages: Java
    Elasticsearch:
      id: 47
      languages: Java
    transport-client:
      id: 48
      languages: Java
    http:
      id: 49
      languages: Java,C#,Node.js
    rpc:
      id: 50
      languages: Java,C#,Node.js
    RabbitMQ:
      id: 51
      languages: Java
    rabbitmq-producer:
      id: 52
      languages: Java
    rabbitmq-consumer:
      id: 53
      languages: Java
    Canal:
      id: 54
      languages: Java
    Gson:
      id: 55
      languages: Java
    Redisson:
      id: 56
      languages: Java
    AspNetCore:
      id: 3001
      languages: C#
    EntityFrameworkCore:
      id: 3002
      languages: C#
    SqlClient:
      id: 3003
      languages: C#
    CAP:
      id: 3004
      languages: C#
    StackExchange.Redis:
      id: 3005
      languages: C#
    SqlServer:
      id: 3006
      languages: C#
    Npgsql:
      id: 3007
      languages: C#
    MySqlConnector:
      id: 3008
      languages: C#
    EntityFrameworkCore.InMemory:
      id: 3009
      languages: C#
    EntityFrameworkCore.SqlServer:
      id: 3010
      languages: C#
    EntityFrameworkCore.Sqlite:
      id: 3011
      languages: C#
    Pomelo.EntityFrameworkCore.MySql:
      id: 3012
      languages: C#
    Npgsql.EntityFrameworkCore.PostgreSQL:
      id: 3013
      languages: C#
    InMemoryDatabase:
      id: 3014
      languages: C#
    AspNet:
      id: 3015
      languages: C#
    # NoeJS components
    # [4000, 5000) for Node.js agent
    HttpServer:
      id: 4001
      languages: Node.js
    express:
      id: 4002
      languages: Node.js
    Egg:
      id: 4003
      languages: Node.js
    Koa:
      id: 4004
      languages: Node.js
    # Component Server mapping defines the server display names of some components
    # e.g.
    # Jedis is a client library in Java for Redis server
    Component-Server-Mappings:
      mongodb-driver: MongoDB
      rocketMQ-producer: RocketMQ
      rocketMQ-consumer: RocketMQ
      kafka-producer: Kafka
      kafka-consumer: Kafka
      activemq-producer: ActiveMQ
      activemq-consumer: ActiveMQ
      rabbitmq-producer: RabbitMQ
      rabbitmq-consumer: RabbitMQ
      postgresql-jdbc-driver: PostgreSQL
      Xmemcached: Memcached
      Spymemcached: Memcached
      h2-jdbc-driver: H2
      mysql-connector-java: Mysql
      Jedis: Redis
      StackExchange.Redis: Redis
      Redisson: Redis
      SqlClient: SqlServer
      Npgsql: PostgreSQL
      MySqlConnector: Mysql
      EntityFrameworkCore.InMemory: InMemoryDatabase
      EntityFrameworkCore.SqlServer: SqlServer
      EntityFrameworkCore.Sqlite: SQLite
      Pomelo.EntityFrameworkCore.MySql: Mysql
      Npgsql.EntityFrameworkCore.PostgreSQL: PostgreSQL
      transport-client: Elasticsearch
[root@k8s01 oap]# kubectl apply -f cm.yaml 
configmap/oap-config created

[root@k8s01 oap]# cat oap.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oap
  namespace: skywalking
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oap
  template:
    metadata:
      labels:
        app: oap
        release: skywalking
    spec:
      imagePullSecrets:
        - name: harborsecret  
      serviceAccountName: skywalking-oap
      initContainers:
      - name: sidecar-init
        image: 10.133.193.241/skywalking/agent-sidecar:6.1.0 # 容器镜像,包含静态资源文件
        command: ["cp", "-r", "/data/agent", "/sidecar"]
        volumeMounts:
        - name: sidecar
          mountPath: /sidecar
      containers:
      - name: oap
        image: 10.133.193.241/skywalking/oap:6.1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 11800
          name: grpc
        - containerPort: 12800
          name: rest
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 2Gi
        env:
        - name: JAVA_OPTS
          value: -Xms256M -Xmx512M
        #从volume载入配置 否则由于挂载config时 会与镜像内路径冲突 报错read only filesystem 
        - name: SW_L0AD_CONFIG_FILE_FROM_VOLUME
          value: "true"
        # - name: SW_STORAGE
        #   value: elasticsearch
        # - name: SW_STORAGE_ES_CLUSTER_NODES
        #   value: "192.168.78.136:9200"
        # - name: SW_CLUSTER
        #   value: kubernetes
        # - name: SW_CLUSTER_K8S_NAMESPACE
        #   value: skywalking
        # - name: SW_SERVICE_MESH_OFFSET_MAX_FILE_SIZE
        #   value: "200"
        # - name: SW_RECEIVER_BUFFER_DATA_MAX_FILE_SIZE
        #   value: "800"
        # - name: SW_RECEIVER_BUFFER_FILE_CLEAN_WHEN_RESTART
        #   value: "true"
        - name: SKYWALKING_oap_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        volumeMounts:
        - name: sidecar
          mountPath: /sidecar
        - name: config
          mountPath: /skywalking/config
      volumes:
      - name: sidecar
        emptyDir: {}
      - name: config
        configMap:
          name: oap-config
---
apiVersion: v1
kind: Service
metadata:
  name: oap
  namespace: skywalking
  labels:
    service: oap
spec:
  ports:
  - port: 12800
    name: rest
  - port: 11800
    name: grpc
  - port: 1234
    name: page
  selector:
    app: oap

#部署oap        
[root@k8s01 oap]# kubectl apply -f oap.yaml 
deployment.apps/oap created
service/oap created
 [root@k8s01 oap]# kubectl get -n skywalking po
NAME                  READY   STATUS    RESTARTS   AGE
elasticsearch-0       1/1     Running   0          24m
oap-dbb4cb494-jgg8f   1/1     Running   0          36s  

部署ui

[root@k8s01 skywalking]# cat ui.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ui-deployment
  namespace: skywalking
  labels:
    app: ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ui
  template:
    metadata:
      labels:
        app: ui
    spec:
      imagePullSecrets:
        - name: harborsecret
      containers:
      - name: ui
        image: 10.133.193.241/skywalking/ui:6.1.0
        ports:
        - containerPort: 8080
          name: page
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 2Gi
        env:
        - name: SW_OAP_ADDRESS
          value: oap:12800
---
apiVersion: v1
kind: Service
metadata:
  name: ui
  namespace: skywalking
  labels:
    service: ui
spec:
  ports:
  - port: 8080
    name: page
    nodePort: 31234
  type: NodePort
  selector:
    app: ui
    
#部署ui   
[root@k8s01 skywalking]# kubectl apply -f ui.yaml 
deployment.apps/ui-deployment created
service/ui created
[root@k8s01 skywalking]# kubectl get -n skywalking po
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-0                  1/1     Running   0          30m
oap-dbb4cb494-jgg8f              1/1     Running   0          6m28s
ui-deployment-64ff5796cd-7jh8g   1/1     Running   0          85s

访问skywalking

本文中使用了nodePort方式访问,当然也可以使用ingress域名访问
默认登录账号密码:admin/admin

sky2.png

sky3.png

测试

部署一个测试程序springcloud-demo

[root@k8s01 springcloud-demo]# cat apm-eureka.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apm-eureka
  namespace: skywalking
  labels:
    deploy: apm-eureka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apm-eureka
  template:
    metadata:
      name: apm-eureka
      labels:
        app: apm-eureka
    spec:
      imagePullSecrets:
        - name: harborsecret
      initContainers:
        - name: sidecar
          image: 10.133.193.241/skywalking/agent-sidecar:6.1.0 # 容器镜像,包含静态资源文件
          imagePullPolicy: IfNotPresent
          command: ["cp", "-r", "/data/agent", "/sidecar"]
          volumeMounts:
            - name: sidecar
              mountPath: /sidecar
      containers:
        - name: apm-eureka
          image: evanxuhe/apm-eureka:6.1.0
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_OPTS
              value: -javaagent:/sidecar/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: apm-eureka
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              value: oap:11800
          ports:
            - name: http
              containerPort: 8761
          volumeMounts:
            - name: sidecar
              mountPath: /sidecar
      volumes:
        - name: sidecar  #共享agent文件夹
          emptyDir: {}
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: apm-eureka
  namespace: skywalking
  labels:
    service: apm-eureka
spec:
  type: NodePort
  selector:
    app: apm-eureka
  ports:
    - port: 8761
      name: register
      targetPort: 8761
      nodePort: 30761
    - port: 8080
      name: index
      targetPort: 8080
      nodePort: 30081
[root@k8s01 springcloud-demo]# kubectl apply -f apm-eureka.yaml 
deployment.apps/apm-eureka created
service/apm-eureka created
[root@k8s01 springcloud-demo]# kubectl get -n skywalking po
NAME                             READY   STATUS    RESTARTS   AGE
apm-eureka-845cb789c8-9qt8b      1/1     Running   0          53s
elasticsearch-0                  1/1     Running   0          57m
oap-dbb4cb494-jgg8f              1/1     Running   0          33m
ui-deployment-64ff5796cd-7jh8g   1/1     Running   0          28m

[root@k8s01 springcloud-demo]# cat apm-item.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apm-item
  namespace: skywalking
  labels:
    app: apm-item
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apm-item
  template:
    metadata:
      name: apm-item
      labels:
        app: apm-item
    spec:
      imagePullSecrets:
        - name: harborsecret   
      initContainers:
        - name: sidecar
          image: 10.133.193.241/skywalking/agent-sidecar:6.1.0 # 容器镜像,包含静态资源文件
          imagePullPolicy: IfNotPresent
          command: ["cp", "-r", "/data/agent", "/sidecar"]
          volumeMounts:
            - name: sidecar
              mountPath: /sidecar
      containers:
        - name: apm-item
          image: evanxuhe/apm-item:6.1.0
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_OPTS
              value: -javaagent:/sidecar/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: apm-item
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              value: oap:11800
            - name: EUREKA_HOSTNAME
              value: apm-eureka
            - name: EUREKA_PORT
              value: '8761'
          ports:
            - name: http
              containerPort: 8082
          volumeMounts:
            - name: sidecar
              mountPath: /sidecar
      volumes:
        - name: sidecar  #共享agent文件夹
          emptyDir: {}
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: apm-item
  namespace: skywalking
  labels:
    service: apm-item
spec:
  type: NodePort
  selector:
    app: apm-item
  ports:
    - port: 8082
      name: item
      targetPort: 8082
      nodePort: 30082
[root@k8s01 springcloud-demo]# kubectl apply -f apm-item.yaml 
deployment.apps/apm-item unchanged
service/apm-item unchanged   
[root@k8s01 springcloud-demo]# kubectl get -n skywalking po
NAME                             READY   STATUS    RESTARTS   AGE
apm-eureka-845cb789c8-9qt8b      1/1     Running   0          53s
apm-item-597b5d49cd-qpnn5        1/1     Running   0          49s
elasticsearch-0                  1/1     Running   0          57m
oap-dbb4cb494-jgg8f              1/1     Running   0          33m
ui-deployment-64ff5796cd-7jh8g   1/1     Running   0          28m

访问http://nodeip:30082/item 会发现UI拓扑图多了两个服务

sky4.png

sky5.png

文章作者: 鲜花的主人
版权声明: 本站所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 爱吃可爱多
链路追踪 Kubernetes Kubernetes 链路追踪
喜欢就支持一下吧
打赏
微信 微信
支付宝 支付宝