系统环境

• rocketmq版本： 4.8.0

• k8s版本： 1.20.0

RocketMQ简介

RocketMQ主要有四大组成部分：NameServer、Broker、Producer、Consumer

Nameserver

NameServer 可以说是 Broker 的注册中心，Broker 在启动的时候，会根据配置信息向所有的 NameServer 进行注册，NameServer 会和每次前来注册的 Broker 保持长连接，并每 30s 检查 Broker 是否还存活，对于宕机的 Broker，NameServer 会将其从列表中剔除。当生产者需要向 Broker 发送消息的时候，就会先从 NameServer 里面获取 Broker 的地址列表，然后负载均衡，选择一台消息服务器进行发送。

RocketMQ的部署方式

• 2m-noslave： 多Master模式，无Slave [双主模式]

• 2m-2s-sync： 多Master多Slave模式，同步双写 [双主双从+同步模式]

• 2m-2s-async：多Master多Slave模式，异步复制 [双主双从+异步模式]

RocketMQ发送消息方式

• 同步发送: 指消息发送方发出数据后会在收到接收方发回响应之后才发下一个数据包

• 异步发送: 指发送方发出数据后，不等接收方发回响应，接着发送下个数据包, 异步方式也需要Broker返回确认信息

• 单向发送: 指只负责发送消息而不等待服务器回应且没有回调函数触发

RocketMQ三种消息发送模式使用场景

具体使用哪种模式，这主要是看应用场景

• 同步发送：主要运用在比较重要一点消息传递/通知等业务

• 异步发送：通常用于对发送消息响应时间要求更高/更快的场景

• 单向发送：适用于某些耗时非常短，但对可靠性要求并不高的场景，例如日志收集，只发送消息不等待服务器响应，只发送请求不等待应答。此方式发送消息的过程耗时非常短，一般在微秒级别

RocketMQ端口

• rocketmq 默认端口：9876（即nameserver端口）

• 非vip通道端口：10911

• vip通道端口：10909

• 10909是VIP通道对应的端口，在JAVA中的消费者对象或者是生产者对象中关闭VIP通道即可无需开放10909端口

本文部署的是RocketMQ的双主双从+同步模式

• broker-a-master 主1

• broker-b-master 主2

• broker-a-slave 从1

• broker-a-slave 从2

• NameServer（注册中心）

• rocketmq-externals（可视化web界面）

部署RocketMQ集群

制作镜像

[root@tools rocketmq]# wget https://repo.huaweicloud.com/apache/rocketmq/4.8.0/rocketmq-all-4.8.0-bin-release.zip
[root@tools rocketmq]# unzip rocketmq-all-4.8.0-bin-release.zip
[root@tools rocketmq]# cd rocketmq-all-4.8.0-bin-release/bin/
[root@tools bin]# cd rocketmq-all-4.8.0-bin-release/bin/
#将runbroker.sh脚本文件里的内存设置调整小点
[root@tools bin]# vim runbroker.sh
...
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"
...
[root@tools rocketmq]# tar -zcvf rocketmq-all-4.8.0-bin-release.tar.gz rocketmq-all-4.8.0-bin-release
[root@tools rocketmq]# rm -fr rocketmq-all-4.8.0-bin-release

Dockerfile文件内容如下：（这里使用的基础镜像里面装了jdk1.8）

[root@tools rocketmq]# cat Dockerfile 
FROM xxx/base/centos7_jdk:latset 
ENV LANG en_US.UTF-8
ADD rocketmq-all-4.8.0-bin-release.tar.gz /usr/local/
RUN mv /usr/local/rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-4.8.0 \
&& mkdir -p /data/rocketmq/store
ENV PATH=/usr/local/rocketmq-4.8.0/bin:$JAVA_HOME/bin:$PATH
CMD ["/bin/bash"]

制作镜像并上传到仓库

[root@tools rocketmq]# docker build -t xxx/tools/rocketmq-4.8.0 .
[root@tools rocketmq]# docker push xxx/tools/rocketmq-4.8.0

RocketMQ配置文件创建

[root@k8s01 rocketmq]# cat cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: rocketmq-config
  namespace: tools-env
data:
  broker-a-slave.properties: |
    brokerClusterName=rocketmq-cluster
    brokerName=broker-a
    brokerId=1
    namesrvAddr=rocketmq.tools-env.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=20911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    diskMaxUsedSpaceRatio=88
    storePathRootDir=/data/rocketmq/store
    maxMessageSize=65536
    brokerRole=SLAVE
    flushDiskType=SYNC_FLUSH
  broker-a-master.properties: |
    brokerClusterName=rocketmq-cluster
    brokerName=broker-a
    brokerId=0
    namesrvAddr=rocketmq.tools-env.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=20911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    diskMaxUsedSpaceRatio=88
    storePathRootDir=/data/rocketmq/store
    maxMessageSize=65536
    brokerRole=MASTER
  broker-b-slave.properties: |
    brokerClusterName=rocketmq-cluster
    brokerName=broker-b
    brokerId=1
    namesrvAddr=rocketmq.tools-env.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=20911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    diskMaxUsedSpaceRatio=88
    storePathRootDir=/data/rocketmq/store
    maxMessageSize=65536
    brokerRole=SLAVE
    flushDiskType=SYNC_FLUSH
  broker-b-master.properties: |
    brokerClusterName=rocketmq-cluster
    brokerName=broker-b
    brokerId=0
    namesrvAddr=rocketmq.tools-env.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=20911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    diskMaxUsedSpaceRatio=88
    storePathRootDir=/data/rocketmq/store
    maxMessageSize=65536
    brokerRole=MASTER
    flushDiskType=SYNC_FLUSH

依据上面rocketmq的4个配置文件创建configmap

[root@k8s01 rocketmq]# kubectl apply -f cm.yaml 
configmap/rocketmq-config created

nameserver安装

[root@k8s01 rocketmq]# vim nameserver.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-nameserver
  name: rocketmq
  namespace: tools-env
spec:
  ports:
  - port: 9876
    targetPort: 9876
    name: rocketmq-nameserver
  selector:
    app: rocketmq-nameserver
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-nameserver
spec:
  serviceName: rocketmq-nameserver
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-nameserver
  template:
    metadata:
     labels:
       app: rocketmq-nameserver
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - rocketmq-nameserver
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: rocketmq-nameserver
        image: xxx/tools/rocketmq-4.8.0
        imagePullPolicy: IfNotPresent
        command: ["sh","/usr/local/rocketmq-4.8.0/bin/mqnamesrv"]
        ports:
        - containerPort: 9876
          protocol: TCP
        env:
        - name: JAVA_OPT
          value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
          #value: "-XX:MaxRAMPercentage=80.0"
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 9876
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
          
[root@k8s01 rocketmq]# kubectl apply -f nameserver.yaml 
service/rocketmq created
statefulset.apps/rocketmq-nameserver created      

[root@k8s01 rocketmq]# kubectl get -n tools-env po |grep rocket
rocketmq-nameserver-0            1/1     Running   0          25s

broker安装

broker-a-master.yaml

[root@k8s01 rocketmq]# vim broker-a-master.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a-master
  name: broker-a-master
  namespace: tools-env
spec:
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
  selector:
    app: broker-a-master
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a-master
  namespace: tools-env
spec:
  serviceName: broker-a-master
  replicas: 1
  selector:
    matchLabels:
      app: broker-a-master
  template:
    metadata:
     labels:
       app: broker-a-master
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-a-master
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: harborsecret
      containers:
      - name: broker-a-master
        image: xxx.cn/tools/rocketmq-4.8.0
        imagePullPolicy: IfNotPresent
        command: ["sh","-c","mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-a-master.properties"]
        env:
        - name: JAVA_OPT
          value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
          #value: "-XX:MaxRAMPercentage=80.0"
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a-master.properties
            subPath: broker-a-master.properties
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

[root@k8s01 rocketmq]# kubectl apply -f broker-a-master.yaml 
service/broker-a-master unchanged
statefulset.apps/broker-a-master created

broker-a-slave.yaml

[root@k8s01 rocketmq]# vim broker-a-slave.yaml            
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a-slave
  name: broker-a-slave
  namespace: tools-env
spec:
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
  selector:
    app: broker-a-slave
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a-slave
  namespace: tools-env
spec:
  serviceName: broker-a-slave
  replicas: 1
  selector:
    matchLabels:
      app: broker-a-slave
  template:
    metadata:
     labels:
       app: broker-a-slave
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-a-slave
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: harborsecret
      containers:
      - name: broker-a-slave
        image: xxx.cn/tools/rocketmq-4.8.0
        imagePullPolicy: IfNotPresent
        command: ["sh","-c","mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-a-slave.properties"]
        env:
        - name: JAVA_OPT
          value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
          #value: "-XX:MaxRAMPercentage=80.0"
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a-slave.properties
            subPath: broker-a-slave.properties
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
          items:
          - key: broker-a-slave.properties
            path: broker-a-slave.properties
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

[root@k8s01 rocketmq]# kubectl apply -f  broker-a-slave.yaml 
service/broker-a-slave created
statefulset.apps/broker-a-slave created

broker-b-master.yaml

[root@k8s01 rocketmq]# vim broker-b-master.yaml             
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b-master
  name: broker-b-master
  namespace: tools-env
spec:
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
  selector:
    app: broker-b-master
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b-master
  namespace: tools-env
spec:
  serviceName: broker-b-master
  replicas: 1
  selector:
    matchLabels:
      app: broker-b-master
  template:
    metadata:
     labels:
       app: broker-b-master
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-b-master
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: harborsecret
      containers:
      - name: broker-b-master
        image: xxx/tools/rocketmq-4.8.0
        imagePullPolicy: IfNotPresent
        command: ["sh","-c","mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-b-master.properties"]
        env:
        - name: JAVA_OPT
          value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
          #value: "-XX:MaxRAMPercentage=80.0"
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b-master.properties
            subPath: broker-b-master.properties
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

[root@k8s01 rocketmq]# kubectl apply -f broker-b-master.yaml    
service/broker-b-master created
statefulset.apps/broker-b-master created

broker-b-slave.yaml

[root@k8s01 rocketmq]# vim broker-b-slave.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b-slave
  name: broker-b-slave
  namespace: tools-env
spec:
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
  selector:
    app: broker-b-slave
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b-slave
  namespace: tools-env
spec:
  serviceName: broker-b-slave
  replicas: 1
  selector:
    matchLabels:
      app: broker-b-slave
  template:
    metadata:
     labels:
       app: broker-b-slave
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-b-slave
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: harborsecret
      containers:
      - name: broker-b-slave
        image: xxx/tools/rocketmq-4.8.0
        imagePullPolicy: IfNotPresent
        command: ["sh","-c","mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-b-slave.properties"]
        env:
        - name: JAVA_OPT
          value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
          #value: "-XX:MaxRAMPercentage=80.0"
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b-slave.properties
            subPath: broker-b-slave.properties
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
          items:
          - key: broker-b-slave.properties
            path: broker-b-slave.properties
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

[root@k8s01 rocketmq]# kubectl apply -f broker-b-slave.yaml            
service/broker-b-slave created
statefulset.apps/broker-b-slave created

部署console

[root@k8s01 rocketmq]# vim console.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rocketmq-console
  name: rocketmq-console
  namespace: tools-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-console
  template:
    metadata:
      labels:
        app: rocketmq-console
    spec:
      containers:
      - image: styletang/rocketmq-console-ng
        name: rocketmq-console
        imagePullPolicy: IfNotPresent
        env: 
        - name: JAVA_OPTS
          value: "-Drocketmq.namesrv.addr=rocketmq.tools-env.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-console
  name: rocketmq-console
  namespace: tools-env
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    nodePort: 31808
  type: NodePort
  selector:
    app: rocketmq-console

[root@k8s01 rocketmq]# kubectl apply -f console.yaml 
deployment.apps/rocketmq-console created
service/rocketmq-console created

查看部署pod

[root@k8s01 rocketmq]# kubectl get -n tools-env po
NAME                                READY   STATUS    RESTARTS   AGE
broker-a-master-0                   1/1     Running   0          22m
broker-a-slave-0                    1/1     Running   0          22m
broker-b-master-0                   1/1     Running   0          22m
broker-b-slave-0                    1/1     Running   0          22m

rocketmq-console-6f649694b5-zz5td   1/1     Running   0          57s
rocketmq-nameserver-0               1/1     Running   0          116m
[root@k8s01 rocketmq]# kubectl get -n tools-env svc
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
broker-a-master        ClusterIP   10.109.29.199    <none>        20911/TCP                       22m
broker-a-slave         ClusterIP   10.104.144.48    <none>        20911/TCP                       22m
broker-b-master        ClusterIP   10.103.254.189   <none>        20911/TCP                       22m
broker-b-slave         ClusterIP   10.97.52.151     <none>        20911/TCP                       22m

rocketmq               ClusterIP   10.100.227.69    <none>        9876/TCP                        117m
rocketmq-console       NodePort    10.104.254.68    <none>        8080:31808/TCP                  82s

访问Web界面

浏览器输出IP:31808