部署 3 节点 controller ，broker 状态 alive 显示为 false #7877

drivebyer · 2024-02-29T08:30:59Z

drivebyer
Feb 29, 2024

我用 operator 部署了一个 5.1.4 版本的 3 个 controller 2 个 broker 1 个 nameserver 的集群，等待一段时间后，在 nameserver 里执行命令：

[root@master0 ~]# kubectl -n mcamel-system exec -it name-service-0 -- ./mqadmin clusterList -n 127.0.0.1:9876
#Cluster Name           #Broker Name            #BID  #Addr                  #Version              #InTPS(LOAD)     #OutTPS(LOAD)  #Timer(Progress)        #PCWait(ms)  #Hour         #SPACE    #ACTIVATED
broker                  broker-0                0     192.168.137.126:10911  V5_1_4                 0.00(0,0ms)       0.00(0,0ms)  0-0(0.0w, 0.0, 0.0)               0  474775.65     0.6800          true
broker                  broker-0                2     192.168.84.199:10911   V5_1_4                 0.00(0,0ms)       0.00(0,0ms)  2-0(0.0w, 0.0, 0.0)               0  474775.65     0.6500         false

看起来是正常的。

在 controller 里执行命令：

[root@master0 ~]# kubectl -n mcamel-system exec -it controller-1 -- ./mqadmin getSyncStateSet -a 127.0.0.1:9878 -c broker -b broker-0

#brokerName	broker-0
#MasterBrokerId	1
#MasterAddr	192.168.137.126:10911
#MasterEpoch	1
#SyncStateSetEpoch	1
#SyncStateSetNums	1

InSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=1, brokerAddress='192.168.137.126:10911', alive=true}

NotInSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=2, brokerAddress='192.168.84.199:10911', alive=false}

显示其中一个 broker alive 为 false。

并且在 192.168.137.126:10911 中发现了下面的错误日志：

2024-02-29 15:50:26 ERROR AutoSwitchHAService_Executor_1 - Error happen when change SyncStateSet, broker:broker-0, masterAddress:192.168.137.126:10911, masterEpoch:1, oldSyncStateSet:[1], newSyncStateSet:[1, 2], syncStateSetEpoch:1
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2006  DESC: Rejecting alter syncStateSet request because the replicas {2} don't alive
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
	at org.apache.rocketmq.broker.out.BrokerOuterAPI.alterSyncStateSet(BrokerOuterAPI.java:1215)
	at org.apache.rocketmq.broker.controller.ReplicasManager.doReportSyncStateSetChanged(ReplicasManager.java:761)
	at org.apache.rocketmq.store.ha.autoswitch.AutoSwitchHAService.lambda$null$0(AutoSwitchHAService.java:263)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.apache.rocketmq.store.ha.autoswitch.AutoSwitchHAService.lambda$notifySyncStateSetChanged$1(AutoSwitchHAService.java:263)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2024-02-29 15:50:30 INFO ReplicasManager_ScheduledService_1 - Update controller leader address to controller-1.controller-svc-headless:9878

ps: 如果使用单节点的 controller 就不会有这个问题

RongtongJin · 2024-03-01T03:06:29Z

RongtongJin
Mar 1, 2024
Collaborator

看着192.168.84.199没有像controller发心跳。。。但单节点controller不会有这个问题比较奇怪。。。可以看下192.168.84.199和controller的日志

5 replies

drivebyer Mar 1, 2024
Author

感谢回复。

刚才我又使用同样的步骤测试了一下，出现了不同的现象。

controller

[root@master0 ~]# kubectl -n mcamel-system exec -it controller-1 -- ./mqadmin getSyncStateSet -a 127.0.0.1:9878 -c broker -b broker-0

#brokerName	broker-0
#MasterBrokerId	null
#MasterAddr	null
#MasterEpoch	2
#SyncStateSetEpoch	1
#SyncStateSetNums	1

InSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=1, brokerAddress='192.168.137.114:10911', alive=false}

NotInSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=2, brokerAddress='192.168.84.226:10911', alive=true}

nameserver:

[root@master0 ~]# kubectl -n mcamel-system exec -it name-service-0 -- ./mqadmin clusterList -n 127.0.0.1:9876
#Cluster Name           #Broker Name            #BID  #Addr                  #Version              #InTPS(LOAD)     #OutTPS(LOAD)  #Timer(Progress)        #PCWait(ms)  #Hour         #SPACE    #ACTIVATED
broker                  broker-0                0     192.168.137.114:10911  V5_1_4                 0.00(0,0ms)       0.00(0,0ms)  0-0(0.0w, 0.0, 0.0)               0  474795.68     0.6800          true

broker(192.168.84.226) 的日志：

2024-03-01 11:41:17 WARN ReplicasManager_ExecutorService_1 - Failed to start replicasManager, retry times:22, current state:REGISTER_TO_CONTROLLER_DONE, try it again
2024-03-01 11:41:17 ERROR ReplicasManager_ExecutorService_1 - Failed to try elect
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2012  DESC: Failed to elect a new master
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
	at org.apache.rocketmq.broker.out.BrokerOuterAPI.brokerElect(BrokerOuterAPI.java:1237)
	at org.apache.rocketmq.broker.controller.ReplicasManager.brokerElect(ReplicasManager.java:378)
	at org.apache.rocketmq.broker.controller.ReplicasManager.startBasicService(ReplicasManager.java:200)
	at org.apache.rocketmq.broker.controller.ReplicasManager.lambda$start$0(ReplicasManager.java:156)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

broker (192.168.137.114) 的日志：

2024-03-01 11:44:49 INFO ReplicasManager_ScheduledService_1 - Update controller leader address to controller-1.controller-svc-headless:9878
2024-03-01 11:44:50 ERROR ReplicasManager_ScheduledService_2 - Failed to try elect
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2012  DESC: Failed to elect a new master
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
	at org.apache.rocketmq.broker.out.BrokerOuterAPI.brokerElect(BrokerOuterAPI.java:1237)
	at org.apache.rocketmq.broker.controller.ReplicasManager.brokerElect(ReplicasManager.java:378)
	at org.apache.rocketmq.broker.controller.ReplicasManager.lambda$schedulingSyncBrokerMetadata$3(ReplicasManager.java:663)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

caigy Mar 11, 2024
Collaborator

broker和controller间的网络正常吗？broker中controller相关的环境变量提供一下。

drivebyer Mar 11, 2024
Author

在两个 broker 中用 nc 命令测试到 controller 的端口都是通的：

bash-4.4# nc -zv 192.168.84.246 9878
192.168.84.246 (192.168.84.246:9878) open
bash-4.4# nc -zv 192.168.137.114 9878
192.168.137.114 (192.168.137.114:9878) open
bash-4.4# nc -zv 192.168.180.10 9878
192.168.180.10 (192.168.180.10:9878) open

bash-4.4# env | grep -i controller
CONTROLLER_SVC_PORT_9878_TCP=tcp://10.105.182.106:9878
enableControllerMode=true
CONTROLLER_SVC_SERVICE_PORT=9878
CONTROLLER_SVC_PORT_9878_TCP_ADDR=10.105.182.106
CONTROLLER_SVC_PORT=tcp://10.105.182.106:9878
CONTROLLER_SVC_PORT_9878_TCP_PORT=9878
CONTROLLER_SVC_SERVICE_HOST=10.105.182.106
controllerAddr=controller-svc:9878
CONTROLLER_SVC_PORT_9878_TCP_PROTO=tcp
CONTROLLER_SVC_SERVICE_PORT_CONTROLLER=9878

anguoming Jun 11, 2024

用operator部署5.2.0的rocketmq，当controller是3节点时也遇到了相同的问题，现象和日志报错跟前边描述的第二个现象相同

drivebyer Jun 11, 2024
Author

用operator部署5.2.0的rocketmq，当controller是3节点时也遇到了相同的问题，现象和日志报错跟前边描述的第二个现象相同

看来我不是一个人 :)

drivebyer · 2024-03-11T06:20:25Z

drivebyer
Mar 11, 2024
Author

BTW, 我用的 yaml 是：

apiVersion: v1
kind: ConfigMap
metadata:
  name: broker-config
data:
  # BROKER_MEM sets the broker JVM, if set to "" then Xms = Xmx = max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
  broker-common.conf: |
    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
    brokerRole=ASYNC_MASTER

---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Broker
metadata:
  # name of broker cluster
  name: broker
spec:
  clusterMode: CONTROLLER
  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
  size: 1
  # nameServers is the [ip:port] list of name service
  nameServers: ""
  # replicaPerGroup is the number of each broker cluster
  replicaPerGroup: 1
  # brokerImage is the customized docker image repo of the RocketMQ broker
  brokerImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-broker:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "2048Mi"
      cpu: "250m"
    limits:
      memory: "12288Mi"
      cpu: "500m"
  # allowRestart defines whether allow pod restart
  allowRestart: true
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: StorageClass
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/broker
  # scalePodName is [Broker name]-[broker group number]-master-0
  scalePodName: broker-0-0-0
  # env defines custom env, e.g. BROKER_MEM
  env:
    - name: BROKER_MEM
      valueFrom:
        configMapKeyRef:
          name: broker-config
          key: BROKER_MEM
  # volumes defines the broker.conf
  volumes:
    - name: broker-config
      configMap:
        name: broker-config
        items:
          - key: broker-common.conf
            path: broker-common.conf
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
---
apiVersion: rocketmq.apache.org/v1alpha1
kind: NameService
metadata:
  name: name-service
spec:
  # size is the the name service instance number of the name service cluster
  size: 1
  # nameServiceImage is the customized docker image repo of the RocketMQ name service
  nameServiceImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-nameserver:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # hostNetwork can be true or false
  hostNetwork: false
  #  Set DNS policy for the pod.
  #  Defaults to "ClusterFirst".
  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
  #  explicitly to 'ClusterFirstWithHostNet'.
  dnsPolicy: ClusterFirstWithHostNet
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1024Mi"
      cpu: "500m"
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: EmptyDir
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/nameserver
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Controller
metadata:
  # name of controller cluster
  name: controller
spec:
  # size is the number of controllers.
  size: 3
  # controllerImage is the customized docker image repo of the RocketMQ Controller
  controllerImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-controller:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: IfNotPresent
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "0.5Gi"
    limits:
      memory: "2Gi"
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: StorageClass
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/controller
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: controller-storage
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

部署 3 节点 controller ，broker 状态 alive 显示为 false #7877

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

部署 3 节点 controller ，broker 状态 alive 显示为 false #7877

drivebyer Feb 29, 2024

Replies: 2 comments · 5 replies

RongtongJin Mar 1, 2024 Collaborator

drivebyer Mar 1, 2024 Author

controller

nameserver:

broker(192.168.84.226) 的日志：

broker (192.168.137.114) 的日志：

caigy Mar 11, 2024 Collaborator

drivebyer Mar 11, 2024 Author

anguoming Jun 11, 2024

drivebyer Jun 11, 2024 Author

drivebyer Mar 11, 2024 Author

drivebyer
Feb 29, 2024

Replies: 2 comments 5 replies

RongtongJin
Mar 1, 2024
Collaborator

drivebyer Mar 1, 2024
Author

caigy Mar 11, 2024
Collaborator

drivebyer Mar 11, 2024
Author

drivebyer Jun 11, 2024
Author

drivebyer
Mar 11, 2024
Author