Hello MQ

外观

Apache RocketMQ 运维与观测 ​

本页结论：看懂两个核心信号（消费堆积 Consume Diff、重试/DLQ 深度）就能覆盖大多数 RocketMQ 日常问题；本仓库 lab 用 mqadmin consumerProgress 的 Consume Diff 做 Broker 侧断言。

管理接口：mqadmin ​

管理命令经 broker 容器内 mqadmin -n namesrv:9876 执行（本仓库拓扑声明都走它）：

命令用途
updateTopic -c <cluster> -t <topic> -a +message.type=...建/改 Topic（声明消息类型）
updateSubGroup -c <cluster> -g <group> [-r 次数] [-p 策略JSON]建/改消费组（重试次数、重试策略）
topicRoute -t <topic>查 Topic 路由（队列分布在哪些 Broker）
consumerProgress -g <group>查消费组进度与 Consume Diff（堆积）

本仓库 lab 框架在断言阶段调用（示例）：

bash
docker compose -p <project> exec broker \
  sh mqadmin -n namesrv:9876 consumerProgress -g orders-basic-group

核心指标 ​

指标含义异常信号
Consume Diff（堆积）队列最新消息位 − 消费组已消费位持续增长 = 消费跟不上（积压），见 背压与积压
重试队列深度%RETRY%<组> 中待重投的消息增长 = 消费失败率上升
DLQ 深度%DLQ%<组> 中耗尽重试的消息>0 = 有毒消息/逻辑缺陷需人工介入
发送 TPS / 失败率Broker 接收能力失败率升高 = 容量或网络异常
Broker 磁盘 / 保留CommitLog 占用逼近磁盘上限前预警（保留期清理）

补充观测点：

  • 端到端延迟：消息产生到消费的时差，比 Consume Diff 更贴近业务感受（fifo-delay 实验的 deliveryDelayMs 即此类）。
  • 事务回查次数：长时间未确认的 Half Message 触发回查，异常升高说明生产者二次确认缺失。

消费堆积（Consume Diff） ​

  • Consume Diff = 未消费消息总量，等价于 Kafka 的 lag。
  • 堆积上涨先查：消费者是否存活、是否卡在单条毒消息（FIFO 队列会因失败阻塞）、处理速度是否低于生产速度。
  • 注意：堆积消息仍占保留期内的日志空间；若保留期先于消费到期，慢消费组会真的错过消息。

Dashboard 简介 ​

RocketMQ Dashboard（原 rocketmq-console）提供 Web 界面：Topic/消费组管理、消费进度、消息查询（按 Key/时间）、Broker 状态。生产建议部署并限制访问来源；本仓库实验未启用，断言直接用 mqadmin。

追踪传播 ​

本仓库 Demo 把 traceId、eventType、aggregateId 写入消息属性（properties），消费端日志原样带出；一条消息在 producer/consumer 两端可用同一 traceId 关联。生产可接 OpenTelemetry：producer 注入属性，consumer 从属性恢复 context。

常见故障速查 ​

现象先查什么
消费停滞、Consume Diff 上涨消费者是否存活；FIFO 队列是否被失败消息阻塞
客户端报路由 40402Topic 刚建、Broker 尚未向 NameServer 注册路由（~30s 心跳）
消息「重复」ack 早于业务完成？发送端重试未配幂等去重？
消息「丢失」异步刷盘+异步复制下 Broker 崩溃；保留期是否先于消费删除
事务消息不投递生产者是否漏了二次确认；回查是否返回 ROLLBACK

安全基线（单节点实验版） ​

  • 本仓库实验不开认证，且仅 Proxy 端口映射到 127.0.0.1，NameServer/Broker 不出 Compose 网络——仅限学习用途。
  • 生产基线：ACL 鉴权 + TLS、按 Topic/Group 最小授权、管理端口与数据端口分离并限制网络访问。
  • 单容器三服务拓扑不等价生产集群；生产应分离角色、规划副本与磁盘（见 存储与高可用）。

cli-tools：纯自带 CLI 实验 ​

bin 目录共 36 项，mqadmin 即可完成收发闭环：clusterList 查状态 → updateTopic 建 ordersCli → sendMessage 连发 3 条全部 SEND_OK → consumeMessage 一次排空收齐 3 条 → topicStatus 复查 maxOffset 合计=3，全程不引入任何客户端 SDK。两处实测坑：Broker 约 30s 心跳注册路由，建 Topic 后需等路由可见再收发；consumeMessage -c 并非可靠条数上限，用默认 messageCount=128 即可。

rocketmq / dockerbash demos/rocketmq/docker/run.sh
assert.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "apache/rocketmq:5.5.0@sha256:7e8f6c9dbd9df742ed26ba69c00d4ad69e2f86a56f3ca7782ff8144dd0798132"
exitCode: 0
---
PASS binCount: 36
PASS brokerInCluster: 1
PASS topicCreated: 1
PASS sentCount: 3
PASS consumedCount: 3
PASS consumedUnique: 3
PASS maxOffsetSum: 3
RESULT: all assertions passed
bin-list.out.txt
[cli-tools] $ sh -c ls /home/rocketmq/rocketmq-5.5.0/bin
cachedog.sh
cleancache.sh
cleancache.v1.sh
controller
dledger
docker-entrypoint.sh
export.sh
mqadmin
mqadmin.cmd
mqbroker
mqbroker.cmd
mqbrokercontainer
mqbroker.numanode0
mqbroker.numanode1
mqbroker.numanode2
mqbroker.numanode3
mqcontroller
mqcontroller.cmd
mqnamesrv
mqnamesrv.cmd
mqproxy
mqproxy.cmd
mqshutdown
mqshutdown.cmd
os.sh
play.cmd
play.sh
README.md
runbroker.cmd
runbroker.sh
runserver.cmd
runserver.sh
setcache.sh
startfsrv.sh
tools.cmd
tools.sh
consume.out.txt
[cli-tools] $ mqadmin consumeMessage -n namesrv:9876 -t ordersCli
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=7] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC120003020113221655749578D80000 MessageExt [brokerName=hello-mq-broker, queueId=6, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498352857, bornHost=/172.18.0.3:58854, storeTimestamp=1787498352875, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F00000000000000ED, commitLogOffset=237, bodyCRC=63934924, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC120003020113221655749578D80000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 50], transactionId='null'}] BODY: order-cli-2
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=6] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=3] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC120003023F13221655749589930000 MessageExt [brokerName=hello-mq-broker, queueId=2, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498357140, bornHost=/172.18.0.3:58862, storeTimestamp=1787498357151, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F00000000000001DA, commitLogOffset=474, bodyCRC=1959305562, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC120003023F13221655749589930000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 51], transactionId='null'}] BODY: order-cli-3
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=2] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=5] print msg finished. status=NO_NEW_MSG, offset=0
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=4] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC12000301C213221655749568150000 MessageExt [brokerName=hello-mq-broker, queueId=1, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498348566, bornHost=/172.18.0.3:35074, storeTimestamp=1787498348608, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F0000000000000000, commitLogOffset=0, bodyCRC=449233014, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC12000301C213221655749568150000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 49], transactionId='null'}] BODY: order-cli-1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=1] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=0] print msg finished. status=NO_NEW_MSG, offset=0
create.out.txt
[cli-tools] $ mqadmin updateTopic -n namesrv:9876 -c DefaultCluster -t ordersCli
create topic to 172.18.0.3:10911 success.
TopicConfig [topicName=ordersCli, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false, attributes={}]
inventory.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "apache/rocketmq:5.5.0@sha256:7e8f6c9dbd9df742ed26ba69c00d4ad69e2f86a56f3ca7782ff8144dd0798132"
exitCode: 0
---
[cli-tools] $ sh -c ls /home/rocketmq/rocketmq-5.5.0/bin
cachedog.sh
cleancache.sh
cleancache.v1.sh
controller
dledger
docker-entrypoint.sh
export.sh
mqadmin
mqadmin.cmd
mqbroker
mqbroker.cmd
mqbrokercontainer
mqbroker.numanode0
mqbroker.numanode1
mqbroker.numanode2
mqbroker.numanode3
mqcontroller
mqcontroller.cmd
mqnamesrv
mqnamesrv.cmd
mqproxy
mqproxy.cmd
mqshutdown
mqshutdown.cmd
os.sh
play.cmd
play.sh
README.md
runbroker.cmd
runbroker.sh
runserver.cmd
runserver.sh
setcache.sh
startfsrv.sh
tools.cmd
tools.sh
produce.out.txt
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-1 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   1     SEND_OK                 AC12000301C213221655749568150000
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-2 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   6     SEND_OK                 AC120003020113221655749578D80000
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-3 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   2     SEND_OK                 AC120003023F13221655749589930000
session.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "apache/rocketmq:5.5.0@sha256:7e8f6c9dbd9df742ed26ba69c00d4ad69e2f86a56f3ca7782ff8144dd0798132"
exitCode: 0
---
## status
[cli-tools] $ mqadmin clusterList -n namesrv:9876
#Cluster Name           #Broker Name            #BID  #Addr                  #Version              #InTPS(LOAD)                   #OutTPS(LOAD)  #Timer(Progress)        #PCWait(ms)  #Hour         #SPACE    #ACTIVATED
DefaultCluster          hello-mq-broker         0     172.18.0.3:10911       V5_5_0                 0.00(0,0ms)               0.00(0,0ms|0,0ms)  0-0(0.0w, 0.0, 0.0)               0  496527.32     0.4300          true

## create
[cli-tools] $ mqadmin updateTopic -n namesrv:9876 -c DefaultCluster -t ordersCli
create topic to 172.18.0.3:10911 success.
TopicConfig [topicName=ordersCli, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false, attributes={}]

## produce
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-1 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   1     SEND_OK                 AC12000301C213221655749568150000
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-2 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   6     SEND_OK                 AC120003020113221655749578D80000
[cli-tools] $ mqadmin sendMessage -n namesrv:9876 -t ordersCli -p order-cli-3 -c TagCli
#Broker Name                      #QID  #Send Result            #MsgId
hello-mq-broker                   2     SEND_OK                 AC120003023F13221655749589930000

## consume
[cli-tools] $ mqadmin consumeMessage -n namesrv:9876 -t ordersCli
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=7] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC120003020113221655749578D80000 MessageExt [brokerName=hello-mq-broker, queueId=6, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498352857, bornHost=/172.18.0.3:58854, storeTimestamp=1787498352875, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F00000000000000ED, commitLogOffset=237, bodyCRC=63934924, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC120003020113221655749578D80000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 50], transactionId='null'}] BODY: order-cli-2
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=6] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=3] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC120003023F13221655749589930000 MessageExt [brokerName=hello-mq-broker, queueId=2, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498357140, bornHost=/172.18.0.3:58862, storeTimestamp=1787498357151, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F00000000000001DA, commitLogOffset=474, bodyCRC=1959305562, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC120003023F13221655749589930000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 51], transactionId='null'}] BODY: order-cli-3
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=2] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=5] print msg finished. status=NO_NEW_MSG, offset=0
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=4] print msg finished. status=NO_NEW_MSG, offset=0
Consume ok
MSGID: AC12000301C213221655749568150000 MessageExt [brokerName=hello-mq-broker, queueId=1, storeSize=237, queueOffset=0, sysFlag=0, bornTimestamp=1787498348566, bornHost=/172.18.0.3:35074, storeTimestamp=1787498348608, storeHost=/172.18.0.3:10911, msgId=AC12000300002A9F0000000000000000, commitLogOffset=0, bodyCRC=449233014, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='ordersCli', flag=0, properties={MSG_REGION=DefaultRegion, UNIQ_KEY=AC12000301C213221655749568150000, CLUSTER=DefaultCluster, MIN_OFFSET=0, TAGS=TagCli, WAIT=true, TRACE_ON=true, MAX_OFFSET=1}, body=[111, 114, 100, 101, 114, 45, 99, 108, 105, 45, 49], transactionId='null'}] BODY: order-cli-1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=1] print msg finished. status=NO_NEW_MSG, offset=1
MessageQueue [topic=ordersCli, brokerName=hello-mq-broker, queueId=0] print msg finished. status=NO_NEW_MSG, offset=0

## verify
[cli-tools] $ mqadmin topicStatus -n namesrv:9876 -t ordersCli
#Broker Name                      #QID  #Min Offset           #Max Offset             #Last Updated
hello-mq-broker                   0     0                     0                       
hello-mq-broker                   1     0                     1                       2026-08-23 15:19:08,608
hello-mq-broker                   2     0                     1                       2026-08-23 15:19:17,151
hello-mq-broker                   3     0                     0                       
hello-mq-broker                   4     0                     0                       
hello-mq-broker                   5     0                     0                       
hello-mq-broker                   6     0                     1                       2026-08-23 15:19:12,875
hello-mq-broker                   7     0                     0                       

Topic Put TPS: 0.0
status.out.txt
[cli-tools] $ mqadmin clusterList -n namesrv:9876
#Cluster Name           #Broker Name            #BID  #Addr                  #Version              #InTPS(LOAD)                   #OutTPS(LOAD)  #Timer(Progress)        #PCWait(ms)  #Hour         #SPACE    #ACTIVATED
DefaultCluster          hello-mq-broker         0     172.18.0.3:10911       V5_5_0                 0.00(0,0ms)               0.00(0,0ms|0,0ms)  0-0(0.0w, 0.0, 0.0)               0  496527.32     0.4300          true
verify.out.txt
[cli-tools] $ mqadmin topicStatus -n namesrv:9876 -t ordersCli
#Broker Name                      #QID  #Min Offset           #Max Offset             #Last Updated
hello-mq-broker                   0     0                     0                       
hello-mq-broker                   1     0                     1                       2026-08-23 15:19:08,608
hello-mq-broker                   2     0                     1                       2026-08-23 15:19:17,151
hello-mq-broker                   3     0                     0                       
hello-mq-broker                   4     0                     0                       
hello-mq-broker                   5     0                     0                       
hello-mq-broker                   6     0                     1                       2026-08-23 15:19:12,875
hello-mq-broker                   7     0                     0                       

Topic Put TPS: 0.0

官方资料 ​

以统一实验验证消息系统语义边界

MIT License