跳转到正文

NATS 运维与观测 ​

本页结论:NATS 的可观测性入口是 HTTP 监控端点(/varz、/connz、/jsz、/healthz)与官方 CLI(nats);JetStream 的积压看 Consumer 的 pending/ack_pending,容量看 Stream 存储与账户配额。

监控端点(-m 8222 开启) ​

端点内容
/healthz就绪检查(本仓库 compose healthcheck 使用)
/varz服务器全局:连接数、入/出消息速率、内存、slow consumers
/connz连接明细:每个客户端的订阅数、缓冲、收发计数
/subsz订阅明细
/jsz?consumers=1JetStream:Stream/Consumer 状态、存储占用、副本信息
/accountz账户与配额(多租户部署时)

本仓库 compose 已将 8222 仅绑定 127.0.0.1。

官方 CLI 速查 ​

bash
nats stream ls                          # Stream 列表与消息数
nats stream info ORDERS                 # 消息数、字节数、首末序列
nats consumer ls ORDERS                 # Consumer 列表
nats consumer info ORDERS orders-first  # 位点、pending、ack_pending、重投统计
nats stream get ORDERS 3 --raw          # 按序列号读取单条(回放排查)
nats pub orders.events '...'            # 手工发布
nats sub 'orders.>'                     # 手工订阅(Core 层)

积压定位决策树 ​

text
消费延迟增大
├── Core 层:/varz slow_consumers 增长
│   └── 订阅者处理不过来被断开 ⇒ 加 Queue Group 成员 / 迁移到 JetStream
├── JetStream:consumer info 的 Num Pending 持续增长
│   ├── ack_pending 高:投递了没确认
│   │   └── AckWait 太短导致重复重投,或业务处理变慢
│   └── ack_pending 低:生产速率 > 消费速率
│       └── 增加共享该 Consumer 的客户端数;评估 Stream 吞吐上限
└── /jsz 存储接近账户 max_bytes
    └── 调整保留策略/限额,或扩容存储

关键指标映射 ​

指标类别NATS 对应
ProducerPublishAck 延迟与错误率(客户端埋点);/varz in_msgs 速率
Broker/varz:connections、in/out_msgs、mem;/jsz:storage、reserved
Consumerconsumer info:delivered、ack_pending、num_redelivered
BacklogConsumer 的 num_pending(未投递)+ ack_pending(未确认)
DLQ 等价物自建 DLQ Stream 的消息数与最老序列年龄
Business幂等表 duplicate_skipped 计数(应用侧埋点)

安全基线 ​

  • 认证:用户名密码(开发)、Token、NKey/签名挑战(推荐)、JWT(多租户运营)。
  • 授权:按用户限制可发布/订阅的 Subject 集合;Account 层做租户隔离与导入导出。
  • TLS:客户端-服务器与路由(cluster)链路均可启用;生产默认开启。
  • 本仓库实验未开启认证(localhost 单节点);默认开放连接是开发配置,不是生产配置。

常见故障 ​

现象常见原因处置
发布报 no space left账户/Stream 存储配额满调整 max_bytes、清理或扩容
消息反复重投AckWait < 业务处理时长调大 AckWait 或拆小批处理
副本长时间降级R3 节点失联/磁盘慢检查节点健康与磁盘 IO;必要时 stream restore
客户端频繁重连网络抖动或服务器过载检查 /varz 负载与客户端重连缓冲配置

cli-tools:纯自带 CLI 实验 ​

nats:2.11.5 官方镜像是 distroless:文件系统仅含 /nats-server 单一二进制,无 shell、无自带收发 CLI(nats CLI 由 nats-io/natscli 独立发行)。因此本实验只做二进制盘点(--version/--help)与宿主机 8222 监控端点体检(/healthz、/varz、/connz、/subsz);CLI 缺口如实记录在 gap.out.txt(容器内 exec nats 实测 executable file not found)。收发需外部客户端,Java 版见 core-pubsub 实验。

nats / dockerbash demos/nats/docker/run.sh
assert.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "nats:2.11.5@sha256:47710a58674c152355730c69fc89d5a86369e3d6d1329cc91767650ce8e0dde6"
exitCode: 0
---
PASS binVersion: 1
PASS healthzOk: ok
PASS varzVersion: 2.11.5
PASS varzHasServerId: true
PASS varzJetstream: true
PASS connzConnections: 0
PASS subszParsed: true
PASS gapRecorded: 1
PASS gapEvidence: 1
RESULT: all assertions passed
bin-list.out.txt
# nats:2.11.5 官方镜像为 distroless:文件系统仅含 /nats-server 单一二进制(无 shell、无其他 CLI),故无 bin 目录可列
[cli-tools] $ compose exec nats /nats-server --version
nats-server: v2.11.5
[cli-tools] $ compose exec nats /nats-server --help

Usage: nats-server [options]

Server Options:
    -a, --addr, --net <host>         Bind to host address (default: 0.0.0.0)
    -p, --port <port>                Use port for clients (default: 4222)
    -n, --name
        --server_name <server_name>  Server name (default: auto)
    -P, --pid <file>                 File to store PID
    -m, --http_port <port>           Use port for http monitoring
    -ms,--https_port <port>          Use port for https monitoring
    -c, --config <file>              Configuration file
    -t                               Test configuration and exit
    -sl,--signal <signal>[=<pid>]    Send signal to nats-server process (ldm, stop, quit, term, reopen, reload)
                                     <pid> can be either a PID (e.g. 1) or the path to a PID file (e.g. /var/run/nats-server.pid)
        --client_advertise <string>  Client URL to advertise to other servers
        --ports_file_dir <dir>       Creates a ports file in the specified directory (<executable_name>_<pid>.ports).

Logging Options:
    -l, --log <file>                 File to redirect log output
    -T, --logtime                    Timestamp log entries (default: true)
    -s, --syslog                     Log to syslog or windows event log
    -r, --remote_syslog <addr>       Syslog server addr (udp://localhost:514)
    -D, --debug                      Enable debugging output
    -V, --trace                      Trace the raw protocol
    -VV                              Verbose trace (traces system account as well)
    -DV                              Debug and trace
    -DVV                             Debug and verbose trace (traces system account as well)
        --log_size_limit <limit>     Logfile size limit (default: auto)
        --max_traced_msg_len <len>   Maximum printable length for traced messages (default: unlimited)

JetStream Options:
    -js, --jetstream                 Enable JetStream functionality
    -sd, --store_dir <dir>           Set the storage directory

Authorization Options:
        --user <user>                User required for connections
        --pass <password>            Password required for connections
        --auth <token>               Authorization token required for connections

TLS Options:
        --tls                        Enable TLS, do not verify clients (default: false)
        --tlscert <file>             Server certificate file
        --tlskey <file>              Private key for server certificate
        --tlsverify                  Enable TLS, verify client certificates
        --tlscacert <file>           Client certificate CA for verification

Cluster Options:
        --routes <rurl-1, rurl-2>    Routes to solicit and connect
        --cluster <cluster-url>      Cluster URL for solicited routes
        --cluster_name <string>      Cluster Name, if not set one will be dynamically generated
        --no_advertise <bool>        Do not advertise known cluster information to clients
        --cluster_advertise <string> Cluster URL to advertise to other servers
        --connect_retries <number>   For implicit routes, number of connect retries
        --cluster_listen <url>       Cluster url from which members can solicit routes

Profiling Options:
        --profile <port>             Profiling HTTP port

Common Options:
    -h, --help                       Show this message
    -v, --version                    Show version
        --help_tls                   TLS help

connz.out.txt
[cli-tools] $ curl -fsS http://127.0.0.1:8222/connz
{
  "server_id": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "now": "2026-08-23T15:17:22.004747792Z",
  "num_connections": 0,
  "total": 0,
  "offset": 0,
  "limit": 1024,
  "connections": []
}
gap.out.txt
# 缺口记录:nats 官方镜像不含 nats CLI(由 nats-io/natscli 独立发行),distroless 镜像连 shell 都没有,
# 纯镜像自带命令无法收发消息;收发需外部客户端(Java 实验见 demos/nats/core-pubsub)。
[cli-tools] $ compose exec nats nats --version   # 尝试在容器内寻找自带 CLI,预期失败
OCI runtime exec failed: exec failed: unable to start container process: exec: "nats": executable file not found in $PATH
exit=127(预期失败:镜像不含 nats 二进制)
inventory.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "nats:2.11.5@sha256:47710a58674c152355730c69fc89d5a86369e3d6d1329cc91767650ce8e0dde6"
exitCode: 0
---
# nats:2.11.5 官方镜像为 distroless:文件系统仅含 /nats-server 单一二进制(无 shell、无其他 CLI),故无 bin 目录可列
[cli-tools] $ compose exec nats /nats-server --version
nats-server: v2.11.5
[cli-tools] $ compose exec nats /nats-server --help

Usage: nats-server [options]

Server Options:
    -a, --addr, --net <host>         Bind to host address (default: 0.0.0.0)
    -p, --port <port>                Use port for clients (default: 4222)
    -n, --name
        --server_name <server_name>  Server name (default: auto)
    -P, --pid <file>                 File to store PID
    -m, --http_port <port>           Use port for http monitoring
    -ms,--https_port <port>          Use port for https monitoring
    -c, --config <file>              Configuration file
    -t                               Test configuration and exit
    -sl,--signal <signal>[=<pid>]    Send signal to nats-server process (ldm, stop, quit, term, reopen, reload)
                                     <pid> can be either a PID (e.g. 1) or the path to a PID file (e.g. /var/run/nats-server.pid)
        --client_advertise <string>  Client URL to advertise to other servers
        --ports_file_dir <dir>       Creates a ports file in the specified directory (<executable_name>_<pid>.ports).

Logging Options:
    -l, --log <file>                 File to redirect log output
    -T, --logtime                    Timestamp log entries (default: true)
    -s, --syslog                     Log to syslog or windows event log
    -r, --remote_syslog <addr>       Syslog server addr (udp://localhost:514)
    -D, --debug                      Enable debugging output
    -V, --trace                      Trace the raw protocol
    -VV                              Verbose trace (traces system account as well)
    -DV                              Debug and trace
    -DVV                             Debug and verbose trace (traces system account as well)
        --log_size_limit <limit>     Logfile size limit (default: auto)
        --max_traced_msg_len <len>   Maximum printable length for traced messages (default: unlimited)

JetStream Options:
    -js, --jetstream                 Enable JetStream functionality
    -sd, --store_dir <dir>           Set the storage directory

Authorization Options:
        --user <user>                User required for connections
        --pass <password>            Password required for connections
        --auth <token>               Authorization token required for connections

TLS Options:
        --tls                        Enable TLS, do not verify clients (default: false)
        --tlscert <file>             Server certificate file
        --tlskey <file>              Private key for server certificate
        --tlsverify                  Enable TLS, verify client certificates
        --tlscacert <file>           Client certificate CA for verification

Cluster Options:
        --routes <rurl-1, rurl-2>    Routes to solicit and connect
        --cluster <cluster-url>      Cluster URL for solicited routes
        --cluster_name <string>      Cluster Name, if not set one will be dynamically generated
        --no_advertise <bool>        Do not advertise known cluster information to clients
        --cluster_advertise <string> Cluster URL to advertise to other servers
        --connect_retries <number>   For implicit routes, number of connect retries
        --cluster_listen <url>       Cluster url from which members can solicit routes

Profiling Options:
        --profile <port>             Profiling HTTP port

Common Options:
    -h, --help                       Show this message
    -v, --version                    Show version
        --help_tls                   TLS help
session.out.txt
---
status: verified
capturedAt: "2026-08-23T15:19:49.324Z"
dockerImage: "nats:2.11.5@sha256:47710a58674c152355730c69fc89d5a86369e3d6d1329cc91767650ce8e0dde6"
exitCode: 0
---
## status
[cli-tools] $ curl -fsS http://127.0.0.1:8222/healthz
{"status":"ok"}

## gap
# 缺口记录:nats 官方镜像不含 nats CLI(由 nats-io/natscli 独立发行),distroless 镜像连 shell 都没有,
# 纯镜像自带命令无法收发消息;收发需外部客户端(Java 实验见 demos/nats/core-pubsub)。
[cli-tools] $ compose exec nats nats --version   # 尝试在容器内寻找自带 CLI,预期失败
OCI runtime exec failed: exec failed: unable to start container process: exec: "nats": executable file not found in $PATH
exit=127(预期失败:镜像不含 nats 二进制)

## varz
[cli-tools] $ curl -fsS http://127.0.0.1:8222/varz
{
  "server_id": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "server_name": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "version": "2.11.5",
  "proto": 1,
  "git_commit": "94fb8b0",
  "go": "go1.24.4",
  "host": "0.0.0.0",
  "port": 4222,
  "max_connections": 65536,
  "ping_interval": 120000000000,
  "ping_max": 2,
  "http_host": "0.0.0.0",
  "http_port": 8222,
  "http_base_path": "",
  "https_port": 0,
  "auth_timeout": 2,
  "max_control_line": 4096,
  "max_payload": 1048576,
  "max_pending": 67108864,
  "cluster": {},
  "gateway": {},
  "leaf": {},
  "mqtt": {},
  "websocket": {},
  "jetstream": {
    "config": {
      "max_memory": 12574804992,
      "max_storage": 68064571392,
      "store_dir": "/tmp/nats/jetstream",
      "sync_interval": 120000000000
    },
    "stats": {
      "memory": 0,
      "storage": 0,
      "reserved_memory": 0,
      "reserved_storage": 0,
      "accounts": 1,
      "ha_assets": 0,
      "api": {
        "level": 1,
        "total": 0,
        "errors": 0
      }
    },
    "limits": {}
  },
  "tls_timeout": 2,
  "write_deadline": 10000000000,
  "start": "2026-08-23T15:17:21.243592603Z",
  "now": "2026-08-23T15:17:21.997112291Z",
  "uptime": "0s",
  "mem": 15831040,
  "cores": 4,
  "gomaxprocs": 4,
  "cpu": 3,
  "connections": 0,
  "total_connections": 0,
  "routes": 0,
  "remotes": 0,
  "leafnodes": 0,
  "in_msgs": 0,
  "out_msgs": 0,
  "in_bytes": 0,
  "out_bytes": 0,
  "slow_consumers": 0,
  "subscriptions": 63,
  "http_req_stats": {
    "/healthz": 2,
    "/varz": 1
  },
  "config_load_time": "2026-08-23T15:17:21.243592603Z",
  "config_digest": "",
  "system_account": "$SYS",
  "slow_consumer_stats": {
    "clients": 0,
    "routes": 0,
    "gateways": 0,
    "leafs": 0
  }
}

## connz
[cli-tools] $ curl -fsS http://127.0.0.1:8222/connz
{
  "server_id": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "now": "2026-08-23T15:17:22.004747792Z",
  "num_connections": 0,
  "total": 0,
  "offset": 0,
  "limit": 1024,
  "connections": []
}

## subsz
[cli-tools] $ curl -fsS http://127.0.0.1:8222/subsz
{
  "num_subscriptions": 63,
  "num_cache": 5,
  "num_inserts": 63,
  "num_removes": 0,
  "num_matches": 6,
  "cache_hit_rate": 0.16666666666666666,
  "max_fanout": 1,
  "avg_fanout": 0.8
}
status.out.txt
[cli-tools] $ curl -fsS http://127.0.0.1:8222/healthz
{"status":"ok"}
subsz.out.txt
[cli-tools] $ curl -fsS http://127.0.0.1:8222/subsz
{
  "num_subscriptions": 63,
  "num_cache": 5,
  "num_inserts": 63,
  "num_removes": 0,
  "num_matches": 6,
  "cache_hit_rate": 0.16666666666666666,
  "max_fanout": 1,
  "avg_fanout": 0.8
}
varz.out.txt
[cli-tools] $ curl -fsS http://127.0.0.1:8222/varz
{
  "server_id": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "server_name": "NBWDRAPRGUPMWTMYXIZKNZRK26AKOZNPIHAIFUMH5ZSY5DN2ZOKGD45T",
  "version": "2.11.5",
  "proto": 1,
  "git_commit": "94fb8b0",
  "go": "go1.24.4",
  "host": "0.0.0.0",
  "port": 4222,
  "max_connections": 65536,
  "ping_interval": 120000000000,
  "ping_max": 2,
  "http_host": "0.0.0.0",
  "http_port": 8222,
  "http_base_path": "",
  "https_port": 0,
  "auth_timeout": 2,
  "max_control_line": 4096,
  "max_payload": 1048576,
  "max_pending": 67108864,
  "cluster": {},
  "gateway": {},
  "leaf": {},
  "mqtt": {},
  "websocket": {},
  "jetstream": {
    "config": {
      "max_memory": 12574804992,
      "max_storage": 68064571392,
      "store_dir": "/tmp/nats/jetstream",
      "sync_interval": 120000000000
    },
    "stats": {
      "memory": 0,
      "storage": 0,
      "reserved_memory": 0,
      "reserved_storage": 0,
      "accounts": 1,
      "ha_assets": 0,
      "api": {
        "level": 1,
        "total": 0,
        "errors": 0
      }
    },
    "limits": {}
  },
  "tls_timeout": 2,
  "write_deadline": 10000000000,
  "start": "2026-08-23T15:17:21.243592603Z",
  "now": "2026-08-23T15:17:21.997112291Z",
  "uptime": "0s",
  "mem": 15831040,
  "cores": 4,
  "gomaxprocs": 4,
  "cpu": 3,
  "connections": 0,
  "total_connections": 0,
  "routes": 0,
  "remotes": 0,
  "leafnodes": 0,
  "in_msgs": 0,
  "out_msgs": 0,
  "in_bytes": 0,
  "out_bytes": 0,
  "slow_consumers": 0,
  "subscriptions": 63,
  "http_req_stats": {
    "/healthz": 2,
    "/varz": 1
  },
  "config_load_time": "2026-08-23T15:17:21.243592603Z",
  "config_digest": "",
  "system_account": "$SYS",
  "slow_consumer_stats": {
    "clients": 0,
    "routes": 0,
    "gateways": 0,
    "leafs": 0
  }
}

以统一实验验证消息系统语义边界