nodetool netstats¶
Displays network statistics including streaming operations, messaging activity, and connection states.
Synopsis¶
nodetool [connection_options] netstats [-H]
Description¶
nodetool netstats shows current network activity including:
- Active streaming sessions (repair, bootstrap, rebuild)
- Pending and completed streams
- Message queues to other nodes
- Read repair statistics
This is essential for monitoring data movement during topology changes, repairs, and other streaming operations.
Options¶
| Option | Description |
|---|---|
-H, --human-readable |
Display bytes in human-readable format |
Output Example¶
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 1234
Mismatch (Blocking): 56
Mismatch (Background): 78
Pool Name Active Pending Completed Dropped
Large messages n/a 0 1234 0
Small messages n/a 12 567890 0
Gossip messages n/a 0 12345 0
With Active Streaming¶
Mode: NORMAL
Receiving from: /192.168.1.102
/var/lib/cassandra/data/my_keyspace/users-a1b2c3d4
Receiving 12 files, 1.2GB total. Already received 8 files, 800MB total
Progress: 66%
Sending to: /192.168.1.103
/var/lib/cassandra/data/my_keyspace/orders-b2c3d4e5
Sending 5 files, 500MB total. Already sent 3 files, 300MB total
Progress: 60%
Read Repair Statistics:
Attempted: 1234
Mismatch (Blocking): 56
Mismatch (Background): 78
Output Fields¶
Mode¶
Current node operation mode:
| Mode | Description |
|---|---|
| NORMAL | Standard operation |
| STARTING | Node is starting up |
| JOINING | Node is bootstrapping |
| LEAVING | Node is decommissioning |
| MOVING | Node is moving to new token |
| DRAINING | Node is draining |
| DRAINED | Node has drained |
Streaming Information¶
| Field | Description |
|---|---|
| Receiving from | Nodes sending data to this node |
| Sending to | Nodes receiving data from this node |
| Files | Number of SSTable files being transferred |
| Progress | Percentage complete |
Message Pools¶
| Pool | Description |
|---|---|
| Large messages | Large inter-node messages (streaming, repair) |
| Small messages | Regular inter-node messages (reads, writes) |
| Gossip messages | Gossip protocol messages |
Pool Statistics¶
| Field | Description |
|---|---|
| Active | Currently processing |
| Pending | Waiting to be processed |
| Completed | Successfully processed |
| Dropped | Timed out and dropped |
Read Repair Statistics¶
| Field | Description |
|---|---|
| Attempted | Total read repairs started |
| Mismatch (Blocking) | Inconsistencies found during blocking reads |
| Mismatch (Background) | Inconsistencies found during background repair |
When to Use¶
Monitor Streaming Operations¶
During bootstrap, decommission, repair, or rebuild:
# Continuous monitoring
watch -n 2 'nodetool netstats'
Check for Stuck Streaming¶
If an operation seems stuck:
nodetool netstats
Progress percentage that doesn't change indicates a problem.
Troubleshoot Slow Operations¶
nodetool netstats -H
Shows how much data remains to transfer.
Investigate Dropped Messages¶
nodetool netstats | grep Dropped
Non-zero dropped messages indicate network or processing issues.
Examples¶
Basic Usage¶
nodetool netstats
Human-Readable Sizes¶
nodetool netstats -H
Monitor Streaming Continuously¶
watch -n 5 'nodetool netstats -H'
Check on All Nodes¶
for node in node1 node2 node3; do
echo "=== $node ==="
ssh "$node" "nodetool netstats"
done
Extract Stream Progress¶
nodetool netstats | grep -E "Progress|Receiving|Sending"
Interpreting Results¶
No Active Streams (Normal)¶
Mode: NORMAL
Not sending any streams.
Normal state when no data movement operations are running.
Bootstrap in Progress¶
Mode: JOINING
Receiving from: /192.168.1.101
Progress: 45%
Receiving from: /192.168.1.102
Progress: 52%
Receiving from: /192.168.1.103
Progress: 38%
New node receiving data from existing nodes.
Repair Streaming¶
Mode: NORMAL
Sending to: /192.168.1.102
streaming_repair_task
Progress: 80%
Repair operation sending data to out-of-sync node.
Decommission Progress¶
Mode: LEAVING
Sending to: /192.168.1.101
Progress: 25%
Sending to: /192.168.1.103
Progress: 30%
Node sending its data to remaining nodes.
Troubleshooting with netstats¶
Streaming Appears Stuck¶
If progress doesn't increase:
- Check network connectivity between nodes
- Check disk space on receiving node
- Check for errors in system.log
- Verify streaming isn't throttled
# Check streaming throughput settings
nodetool getstreamthroughput
nodetool getinterdcstreamthroughput
High Dropped Messages¶
Pool Name Active Pending Completed Dropped
Small messages n/a 50 567890 125
Dropped Messages
Dropped messages indicate:
- Network timeouts
- Overloaded nodes
- Processing delays
Investigate immediately - affects data consistency.
High Pending Messages¶
Pool Name Active Pending Completed Dropped
Small messages n/a 500 567890 0
Message Backlog
High pending indicates:
- Node is overloaded
- Network backpressure
- May lead to dropped messages
Large Read Repair Mismatch¶
Read Repair Statistics:
Attempted: 10000
Mismatch (Blocking): 5000
50% mismatch rate indicates:
- Recent node outage
- Repair hasn't run recently
- Write issues during failures
Streaming Throughput¶
Control streaming speed to reduce impact:
# Get current setting
nodetool getstreamthroughput
# Set throughput (MB/s)
nodetool setstreamthroughput 200
# Inter-DC streaming
nodetool getinterdcstreamthroughput
nodetool setinterdcstreamthroughput 50
Throttling Trade-offs
- Higher throughput = faster operations, more impact on production
- Lower throughput = slower operations, less impact
- Default is often conservative; increase for faster streaming
Related Commands¶
| Command | Relationship |
|---|---|
| status | Check node states |
| tpstats | Thread pool statistics |
| compactionstats | Compaction progress |
| repair | Triggers streaming |
| decommission | Triggers streaming |
| rebuild | Triggers streaming |
| setstreamthroughput | Control streaming rate |