Skip to content

nodetool setstreamthroughput

Sets the streaming throughput limit for inter-node data transfers.

Synopsis

nodetool [connection_options] setstreamthroughput <throughput_mb_per_sec>

Description

nodetool setstreamthroughput controls the maximum rate at which data can be streamed between Cassandra nodes. Streaming occurs during:

  • Repair operations - Transferring data to synchronize replicas
  • Bootstrap - New nodes receiving data from existing nodes
  • Rebuild - Nodes rebuilding data from other data centers
  • Decommission - Nodes transferring data before leaving cluster
  • Node replacement - Replacement nodes receiving data

This setting affects outgoing streams from the node where the command is executed.

Non-Persistent Setting

This setting is applied at runtime only and does not persist across node restarts. After a restart, the value reverts to the stream_throughput_outbound setting in cassandra.yaml (default: 200 MB/s).

To make the change permanent, update cassandra.yaml:

stream_throughput_outbound: 200MiB/s

Arguments

Argument Description
throughput_mb_per_sec Maximum streaming rate in MB/s. Use 0 for unlimited

Examples

Set Throughput

nodetool setstreamthroughput 200

Unlimited Streaming

nodetool setstreamthroughput 0

Unlimited Streaming

Setting to 0 removes all throttling. Use with caution as it may saturate network bandwidth and impact other operations.

Verify Setting

nodetool setstreamthroughput 200
nodetool getstreamthroughput
# Output: Current stream throughput: 200 MB/s

When to Use

Speed Up Maintenance Operations

During scheduled maintenance windows:

# Increase throughput for faster repair
nodetool setstreamthroughput 400

# Run repair
nodetool repair -pr my_keyspace

# Restore normal throughput
nodetool setstreamthroughput 200

Reduce Network Impact

During high traffic periods:

# Reduce streaming to prioritize client traffic
nodetool setstreamthroughput 100

Bootstrap Optimization

When adding new nodes:

# On seed nodes (streaming sources)
nodetool setstreamthroughput 300

# Monitor bootstrap progress
nodetool netstats

Decommission Acceleration

Speed up node removal:

# Before decommissioning
nodetool setstreamthroughput 400

# Start decommission
nodetool decommission

# Monitor progress
watch nodetool netstats

Multi-DC Considerations

Adjust for cross-datacenter rebuilds:

# Lower throughput for WAN links
nodetool setstreamthroughput 50

# Start rebuild
nodetool rebuild dc1

# Monitor
nodetool netstats

By Scenario

Scenario Throughput Rationale
Default 200 MB/s Balanced default
High-performance network 400-800 MB/s Fast 10GbE+ networks
Shared network 100-200 MB/s Avoid impacting other traffic
Maintenance window 400+ MB/s Prioritize streaming
Cross-DC (WAN) 50-100 MB/s Limited bandwidth
During high load 100 MB/s Prioritize client traffic

By Network Type

Network Recommended Range
1 GbE 50-100 MB/s
10 GbE 200-500 MB/s
25 GbE 400-800 MB/s
AWS (same AZ) 200-400 MB/s
AWS (cross AZ) 100-200 MB/s
Cross region/WAN 50-100 MB/s

Workflow: Rolling Repair with Optimized Streaming

#!/bin/bash
# repair_with_tuned_streaming.sh

# Store original value
original=$(nodetool getstreamthroughput | awk '{print $4}')
echo "Original streaming throughput: $original MB/s"

# Increase for repair
echo "Increasing streaming throughput to 400 MB/s"
nodetool setstreamthroughput 400

# Run repair
echo "Starting repair..."
nodetool repair -pr my_keyspace

# Restore original
echo "Restoring original throughput: $original MB/s"
nodetool setstreamthroughput $original

echo "Repair complete"

Workflow: Bootstrap New Node

On existing nodes (streaming sources):

# 1. Check current settings
nodetool getstreamthroughput

# 2. Increase for bootstrap
nodetool setstreamthroughput 400

# 3. (Bootstrap starts on new node)

# 4. Monitor streaming
watch 'nodetool netstats | head -20'

# 5. After bootstrap completes, restore
nodetool setstreamthroughput 200

Relationship with Other Settings

Streaming vs Compaction

Both affect disk I/O and should be considered together:

Setting Affects Command
Stream throughput Node-to-node transfers setstreamthroughput
Compaction throughput Local SSTable processing setcompactionthroughput 
# Check both settings
echo "Stream: $(nodetool getstreamthroughput)"
echo "Compaction: $(nodetool getcompactionthroughput)"

Total I/O Budget

Consider combined impact:

# During heavy streaming
nodetool setstreamthroughput 300
nodetool setcompactionthroughput 32  # Reduce to free I/O for streaming

# After streaming completes
nodetool setstreamthroughput 200
nodetool setcompactionthroughput 64  # Restore normal compaction

Monitoring Streaming

Check Current Activity

nodetool netstats

Output: 

Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 1234
Mismatch (Blocking): 56
Mismatch (Background): 78
Pool Name                    Active   Pending      Completed   Dropped
Large messages                    0         0          12345         0
Small messages                    0         0         678901         0

During Active Streaming

nodetool netstats

Output during bootstrap/repair: 

Mode: NORMAL
Receiving from: /192.168.1.101
  /192.168.1.101
    Receiving 15 files, 2.3 GB total. Already received 1.1 GB, at 45.2 MB/s
      my_keyspace/my_table
Sending to: /192.168.1.102
  /192.168.1.102
    Sending 10 files, 1.8 GB total. Already sent 900 MB, at 38.5 MB/s

Monitor Progress

# Continuous monitoring
watch -n 5 'nodetool netstats | grep -E "Receiving|Sending|MB/s"'

Troubleshooting

Streaming Too Slow

If bootstrap/repair taking too long:

# Check current limit
nodetool getstreamthroughput

# Check actual rate
nodetool netstats | grep "MB/s"

# If rate << limit, bottleneck is elsewhere
# Check: disk I/O, network, source node load

# Increase limit
nodetool setstreamthroughput 400

Network Saturation

If streaming impacts client traffic:

# Reduce streaming throughput
nodetool setstreamthroughput 100

# Verify client latencies improve
nodetool proxyhistograms

Streaming Fails

If streams fail or timeout:

# Check streaming status
nodetool netstats

# Check for errors in logs
grep -i stream /var/log/cassandra/system.log | tail -20

# Try reducing throughput (too fast can cause issues)
nodetool setstreamthroughput 100

Cluster-Wide Management

Apply to All Nodes

for node in node1 node2 node3; do
    echo "Setting streaming throughput on $node"
    ssh "$node" "nodetool setstreamthroughput 200"
done

Verify Cluster Settings

for node in node1 node2 node3; do
    echo -n "$node: "
    ssh "$node" "nodetool getstreamthroughput"
done

Automation Script

#!/bin/bash
# adjust_streaming.sh - Adjust streaming throughput cluster-wide

THROUGHPUT="${1:-200}"
# Get list of node IPs from local nodetool status
NODES=$(nodetool status | grep "^UN" | awk '{print $2}')

echo "Setting streaming throughput to $THROUGHPUT MB/s on all nodes"
echo "=============================================="

for node in $NODES; do
    echo -n "$node: "
    ssh "$node" "nodetool setstreamthroughput $THROUGHPUT" 2>/dev/null
    if [ $? -eq 0 ]; then
        echo "OK"
    else
        echo "FAILED"
    fi
done

echo ""
echo "Verification:"
for node in $NODES; do
    echo -n "$node: "
    ssh "$node" "nodetool getstreamthroughput" 2>/dev/null | awk '{print $4 " " $5}'
done

Best Practices

Streaming Throughput Guidelines

  1. Know your network - Set appropriate limits for your bandwidth
  2. Adjust for operations - Increase during maintenance windows
  3. Monitor actively - Watch netstats during streaming operations
  4. Balance with compaction - Consider total I/O budget
  5. Test before production - Validate settings in non-production first
  6. Document changes - Log when and why throughput was adjusted
  7. Cluster consistency - Use similar settings across nodes
  8. WAN awareness - Use lower settings for cross-DC operations
Command Relationship
netstats Monitor streaming activity
repair Triggers streaming
rebuild Triggers cross-DC streaming
decommission Triggers outgoing streams
setcompactionthroughput Related I/O throttle
getcompactionthroughput View compaction throttle