Skip to content

Compaction Issues

Compaction merges SSTables to maintain read performance and reclaim space. When compaction falls behind or behaves unexpectedly, read performance degrades and disk usage grows.

Symptoms

  • Growing pending compaction count
  • Increasing SSTable count per table
  • Degrading read latency over time
  • Disk space not being reclaimed after deletes
  • High I/O during compaction spikes
  • "Compaction stuck" or very slow progress

Diagnosis

Step 1: Check Compaction Status

nodetool compactionstats

Problem indicators: - High pending count (> 50 per table) - Same compaction running for hours - No active compactions despite pending

Step 2: Check SSTable Counts

nodetool tablestats my_keyspace | grep -E "Table:|SSTable count"

Target: Generally < 20 SSTables per table (varies by strategy).

Step 3: Check Compaction Throughput

nodetool getcompactionthroughput

Default: 64 MB/s. May need increase for high-write workloads.

Step 4: Check Disk I/O

iostat -x 1 10

Problem indicators: - Disk utilization at 100% - High await times

Step 5: Check Compaction Strategy

cqlsh -e "SELECT table_name, compaction FROM system_schema.tables WHERE keyspace_name = 'my_keyspace';"

Resolution

Case 1: Compaction Falling Behind

Increase compaction throughput:

# Check current
nodetool getcompactionthroughput

# Increase (MB/s)
nodetool setcompactionthroughput 128

Increase concurrent compactors:

# Check current
nodetool getconcurrentcompactors

# Increase
nodetool setconcurrentcompactors 4

Case 2: Compaction Disabled

# Check if auto-compaction is enabled
nodetool statusautocompaction my_keyspace my_table

# Enable if disabled
nodetool enableautocompaction my_keyspace my_table

Case 3: Wrong Compaction Strategy

For read-heavy workloads (many updates):

ALTER TABLE my_table WITH compaction = {
    'class': 'LeveledCompactionStrategy',
    'sstable_size_in_mb': 160
};

For time-series data:

ALTER TABLE my_table WITH compaction = {
    'class': 'TimeWindowCompactionStrategy',
    'compaction_window_unit': 'DAYS',
    'compaction_window_size': 1
};

For write-heavy workloads:

ALTER TABLE my_table WITH compaction = {
    'class': 'SizeTieredCompactionStrategy',
    'min_threshold': 4,
    'max_threshold': 32
};

Case 4: Force Manual Compaction

For specific table:

nodetool compact my_keyspace my_table

For major compaction (use sparingly):

# Warning: Resource intensive
nodetool compact --split-output my_keyspace my_table

Major Compaction

Major compaction creates one large SSTable and can cause significant I/O. Use --split-output to create multiple smaller SSTables instead.

Case 5: Compaction Stuck on Large SSTable

Check what's happening:

nodetool compactionstats -H

If stuck on validation:

# May need to restart node if truly stuck
# First try waiting - large SSTables take time

# If necessary, cancel specific compaction
# (Requires identifying compaction ID from logs)

Case 6: Disk Full Preventing Compaction

See Handle Full Disk.

# Quick space recovery
nodetool clearsnapshot --all

# Check space
df -h /var/lib/cassandra

Compaction Strategy Selection

Strategy Best For SSTable Behavior
STCS Write-heavy, space-efficient Many SSTables, tiered by size
LCS Read-heavy, consistent performance Fixed-size levels
TWCS Time-series, TTL data Time-based windows
UCS Flexible, Cassandra 5.0+ Unified approach

Migration Between Strategies

-- Check current strategy
SELECT compaction FROM system_schema.tables
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';

-- Change strategy (takes effect gradually)
ALTER TABLE my_table WITH compaction = {
    'class': 'LeveledCompactionStrategy'
};

-- Force migration
nodetool compact my_keyspace my_table

Tuning Parameters

Size-Tiered (STCS)

ALTER TABLE my_table WITH compaction = {
    'class': 'SizeTieredCompactionStrategy',
    'min_threshold': 4,        -- Min SSTables to compact
    'max_threshold': 32,       -- Max SSTables to compact
    'min_sstable_size': 50     -- Min size to consider (MB)
};

Leveled (LCS)

ALTER TABLE my_table WITH compaction = {
    'class': 'LeveledCompactionStrategy',
    'sstable_size_in_mb': 160  -- Target SSTable size
};

Time-Window (TWCS)

ALTER TABLE my_table WITH compaction = {
    'class': 'TimeWindowCompactionStrategy',
    'compaction_window_unit': 'DAYS',
    'compaction_window_size': 1,
    'max_threshold': 32
};

Recovery

Verify Compaction Health

# Pending should decrease
watch -n 30 'nodetool compactionstats | head -10'

# SSTable count should stabilize
nodetool tablestats my_keyspace.my_table | grep "SSTable count"

Verify Read Performance

# Latencies should improve
nodetool tablehistograms my_keyspace my_table

Monitoring

Key Metrics

Metric Warning Critical
Pending compactions > 20 > 100
SSTable count > 20 > 50
Compaction throughput Near limit At 0
Disk usage > 70% > 85%

Monitoring Script

#!/bin/bash
# monitor_compaction.sh

while true; do
    clear
    echo "=== $(date) ==="
    echo ""
    echo "--- Compaction Stats ---"
    nodetool compactionstats | head -20
    echo ""
    echo "--- SSTable Counts ---"
    nodetool tablestats 2>/dev/null | grep -E "Table:|SSTable count" | head -20
    echo ""
    echo "--- Disk Usage ---"
    df -h /var/lib/cassandra
    sleep 60
done

Prevention

  1. Choose appropriate strategy - Match to workload pattern
  2. Monitor pending compactions - Alert early
  3. Adequate disk space - Keep < 70% utilization
  4. Sufficient I/O capacity - SSD recommended
  5. Tune throughput - Match to hardware capability
  6. Regular table maintenance - Monitor SSTable counts
Command Purpose
nodetool compactionstats Current compaction status
nodetool compact Force compaction
nodetool setcompactionthroughput Adjust throughput
nodetool getconcurrentcompactors View compactor count
nodetool enableautocompaction Enable auto-compaction
nodetool disableautocompaction Disable auto-compaction