Cassandra Logging Configuration (logback.xml)¶

Cassandra uses Logback for logging, configured via logback.xml. This file controls log levels, output destinations, formats, and rotation policies.

Configuration File¶

File	Location
`logback.xml`	`/etc/cassandra/`

Changes to logback.xml are detected automatically within 60 seconds—no restart required.

Log Files¶

Cassandra produces several log files by default:

Log File	Purpose	Default Location
`system.log`	Main operational log	`/var/log/cassandra/`
`debug.log`	Detailed debug output	`/var/log/cassandra/`
`gc.log`	Garbage collection (JVM-configured)	`/var/log/cassandra/`

Basic Structure¶

<configuration scan="true" scanPeriod="60 seconds">

  <!-- Appenders define where logs go -->
  <appender name="..." class="...">
    <!-- appender configuration -->
  </appender>

  <!-- Loggers define what gets logged -->
  <logger name="..." level="..."/>

  <!-- Root logger catches everything not matched by specific loggers -->
  <root level="INFO">
    <appender-ref ref="..."/>
  </root>

</configuration>

Appenders¶

Console Appender¶

Output to stdout (useful for containers):

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
  <encoder>
    <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
  </encoder>
</appender>

Rolling File Appender (system.log)¶

Standard file appender with rotation:

<appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${cassandra.logdir}/system.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
    <!-- Roll daily -->
    <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
    <!-- Max 50MB per file -->
    <maxFileSize>50MB</maxFileSize>
    <!-- Keep 7 days of history -->
    <maxHistory>7</maxHistory>
    <!-- Cap total size at 5GB -->
    <totalSizeCap>5GB</totalSizeCap>
  </rollingPolicy>
  <encoder>
    <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
  </encoder>
</appender>

Debug Log Appender¶

Separate file for DEBUG level messages:

<appender name="DEBUGLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${cassandra.logdir}/debug.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
    <fileNamePattern>${cassandra.logdir}/debug.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
    <maxFileSize>50MB</maxFileSize>
    <maxHistory>7</maxHistory>
    <totalSizeCap>5GB</totalSizeCap>
  </rollingPolicy>
  <encoder>
    <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
  </encoder>
</appender>

Async Appender¶

Wrap appenders for non-blocking logging:

<appender name="ASYNCSYSTEMLOG" class="ch.qos.logback.classic.AsyncAppender">
  <queueSize>1024</queueSize>
  <discardingThreshold>0</discardingThreshold>
  <includeCallerData>true</includeCallerData>
  <appender-ref ref="SYSTEMLOG"/>
</appender>

Log Levels¶

Level	Description	Use Case
`TRACE`	Most detailed	Deep debugging
`DEBUG`	Detailed information	Development, troubleshooting
`INFO`	Normal operations	Production default
`WARN`	Potential issues	Production monitoring
`ERROR`	Errors requiring attention	Always enabled
`OFF`	Disable logging	Specific loggers only

Logger Configuration¶

Root Logger¶

Catches all log messages not matched by specific loggers:

<root level="INFO">
  <appender-ref ref="SYSTEMLOG"/>
  <appender-ref ref="STDOUT"/>
</root>

Package-Specific Loggers¶

Control verbosity for specific components:

<!-- Reduce noisy components -->
<logger name="org.apache.cassandra.db.compaction" level="INFO"/>
<logger name="org.apache.cassandra.service.StorageService" level="INFO"/>

<!-- Increase verbosity for troubleshooting -->
<logger name="org.apache.cassandra.db.commitlog" level="DEBUG"/>
<logger name="org.apache.cassandra.streaming" level="DEBUG"/>

Common Logger Configurations¶

Logger	Level	Purpose
`org.apache.cassandra`	INFO	All Cassandra components
`org.apache.cassandra.db.compaction`	INFO	Compaction operations
`org.apache.cassandra.db.commitlog`	INFO	Commit log operations
`org.apache.cassandra.streaming`	INFO	Streaming operations
`org.apache.cassandra.hints`	INFO	Hinted handoff
`org.apache.cassandra.repair`	INFO	Repair operations
`org.apache.cassandra.gms`	INFO	Gossip protocol
`org.apache.cassandra.net`	INFO	Internode messaging
`org.apache.cassandra.cql3`	INFO	CQL processing
`org.apache.cassandra.auth`	INFO	Authentication/authorization

Log Pattern Format¶

Pattern Elements¶

Element	Description	Example Output
`%level`	Log level	`INFO`
`%-5level`	Log level, left-padded to 5 chars	`INFO`
`%thread`	Thread name	`main`
`%date{ISO8601}`	ISO timestamp	`2024-01-15 10:30:45,123`
`%logger`	Logger name	`org.apache.cassandra.db.Memtable`
`%F`	Source file name	`Memtable.java`
`%L`	Line number	`234`
`%M`	Method name	`flush`
`%msg`	Log message	`Flushing memtable`
`%n`	Newline
`%X{key}`	MDC value	Custom context

Standard Pattern¶

<pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>

Output:

INFO  [main] 2024-01-15 10:30:45,123 StorageService.java:1234 - Starting listening for CQL clients

JSON Pattern (for log aggregation)¶

<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
  <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
    <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter"/>
    <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</timestampFormat>
    <appendLineSeparator>true</appendLineSeparator>
  </layout>
</encoder>

Rolling Policies¶

Size and Time Based¶

Roll logs based on both size and time:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
  <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
  <maxFileSize>50MB</maxFileSize>
  <maxHistory>7</maxHistory>
  <totalSizeCap>5GB</totalSizeCap>
</rollingPolicy>

Time Based Only¶

Roll logs daily without size limit:

<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
  <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.zip</fileNamePattern>
  <maxHistory>30</maxHistory>
</rollingPolicy>

Size Based Only¶

Roll logs based on size only:

<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
  <fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern>
  <minIndex>1</minIndex>
  <maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
  <maxFileSize>100MB</maxFileSize>
</triggeringPolicy>

Complete Examples¶

Production Configuration¶

<configuration scan="true" scanPeriod="60 seconds">

  <!-- JMX configuration -->
  <jmxConfigurator/>

  <!-- System log -->
  <appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/system.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
      <maxFileSize>50MB</maxFileSize>
      <maxHistory>7</maxHistory>
      <totalSizeCap>5GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- Debug log -->
  <appender name="DEBUGLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/debug.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/debug.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
      <maxFileSize>50MB</maxFileSize>
      <maxHistory>7</maxHistory>
      <totalSizeCap>5GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
    </encoder>
    <!-- Only DEBUG level -->
    <filter class="ch.qos.logback.classic.filter.LevelFilter">
      <level>DEBUG</level>
      <onMatch>ACCEPT</onMatch>
      <onMismatch>DENY</onMismatch>
    </filter>
  </appender>

  <!-- Async wrappers -->
  <appender name="ASYNCSYSTEMLOG" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>1024</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <includeCallerData>true</includeCallerData>
    <appender-ref ref="SYSTEMLOG"/>
  </appender>

  <appender name="ASYNCDEBUGLOG" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>1024</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <includeCallerData>true</includeCallerData>
    <appender-ref ref="DEBUGLOG"/>
  </appender>

  <!-- Reduce noisy loggers -->
  <logger name="org.apache.cassandra.db.compaction.CompactionTask" level="INFO"/>
  <logger name="org.apache.cassandra.service.StorageProxy" level="INFO"/>

  <!-- Root logger -->
  <root level="INFO">
    <appender-ref ref="ASYNCSYSTEMLOG"/>
    <appender-ref ref="ASYNCDEBUGLOG"/>
  </root>

</configuration>

Container/Kubernetes Configuration¶

<configuration scan="true" scanPeriod="60 seconds">

  <!-- Console output for container logs -->
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- Async wrapper -->
  <appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>1024</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <appender-ref ref="STDOUT"/>
  </appender>

  <!-- Reduce verbosity -->
  <logger name="org.apache.cassandra.db.compaction" level="WARN"/>
  <logger name="org.apache.cassandra.gms.Gossiper" level="WARN"/>

  <root level="INFO">
    <appender-ref ref="ASYNCSTDOUT"/>
  </root>

</configuration>

Debug/Troubleshooting Configuration¶

<configuration scan="true" scanPeriod="60 seconds">

  <appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/system.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
      <maxFileSize>100MB</maxFileSize>
      <maxHistory>3</maxHistory>
      <totalSizeCap>10GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %logger{36}:%L - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- Enable debug for specific subsystems -->
  <logger name="org.apache.cassandra.db.compaction" level="DEBUG"/>
  <logger name="org.apache.cassandra.streaming" level="DEBUG"/>
  <logger name="org.apache.cassandra.repair" level="DEBUG"/>
  <logger name="org.apache.cassandra.hints" level="DEBUG"/>
  <logger name="org.apache.cassandra.net" level="DEBUG"/>
  <logger name="org.apache.cassandra.gms" level="DEBUG"/>

  <root level="DEBUG">
    <appender-ref ref="SYSTEMLOG"/>
  </root>

</configuration>

Runtime Log Level Changes¶

Modify log levels without restart using nodetool:

# View current levels
nodetool getlogginglevels

# Set level for specific logger
nodetool setlogginglevel org.apache.cassandra.db.compaction DEBUG

# Reset to configuration file settings
nodetool setlogginglevel org.apache.cassandra.db.compaction RESET

Runtime vs Persistent Changes

Changes via nodetool are temporary and reset on restart. For persistent changes, modify logback.xml.

Audit Logging¶

Cassandra 4.0+ includes built-in audit logging:

<!-- Audit log appender -->
<appender name="AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${cassandra.logdir}/audit/audit.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
    <fileNamePattern>${cassandra.logdir}/audit/audit.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
    <maxFileSize>50MB</maxFileSize>
    <maxHistory>30</maxHistory>
    <totalSizeCap>10GB</totalSizeCap>
  </rollingPolicy>
  <encoder>
    <pattern>%-5level [%thread] %date{ISO8601} - %msg%n</pattern>
  </encoder>
</appender>

<logger name="org.apache.cassandra.audit" level="INFO" additivity="false">
  <appender-ref ref="AUDIT"/>
</logger>

Enable in cassandra.yaml:

audit_logging_options:
  enabled: true
  logger:
    - class_name: BinAuditLogger
  included_keyspaces: my_keyspace
  included_categories: QUERY,DML,DDL,AUTH

Full Query Logging¶

Log all CQL queries for debugging:

# Enable via nodetool
nodetool enablefullquerylog --path /var/log/cassandra/fql

# Disable
nodetool disablefullquerylog

# Read logs
fqltool dump /var/log/cassandra/fql

Slow Query Logging¶

Log queries exceeding threshold:

# cassandra.yaml
slow_query_log_timeout_in_ms: 500

Slow queries appear in system.log at WARN level.

Troubleshooting¶

Log File Not Created¶

Check directory permissions:
```
ls -la /var/log/cassandra/
```

Verify cassandra.logdir property:

grep logdir /etc/cassandra/cassandra-env.sh

Check for XML syntax errors:

xmllint --noout /etc/cassandra/logback.xml

Disk Space Issues¶

Configure aggressive rotation:

<maxFileSize>25MB</maxFileSize>
<maxHistory>3</maxHistory>
<totalSizeCap>1GB</totalSizeCap>

Performance Impact¶

Use async appenders to minimize logging overhead:

<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
  <queueSize>1024</queueSize>
  <discardingThreshold>0</discardingThreshold>
  <neverBlock>true</neverBlock>
  <appender-ref ref="SYSTEMLOG"/>
</appender>

Missing Stack Traces¶

Ensure includeCallerData is enabled for async appenders:

<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
  <includeCallerData>true</includeCallerData>
  <appender-ref ref="SYSTEMLOG"/>
</appender>

Best Practices¶

Practice	Recommendation
Production log level	INFO for root, WARN for noisy components
Async appenders	Always use for production
Rotation	Size + time based with totalSizeCap
Debug logs	Separate file, enable only when needed
Container deployments	Console output, let orchestrator handle rotation
Disk monitoring	Alert when log partition exceeds 80%
Retention	7-30 days depending on compliance requirements

cassandra.yaml - Main configuration
JVM Options - GC logging configuration
Monitoring - Log aggregation and alerting