Skip to content

Agent setup kafka

Install Kafka Agent

Select the Kafka Version

Select the Java Version.

Install the AxonOps Kafka Agent and its dependency axon-agent:

sudo apt-get install axon-kafka2-agent

sudo apt-get install axon-kafka3-agent

sudo apt-get install axon-kafka2-agent

sudo apt-get install axon-kafka3-agent

Configuration File Locations

AxonOps Kafka Agent

The AxonOps Kafka Agent is the jar that is directly loaded by Kafka. The AxonOps Kafka Agent then reaches out directly to the AxonOps Agent binary which contacts the AxonOps Server directly.

  • Configuration File: /etc/axonops/axon-agent.yml
  • Binary: /usr/share/axonops/axon-kafka{version}-agent.jar
  • Version number: /usr/share/axonops/axon-kafka{version}-agent.version
  • Copyright: /usr/share/doc/axonops/axon-kafka{version}-agent/copyright
  • Licenses: /usr/share/axonops/licenses/axon-kafka{version}-agent/

AxonOps Agent

The AxonOps Agent is a dependency of the AxonOps Kafka Agent. This binary contacts the AxonOps Server directly while minimizing the memory footprint and CPU utilization of the Kafka process.

  • Configuration File: /etc/axonops/axon-agent.yml
  • Binary: usr/share/axonops/axon-agent
  • Logs: /var/log/axonops/axon-agent.log
  • Systemd service: /usr/lib/systemd/system/axon-agent.service

Agent Configuration

Update the highlighted lines in /etc/axonops/axon-agent.yml.

These need to match the config that you have in your axon-server.yml setup.



axon-server:
  hosts: "agents.axonops.cloud" # AxonOps SaaS
  # hosts: "${AXONOPS_SERVER_HOSTS}" # AxonOps Server On-Premise Endpoint
  # port: 1888 # AxonOps Server On-Premise Port (Default is 1888)

axon-agent:
  key: "<THIS_IS_A_DUMMY_KEY_PLEASE_UPDATE>"
  org: "<THIS_IS_A_DUMMY_ORG_NAME_PLEASE_UPDATE>"
  cluster_name: "<THIS_IS_A_DUMMY_CLUSTER_NAME_PLEASE_UPDATE>"
  tls:
    mode: "TLS" # disabled, TLS
    #skipVerify: false # Disables CA and Hostname verification
    #caFile: "path_to_certs_on_axon_agent_node.crt" # required if skipVerify is not set and you are using a self-signed cert
    #certFile: "path_to_certs_on_axon_agent_node.crt"
    #keyFile: "path_to_key_file_on_axon_agent_node.key"
# Specify the NTP server IP addresses or hostnames configured for your hosts
# The port defaults to 123 if not specified.
# NTP:
#    hosts:
#        - "x.x.x.x:123"
# Optionally restrict which commands can be executed by axon-agent.
# If "true", only scripts placed in scripts_location can be executed by axon-agent.
# disable_command_exec: false
# If disable_command_exec is true then axon-agent is only allowed to execute scripts
# under this path
# scripts_location: /var/lib/axonops/scripts/

kafka:
  node_type: "broker" # broker, kraft-broker, kraft-controller, zookeeper, connect
  # rack: "testrack" # Auto-detected from Kafka config, optionally override the rack to group nodes in AxonOps
  kafka_client:
    brokers: ["<host_listener_ip_address_or_fqdn>:<port>>"] # 10.0.0.2:9092 or 10.20.30.40:9094 or this_is_my_server.domain.com:9093
    # Authentication Settings
    sasl:
      username: <THIS_IS_A_DUMMY_USERNAME_PLEASE_UPDATE>
      password: <THIS_IS_A_DUMMY_PASSWORD_PLEASE_UPDATE>
      mechanism: PLAIN # SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, GSSAPI
      # For oauth support
      # oauth:
      #   token:
      #   clientId:
      #   clientSecret:
      #   tokenEndpoint:
      #   scope:
      # For gssapi support
      # gssapi:
      #   authType:
      #   keyTabPath:
      #   kerberosConfigPath:
      #   serviceName:
      #   username:
      #   password:
      #   realm:
      #   enableFast: true
    # SSL settings for connection to Kafka
    tls:
      enabled: true
      caFilepath: <THIS_IS_A_DUMMY_CA_PATH_PLEASE_UPDATE>
      insecureSkipTlsVerify: false

Set file permissions on /etc/axonops/axon-agent.yml file by executing the following command

sudo chmod 0640 /etc/axonops/axon-agent.yml

Configure Kafka

Edit kafka-server-start.sh, usually located in your Kafka install path such as:

/<Kafka_Home>/bin/kafka-server-start.sh

Insert the line . /usr/share/axonops/axonops-jvm.options right before the final line in your config file. Look for the last line that starts with exec, as demonstrated in the example below.

NB. Please note the period(.) at the beginning of the config line.

Example:

. /usr/share/axonops/axonops-jvm.options
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

  EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc -javaagent:/usr/share/axonops/axon-kafka2.0-agent.jar=/etc/axonops/axon-agent.yml --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED  --add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED  --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-opens=java.management/com.sun.jmx.interceptor=ALL-UNNAMED --add-exports=java.management/com.sun.jmx.interceptor=ALL-UNNAMED'}

. /usr/share/axonops/axonops-jvm.options
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

  EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc -javaagent:/usr/share/axonops/axon-kafka3.0-agent.jar=/etc/axonops/axon-agent.yml --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED  --add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED  --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-opens=java.management/com.sun.jmx.interceptor=ALL-UNNAMED --add-exports=java.management/com.sun.jmx.interceptor=ALL-UNNAMED'}

NB. Make sure that this configuration will not get overridden by an automation tool.

Add axonops user to Kafka user group and Kafka user to axonops group

sudo usermod -aG <your_kafka_group> axonops
sudo usermod -aG axonops <your_kafka_user>

Start/Restart Kafka

To load the AxonOps Java Agent and Kafka config changes please either start the Kafka service if stopped restart the Kafka service if already running.