AxonOps Server installation¶
Step 1 - Prerequisites¶
Metrics Storage Engine¶
Elasticsearch¶
Elasticsearch is the default data storage engine for all Cassandra metrics and AxonOps config. AxonOps is currently compatible with Elasticsearch 7.x, we recommend installing the latest available 7.x release.
Cassandra¶
You can use Cassandra as a metrics store instead of Elasticsearch for better performance. Elasticsearch is still required in conjunction with the dedicated AxonOps Cassandra cluster, Elasticsearch is used to store all the AxonOps configuration, logs data and metrics metadata.
For more information on setting up Cassandra as a metric store: Cassandra as Metrics Database
Installing Elasticsearch¶
Select the OS Family.¶
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.16-amd64.deb
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.16-amd64.deb.sha512
shasum -a 512 -c elasticsearch-7.17.16-amd64.deb.sha512
sudo dpkg -i elasticsearch-7.17.16-amd64.deb
elasticsearch-7.17.16-amd64.deb: OK
Increase the bulk queue size of Elasticsearch by running the following command:
sudo echo 'thread_pool.write.queue_size: 2000' >> /etc/elasticsearch/elasticsearch.yml
Increase the default heap size of elasticsearch by editing /etc/elasticsearch/jvm.options.
Set Xmx and Xms to no more than 50% of your physical RAM.
Example:
From:
-Xms1g
-Xmx1g
-Xms8g
-Xmx8g
Elasticsearch requires memory for purposes other than the JVM heap and it is important to leave available memory(RAM) space for this.
Set the following index codec by running the following command:
sudo echo 'index.codec: best_compression' >> /etc/elasticsearch/elasticsearch.yml
Elasticsearch uses an mmapfs directory by default to store its indices.
The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.
You can increase the limits by running the following command:
sudo sysctl -w vm.max_map_count=262144
To make this change persist across reboots run this command:
echo "vm.max_map_count = 262144" | sudo tee /etc/sysctl.d/10-elasticsearch.conf > /dev/null
Elasticsearch needs max file descriptors system settings at least to 65536.
echo 'elasticsearch - nofile 65536' | sudo tee --append /etc/security/limits.conf > /dev/null
Start Elasticsearch¶
sudo systemctl start elasticsearch.service
After a short period of time, you can verify that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:
curl "localhost:9200"
Securing Elasticsearch¶
Step 2 - axon-server installation¶
Execute the following command to setup the AxonOps repository for your OS using the built in Package manager.
sudo apt-get update
sudo apt-get install -y curl gnupg ca-certificates
curl -L https://packages.axonops.com/apt/repo-signing-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/axonops.gpg
echo "deb [arch=arm64,amd64 signed-by=/usr/share/keyrings/axonops.gpg] https://packages.axonops.com/apt axonops-apt main" | sudo tee /etc/apt/sources.list.d/axonops-apt.list
sudo apt-get update
sudo apt-get install axon-server
Step 3 - axon-server configurations¶
Make sure elastic_host and elastic_port are corresponding to your Elasticsearch instance.
Basic Auth in Elasticsearch
- Create a user that has a dedicated role and username password.
- Please dont use any of the built in users for Elasticsearch.
To create users please refer to the Elasticsearch docs here
AxonOps Server configuration file location : /etc/axonops/axon-server.yml
host: 0.0.0.0 # axon-server listening address (used by axon-agents for connections) (env variable: AXONSERVER_HOST)
agents_port: 1888 # axon-server listening port for agent connections
api_host: 127.0.0.1 # axon-server listening address (used by axon-dash for connections)
api_port: 8080 # axon-server HTTP API listening port (used by axon-dash) (AXONSERVER_PORT)
elastic_hosts: # Elasticsearch endpoint (env variable:ELASTIC_HOSTS, comma separated list)
- http://localhost:9200
# Configure multiple Elasticsearch hosts with username and password authentication
# elastic_hosts:
# - https://username:[email protected]:port
# - https://username:[email protected]:port
# - https://username:[email protected]:port
# SSL/TLS config for Elasticsearch
# elastic_skipVerify: true # Disables CA and Hostname verification
# Configure the number of shards per index. The default value of 1 is recommended for most use cases
elastic_shards: 1
# Configure the number of replicas per shard. Defaults to 0 if not specified.
elastic_replicas: 0
# Enable/disable Elasticsearch cluster discovery (sniffing). Defaults to true, set to false to disable
# Allows more nodes to be added to Elasticsearch for Metrics storage without having to restart Axon-Server and update elastic_hosts with all the ELK node values.
elastic_discover_nodes: true
# How often to perform cluster discovery. Default is every 5 minutes if this is omitted
elastic_discover_node_interval: 5m
#integrations_proxy: # proxy endpoint for integrations. (INTEGRATIONS_PROXY)
# AxonOps licensing
license_key: license-key
org_name: my-company
# SSL/TLS Settings for AxonOps Agent connections
tls:
mode: "disabled" # disabled, TLS
# Only set below if mode is TLS
skipVerify: false # Disables CA and Hostname verification
caFile: "path_to_certs_on_axonops_server.crt" # required if skipVerify is not set and you are using a self-signed cert
certFile: "path_to_certs_on_axonops_server.crt"
keyFile: "path_to_key_file_on_axonops_server.key"
# For better performance on large clusters, you can use a CQL store for the metrics.
# To opt-in for CQL metrics storage, just specify at least one CQL host.
# We do recommend to specify a NetworkTopologyStrategy for cql_keyspace_replication
#cql_hosts: # (CQL_HOSTS, comma separated list)
# - 192.168.0.10:9042
# - 192.168.0.11:9042
#cql_username: "cassandra" # (CQL_USERNAME)
#cql_password: "cassandra" # (CQL_PASSWORD)
#cql_local_dc: datacenter1 # (CQL_LOCAL_DC)
#cql_ssl: false # (CQL_SSL)
#cql_skip_verify: false # (CQL_SSL_SKIP_VERIFY)
#cql_ca_file: /path/to/ca_file # (CQL_CA_FILE)
#cql_cert_file: /path/to/cert_file # (CQL_CERT_FILE)
#cql_key_file: /path/to/key_file # (CQL_KEY_FILE)
#cql_proto_version: 4 # (CQL_PROTO_VERSION)
#cql_max_concurrent_reads: 1000 # (CQL_MAX_CONCURRENT_READS)
#cql_batch_size: 1 # (CQL_BATCH_SIZE)
#cql_page_size: 10 # (CQL_PAGE_SIZE)
#cql_autocreate_tables: true # (CQL_AUTO_CREATE_TABLES) this will tell axon-server to automatically create the metrics tables (true is recommended)
#cql_keyspace_replication: "{ 'class' : 'SimpleStrategy', 'replication_factor' : 1 }" # (CQL_KS_REPLICATION) keyspace replication for the metrics tables
#cql_retrypolicy_numretries: 3 # (CQL_RETRY_POLICY_NUM_RETRIES)
#cql_retrypolicy_min: 1s # (CQL_RETRY_POLICY_MIN)
#cql_retrypolicy_max: 10s # (CQL_RETRY_POLICY_MAX)
#cql_reconnectionpolicy_maxretries: 10 # (CQL_RECONNECTION_POLICY_MAX_RETRIES)
#cql_reconnectionpolicy_initialinterval: 1s # (CQL_RECONNECTION_POLICY_INITIAL_INTERVAL)
#cql_reconnectionpolicy_maxinterval: 10s # (CQL_RECONNECTION_POLICY_MAX_INTERVAL)
#cql_metrics_cache_max_size_mb: 100 #MB # (CQL_METRICS_CACHE_MAX_SIZE_MB)
#cql_read_consistency: "LOCAL_ONE" # (CQL_READ_CONSISTENCY) #One of the following: ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, LOCAL_ONE
#cql_write_consistency: "LOCAL_ONE" # (CQL_WRITE_CONSISTENCY) #One of the following: ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, LOCAL_ONE
#cql_lvl1_compaction_window_size: 12 # (CQL_LVL1_COMPACTION_WINDOW_SIZE)
#cql_lvl2_compaction_window_size: 1 # (CQL_LVL2_COMPACTION_WINDOW_SIZE)
#cql_lvl3_compaction_window_size: 1 # (CQL_LVL3_COMPACTION_WINDOW_SIZE)
#cql_lvl4_compaction_window_size: 10 # (CQL_LVL4_COMPACTION_WINDOW_SIZE)
#cql_lvl5_compaction_window_size: 120 # (CQL_LVL5_COMPACTION_WINDOW_SIZE)
axon-dash: # This must point to the axon-dash address accessible from axon-server
host: 127.0.0.1
port: 3000
https: false
alerting:
# How long to wait before sending a notification again if it has already
# been sent successfully for an alert. (Usually ~3h or more).
notification_interval: 3h
# Default retention settings, most can be overridden from the frontend
retention:
events: 8w # logs and events retention. Must be expressed in weeks (w)
metrics:
high_resolution: 14d # High frequency metrics. Must be expressed in days (d)
med_resolution: 12w # Must be expressed in weeks (w)
low_resolution: 12M # Must be expressed in months (M)
super_low_resolution: 2y # Must be expressed in years (y)
backups: # Those are use as defaults but can be overridden from the UI
local: 10d
remote: 30d
# Storage options for PDF reports
# Override the default local path of /var/lib/axonops/reports
#report_storage_path: /my/reports/storage/directory
# Alternatively store PDF reports in an object store by providing report_storage_config
#report_storage_path: my-reports-s3-bucket/reports-folder
#report_storage_config:
# type: s3
# provider: AWS
# access_key_id: MY_ACCESS_KEY_ID
# secret_access_key: MY_SECRET_ACCESS_KEY
# region: us-east-1
# acl: private
# server_side_encryption: AES256
# storage_class: STANDARD
For better performances on large clusters (100+ nodes), you can use a CQL store for the metrics such as Cassandra. To opt-in for CQL metrics storage, specify at least one CQL host with axon-server configuration.
Step 4 - Start the server¶
sudo systemctl daemon-reload
sudo systemctl start axon-server
sudo systemctl status axon-server
This will start the axon-server process as the axonops user, which was created during the package installation. The default listening address is 0.0.0.0:8080.
Package details¶
- Configuration:
/etc/axonops/axon-server.yml - Binary:
/usr/share/axonops/axon-server - Logs:
/var/log/axonops/axon-server.log - Systemd service:
/usr/lib/systemd/system/axon-server.service - Copyright :
/usr/share/doc/axonops/axon-server/copyright - Licenses :
/usr/share/axonops/licenses/axon-server/
Step 5 - Installing axon-dash¶
Now axon-server is installed, you can start installing the GUI for it: axon-dash