Sensu Go Elasticsearch Handler

Table of Contents

• Overview
  • Payload
  • Metrics handling
  • Visualizing metrics
• Usage examples
• Configuration
  • Asset registration
  • Handler definition
  • Proxy support
  • Annotations

Overview

sensu-elasticsearch-handler is a Sensu Handler for sending Sensu events
and metrics to Elasticsearch for indexing and/or visualization in Kibana.

Payload

Regargless of whether this handler is used for normal events processing
(specified in a check's handlers list) or for metrics (specified in a check's
output_metric_handlers list) the full event payload is sent to Elasticsearch
to be indexed and the event's top-level timestamp is used as the index's
timestamp.

If the --omit-redundant-fields argument is provided, the following event
attributes are dropped from the event payload prior to being sent to
Elasticsearch for indexing:

• event.entity.redact
• event.check.history
• event.check.runtimeAssets
• event.entity.system.network.interfaces

Metrics handling

When this handler is used as a metrics handler and the event contains metrics,
the metrics points are mutated by the handler and made into a top-level
object named measurements with each entry within it being a key/value pair of
the metric point name and its associated value. The top-level metrics
object is dropped and not sent to Elasticsearch.

Also, any tags for a metrics point are mutated into labels in the event
metadata. This allows for them to still be queried from within Elasticsearch.

The snippets below demonstrate these mutations.

Before

"metadata": {
  "namespace": "production",
},
"metrics": {
  "handlers": [
    "elasticsearch"
  ],
  "points": [
    {
      "name": "sensu-go-sandbox.curl_timings.time_total",
      "tags": [
        {
          "name": "site",
          "value": "https://sensu.io"
        },
        {
          "name": "environment",
          "value": "production"
        }
      ],
      "timestamp": 1552506033,
      "value": 0.005
    },
    {
      "name": "sensu-go-sandbox.curl_timings.time_namelookup",
      "tags": [
        {
          "name": "site",
          "value": "https://sensu.io"
        },
        {
          "name": "environment",
          "value": "production"
        }
      ],
      "timestamp": 1552506033,
      "value": 0.004
    }
  ]
}

After

"metadata": {
  "namespace": "production",
  "labels": {
    "sensu-go-sandbox.curl_timings.time_total.site": "https://sensu.io",
    "sensu-go-sandbox.curl_timings.time_total.environment": "production",
    "sensu-go-sandbox.curl_timings.time_namelookup.site": "https://sensu.io",
    "sensu-go-sandbox.curl_timings.time_namelookup.environment": "production"
  }
},
"measurements": {
  "sensu-go-sandbox.curl_timings.time_total": 0.005,
  "sensu-go-sandbox.curl_timings.time_namelookup": 0.004
}

Visualizing metrics

With the metrics payload in the above format, they can be easily visualized in
Kibana. In the following example I will use a metric from http-perf command
(in the http-checks asset) to graph the total request duration
for two sites being monitored. In this case, the events for each site have a
label in the check metatadata that will be used to filter the metrics.

Usage Examples

Help:

The Sensu Go handler for event/metric logging in Elasticsearch

Usage:
  sensu-elasticsearch-handler [flags]
  sensu-elasticsearch-handler [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -u, --url strings              URL(s) for Elasticsearch server(s), accepts multiple iterations or comma separated list
  -i, --index string             Index to be used for Sensu events (default "sensu_events")
  -r, --index-rotation string    Postfix index with current date based on rotation schedule (e.g. daily=index-2020-08-20, weekly=index-2020-W34, monthly=index-2020-08, yearly=index-2020)
  -U, --username string          Username, if required, to authenticate requests to Elasticsearch
  -P, --password string          Password, if required, to authenticate requests to Elasticsearch
  -o, --omit-redundant-fields    Omit certain fields from the event before sending to Elasticsearch for indexing
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -s, --insecure-skip-verify     Skip TLS certificate verification (not recommended!)
  -h, --help                     help for sensu-elasticsearch-handler

Use "sensu-elasticsearch-handler [command] --help" for more information about a command.

Configuration

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please
consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the
following command to add the asset:

sensuctl asset add sensu/sensu-elasticsearch-handler

If you're using an earlier version of sensuctl, you can find the asset on the Bonsai Asset Index

Handler definition

---
type: Handler
api_version: core/v2
metadata:
  name: elasticsearch
  namespace: default
spec:
  command: sensu-elasticsearch-handler --index-rotation daily
  type: pipe
  runtime_assets:
    - sensu/sensu-elasticsearch-handler
  secrets:
    - name: ELASTICSEARCH_URL
      secret: elasticsearch_url
    - name: ELASTICSEARCH_USERNAME
      secret: elasticsearch_username
    - name: ELASTICSEARCH_PASSWORD
      secret: elasticsearch_password
  filters:
    - has_metrics

Note: The handler definition above assumes it is being defined as a metrics
handler. It uses the has_metrics filter to only send events that contain
metrics. If you want all events, remove this filter and make sure it is
being used as a regular handler, not a metrics handler.

Environment Variables

Many arguments for this handler are available to be set via environment
variables. However, any arguments specified directly on the command line
override the corresponding environment variable.

Given that the username/password may be included in the URL
(e.g. http://user:pass@localhost:9200),it is suggested to make use of
secrets management to surface it. The handler definition
above references it as a secret. Below is an example secret definition that
makes use of the built-in env secrets provider.

---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_url
spec:
  provider: env
  id: ELASTICSEARCH_URL

If you are specifying the username and password outside of the URL, you should
make use of secrets management to surface them as well. Below are example
secret definitions.

---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_username
spec:
  provider: env
  id: ELASTICSEARCH_USERNAME
---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_password
spec:
  provider: env
  id: ELASTICSEARCH_PASSWORD

Proxy support

This handler supports the use of the environment variables HTTP_PROXY,
HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). HTTPS_PROXY takes
precedence over HTTP_PROXY for https requests. The environment values may be
either a complete URL or a "host[:port]", in which case the "http" scheme is assumed.

Annotations

All arguments for this handler are tunable on a per entity or check basis based on annotations. The
annotations keyspace for this handler is sensu.io/plugins/sensu-elasticsearch-handler/config.

Examples

To change the index argument for a particular entity, in that entity's agent.yml add the following:

[...]
annotations:
  sensu.io/plugins/sensu-elasticsearch-handler/config/index: "dev_index"
[...]