Sensu Go Elasticsearch Handler

Table of Contents

• Overview
  • Payload
  • Metrics handling
  • Visualizing metrics
• Usage examples
  • Help output
  • Index rotation
• Configuration
  • Asset registration
  • Handler definition
  • Proxy support
  • Annotations

Overview

sensu-elasticsearch-handler is a Sensu Handler for sending Sensu events
and metrics to Elasticsearch for indexing and/or visualization in Kibana.

Payload

Regargless of whether this handler is used for normal events processing
(specified in a check's handlers list) or for metrics (specified in a check's
output_metric_handlers list) the full event payload is sent to Elasticsearch
to be indexed and the event's top-level timestamp is used as the index's
timestamp.

If the --omit-redundant-fields argument is provided, the following event
attributes are dropped from the event payload prior to being sent to
Elasticsearch for indexing:

• event.entity.redact
• event.check.history
• event.check.runtimeAssets
• event.entity.system.network.interfaces

Metrics handling

When this handler is used as a metrics handler and the event contains metrics,
the metrics points are mutated by the handler and made into a top-level
object named measurements with each entry within it being a key/value pair of
the metric point name and its associated value. The top-level metrics
object is dropped and not sent to Elasticsearch.

Also, any tags for a metrics point are mutated into labels in the event
metadata. This allows for them to still be queried from within Elasticsearch.

The snippets below demonstrate these mutations.

Before

"metadata": {
  "namespace": "production",
},
"metrics": {
  "handlers": [
    "elasticsearch"
  ],
  "points": [
    {
      "name": "sensu-go-sandbox.curl_timings.time_total",
      "tags": [
        {
          "name": "site",
          "value": "https://sensu.io"
        },
        {
          "name": "environment",
          "value": "production"
        }
      ],
      "timestamp": 1552506033,
      "value": 0.005
    },
    {
      "name": "sensu-go-sandbox.curl_timings.time_namelookup",
      "tags": [
        {
          "name": "site",
          "value": "https://sensu.io"
        },
        {
          "name": "environment",
          "value": "production"
        }
      ],
      "timestamp": 1552506033,
      "value": 0.004
    }
  ]
}

After

"metadata": {
  "namespace": "production",
  "labels": {
    "sensu-go-sandbox.curl_timings.time_total.site": "https://sensu.io",
    "sensu-go-sandbox.curl_timings.time_total.environment": "production",
    "sensu-go-sandbox.curl_timings.time_namelookup.site": "https://sensu.io",
    "sensu-go-sandbox.curl_timings.time_namelookup.environment": "production"
  }
},
"measurements": {
  "sensu-go-sandbox.curl_timings.time_total": 0.005,
  "sensu-go-sandbox.curl_timings.time_namelookup": 0.004
}

Visualizing metrics

With the metrics payload in the above format, they can be easily visualized in
Kibana. In the following example I will use a metric from http-perf command
(in the http-checks asset) to graph the total request duration
for two sites being monitored. In this case, the events for each site have a
label in the check metatadata that will be used to filter the metrics.

Usage Examples

Help output

The Sensu Go handler for event/metric logging in Elasticsearch

Usage:
  sensu-elasticsearch-handler [flags]
  sensu-elasticsearch-handler [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -u, --url strings                         URL(s) for Elasticsearch server(s), accepts multiple iterations or comma separated list
  -i, --index string                        Index to be used for Sensu events (default "sensu_events")
  -r, --index-rotation string               Postfix index with current date based on rotation schedule (e.g. daily=index-2020-08-20, weekly=index-2020-W34, monthly=index-2020-08, yearly=index-2020)
  -f, --index-rotation-date-format string   Format for date appended with index rotation, YYYY for year, MM for month, DD for day, and WW for week (e.g. "YYYY.WW" would yield 2020.45)
  -U, --username string                     Username, if required, to authenticate requests to Elasticsearch
  -P, --password string                     Password, if required, to authenticate requests to Elasticsearch
  -S, --strip-hostname                      Strip the hostname/entity name from the metrics point names as inserted by certain metrics checks (e.g. hostname.example.com.cpu_idle becomes cpu_idle)
  -T, --truncate-check-output int           Truncate the check output to this length, if zero (0) do not truncate
  -o, --omit-redundant-fields               Omit certain fields from the event before sending to Elasticsearch for indexing
  -t, --trusted-ca-file string              TLS CA certificate bundle in PEM format
  -s, --insecure-skip-verify                Skip TLS certificate verification (not recommended!)
  -h, --help                                help for sensu-elasticsearch-handler

Use "sensu-elasticsearch-handler [command] --help" for more information about a command.

Index rotation

This handler supports specifying the index rotation scheme on a daily, weekly,
monthly, or yearly basis. By default the index name (--index) is suffixed
with the appropriate date for the rotation schemem in ISO 8601 format. Assuming
the default index name of "sensu_events" and the date of November 6, 2020, this
is what the index would look like for each rotation.

The option exists to change the date formatting using
--index-rotation-date-format. The formatting options are YYYY for the year,
WW for the week number, MM for the month, and DD for the day. Here are some
examples using the index name and date from above.

Caveats:

• Shortened formats are not supported (e.g. using YY will not yield the
  two-digit year).
• For weekly, you cannot place the week before the year. You can change
  the order for any other rotation period.

Configuration

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please
consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the
following command to add the asset:

sensuctl asset add sensu/sensu-elasticsearch-handler

If you're using an earlier version of sensuctl, you can find the asset on the Bonsai Asset Index

Handler definition

---
type: Handler
api_version: core/v2
metadata:
  name: elasticsearch
  namespace: default
spec:
  command: sensu-elasticsearch-handler --index-rotation daily
  type: pipe
  runtime_assets:
    - sensu/sensu-elasticsearch-handler
  secrets:
    - name: ELASTICSEARCH_URL
      secret: elasticsearch_url
    - name: ELASTICSEARCH_USERNAME
      secret: elasticsearch_username
    - name: ELASTICSEARCH_PASSWORD
      secret: elasticsearch_password
  filters:
    - has_metrics

Note: The handler definition above assumes it is being defined as a metrics
handler. It uses the has_metrics filter to only send events that contain
metrics. If you want all events, remove this filter and make sure it is
being used as a regular handler, not a metrics handler.

Environment Variables

Many arguments for this handler are available to be set via environment
variables. However, any arguments specified directly on the command line
override the corresponding environment variable.

Given that the username/password may be included in the URL
(e.g. http://user:pass@localhost:9200),it is suggested to make use of
secrets management to surface it. The handler definition
above references it as a secret. Below is an example secret definition that
makes use of the built-in env secrets provider.

---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_url
spec:
  provider: env
  id: ELASTICSEARCH_URL

If you are specifying the username and password outside of the URL, you should
make use of secrets management to surface them as well. Below are example
secret definitions.

---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_username
spec:
  provider: env
  id: ELASTICSEARCH_USERNAME
---
type: Secret
api_version: secrets/v1
metadata:
  name: elasticsearch_password
spec:
  provider: env
  id: ELASTICSEARCH_PASSWORD

Proxy support

This handler supports the use of the environment variables HTTP_PROXY,
HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). HTTPS_PROXY takes
precedence over HTTP_PROXY for https requests. The environment values may be
either a complete URL or a "host[:port]", in which case the "http" scheme is assumed.

Annotations

All arguments for this handler are tunable on a per entity or check basis based on annotations. The
annotations keyspace for this handler is sensu.io/plugins/sensu-elasticsearch-handler/config.

Examples

To change the index argument for a particular entity, in that entity's agent.yml add the following:

[...]
annotations:
  sensu.io/plugins/sensu-elasticsearch-handler/config/index: "dev_index"
[...]