Select Supported Platforms

Select Supported Architectures

Select Tier

Github Name


README
Asset Definition
Release Notes
Release Assets

Sensu Bonsai Asset
Go Test
goreleaser

sensu-grafana-mutator

Table of Contents

Overview

The sensu-grafana-mutator is a Sensu Mutator created to parse event labels and generate one or more event.check.annotations ending in _url with a time range to make sysadmin's life easier to start his troubleshooting. Very basic usage is to parse label and match a Grafana Dashboard with these labels.
It contains 2 builtin integrations:

Usage

Sensu grafana mutator add Grafana Dashboards or Grafana Explore Links in event annotations

Usage:
  sensu-grafana-mutator [flags]
  sensu-grafana-mutator [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -a, --alertmanager-events-integration             Grafana Mutator parser for sensu-alertmanager-events plugin
  -A, --alertmanager-integration-label string       Label used to identify sensu-alertmanager-events plugin events (default "sensu-alertmanager-events")
      --default-loki-label-hostname string          Default hostname label for Grafana Loki Stream. {hostname=value} (default "hostname")
      --default-loki-label-namespace string         Default namespace label for Grafana Loki Stream. {namespace=value} (default "namespace")
  -d, --grafana-dashboard-suggested string          Suggested Dashboard based on Labels (json format). e. [{"grafana_annotation":"kubernetes_namespace","dashboard_url":"https://grafana.example.com/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=thanos","labels":["namespace"]}]
  -e, --grafana-explore-link-enabled                Enable Grafana Loki Explore Links
  -D, --grafana-loki-datasource string              An Grafana Loki Datasource name. e. -d loki  (default "loki")
  -r, --grafana-mutator-time-range int              Time range in seconds to create grafana URLs (default 300)
  -g, --grafana-url string                          An grafana complete URL. e. https://grafana.com/?orgId=1 
  -h, --help                                        help for sensu-grafana-mutator
  -k, --kubernetes-events-integration               Grafana Mutator parser for sensu-kubernetes-events plugin
  -P, --kubernetes-events-pipeline string           Grafana Loki pipeline to match. e. {app=eventrouter} |= io.kubernetes.event.id (default "io.kubernetes.event.id")
  -L, --kubernetes-events-stream-label string       Grafana Loki stream label. e. {app=eventrouter} (default "app")
  -N, --kubernetes-events-stream-namespace string   Grafana Loki stream namespace. e. {app=eventrouter,namespace=io.kubernetes.event.namespace} (default "io.kubernetes.event.namespace")
  -S, --kubernetes-events-stream-selector string    Grafana Loki stream selector. e. {app=eventrouter} (default "eventrouter")
  -s, --sensu-label-selector string                 Sensu Label Selector to create Grafana Explore URL using loki as Datasource. {namespace=kubernetes_namespace.value} (default "kubernetes_namespace")

Use "sensu-grafana-mutator [command] --help" for more information about a command.


Configuration

Basic usage sensu-grafana-mutator should be:

cat event.json | ./sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e

Output annotation: event.check.annotations["grafana_loki_url"].

To change sensu label selector, use:

cat event.json | ./sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e -s namespace

Requirements

You should have Grafana installed and configured. If you want to use --grafana-explore-link-enabled you should have a Grafana Loki installed and receiving logs.

sensu-kubernetes-events

Using Grafana Explore tab and Grafana Loki as datasource:

  • Search as example: {app="eventrouter",namespace="default"}|= "nginx-deployment-78dc4549b8-kkxnf.164c27e81b96bdc8" where "nginx-deployment-78dc4549b8-kkxnf.164c27e81b96bdc8" cames from the value from sensu event.label["io.kubernetes.event.id"].

Then sensu-grafana-mutator should be:

cat event.json | ./sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e -k

Output annotation: event.check.annotations["grafana_loki_url"].

sensu-alertmanager-events

It will try to find the label in event.check.Label with name sensu-alertmanager-events and value owner then it will create a grafana loki URL using only namespace in stream. Example: {namespace="Value"}. Only change --alertmanager-integration-label if the sensu-alertmanager-events plugin changed it.

Then sensu-grafana-mutator should be:

cat event.json | ./sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e -a

Output annotation: event.check.annotations["grafana_loki_url"].

grafana-dashboard-suggested

You can include multiples grafana_annotations inside this flag. But we don't have a benchmark about it. Then keep it simple and it will work as expected. We used one example dashboard from kubernetes-mixin called kubernetes-compute-resources-namespace-pods.

[
  {
    "grafana_annotation": "kubernetes_namespace",
    "dashboard_url": "https://grafana.example.com/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=thanos",
    "labels": [
      "namespace",
      "cluster"
    ]
  }
]

But in Sensu yaml configuration should be in one line with scapes:

cat event.json | ./sensu-grafana-mutator -d "[{\"grafana_annotation\":\"kubernetes_namespace\",\"dashboard_url\":\"https://grafana.example.com/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=thanos\",\"labels\":[\"namespace\",\"cluster\"]}]"

Output annotation: event.check.annotations["grafana_kubernetes_namespace_url"].

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please
consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the
following command to add the asset:

sensuctl asset add betorvs/sensu-grafana-mutator

If you're using an earlier version of sensuctl, you can find the asset on the [Bonsai Asset Index][https://bonsai.sensu.io/assets/betorvs/sensu-grafana-mutator].

Mutator definition

Basic usage will parse sensu label kubernetes_namespace as namespace in Grafana Loki Explore URL.

---
type: Mutator
api_version: core/v2
metadata:
  name: sensu-grafana-mutator
  namespace: default
spec:
  command: sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e
  runtime_assets:
  - betorvs/sensu-grafana-mutator

Full example

Will parse sensu events with label kubernetes_namespace and events from these plugins sensu-kubernetes-events and sensu-alertmanager-events and if found labels namespace and cluster will add a dashboard kubernetes-compute-resources-namespace-pods link and if found labels node and cluster will add dashboard kubernetes-compute-resources-node-pods link.

---
type: Mutator
api_version: core/v2
metadata:
  name: sensu-grafana-mutator
  namespace: default
spec:
  command: >-
    sensu-grafana-mutator -g https://grafana.example.com/?orgId=1 -e -k -a
    -d "[{\"grafana_annotation\":\"kubernetes_namespace\",\"dashboard_url\":\"https://grafana.example.com/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=thanos\",\"labels\":[\"namespace\",\"cluster\"]},{\"grafana_annotation\":\"kubernetes_nodes\",\"dashboard_url\":\"https://grafana.example.com/d/200ac8fdbfbb74b39aff88118e4d1c2c/kubernetes-compute-resources-node-pods?orgId=1&var-datasource=thanos\",\"labels\":[\"node\",\"cluster\"]}]"
  runtime_assets:
  - betorvs/sensu-grafana-mutator

Installation from source

The preferred way of installing and deploying this plugin is to use it as an Asset. If you would
like to compile and install the plugin from source or contribute to it, download the latest version
or create an executable script from this source.

From the local path of the sensu-grafana-mutator repository:

go build

Additional notes

This mutator was design to work with handlers that can parse these grafana annotations as link. Example sensu-opsgenie-handler and sensu-hangouts-chat-handler.

Contributing

For more information about contributing to this plugin, see Contributing.

Are you sure you want to report this asset?

Please describe the reason for reporting this asset. Our moderators will be notified and will disable the asset if it is found to be inappropriate.

×

You must be signed in to report this asset.

Sign In with Github

Download

×

Either download the source code:

Download Source

Or download the asset definition: