sensu-dynamic-check-plugin

Table of Contents

• Overview
• Usage
• Configuration
  • kubectl as asset
  • http-nginx check example
  • Json details
• Asset registration
• Mutator definition
• Installation from source
• Additional notes
• Contributing

Overview

The sensu-dynamic-check-plugin is a Sensu Mutator that parse labels from sensu events and creates a dynamic check in Sensu Backend and add event.check.annotation io.sensu.remediation.config.actions with these values then it can be scheduled by sensu-remediation-handler

Usage

Sensu Dynamic Check Mutator creates sensu check based on template

Usage:
  sensu-dynamic-check-mutator [flags]
  sensu-dynamic-check-mutator [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -B, --api-backend-host string                  Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -k, --api-backend-key string                   Sensu Go Backend API Key
  -P, --api-backend-pass string                  Sensu Go Backend API Password (default "P@ssw0rd!")
  -p, --api-backend-port int                     Sensu Go Backend API Port (e.g. 4242) (default 8080)
  -u, --api-backend-user string                  Sensu Go Backend API User (default "admin")
  -c, --check-config string                      Json template for Sensu Check
      --command-arguments-template string        Template for Sensu Check Command (default "{{ range $key, $value := . }} {{ $key }} {{ $value }}{{ end }}")
      --command-bool-arguments-template string   Template for Sensu Check Command (default "{{ range $value := . }} {{ $value }}{{ end }}")
      --command-handler string                   Handler used to post the result (default "default")
      --default-check-suffix-name string         Default suffix name for unpublished checks (default "dynamic")
  -h, --help                                     help for sensu-dynamic-check-mutator
  -i, --insecure-skip-verify                     skip TLS certificate verification (not recommended!)
      --remediation-event-annotation string      Add an annotation in dynamic check created like remediation-event-alias: event.Entity.Name/event.Check.Name (default "remediation-event-alias")
  -s, --secure                                   Use TLS connection to API
  -t, --trusted-ca-file string                   TLS CA certificate bundle in PEM format

Use "sensu-dynamic-check-mutator [command] --help" for more information about a command.

Configuration

We add a json inside --check-config:

[
  {
    "name": "describe-resource",
    "command": "${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
    "bool_args": [
      "--no-headers"
    ],
    "arguments": ["daemonset","deployment","pod","statefulset"],
    "options": {
        "--namespace": "namespace"
    },
    "match_labels": {
        "sensu-alertmanager-events": "owner"
    },
    "exclude_labels": [
      {
        "alertname": "TargetDown"
      },
      {
        "alertname": "KubeVersionMismatch"
      }
    ],
    "sensu_assets": [
        "kubectl"
    ],
    "occurrences": [1],
    "severities": [2],
  },
  {
    "name": "systemctl-status",
    "command": "sudo systemctl",
    "options": {
        "status": "application"
    },
    "match_labels": {
        "systemd": "true"
    },
    "occurrences": [1]
  },
  {
    "name": "systemctl-restart",
    "command": "sudo systemctl",
    "options": {
        "restart": "application"
    },
    "match_labels": {
        "systemd": "true"
    },
    "occurrences": [3]
  }
]

In this example, to change the event, this mutator need to find a label called namespace, and need to find at least one of the arguments array, like label deployment. Then it will create a check.command: ${{assetPath "kubectl"}}/kubernetes/client/bin/kubectl describe --namespace default deployment nginx.

And it will create one annotation like:

"io.sensu.remediation.config.actions": "[{\"request\":\"KubeDeploymentReplicasMismatch-default-nginx-describe-resource-dynamic\",\"occurrences\":[1],\"severities\":[2],\"subscriptions\":[\"entity:k8s.dev.local\"]}]"

And one check called: KubeDeploymentReplicasMismatch-default-nginx-describe-resource-dynamic with command ${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe --namespace default daemonset nginx.

In systemctl-status and systemctl-restart if this mutator found two labels systemd:true and application:nginx as example for a check called http-nginx, it will create two checks in Sensu Backend called http-nginx-systemctl-status-dynamic and http-nginx-systemctl-restart-dynamic both running a sudo systemctl [status|restart] nginx command and it will add the following annotation:

"io.sensu.remediation.config.actions": "[{\"request\":\"http-nginx-systemctl-status-dynamic\",\"occurrences\":[1],\"severities\":[2],\"subscriptions\":[\"entity:systemd-ubuntu\"]},{\"request\":\"http-nginx-systemctl-restart-dynamic\",\"occurrences\":[3],\"severities\":[2],\"subscriptions\":[\"entity:systemd-ubuntu\"]}]"

kubectl as asset

In these example we use one event imported by sensu-alertmanager-events and we installed kubectl using assets.

type: Asset
api_version: core/v2
metadata:
  name: kubectl
  namespace: default
spec:
  sha512:  081472833601aa4fa78e79239f67833aa4efcb4efe714426cd01d4ddf6f36fbf304ef7e1f5373bff0fdff44a845f7560165c093c108bd359b5ab4189f36b1f2f
  url: https://dl.k8s.io/v1.20.0/kubernetes-client-linux-amd64.tar.gz

http-nginx

type: Check
api_version: core/v2
metadata:
  name: http-nginx
  namespace: default
spec:
  command: check-http.rb -u http://127.0.0.1 -t 5
  handlers:
  - default
  - remediation
  interval: 60
  publish: true
  runtime_assets:
  - sensu-ruby-runtime
  - sensu-plugins-http
  subscriptions:
  - ubuntu

Json details

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please
consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the
following command to add the asset:

sensuctl asset add betorvs/sensu-dynamic-check-mutator

If you're using an earlier version of sensuctl, you can find the asset on the [Bonsai Asset Index][https://bonsai.sensu.io/assets/betorvs/sensu-dynamic-check-mutator].

Mutator definition

Maybe is important to add authetication configs -u dynamic -P ${MUTATOR_PASS} -B sensu-api.example.com -s -t /$PATH_TO_CERTIFICATE/ca.pem

---
type: Mutator
api_version: core/v2
metadata:
  name: sensu-dynamic-check-mutator
spec:
  command: >-
    sensu-dynamic-check-mutator -P ${SENSU_API_PASS} -B sensu-api.example.com -s -t /certs/sensu-ca.pem 
    --remediation-event-annotation "sensu.io/plugins/sensu-opsgenie-handler/config/remediation-event-alias"
    -c '[
      { "name": "describe-resource",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
        "arguments":["daemonset","deployment","pod","statefulset"],
        "options":{"--namespace":"namespace"},
        "match_labels":{"sensu-alertmanager-events":"owner"},
        "exclude_labels":[
          {"alertname":"TargetDown"},
        ],
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[1,2],
        "severities":[2]},
      { "name": "describe-node",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
        "arguments":["node"],
        "match_labels":{"sensu-alertmanager-events":"owner","alertname":"KubeNodeUnreachable","alertname":"KubeNodeNotReady"},
        "exclude_labels":[
          {"alertname":"TargetDown"},
        ],
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[1,2],
        "severities":[2]}
      {"name":"pod-termination-stuck",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl delete ",
        "bool_options":["--grace-period=0","--force"],
        "arguments":["pod"],
        "options":{"--namespace":"namespace"},
        "match_labels":{"alertname":"PodTerminationStuck"},
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[3,4],
        "severities":[2]}
    ]'
  runtime_assets:
  - sensu-dynamic-check-mutator

Using sensu-dynamic-check-mutator with --remediation-event-annotation and sensu-opsgenie-handler as a opsgenie_remediation handler we can add remediation action inside the same alert in opsgenie as extra properties and as note.

Example of remediation handler:

---
type: Handler
api_version: core/v2
metadata:
  name: remediation
spec:
  type: pipe
  command: sensu-remediation-handler
  timeout: 10
  mutator: "sensu-dynamic-check-mutator"
  runtime_assets:
  - sensu-remediation-handler
  env_vars:
  - "SENSU_API_URL=https://sensu-api.sensu.svc.cluster.local:8080"
  - "SENSU_API_CERT_FILE=/etc/sensu/tls/ca.pem"

Installation from source

The preferred way of installing and deploying this plugin is to use it as an Asset. If you would
like to compile and install the plugin from source or contribute to it, download the latest version
or create an executable script from this source.

From the local path of the sensu-dynamic-check-mutator repository:

go build

Additional notes

Contributing

For more information about contributing to this plugin, see Contributing.