Select Supported Platforms

Select Supported Architectures

Select Tier

Github Name


README
Asset Definition
Release Notes
Release Assets

sensu-dynamic-check-plugin

Sensu Bonsai Asset
Go Test
goreleaser

Table of Contents

Overview

The sensu-dynamic-check-plugin is a Sensu Mutator that parse labels from sensu events and creates a dynamic check in Sensu Backend and add event.check.annotation io.sensu.remediation.config.actions with these values then it can be scheduled by sensu-remediation-handler

Usage

Sensu Dynamic Check Mutator creates sensu check based on template

Usage:
  sensu-dynamic-check-mutator [flags]
  sensu-dynamic-check-mutator [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -B, --api-backend-host string                  Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -k, --api-backend-key string                   Sensu Go Backend API Key
  -P, --api-backend-pass string                  Sensu Go Backend API Password (default "P@ssw0rd!")
  -p, --api-backend-port int                     Sensu Go Backend API Port (e.g. 4242) (default 8080)
  -u, --api-backend-user string                  Sensu Go Backend API User (default "admin")
  -c, --check-config string                      Json template for Sensu Check
      --command-arguments-template string        Template for Sensu Check Command (default "{{ range $key, $value := . }} {{ $key }} {{ $value }}{{ end }}")
      --command-bool-arguments-template string   Template for Sensu Check Command (default "{{ range $value := . }} {{ $value }}{{ end }}")
      --command-handler string                   Handler used to post the result (default "default")
      --default-check-suffix-name string         Default suffix name for unpublished checks (default "dynamic")
  -h, --help                                     help for sensu-dynamic-check-mutator
  -i, --insecure-skip-verify                     skip TLS certificate verification (not recommended!)
      --remediation-event-annotation string      Add an annotation in dynamic check created like remediation-event-alias: event.Entity.Name/event.Check.Name (default "remediation-event-alias")
  -s, --secure                                   Use TLS connection to API
  -t, --trusted-ca-file string                   TLS CA certificate bundle in PEM format

Use "sensu-dynamic-check-mutator [command] --help" for more information about a command.

Configuration

We add a json inside --check-config:

[
  {
    "name": "describe-resource",
    "command": "${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
    "bool_args": [
      "--no-headers"
    ],
    "arguments": ["daemonset","deployment","pod","statefulset"],
    "options": {
        "--namespace": "namespace"
    },
    "match_labels": {
        "sensu-alertmanager-events": "owner"
    },
    "exclude_labels": [
      {
        "alertname": "TargetDown"
      },
      {
        "alertname": "KubeVersionMismatch"
      }
    ],
    "sensu_assets": [
        "kubectl"
    ],
    "occurrences": [1],
    "severities": [2],
  },
  {
    "name": "systemctl-status",
    "command": "sudo systemctl",
    "options": {
        "status": "application"
    },
    "match_labels": {
        "systemd": "true"
    },
    "occurrences": [1]
  },
  {
    "name": "systemctl-restart",
    "command": "sudo systemctl",
    "options": {
        "restart": "application"
    },
    "match_labels": {
        "systemd": "true"
    },
    "occurrences": [3]
  }
]

In this example, to change the event, this mutator need to find a label called namespace, and need to find at least one of the arguments array, like label deployment. Then it will create a check.command: ${{assetPath "kubectl"}}/kubernetes/client/bin/kubectl describe --namespace default deployment nginx.

And it will create one annotation like:

"io.sensu.remediation.config.actions": "[{\"request\":\"KubeDeploymentReplicasMismatch-default-nginx-describe-resource-dynamic\",\"occurrences\":[1],\"severities\":[2],\"subscriptions\":[\"entity:k8s.dev.local\"]}]"

And one check called: KubeDeploymentReplicasMismatch-default-nginx-describe-resource-dynamic with command ${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe --namespace default daemonset nginx.

In systemctl-status and systemctl-restart if this mutator found two labels systemd:true and application:nginx as example for a check called http-nginx, it will create two checks in Sensu Backend called http-nginx-systemctl-status-dynamic and http-nginx-systemctl-restart-dynamic both running a sudo systemctl [status|restart] nginx command and it will add the following annotation:

"io.sensu.remediation.config.actions": "[{\"request\":\"http-nginx-systemctl-status-dynamic\",\"occurrences\":[1],\"severities\":[2],\"subscriptions\":[\"entity:systemd-ubuntu\"]},{\"request\":\"http-nginx-systemctl-restart-dynamic\",\"occurrences\":[3],\"severities\":[2],\"subscriptions\":[\"entity:systemd-ubuntu\"]}]"

kubectl as asset

In these example we use one event imported by sensu-alertmanager-events and we installed kubectl using assets.

type: Asset
api_version: core/v2
metadata:
  name: kubectl
  namespace: default
spec:
  sha512:  081472833601aa4fa78e79239f67833aa4efcb4efe714426cd01d4ddf6f36fbf304ef7e1f5373bff0fdff44a845f7560165c093c108bd359b5ab4189f36b1f2f
  url: https://dl.k8s.io/v1.20.0/kubernetes-client-linux-amd64.tar.gz

http-nginx

type: Check
api_version: core/v2
metadata:
  name: http-nginx
  namespace: default
spec:
  command: check-http.rb -u http://127.0.0.1 -t 5
  handlers:
  - default
  - remediation
  interval: 60
  publish: true
  runtime_assets:
  - sensu-ruby-runtime
  - sensu-plugins-http
  subscriptions:
  - ubuntu

Json details

Field What it does Example
name just a name for that config
command which command should run
bool_args add flags without any argument. Always include any configured flags -k
arguments add label.key label.value inside command. Should match at least one. If not, will return event without any change deployment ingress-nginx
options should match all configured to change the event. Use it when you need to use a different flag but with some content from a label To use a label.value in the flag --namespace, use it: {"--namespace": "namespace"}
match_labels If found these label.key=label.value it will change the event -
exclude_labels Use this array to exclude some label.key=label.value that doesnt match with your dynamic check -
sensu_assets Use to include an runtime assets in check created by dynamic check -
occurrences same occurrences field in sensu-remediation-handler default: []int{1}
severities same severities field in sensu-remediation-handler default: []int{2}
publish bool field. If it is enabled it will not send any information to sensu-remediation-handler and create a new check with interval default: false
interval integer field default: 10
subscription string field used to overwrite subscription used in check definition created by sensu-dynamic-check-mutator default: ""
name_suffix string field append in check name a label.value default: ""
proxy_entity_id string field used in check.proxy_entity_id based on label.value default: ""
sensu_handlers []string used to send handler with dynamic check created default: []string{"default"}

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please
consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the
following command to add the asset:

sensuctl asset add betorvs/sensu-dynamic-check-mutator

If you're using an earlier version of sensuctl, you can find the asset on the [Bonsai Asset Index][https://bonsai.sensu.io/assets/betorvs/sensu-dynamic-check-mutator].

Mutator definition

Maybe is important to add authetication configs -u dynamic -P ${MUTATOR_PASS} -B sensu-api.example.com -s -t /$PATH_TO_CERTIFICATE/ca.pem

---
type: Mutator
api_version: core/v2
metadata:
  name: sensu-dynamic-check-mutator
spec:
  command: >-
    sensu-dynamic-check-mutator -P ${SENSU_API_PASS} -B sensu-api.example.com -s -t /certs/sensu-ca.pem 
    --remediation-event-annotation "sensu.io/plugins/sensu-opsgenie-handler/config/remediation-event-alias"
    -c '[
      { "name": "describe-resource",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
        "arguments":["daemonset","deployment","pod","statefulset"],
        "options":{"--namespace":"namespace"},
        "match_labels":{"sensu-alertmanager-events":"owner"},
        "exclude_labels":[
          {"alertname":"TargetDown"},
        ],
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[1,2],
        "severities":[2]},
      { "name": "describe-node",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl describe",
        "arguments":["node"],
        "match_labels":{"sensu-alertmanager-events":"owner","alertname":"KubeNodeUnreachable","alertname":"KubeNodeNotReady"},
        "exclude_labels":[
          {"alertname":"TargetDown"},
        ],
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[1,2],
        "severities":[2]}
      {"name":"pod-termination-stuck",
        "command":"${{assetPath \"kubectl\"}}/kubernetes/client/bin/kubectl delete ",
        "bool_options":["--grace-period=0","--force"],
        "arguments":["pod"],
        "options":{"--namespace":"namespace"},
        "match_labels":{"alertname":"PodTerminationStuck"},
        "sensu_assets":["kubectl"],
        "sensu_handlers":["opsgenie_remediation"],
        "occurrences":[3,4],
        "severities":[2]}
    ]'
  runtime_assets:
  - sensu-dynamic-check-mutator

Using sensu-dynamic-check-mutator with --remediation-event-annotation and sensu-opsgenie-handler as a opsgenie_remediation handler we can add remediation action inside the same alert in opsgenie as extra properties and as note.

Example of remediation handler:

---
type: Handler
api_version: core/v2
metadata:
  name: remediation
spec:
  type: pipe
  command: sensu-remediation-handler
  timeout: 10
  mutator: "sensu-dynamic-check-mutator"
  runtime_assets:
  - sensu-remediation-handler
  env_vars:
  - "SENSU_API_URL=https://sensu-api.sensu.svc.cluster.local:8080"
  - "SENSU_API_CERT_FILE=/etc/sensu/tls/ca.pem"

Installation from source

The preferred way of installing and deploying this plugin is to use it as an Asset. If you would
like to compile and install the plugin from source or contribute to it, download the latest version
or create an executable script from this source.

From the local path of the sensu-dynamic-check-mutator repository:

go build

Additional notes

Contributing

For more information about contributing to this plugin, see Contributing.

Are you sure you want to report this asset?

Please describe the reason for reporting this asset. Our moderators will be notified and will disable the asset if it is found to be inappropriate.

×

You must be signed in to report this asset.

Sign In with Github

Download

×

Either download the source code:

Download Source

Or download the asset definition: