Sensu Remediation Handler

• Overview
• Configuration
  • Environment Variables
  • Annotations
    • Annotation Specification
    • Remediation Action Specification
• Setup
• Examples
  • Example "Unscheduled" Check (Remediation Action)
  • Example Check Definition and Remediation Request Configuration
• Acknowledgements

Overview

The Sensu Remediation Handler is a Sensu Go event handler
that enables you to build self-healing workflows in Sensu.

The Sensu Remediation Handler – and other similar self-healing
workflows in Sensu – combine a few Sensu features:

• An "unscheduled check" configuration (i.e. a Sensu check with the "publish": false attribute set).
• The Sensu agent's built-in entity subscriptions (e.g. entity:web-server-01),
  which make it possible to target a single agent with a check execution
  request.
• The Sensu Checks API POST /checks/:check/execute endpoint, which allows
  this handler to issue ad hoc check requests.

Configuration

Environment Variables

The Sensu Remediation Handler does not honor any command line flags. Instead, it requires environment variables, either in the handler definition or in the Sensu backend service environment.

Annotations

Although environment variables provide connection details for the Sensu API, you'll use the io.sensu.remediation.config.actions check annotation to provide the configuration that defines remediation activities.

Annotation Specification

The Sensu Remediation Handler uses the string value of the io.sensu.remediation.config.actions check annotation to determine which remediation actions, if any, should be scheduled for a given event.

When present, the value of the io.sensu.remediation.config.actions check annotation must be a array of objects containing key/value pairs. Each object element in the array must conform to the remediation action specification.

Remediation Action Specification

Setup

1. Create a dedicated Sensu user and role for the remediation handler.

   sensuctl role create remediation-handler --namespace=default --verb=create,update --resource checks
   sensuctl role-binding create remediation-handler --role=remediation-handler --user=remediation-handler
   sensuctl user create remediation-handler --password REPLACEME
   

2. Register the remediation handler asset.

   sensuctl asset add sensu/sensu-remediation-handler --rename sensu-remediation-handler
   

3. Configure the remediation handler.

   ---
   type: Handler
   api_version: core/v2
   metadata:
     name: remediation
     namespace: default
   spec:
     type: pipe
     command: sensu-remediation-handler
     timeout: 10
     runtime_assets:
     - sensu-remediation-handler
     env_vars:
     - "SENSU_API_URL=http://127.0.0.1:8080"
     - "SENSU_API_CERT_FILE="
     - "SENSU_API_USER=remediation-handler"
     - "SENSU_API_PASS=REPLACEME"
   

   Save this definition to a file named sensu-remediation-handler.yaml and
   run:

   sensuctl create -f sensu-remediation-handler.yaml
   

Examples

Example "Unscheduled" Check (Remediation Action)

---
type: CheckConfig
api_version: core/v2
metadata:
  name: systemd-start-nginx
  namespace: default
spec:
  command: sudo systemctl start nginx
  publish: false
  interval: 10 # interval is required but not used
  subscriptions: []

Example Check Definition and Remediation Request Configuration

---
type: CheckConfig
api_version: core/v2
metadata:
  name: check-nginx
  namespace: default
  labels:
    foo: bar
  annotations:
    io.sensu.remediation.config.actions: |
      [
        {
          "description": "Perform this action once after Nginx has been down for 30 seconds.",
          "request": "systemd-start-nginx",
          "occurrences": [ 3 ],
          "severities": [ 1,2 ]
        },
        {
          "description": "Perform this action once after Nginx has been down for 120 seconds.",
          "request": "systemd-restart-nginx",
          "occurrences": [ 12 ],
          "severities": [ 1,2 ]
        }
      ]
spec:
  command: check_http -H 127.0.0.1 -P 80 -N
  publish: true
  interval: 10
  handlers:
  - remediation
  subscriptions:
  - nginx

Acknowledgements

This handler implements a pattern first implemented in Nick Stielau's Sensu Remediator circa 2012. Thanks, Nick!