Sensu SaltStack Handler

Table of Contents

• Overview
• Enterprise plugin
• Usage examples
• Configuration
  • Asset registration
  • Handler definition
  • Minion targeting
  • Environment variables
  • Annotations
    • Annotations for remediation actions
      • Available functions
    • Annotations for arguments
  • Proxy support

Overview

The Sensu SaltStack Handler is a Sensu Handler for launching
SaltStack Jobs for automated remediation. This handler requires
access to either the SaltStack Enterprise API Server (RaaS) or the open
source CherryPy REST API Server.

Enterprise plugin

The Sensu SaltStack Handler is a Sensu plugin that requires a valid Sensu
license to run. Sensu Go >= 5.21 will add the SENSU_LICENSE_FILE
environment variable to the handler execution. To run the plugin independently
of Sensu (e.g. test/dev), you must set the environment variable:

SENSU_LICENSE_FILE=$(sensuctl license info --format json)

Usage examples

Sensu SaltStack Handler

Usage:
  sensu-saltstack-handler [flags]
  sensu-saltstack-handler [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -a, --api-url string             URL for SaltStack API (default "https://localhost")
  -u, --username string            Username for connecting to SaltStack API
  -p, --password string            Password for connecting to SaltStack API
  -o, --open-source                Use open source API (rest_cherrypy)
  -e, --open-source-eauth string   External Auth System to use with open source API (default "pam")
  -m, --minion-template string     The template to use for naming the minion to be targeted (default "{{.Entity.Name}}")
  -t, --trusted-ca-file string     TLS CA certificate bundle in PEM format
  -i, --insecure-skip-verify       Skip TLS certificate verification (not recommended!)
  -h, --help                       help for sensu-saltstack-handler

Use "sensu-saltstack-handler [command] --help" for more information about a command.

Configuration

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not
using an asset, please consider doing so! If you're using sensuctl 5.13 with
Sensu Backend 5.13 or later, you can use the following command to add the asset:

sensuctl asset add sensu/sensu-saltstack-handler

If you're using an earlier version of sensuctl, you can find the asset on the
Bonsai Asset Index.

Handler definition

---
type: Handler
api_version: core/v2
metadata:
  name: sensu-saltstack-handler
  namespace: default
spec:
  command: sensu-saltstack-handler  --api-url https://sse.example.com
  type: pipe
  runtime_assets:
    - sensu/sensu-saltstack-handler
  filters:
    - not_silenced
  secrets:
    - name: SALTSTACK_USERNAME
      secret: saltstack_username
    - name: SALTSTACK_PASSWORD
      secret: saltstack_password

Minion targeting

By default the handler uses the entity name as the minion to target for the
job(s). In the event this does not match up with the minion name in
SaltStack, it is configurable via a template for the event using the
--minion-template argument. The default template is {{.Entity.Name}}.
See the Sensu Docs for more information on handler templates.

Environment variables

Most arguments for this handler are available to be set via environment
variables. However, any arguments specified directly on the command line
override the corresponding environment variable.

Security Note: Care should be taken to not expose the username/password
information for this handler by specifying it on the command line or by directly
setting the environment variable in the handler definition. It is suggested to
make use of secrets management to surface them as environment variables.
The handler definition above references them as secrets. Below is an example
secrets definition that make use of the built-in env secrets provider.

---
type: Secret
api_version: secrets/v1
metadata:
  name: saltstack_username
spec:
  provider: env
  id: SALTSTACK_USERNAME
---
type: Secret
api_version: secrets/v1
metadata:
  name: saltstack_password
spec:
  provider: env
  id: SALTSTACK_PASSWORD

Annotations

The annotations keyspace for this handler is
sensu.io/plugins/sensu-saltstack-handler/config.

Annotations for SaltStack Jobs

This handler accomplishes remediation by submitting jobs that are the
equivalent of various SaltStack CLI commands like state.apply and
service.restart via the SaltStack Enterprise API server (RaaS) or
the open source CherryPY REST API server.

To specify the remediation action(s) for a check, assign a JSON similar to the
following to the actions annotation in the handler's keyspace (noted above).

[
  {
    "fun": "service.restart",
    "arg": [ "nginx" ],
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  }
]

The available configuration attributes are:

• fun is the SaltStack function to run
• arg contains one or more arguments for the SaltStack function
• kwarg (not shown) contains one or more keyword args for the SaltStack function
• occurrences is one or more occurrences of the Sensu event when you would attempt this remediation
• severities is one or more check statuses from the Sensu event for when to attempt this remediation

For example, if you have an NGINX web server running and you wanted to attempt
remediation by using the SaltStack service.restart function for the nginx
service on the the third occurrence of the check returning 1 (warning) or 2
(critical), you would specify it as so:

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io/plugins/sensu-saltstack-handler/config/actions: |
      [
        {
          "fun": "service.restart",
          "arg": [ "nginx" ],
          "occurrences": [ 3 ],
          "severities": [ 1, 2 ]
        }
      ]
spec:
  handlers:
    - sensu-saltstack-handler

The actions can contain multiple functions to run on the same or different
combination of occurrences and severities. For example, if you wanted to
extend the configuration above by attempting a state.apply on the
deploy_web_app state if the service.restart of nginx fails to
rememediate and the occurrences then reaches 6, you would specify the
following:

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io/plugins/sensu-saltstack-handler/config/actions: |
      [
        {
          "fun": "service.restart",
          "arg": [ "nginx" ],
          "occurrences": [ 3 ],
          "severities": [ 1, 2 ]
        },
        {
          "fun": "state.apply",
          "arg": [ "deploy_web_app" ],
          "occurrences": [ 6 ],
          "severities": [ 1, 2 ]
        }
      ]
spec:
  handlers:
    - sensu-saltstack-handler

Available functions

You should be able to use most standard SaltStack functions and arguments for
the fun, arg, and/or kwarg definitions. Below are some examples that
have been tested to work with this handler.

Function: saltutil

[
  {
    "fun": "saltutil.refresh_modules",
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  },
  {
    "fun": "saltutil.sync_grains",
    "arg": [ "refresh=True" ],
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  }
]

Function: service

[
  {
    "fun": "service.restart",
    "arg": [ "nginx" ],
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  }
]

Function: grains

[
  {
    "fun": "grains.item",
    "arg": [ "os", "osrelease" ],
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  }
]

Function: state (in addition to state.apply in prior examples)

[
  {
    "fun": "state.sls",
    "kwarg": { "mods": "webapp", "pillar": { "name": "mywebapp" }, "exclude": { "sls": "nginx" }, "queue": true},
    "occurrences": [ 3 ],
    "severities": [ 1, 2 ]
  }
]

Annotations for arguments

All arguments for this handler are tunable on a per entity or check basis based
on annotations.

Examples

To change the SaltStack API URL argument (--api-url) for a particular check,
in that checks's metadata add the following (building from the example above):

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io/plugins/sensu-saltstack-handler/config/api-url: "https://salt-api.example.com"
    sensu.io/plugins/sensu-saltstack-handler/config/actions: |
      [
        {
          "state_to_apply": "nginx",
          "occurrences": [ 3 ],
          "severities": [ 1, 2 ]
        }
      ]
spec:
  handlers:
    - sensu-saltstack-handler
[...]

Proxy Support

This handler supports the use of the environment variables HTTP_PROXY,
HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). HTTPS_PROXY takes
precedence over HTTP_PROXY for https requests. The environment values may be
either a complete URL or a "host[:port]", in which case the "http" scheme is
assumed.

For more information about contributing to this plugin, see Contributing.