Playing with SaltStack and external classifiers

We have been discussing this a lot lately. How do we structure our SaltStack-config in a way that lets us do changes without possibly breaking abseloutly everything. Finding a good hierarchy is not always easy. How do we build it so that we can open it up later…. How can we manage to mostly leave the top.sls file alone and how do we include the right config for the right minions without having to maintain a list of minions in the config and make it so large its unreadable.

Turns out… the solution was pretty easy for us, as soon as we came up with the idea.

We allready have an internally written CMDB-solution and we wanted to use that as an external classifier.
First we had to write a simple module that made pillars from the data in our CMDB. More about that some other time. This post is about the structure we went for.
Anyway, our cmdb-module creates pillars for all hosts containing hostname, status (dev,qa,prod) and product (or role if you prefer).
This will typically look like this for a dev-server:

ourservername:
status: development
product: ourwebsite

So what we ended up with, was using file_roots in salt, matching each our environments like this:
(top.sls)
base:
"*"
- somebasestuff
"cmdb:status:prod*":
- match: pillar
- role
"cmdb:status:qa":
- match: pillar
- role
"cmdb:status:dev*":
- match: pillar
- role

This will match the role.sls file in all 3 environments.
In all 3 environments we have 2 subfolders. “products” and “services”.
The “product” folders contain the state of the final product, using services
from the “services” folder. For instance, say you have a product called “yourwebsite”.
It will probably contain installation and configuration of web, cache and db. Those 3 are
reusable services under the services folder and doesnt change much.
In our role.sls we are now matching on the pillar “product” in our CMDB like this:

include:
- products/{{ pillar.get('cmdb', {}).get('product') }}

What this will do, is look for the CMDB-value for “product” and then include the matching item in the products-folder …and so, we do not need to maintain the top.sls OR any hostnames in the salt-config. So far we think it is a good idea, but we will see in a few weeks if it actually lives up to our expectations.

Anyways, figured I should share our thoughts.

Tags: , ,

running nagiosplugins via saltstacks peer communicationsystem

So …my previous post was  similar to this, but you most likely dont want to run the salt-master and nagios on the same server, so I had to find a way to let the nagios-server execute its plugins on hosts via the salt-master. This can be done using the python client api and saltstacks own peer communication system.

First of all, read this : http://docs.saltstack.com/ref/peer.html

Then check out my wrapper here : https://github.com/mortis1337/nagios-plugins/blob/master/check_by_saltpeer.py

Yay! Now you can throw away NRPE forever and stop using ssh-keys for the nagiosuser if you are doing that allready.

Nagiosplugins over zmq? I like it :)

Tags: , , , , ,

Running nagios-plugins via saltstack

I’m so sick of maintaining NRPE-config on my servers, and I dont really want root-sshkeys all over the place. Recently I discovered saltstack and started to play with it a bit. I came up with the idea of running Nagios(or Icinga) on the same server as my salt-master and so I created a little wrapper that lets me run nagios-checks via saltstack.

Here’s how it works.

This is my little wrapper-script written in python: https://github.com/mortis1337/nagios-plugins/blob/master/check_by_salt.py

The wrapper takes hostname, plugin and a timeoutvalue as arguments:

$ python check_by_salt.py -H examplehost -p “/path/to/existing/nagiosplugin arg1 arg2″ -t 10

The wrapper imports salt and runs commands on minions with cmd.run_all and returns the output and the exitcode.

For this to work as the nagios/icinga user, you will have to configure the client_acl for the user in the salt-master config, so go ahead and edit the master-configfile (default: /etc/salt/master)

Search for “client_acl” in the file and add this :

client_acl:
icinga:
- cmd.*

Yeeaaaap, thats quite the security risk right there, but read up on how to limit what can be done with the cmd-state in salt and atleast it will be safer than using ssh-keys :)

check_by_salt in combination with https://github.com/mortis1337/nagios-plugins/blob/master/check_disk_generic.py will instantly give you monitoring of all your disks with no clientside-configuration.

Use it if you like it and feel free to improve it.

 

 

 

Tags: , , , ,