Playing with SaltStack and external classifiers

We have been discussing this a lot lately. How do we structure our SaltStack-config in a way that lets us do changes without possibly breaking abseloutly everything. Finding a good hierarchy is not always easy. How do we build it so that we can open it up later…. How can we manage to mostly leave the top.sls file alone and how do we include the right config for the right minions without having to maintain a list of minions in the config and make it so large its unreadable.

Turns out… the solution was pretty easy for us, as soon as we came up with the idea.

We allready have an internally written CMDB-solution and we wanted to use that as an external classifier.
First we had to write a simple module that made pillars from the data in our CMDB. More about that some other time. This post is about the structure we went for.
Anyway, our cmdb-module creates pillars for all hosts containing hostname, status (dev,qa,prod) and product (or role if you prefer).
This will typically look like this for a dev-server:

ourservername:
status: development
product: ourwebsite

So what we ended up with, was using file_roots in salt, matching each our environments like this:
(top.sls)
base:
"*"
- somebasestuff
"cmdb:status:prod*":
- match: pillar
- role
"cmdb:status:qa":
- match: pillar
- role
"cmdb:status:dev*":
- match: pillar
- role

This will match the role.sls file in all 3 environments.
In all 3 environments we have 2 subfolders. “products” and “services”.
The “product” folders contain the state of the final product, using services
from the “services” folder. For instance, say you have a product called “yourwebsite”.
It will probably contain installation and configuration of web, cache and db. Those 3 are
reusable services under the services folder and doesnt change much.
In our role.sls we are now matching on the pillar “product” in our CMDB like this:

include:
- products/{{ pillar.get('cmdb', {}).get('product') }}

What this will do, is look for the CMDB-value for “product” and then include the matching item in the products-folder …and so, we do not need to maintain the top.sls OR any hostnames in the salt-config. So far we think it is a good idea, but we will see in a few weeks if it actually lives up to our expectations.

Anyways, figured I should share our thoughts.

Tags: , ,

Ideas of CMDB, cfengine and nagios integration

For a while we’ve been discussing how we can become as lazy as possible when it comes to systemadministration, and this time we’ve made a quite neat integration between our homemade CMDB, cfengine and nagios.

Here’s the idea:

First of all, nobody likes to manually update a CMDB. Also its never really possible to maintain it in a way that makes its info become obsolete after some time. This is why we made a script, cfcmdb, that is triggered from cfengine on every host. This script fills the CMDB database with all sort of info from tools like dmidecode and also from standard commandline tools. (Memory, networkingcards, os version, cpu, vendor etc etc). So now our CMDB pretty much keeps itself up to date.

Lately we came up with the idea to fill our CMDB with cfengine classes information. So adding to the cfcmdb script mentioned above :

cfagent –no-splay -p -v | grep Defined

..and a little perl split and join, we now have all the classes in our CMDB bount to hostid’s.

Cool. On our nagios-server, we made another script, cmdb2nagios, which takes the parameters “hosts”, “hostgroups” or “services”.

cmdb2nagios hosts : creates the nagios host-config file

cmdb2nagios hostgroups : creates the nagios hostgroup-config file

cmdb2nagios services : creates the servicefile

The services parsing is quite nice now, cause we can automatically monitor any services set up with cfengine. Lets say we have a bunch of hosts installed with cfengine and cfengine tells them to have apache2 running. That means that this will be part of a cfengine class, that will be available in our CMDB.

Example of cmdb2nagios service parsing :

[snip]

$sql = “select hosts.name from hosts,classes where classes.name = ‘class_apache’ and hosts.hostid = classes.hostid”;
$execute = $connect->query($sql) or die “wtf? it didnt work …check syntax.”;
my @servicehosts;
while (@results = $execute->fetchrow()) {
push(@servicehosts, $results[0]);
}

$hosts = join(”,”,@servicehosts);
print “define service{\n”;
print “\tuse\t\t\tgeneric-service\n”;
print “\thost_name\t\t” . $hosts . “\n”;
print “\tservice_description\tcfg_CHECK_APACHE\n”;
print “\tis_volatile\t\t0\n”;
print “\tmax_check_attempts\t1\n”;
print “\tnormal_check_interval\t5\n”;
print “\tretry_check_interval\t1\n”;
print “\tcontact_groups\t\tlinux-admins\n”;
print “\tnotification_period\t24×7\n”;
print “\tnotification_options\tc,w,r\n”;
print “\tprocess_perf_data\t1\n”;
print “\tcheck_command\t\tcheck_apache\n”;
print “\t}\n\n”;
[snip]

As you can see, monitoring apache will be applied to all hosts running apache.

This leaves us to really only having to maintain our cfengine configuration, while the CMDB is auto-updated and the nagios-config is auto-parsed.

Also our eventhandlers in nagios tells cfengine to do this and that, so now we can sit back, enjoy a coffee and watch this show.

(see previous post about eventhandlers and cfengine : http://www.sladder.org/?p=261)

Tags: , ,