puppet tricks: debugging

Update: (2012/9/30) I came up with this around the time I was using 0.25.  Apparently now you can do similar utilizing the –debug switch on the client along with debug() calls. I thought the function was only part of PuppetLab‘s stdlib, but apparently its in base, at least in 2.7+. I’ll probably do a part 2 to this with more info, although there isn’t much more.

Update: (2012/12/20) So the debug() function from stdlib is lame. I spent a while troubleshooting my new environment not getting messages and realized that rolling back to notice() worked. Could have sworn I tested it when I posted that. I did also run into an issue that naming the fact debug is actually a bad idea and so have updated this blog accordingly.

Update: Found this bug that talks about the facts not returning as the appropriate types.

Disclaimer: I am not a ruby programmer… so there might be “easier” or “shorter” ways to do some of the things I do with ruby, but my aim is for readability, comprehensibility by non-programmers, and consistency.

In my time playing with puppet I have had to do a few things I was not pleased with.  Mainly I had to write several hundred lines of custom facts and functions.  Debugging was one of the biggest pains, until I found a wonderful blog post that helped me out with that.  Actually, when he helped me out with debugging I had already been to the site once because I ran into a bug related to the actual topic of his post, “calling custom functions from inside other custom functions”.  Back to the matter at hand… when I first started working on custom functions I would leave exceptions all over my code and use them to step through the functions during debugging sessions.  While the code itself was short, this a tedious process as I would have to comment out each exception to move to the next one and then re-run the test.  It looked like this:

Then I found function_notice, which got rid of the commenting of exceptions by allowing me to log debug statements.  So I replaced all of my exceptions with if wrapped function_notice calls, resulting with:

An important thing to remember about function_notice in a custom function is that the variable you pass to function_notice must be a list.  I have not done anything other than send a single string inside a single list, so I could not speak to its other behaviors.  The length of the code increases greatly, and I do not actually do a debug for everything.  Overall this is a much better place to be.  However, now to enable debug I have to edit the custom functions on the puppet master which requires a restart the service (puppetmasterd, apache, etc), and logs are generated for every client.  That is still a pain.  This is when I had a “supposed to be sleeping” late at night revelation.  You can lookup facts and variables inside your custom functions!  So I created a very simple fact named debug.rb that looks like this:

So what that means is that on any of my puppet clients I can enable debugging of my puppet setup by touching the file /etc/puppet/debug, and disable it by deleting that file.  To enable this in my custom function I change the definition of debug.

Now, this may seem like a kinda odd way to go about setting the debug value, but while the code in the custom fact is working with the boolean value of true/false, when called as a fact it returns the string “true” or “false”.  Since the string “false” is true from a boolean sense you could end up getting flooded with logs if you do a simply true/false check against the lookup() result.  Thus, we default to false as that should be our normal working mode, and if the fact returns the string “true”, we set debug to true.  Now there is a custom fact providing debug, and a custom function utilizing it to log messages on the puppet server. Yay!  But wait, there is more!  Now that you have the custom fact defined, you can utilize it inside your puppet manifests in the same way!  Let take a look:

Wait, what? Sorry.. threw a few curve balls at you. The notify call, which is not a local function, logs on the client side. Then I wrapped it in a define called print, because I was going to pass an array to it. By wrapping it in the define it takes the array and performs the notify call on each object in the array. You can read more about this on this page, under the sections What is the value of a variable? and Whats in an array?.  The article has some nice explanations of a few other things as well.

Also, if you’d rather check for $::debug than $::puppet_debug then add the following to your site.pp:

$::debug = $::puppet_debug

puppet tricks: staging puppet

As I have been learning puppet @dayjob one of the things I have been striving to deal with is order of operations.  Puppet supports a few resource references, such as before, after, notify, and subscribe. But my classes were quickly becoming slightly painful to define all these in, when the reality was there was not always hard dependencies so much as a preferred order.  After having issues with this for a while and researching other parts of puppet I stumbled across some mention of run stages, which were added in the 2.6.0 release of puppet.  If you read through the language guide they are mentioned.  There has always been a single default stage, main.  But now you add as many as you want.  To define a stage you have to go into a manifest such as your site.pp and define the stages, like so:

That defines the existence of two stages, a pre stage for before main and a post for after main.  But I have not defined any ordering.  To do that we can do the following, still in site.pp:

Thus telling puppet how to order these stages.  An alternate way would be:

It all depends on your style. So now that we have created the alternate stages, and told puppet what the ordering of these stages is, how do we associate our classes inside them?  It is fairly simple, when you are including a class or module you pass it in as a class parameter.  To do this they introduced an alternate method of “including” a class.  Before you would use one of these two methods:

In this the base class requires that the users class is done before it, and then includes the packages class. Its fairly basic. Transitioning this to stages comes out like this:

It is very similar to calling a define.  In production I ended up where adding my base class in the pre stage of a lot of classes, and which became kinda burdensome. I knew that there were universal bits that belonged in the pre stage, and universal bits that did not. To simplify I settled on the following:

With this setup I do not have to worry about defining the stages multiple times. I even took it further by doing the same concept for the different groups that are also applied to systems, so the universal base and the group base are both configured as in the last example. I have not tried it with the post stage, as I do not use one yet, but I would imagine it would work just as above. Here is an untested example:

Maybe this seems fairly obvious to people already using stages, but it took me a bit to arrive here, so hopefully it helps you out.


UPDATE: PuppetLabs’ stdlib module provides a ‘deeper’ staging setup.  Here is the manifest in github.

call to software vendors… package it right

One of the tasks that I have been responsible for performing over the last several years is packaging software into RPM Package Manager (RPM) packages.  All of our internal RPMs are fairly simple, the tricky part is the 3rd party software.  There are several problems with the distribution of commercial off the shelf (COTS) software in the Linux ecosystem.

  • They are rarely RPMs
  • Sometimes they use InstallAnywhere installers (more on that later)
  • When they are RPMs they either think they know better than RPM (can be explained as a lack of understanding as well) or they try too hard to make a single RPM that works on all RPM-based distributions.

Before I go further I would like to say, if you are packaging your software as a real RPM (or any other native Linux packaging system), even if it is not perfect, THANK YOU. We appreciate it.  Please take my commentary as constructive criticism.  I am not angry with you, and will gladly help you with packaging issues if I can.  I am not the best either, but I have a fair bit of practice.  I am sure others would gladly assist as well.

Moving on… So today I got stuck attempting to automate the installation and configuration of some of unnamed vendor’s system management RPMs via puppet.  I made the mistake of looking into the scriptlets and was frustrated by some of their practices.  I started to write a package by package evaluation of the scriptlets, but one package in particular would have taken forever (it attempted to cover every possible RPM-based Linux distribution via the scriptlets).  I recalled a conversation I had a year or two ago with the individual who heads up this company’s Linux packaging group, and they had expressed interest in feedback.  At the time I had a few points to provide, but I did not have time for a more in depth analysis.  So I decided to finally write up a general set of bullet points to pass their way, if I am not trying to help then I am part of the problem, right?.  I figured it would not hurt to put it out here as well.  So I am re-wording a touch to make it less specific to just them, and more of a general call to all software vendors.  Also, I would imaging most of what is stated directly translates to Debian, conary, and other native packaging systems; but it is not intended to be a definitive guide.

Things to keep in mind when packaging software for native Linux distribution:

  • It helps to build (compile) your software from source using the packaging tools (RPM), instead of just packaging up the binaries.  You do not have to distribute the source (SRPM, tarball, etc), but the build process can potentially be cleaner and more manageable.  Yes, I know you paid all kinds of money for some fancy handle everything build system.  Are your customers (the system administrators) happy with the output?  I know I am not.
  • With the modern build processes and systems available there really is no good reason to build a mangled cluster of scripts that attempt to make one platform independent RPM instead of building distribution specific RPMs.  They can even still come from the same consolidate spec files, thus allowing you to reduce duplicate work.
  • Scriptlets
    • Should be short and concise
    • Almost the entire build and installation of the software should occur in the %build and %install sections, respectively.
    • Any setup or file layout should be handled in %build and %install sections, if its host specific it should be a documented post-install exercise for the admin.
    • User/group creation, symlinking files, chkconfig and service commands are all acceptable.
    • Should never touch a file owned by another package.  If it needs a setting or switch flipped, document it for the administrator. At worst, include a “check” or “setup” script that the administrator can run manually if they want you to do the work for them, cause the majority of us don’t.
  • Files should only be placed on the system via RPM’s built in methods.
    • Symlinks can be a caveat of this, but should not be abused. UPDATE 2014.03.19 – Retracting since I just built a package that managed symlinks right. No excuses. :)
    • File ownership and permissions should be set by the RPM, not in the scriptlets.
    • Do not provide library files that the OS already provides.  Get past the “i need this specific version, and no other will do” or “well we used that one but slightly modified” mentality, and when you can’t, then require the correct compatible library.  Most distributions do provide them already.  If you needed to modify it, are you properly following the licensing? Wouldn’t it be better to just submit a patch and stop having to maintain it and not worry about licensing issues?
  • You should not be adjusting security settings on the system for the administrator, you can provide them (i.e. SELinux policy files, default firewall rules, a file for /etc/sudoers.d/, etc), but implementing them for me is bad security at its worst.
  • If you provide an SELinux policy that does not change any existing policies on the system directly, you can implement that.  But if you change something existing, let us do the work so that we are aware.
  • Do not flip SELinux booleans related to other bits, let the admin or find the right way.
  • Get help… it is out there.  There is selinux mailing lists, and ya know what? Call Red Hat.  They helped get SELinux going, they know what to do.  You are not in charge of my system’s security, I am.  If you need a change to an existing policy, talk to that policy’s maintainer or implement it in your own policy.
  • Users and groups – for the most part this hasn’t looked bad except:
    • Deleting and re-creating a user is a annoying thing to do.  If the admin changed something about the user and everything still functions, don’t touch it.  If I need to fix it I can delete the user myself and then let you create it by re-installing, or just look at your newly clean scriptlets to discover the exact syntax.
    • Technically, you should not be deleting any user you create on the system… another exercise for the admin to avoid stranded files.
  • If your company decides not to supply good RPMs, consider a tarball(s) with an simple install script, we can do the rest.  I was avoiding naming names, and I hate to give this software credit, but IBM’s DB2 has got to be the easiest 3rd Party software I have ever packaged and deserves credit for the fact that there was just a bunch of tarballs and a few commands to install.  Setup was another matter. heh.

To summarize, write the RPMs according to a public packaging guideline, such as the Fedora Packaging Guidelines.  If Fedora would accept it into the EPEL repositories then you have succeeded.  I realize you want your packages to support other distributions, but odds are that if the spec file for the RPMs are cleaned up to meet Fedora’s guidelines, the other distributions should be easy to support, and us administrators would be ecstatic.  Plus the use of tools like mock, koji, OpenSuSE Build Service, etc can greatly ease build and distribution issues.

Since I already named one name, might as well point out a negative one.  Flexera’s InstallAnywhere… So InstallAnywhere is a universal installer with an interesting feature.  Their site claims: “Install native packages, such as RPM, on Linux, Solaris, and HP-UX“.  This is inaccurate, at least towards RPM.  What they produce is a Java-based installer that injects RPM metadata into your system’s RPM database.  This is not an RPM.  We can not distribute this software via Yum or Spacewalk or Red Hat Network Satellite.  They should be ashamed. :(

So some useful reading:

UPDATE: Some basic grammar fixes. 2011.10.19