Managing your personal servers with Puppet
At GitHub, we use Puppet to manage all our physical and virtual hosts’ configurations based on app-role combinations. Having become familiar with Puppet over my tenure there, I finally decided last month that I should be using Puppet to manage my own servers.
When I was just graduating from college and had a steady income, I finally decided it was time to have my own cloud server. Rackspace was the easiest choice back then, and I spun up a server with CentOS 6. Being new to server administration, I spent a long time organizing the packages and apps I installed. I ran a few Ruby web apps and hosted my static site built with Jekyll. I even went through the arduous process of manually configuring postfix to receive email for me on this host. Needless to say, I learned a lot.
GitHub switched to Debian in the last year and I figured I would migrate my server to Debian, too. During my internship at 6Wunderkinder / Wunderlist, I saw first-hand how nice immutable deploys were. The concept was fairly novel for a Ruby on Rails shop: with each deploy, spin up brand new VM’s, install your code, add them to the load balancer, and spin down the old ones. This is very much how Kubernetes rolling deploys happen today. After several years of my CentOS server running and collecting cruft, I decided to follow in 6Wunderkinder’s footsteps and spun up a fresh Debian server.
In the interim, I had, like millions of other developers, become enammored with Docker. The idea of packaging up code and dependencies in a language-agnostic container was immediately appealing. I began experimenting with running my apps inside Docker containers, and proxying traffic from nginx to the Docker containers. After a while, I found a system I liked: run the docker container in an upstart script, proxy traffic received by nginx to the docker container. When I wanted to update the app, I pushed a new revision of the Docker image, modified my upstart script to use the new tag, and ran restart <app> on my VM.
Over time, I made some improvements to this: I migrated all my apps from upstart to systemd when moving to Debian, and I generated nginx configurations for static and proxy sites instead of manually creating them.
In the last month or so, I decided that spending $35/mo on a VM didn’t make much sense when there were equivalent computing environments for much less money. I signed up for Linode, which offers 1GB VM’s for $5/mo, and had to migrate all my apps again. Time for a configuration management system! I had never much liked Chef, so I chose to give Puppet a try. What a massive difference this change has made.
I began by setting up my development environment: creating my user, adding my SSH public keys, and cloning my dotfiles. This was a fairly straight-forward process. I pushed up this new code to a repository. In order to get started with Puppet on the new host, I had to install Puppet. I went with v4.8, cloned down my repository to /etc/puppet, and ran puppet apply. It worked! I setup a script/apply bash script and set it up to run every hour.
The next piece was to setup hieradata and a parker::app resource which I could use to setup my above flow. First piece was the docker image. I installed the puppetlabs-docker Puppet module (using puppet-librarian, of course!), and used the docker::image resource to download the image of my choosing. Next, I created a systemd::unit_file for my app. The unit file is based on a common template using docker run and writing all environment variables to a file and using the --env-file flag when running the docker container. It exposes the server to a port of my choosing. I used the camptocamp-systemd Puppet module to get systemd::systemctl::daemon_reload, which I notify when a systemd unit file changes. Last, I used the service resource to ensure the service is running when I apply my puppet catalog.
This made this very easy! I setup a new .pp file for each of the servers I wanted to run. For those which used a database, I used the puppetlabs-mysql module and used the mysql::db resource to create a database, user, and password specific to each app. If I needed any other resources (for example, a specific unix user for the app), then I could add these in this .pp file.
To install each app, I used hieradata. Hiera is basically the greatest thing ever, and makes managing your puppet code so much nicer. All my parameters to each class go in the hiera YAML file for each host. And at the top, I specify a classes array, which essentially includes each class. When I want a new app, I create the class for it, and include it in this list for the host I want to install it on.
I also run a few static sites from this server (like this one!) and wanted to set those up, too. I created a site resource and created a new class for each static site I wanted to host. These are simple webroot nginx configurations so all I do is create directories, symlink a file in sites-available to sites-enabled, and let my jekyll-build-server handle the creation of the site.
This has been working great so far! Setting up new apps and static sites is super simple now. I’m in the process of both manging nginx with this (I compile manually using the versions of nginx, zlib, openssl, and pcre that I want), and setting up my home NUC to use this, too. I have a bunch of little command-line utilities which do my bidding and it’d be nice to get those all installed via this new method, too.
One open question I have is managing secrets. I just write these in plain text in my hieradata YAML file, which isn’t ideal. Anyone with the proper unix permissions who gets onto my server can see all my credentials. One thing I’ve done is setup a separate user on GitHub.com with access to my static sites and other repos of mine and use personal access tokens for that user instead of for my user. This helps reduce the likelihood that a security breach on my servers would result in any meaningful access to private code that I didn’t explicitly allow my server to access. Using GitHub Apps seems like a better next step, but I haven’t quite gotten there. If you have any recommendations for how I can better organize this, please drop me an email!
Managing my servers with Puppet has been a great pleasure so far and I’m really pleased with how it’s helped make managing my servers even easier. 