Monthly Archives: December 2011

Turn ruby code into simple daemons with Dante

Context

At Miso, the engineering team has recently been working on a major shift to our architecture. Namely, moving to a much purer service oriented approach. Our new strategy as a whole is going to be the subject of many blog posts in the future. This transition has required us to setup a lot of new services. These services range from small API services written in Sinatra and Renee to front-end heavy services in Padrino, even to internal services communicating with protobuf messages.

Every one of these services has a lot of shared behaviors in common; each needs to log output, be monitored, be daemonized and easily start / stoppable, have automated deployment, etc. These shared commonalities between services and the quantity of these services led us to begin our journey to create a ‘unified’ services architecture that will manage all of this for us. While that project is still a work in progress, today I would like to introduce one tiny library that we built along the way.

Introduction

One of the common behaviors each ruby service needs is a simple executable that can reliably start / stop a process. More-over, each executable needs to be able to start the process in non-daemon and daemonized mode, allow a port to be specified, allow a pid file to be specified, and the executable should be able to stop daemonized processes that had been started. Ideally, this same interface could be used with God or Monit to monitor the process.

We started by looking at existing daemonizing gems such as daemon-kit, and while these are excellent libraries, they didn’t really achieve quite our purposes. In the end, we decided to whip up a gem that would provide this in the simplest way that could possibly work. The result is an extremely tiny but well-tested daemonize gem called Dante.

Usage

First, let’s start with how to integrate Dante into a gem or ruby code base.

Since dante will become a dependency to your gem’s executable, add to your gemspec:

# mygem.gemspec

Gem::Specification.new do |s|
  # ...
  s.add_dependency "dante"
end

or alternatively to your Gemfile:

# Gemfile

gem "dante"

Now you can start using dante for your executables. Dante’s interface is extremely simple. You wrap your code in a Dante block and then the gem handles the rest and passes you the relevant options.

Let’s say we want to daemonize arbitrary ruby code, the simplest way would be:

# Starts the code as a daemonized process
Dante::Runner.new('foo').execute(
  :daemonize => true, :pid_path => "/tmp/path.pid") { do_something! }

You can then also stop based on the pid:

# Stops the daemonized process
Dante::Runner.new('foo').execute(
  :kill => true, :pid_path => "/tmp/path.pid")

Now, let’s say we wanted to boot up a daemonized thin api service using an executable. First we would create the executable in bin/blog and then wrap the thin bootup in a Dante#run block:

#!/usr/bin/env ruby

require File.expand_path("../blog.rb", __FILE__)

Dante.run('blog') do |opts|
  # opts available: host, pid_path, port, daemonize, user, group
  Thin::Server.start('0.0.0.0', opts[:port]) do
    use Rack::CommonLogger
    use Rack::ShowExceptions
    run Blog
  end
end

Notice how simple the Dante part is. Dante just accepts a daemon name and yield options that were passed on the command line to your program. In this example, the thin process uses those options to bootup thin on the correct port. So what does wrapping your executable code in Dante give you?

Functionality

With your executable daemonized with Dante, you get a familiar and standardized interface for controlling your process. The simplest way to start the thin server now is just to run the executable with no options:

./bin/blog

That will boot up a non-daemonized process in the command line. Now, let’s say we want to get help for this process:

./bin/blog --help

will print a nice auto-generated help message showing the daemons commands as a reference. Now, let’s start the process as a daemon, setting the port and pidfile:

./bin/blog -d -p 8080 -P /var/run/blog.pid

This will start the blog service on port 8080 storing the pid file at /var/run/blog.pid, and with that your process will be running. Now let’s stop the process:

./bin/blog -k -P /var/run/blog.pid

The -k flag will find the process from the pid and then kill the daemonized process for you.

Customization

In many cases, you will need to custom flags/options or a custom description for your executable. You can do this by using Dante::Runner explicitly:

#!/usr/bin/env ruby

require File.expand_path("../../myapp.rb", __FILE__)

# Set executable name to 'myapp' and default port to 8080
runner = Dante::Runner.new('myapp', :port => 8080)

# Sets the description in 'help'
runner.description = "This is awesome myapp!"

# Setup custom 'test' option flag
runner.with_options do |opts|
  opts.on("-t", "--test TEST", String, "Test this thing") do |test|
    options[:test] = test
  end
  # ... more opts ...
end

# Parse command-line options and execute the process
runner.execute do |opts|
  # opts: host, pid_path, port, daemonize, user, group
  Thin::Server.start('0.0.0.0', opts[:port]) do
    puts opts[:test] # Referencing my custom option
    use Rack::CommonLogger
    use Rack::ShowExceptions
    run MyApp
  end
end

Now you would be able to do:

./bin/myapp -t custom

and the opts would contain the :test option for use in your script. In addition, help will now contain your customized description in the banner.

That’s all you need to know to use Dante although the process also supports starting on alternate hosts and a handful of other options. This is all Dante does, the simplest daemonizer that could possibly work. But it has been extremely handy for us for controlling our services.

Process Monitoring

One easy way to monitor a daemonized process is to use God to watch the process. Thankfully, God works perfectly with Dante and they cooperate quite nicely. God is supposed to be able to daemonize processes automatically but in my experience this can be unreliable and having a consistent interface for the executable can be convenient for testing.

Setting up a god file for a dante process is simple:

# /etc/god/blog.rb

God.watch do |w|
  pid_file = "/var/run/blog.pid"
  w.name            = "blog"
  w.interval        = 30.seconds
  w.start           = "ruby /path/to/blog/bin/blog -d -P #{pid_file}"
  w.stop            = "ruby /path/to/blog/bin/blog -k -P #{pid_file}"
  w.start_grace     = 15.seconds
  w.restart_grace   = 15.seconds
  w.pid_file        = pid_file
  w.behavior(:clean_pid_file)
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end
end

Conclusion

Our team will have a lot more to talk about soon as we begin open-sourcing various parts of our new services architecture. However, I hope Dante can be useful as a standalone utility. I know that it has been for us. Be sure to checkout Dante for more information and to try this out for yourself.

For an example of our usage of Dante in a ruby gem, check out Gitdocs, our open-source dropbox clone using Ruby and Git. In particular, check out the cli.rb class for advanced usage.