Last month we released YAMLRecord, a lightweight way to persist a small dataset into a simple YAML file which is fine if you’ve only one app server and you keep the YAML file stored locally.

As pointed out by Nelson Hernandez, YAMLRecord presents a problem when you have multiple application servers because the file can no longer be stored locally on a single application server. The problem is that if a change occurs in one of theses servers, the YAML file would have to be updated on all the others or stored in a new location accessible by all the servers.

To address this issue, the goal was to find a way to move the YAML file away from the app server and store it somewhere else that can handle access from multiple instances. With Nathan, we brainstormed and listed several solutions we could implement to allow us to scale out with YAML Record:

1. Store YAML file on S3 and cache the data locally
2. Use NFS and mount a shared volume and store the YAML on that volume
3. Serialize YAML content in an existing persistence store such as PostgreSQL or Redis

Why Redis?

We chose to augment YAMLRecord with a pluggable “adapter” system that supports storing the YAML content on Redis. We would then move the YAML data from the file system to a redis-backed store. This may seem to be an odd choice but we have a number of simple lightweight YAMLRecord resources setup and we wanted to keep this system in tact for the time being.

The first solution with S3 and memcached seemed relatively sound in which we store the YAML files on S3 and then update them there each time a change is made. This solution seemed to be an issue because we would be forced to leave our LAN and access S3 each and every time we wanted to update the YAML data. Also, to retrieve the YAML data from a remote location each time seemed a bit overkill.

Using NFS shared volumes is the more “traditional” solution. Simply setup a shared volume and mount the YAML files to each application server. We opted not to use this approach because this would be yet another “moving part” in our system that we would have to manage as we deploy servers. We wanted to use an existing system if possible so that additional complexity wouldn’t be introduced.

We ultimately felt that the last solution where we would allow YAMLRecord to support different storage adapters would be a decent approach. We felt this made sense for our infrastructure which was already set up for redundancy and backups on PostgreSQL and Redis. When we built YAML Record, it was to avoid the heavy-ness of a SQL Database, it didn’t really make sense to pursue a storage adapter in that direction yet. We felt Redis would be an easy to use storage adapter with the key being the YAML file name and the value being the serialized YAML data.

What’s new on YAML Record?

With Redis as a solution to scale out YAML Record data, we also wanted to keep the option for a local store with a simple file. We came up with this idea of swappable adapters which allows YAML Record to be modular.

In order to use the Redis adapter it’s quite simple, you just need to pick which adapter you want to use in a model:

class Team < YamlRecord::Base
  # Declare which adapter you want to use
  adapter :redis, $redis # $redis is your redis client

  # Declare your properties
  properties :name, :role

  source "team," # Will store data in a key "yaml_record:team"
end

What’s next?

Here’s some ideas we want to achieve soon:

• Be able to specify type for properties
• Add validations
• Add timestamps magic fields like on activerecord
• New adapters build by you? :)