If you’re using to_json, you’re doing it wrong

At Miso, we have been very busy in the last few months building out a large number of public APIs for our Developer Platform. In a short time, we have already seen early versions of applications built on our platform for Chrome, Windows Mobile 7, Blackberry, Playbook, XBMC among others. This has been very exciting to see the community embrace our platform and leverage our data to power additional services or bring our service to a new group of users. In this post, we will discuss how we started out building our APIs using Rails and ‘to_json’, why we became frustrated with that approach and how we ended up building our own library for API generation.

Our public APIs are designed to be unsurprising and intuitive for a developer. We chose OAuth 1.0a (soon to support OAuth 2) because this is already familiar to developers and there is rich library support across languages for this authentication strategy. The endpoints are for the most part RESTful with GET retrieving and POST / DELETE used to modify the data associated with a user. We also tried to design our API responses as simply as possible, giving every attribute a readable name, keeping node hierarchies relatively flat and not including unnecessary information from our database schema. While this may seem like good design goals when building a Public API, you might find yourself surprised at how difficult this can be using Rails and the baked in ‘to_json’ serialization that the framework provides.

Rails ‘to_json’ API generation

Let’s start by discussing the canonical approach Rails provides for generating APIs. The idea is to have the JSON and XML responses be deeply tied to the model schemas with one-to-one mappings in most cases between the database columns and the api output. This is great if you are working on internal APIs and you wish to dump the data directly into a response but less great for well-designed public APIs. Let’s look at how rendering with this to_json approach works in practice:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts)
  end
#...

Now, when this controller action is invoked the @post object automatically has the ‘to_json’ method called which takes the ActiveRecord model and converts seamlessly to JSON output. The model can also optionally be given parameters in the model by overriding the as_json method with options:

# app/models/post.rb
class Post
  def as_json(options={})
    super(options.merge(:methods => [...], :only => [...], :include => [...])
  end
end

This would then render the associated JSON response based on the options specified. In the simplest cases this would be all you need and the Rails way works without a problem. As long as your database schema is deeply coupled with the API output then all is well. As mentioned there are also a few choices to customize the to_json output and alter the output:

  • only – Only show column names in the output as specified in this list
  • except – Show all column names except the ones specified in this list
  • methods – Include these methods nodes (without any arguments) as nodes in the output
  • include – Add child nodes (potentially nested) based on associations within the object

Alright, to recap: There is a model which contains a specific database schema and some methods. There are limited options to transforming that into JSON as described above by passing a hash of options. These options can be passed in as defaults with the ‘as_json’ method in the model itself or through the controller action.

The unravelling of ‘to_json’ begins

Well wait just a minute, what if in different API responses, you want to include different api output options? What if you want to override the ‘as_json’ defaults? Not too bad you can just do:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts) do |format|
      format.json { render :json => @posts.to_json(:include => [...], :methods => [...]) }
    end
  end
#...

Using those settings you can change the JSON output on a per action basis. This system as described is a good high level overview of the Rails JSON generation approach. This can work in very simple and naive applications, but it should not be hard to imagine how this approach can be restrictive as well as verbose. Suppose I want to render a JSON output with only a few columns, also adding several methods and including nested options for multiple associations? That might look something like this:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts) do |format|
      format.json { render :json => @posts.to_json(
         :only => [:title, :body, :created_at, :tags, :category],
         :include => [
            :likes => { :only => [:created_at], :include => [:author] },
            :comments => { only => [:created_at, :body], :include => [:author]  },
            :user => { :only => [:first_name, :last_name}, :methods => [:full_name] },
         :methods => [:likes_count, :comments_count])
      }
    end
  end
#...

This action code is already starting to smell a bit funny even here. This is a lot of bulk in the controller and quite redundant. This doesn’t even really seem like this belongs in the controller. This is more of a view or template concern discussing the details of a particular JSON representation. You could move that all to the model inside a method, but that actually makes things harder to follow. Already this method of generating JSON doesn’t feel quite right and begins to break down.

You may think this example is a contrived case or poor API design but consider that there’s actually not that much going on here. This type of response is commonplace in almost any public API you will see on the web. In fact, it is actually much simpler then many in the wild. Compare the above to the Instagram API.

More Frustrations with API Generation

The issues above were just the beginning of the issues we ran up against using the ‘to_json’ method because that approach is interested in ‘serializing’ a database object while we are interested in creating a relevant representation for our public API platform. The ‘serialization’ of the object so directly just didn’t quite fit what we were trying to do.

The easiest way to demonstrate the limitations that frustrated us is to show relevant examples. Let’s start with a simple idea. In our system we have ‘posts’ and we have the idea of ‘liking a post’. In our API we want to return if the authenticated user ‘liked’ a particular post in the feed. Something like:

[ { post : { title : "...", liked_by_user : true }, ...]

Notice the node ‘liked_by_user’ which contains whether or not a user has liked the given post. Assuming we have this method in the model:

class Post
  # Returns true if given user has liked the post.
  # @user.liked_by_user?(@user) => true
  def liked_by_user?(user)
    self.likes.exists?(:user_id => user.id)
  end
end

We simply want to get this boolean value into the API response with the node name ‘liked_by_user’. How would we do this in ‘to_json’? How do we pass an argument to a method? After doing some research, it was apparent that this was not particularly easy or intuitive. It would be nice to have a simple way to pass multiple arguments to a method in the model without jumping through hoops.

Let’s move onto another example. Suppose we want to change the ‘user’ association to be aliased as an ‘author’ node in the output. Let’s say we have:

class Post
  belongs_to :user
end

and we want to have the output be:

[ { post : { title : "...", author : { first_name : "...", last_name : "..." }  }, ...]

What if I just need a minor change to the value of an attribute before inserting it into the JSON? What if I need a custom node in the JSON that is not needed in the model directly? What if I want to include the value of a method only if a condition on the record is met? What if I want to reduce duplication and render a JSON hash as a child of the parent response? What if I want to glue a couple of attributes from the user to the post? Change the model and fill it with this display logic every time? Fill our controllers with complicated JSON display options? Workaround the problems by fighting with ‘to_json’ and/or monkeypatching it?

Perhaps a better approach

As we came against these issues and many more while we designed and implemented our public APIs, we butted our heads against ‘to_json’ again and again. Often we wanted the attributes defined in the schema to be renamed or modified for the representation, or we wanted to omit attributes, or we wanted to include attributes if a condition was met, we wanted to handle polymorphic associations in a clean and easy way, we wanted to keep a flat hierarchy by ‘gluing’ attributes from the child to the parent.

Furthermore, the model and/or controller was getting filled up with tons of json specific details that had nothing to do with model or business logic. In fact, these JSON responses and verbose declarations didn’t seem to belong in the model or the controller at all and were cluttering up our code. In fact, true to MVC these details of the response seemed much more appropriate in a view of some kind. This idea of storing the JSON in a view sparked an experiment. Why not just generate the JSON in a template and move all of the display details out of the model and the controller. What if the API could be crafted easily in the view where a JSON representation belongs?

We agreed that implementing APIs in a view made the most sense both conceptually and practically. The next question becomes what templating language to use to generate these APIs? Forming the XML or JSON manually in ‘erb’ seemed verbose and error-prone. Using builder seemed silly since we wanted to build APIs that work primarily in JSON. Indeed, none of the default templating languages we grab for seemed to fit. We didn’t want to painstakingly handcraft nodes manually, we just wanted a simple way to declare how our APIs should look that afforded us the flexibility we needed.

We investigated a wealth of different libraries that seemed to fit the bill from tequila, to json_builder, to argonaut and many more attempts to solve this problem. Clearly we weren’t the only ones that had experienced the pain of ‘to_json’. Perusing the READMEs of any of these libraries quickly revealed people fed up with the limitations same as we had become. Problem was every option we could find didn’t work for one reason or another. Either the syntax was awkward, the libraries weren’t maintained, there were too many bugs, or the templates became verbose and difficult to manage. After reviewing the available options, we decided to try and design our own library for creating APIs. One that would solve all the problems we had encountered thus far.

The Ruby API Builder Language

We embarked on a thought experiment before building the library. What were our frustrations with existing libraries and tools? Where do we want the JSON options to live? How did we want to specify them? What options did we want to have? What language or syntax should we use to define the output? How do we keep the options DRY and intuitive?

Early on we decided we wanted the JSON output to be defined in the views. Logic that belonged in the models would stay there where it belonged, but this was rarely the case. Most of the options were simply crafting the JSON response and clearly belongs in a template. So that meant a file living in the views folder within Rails. We also decided we didn’t want to learn a new language and that Ruby was as good an API builder as any. Why not just leverage a simple Ruby DSL to build our APIs? Why not support inheritance and partials for our APIs? Why not allow the same template to describe both the JSON and XML responses for our API?

From these design questions and several days of work, the RABL gem was born. We started using this approach and fell in love with it immediately. All of a sudden, generating APIs was easy and intuitive. Even the most complex or custom API output was very simple and maintainable through the use of inheritance, partials and custom nodes. All of this was kept neatly tucked away in a view template where it belonged without requiring any extra code in the models or worse the controller actions.

Stay Tuned

Since we built RABL, we have gotten excellent feedback from the community. We have deployed in production all of our Public APIs using RABL and we couldn’t be happier. Please checkout the README and let us know what you think! We would love to hear your experiences with building APIs on Rails or Sinatra. This post is a setup for a thorough step-by-step tutorial we plan to publish soon on generating clean JSON and XML APIs in Rails 3 using RABL.

This entry was posted in All, Engineering and tagged , , , on by .

About Nathan Esquenazi

I've been working professionally at early-stage startups for the last 10 years developing for web, mobile and desktop across many platforms. I have always been passionate about education, actively tutoring in college, holding workshops for students and volunteering to teach practical technical skills to those that want to learn them.

34 thoughts on “If you’re using to_json, you’re doing it wrong

  1. Ken Collins

    When Rails 3.0.0.beta1 came out I did a talk on integrating rails applications with RESTful services to Core Data. One thing I made heavy use of was the #as_json to return nothing more than a simple primitive hash.

    My opinion is that if you are using the :only, :except, :methods, and :include options – you are doing it wrong. I think these options are a throw backs to the #to_json and #to_xml way of doing things and have no place when using #as_json. The options are only there for hot different backends (if used) as provided by ActiveSupport. Here is an example:

    https://gist.github.com/977846

    So basically you want to set a sane default in your models using #as_json and make sure to return a ruby primitive (a hash) and any objects in the hash need to understand how to respond to #as_json. And then it just all works. From here, and if you need to, you can use a presenter pattern for controllers that need to give other json attributes. So you would have a collection of presenter object for each that in turn would not how to respond to #as_json and wrap said model objects. Or you could easily add accessors that would return an array of attributes on each model. The results for this would be passed conditionally to the attributes.slice. Properly abstracted, you could get the best of #as_json and different attributes all thru a standard interface.

    1. nesquena Post author

      I think this approach can work using as_json with a primitive hash combined with the presenter pattern. We even tried a similar approach in a different project. I don’t have anything against that, although I often found I wanted different representations in different cases some of the time and manipulating all of these hashes manually felt kind of unnecessary. I often had to pass other data in the controller back into the presenter awkwardly in order to calculate node values. I actually think using a presenter is a viable alternative although I found (when researching it and trying it myself) it was a bit verbose and required me to kind of create a bunch of my own helpers and reusable methods to keep the declarations lean. Ultimately we just decided we would rather declare our output in declarative view templates that can be used and reused to form XML and JSON and didn’t require manual hash manipulation. In most cases I didn’t find doing manual hash manipulation in a presenter to have many benefits over our approach. To each his own though and I agree presenters can be a valid alternative. Thanks for commenting.

      1. Ken Collins

        Agreed on a presenter pattern. It may be more pragmatic to just have a constant like JSON_ATTRS for each model with a hash of options and a base attribute in say ActiveRecord to get the array of which attributes to slice. Namespaced api controllers would be easy to have an around filter that set that so you did not have to make lots of presenter code or manually coerce the models. Something like.

        https://gist.github.com/977890

        That aside, I think the important thing to point out to people is that those legacy options that were typical for #to_json and more so #to_xml have no real meaning anymore. Just return an objects or collections of object that know how to respond to #as_json. Thanks for writing me back!

    2. nesquena Post author

      Also for anyone interested in this approach, I think this blog post lays out a decent alternative for how to generate APIs using a presenter. As I said, to each his own, but I tried this approach and found it didn’t afford me anything that I didn’t already have using RABL.

  2. Chris Kimpton

    Hi,

    Great post – just switched my new app to using RABL – very nice.

    Any plans to cover what you did for the OAuth provider side of things? Or a one liner if it was that simple…

    Thanks, Chris

    1. nesquena Post author

      Hey Chris,

      Glad to hear you are using RABL! Good question regarding the OAuth provider side of things. I actually plan to do a full post related to that very topic. My current plans are to first write a detailed step by step tutorial to constructing APIs with RABL and Rails 3. Then circle back and outline how we created the authentication mechanisms. The one liner is using https://github.com/pelle/oauth-plugin and/or https://github.com/flowtown/rack-oauth2-server

      1. Chris Kimpton

        Thanks for the pointers.

        Just had a thought – any plans to extends RABL to help with POST type calls.

        For example, I have a pricing call, that takes several parameters. It then returns the pricing details. I can see how I’d use RABL for the response, but was wondering if there needs to be any help on the parameters in part – although perhaps the existing support in Rails (etc) for getting parameters is adequate.

        Thanks, Chris

        1. nesquena Post author

          I would be interesting in exploring the use case further. We have a lot of API calls that accept parameters but I have never ran into a situation yet where we needed help beyond that provided by Rails params parsing. Can you elaborate on what you would be interested in having RABL augment. Maybe a gist of example usage? But yeah when it comes to the response I would take the parameters, retrieve the values and use RABL to display the response. The great thing about RABL which I didn’t touch on yet is that the same template can be used for XML and JSON formats without any extra work.

  3. Sterling Cobb

    Hey thanks for the post .

    I just spend a bunch of time creating custom as_json to_json blah blah includes garbage with custom methods for names etc and kind of wish you posted this 3 days earlier! ha jk. I’m ganna check out RABL but thanks for the post!

    - Sterling

  4. Luiz Rocha

    Here @Abril, we ran in a very similar problem and came up with — basically — the same kind of solution. Our DSL gem is called “Tokamak” and is avaliable at Github.

    Maybe we could collaborate? :-)

    1. nesquena Post author

      Interesting, yeah I feel like this is a pain point felt by many people when they try to build real well defined JSON representations. Very cool Gem, thanks for sharing. I like what you guys are trying to do.

  5. Paul

    Where were you six weeks ago when I embarked on a very similar exercise, but with to_xml, and hit pretty much every problem you mention here!?!
    ;)
    Sadly I didn’t have the resources to write a gem, but I’ll definitely check RABL out now that I know it exists.

    1. nesquena Post author

      RABL works with JSON as well as XML and you can even share templates between the two to reduce duplication. Let me know what you think :)

  6. Pingback: Wilcox Development Solutions Blog » Rails, REST, and .js (TL;DR: We’re doing it wrong)

  7. bobek

    Thank you for sharing RABL, I totally agree on pushing json/xml generation to view layer. I’ve done something similar but in crude way, so RABL is a great addition to toolbox.

    I have one remotely related question – what are you using for API documentation and generation of your on-line reference? I’ve learned that API documentation is the hardest part :) and still looking for nice tools for maintaining documentation.

    1. nesquena Post author

      Good question regarding API documentation and generation, which are definitely an important part of a good public API. Might do a separate blog post on that in the future. Glad you like RABL!

  8. Mike Williams

    Amen brother! Depending on #to_json and #to_xml ties API responses much to closely to the model.

    It’s great to see the wealth of solutions starting to appear to this problem.

    My own attempt at a solution is “Representative”, and it’s Rails-specific sidekick “RepresentativeView”. Like RABL, it support generation of JSON and XML from the same template, but it’s DSL is quite different, being modelled on the FormBuilder pattern.

    https://github.com/mdub/representative
    https://github.com/mdub/representative_view

    1. nesquena Post author

      Awesome, thanks for sharing. Love to see other solutions by people bitten by this same problem. Fascinating more people don’t talk about this widespread problem. So many people have responded to this expressing the same frustrations with to_json that we felt and yet when I googled this issue I didn’t see much discussion. Your library looks like a great alternative

    1. nesquena Post author

      Yeah we thought so too. Been awhile since we took the approach now and haven’t run into any issues with it yet.

  9. Pingback: Rails JSON templates through RABL | DCXN

  10. Pingback: Building a Platform API on Rails | Miso Engineering

  11. Pingback: Rails API’s – Rendering Flexible XML and JSON | Web Tempest

  12. Matthew Platts

    I was in a similar position to you guys as well, except my xml to render came from an object quite different from my ActiveRecord models. I like the flexibility of the builder gem, which also separates the xml building into the views. Due to lack of a json equivalent, I just converted the xml into json. This approach may appeal to some who don’t want to learn your new gem or who like the builder gem.

    See my approach here: http://webtempest.com/rails-apis-rendering-flexible-xml-and-json/

  13. Sven Felix Oberquelle

    I like the idea behind RABL. Actually I already had though about writing something like this a few times.

    But the syntax feels a little bit wrong in my opinion:
    object @user => :person

    I would write it just the other way round, like:
    object :person => @user

    And I would include blocks to make clear were attributes should belong to. For example like this:
    collection @posts do
    attributes :id, :title, :subject
    child(:user) { attributes :full_name }
    node(:read) { |post| post.read_by?(@user) }
    end

  14. Bryan (@mrpunkin)

    Still a ways out from implementing any sort of full API for our web app but I was researching some to_json stuff and ran across this. So glad I stumbled here, as this knowledge will be invaluable going forward as we provide ways to interact with our app, and maybe in the future provide a full API. Thanks for posting this.

  15. Pingback: Ginkgo-API: Technische Aspekte « Studentenblogs DDI@UPB

  16. Chris

    It is a lot easier and faster to put that logic into a ruby object. I think the existing as_json syntax could’ve been tweaked slightly to achieve an 80% situation.

Comments are closed.