Category Archives: Engineering

Asynchronous Processing in Web Applications, Part 2: Developers Need to Understand Message Queues

In the first part of this series, we explained asynchronous processing, when you might need to use it and why leveraging a database for that purpose is not necessarily the best option. In this post, we will explore a smarter approach to asynchronous processing using “message queues”.

Message Queues

Given the reasons that a traditional database is not the right approach for asynchronous processing, let’s take a look at the why a message queue might be a smarter choice. With a message queue, we can efficiently support a significantly higher volume of concurrent messages, messages are pushed in real-time instead of periodically polled, messages are automatically cleaned up after being received and we don’t need to worry about any pesky deadlocks or race conditions. In short, message queues present a fundamentally sound strategy for handling high-volume asynchronous processing.

Before we continue, it is important to mention that I am not claiming that a database can never be used as a queue. In fact, every database can be made to work at a low or medium volume as a queue given enough time and effort of a clever developer. Moreover, there are actually databases such as PostgreSQL that have additional support for queueing and solid libraries that can make this a manageable solution. If you are processing a low to medium volume of messages largely intended to be used as a job queue and you already use postgreSQL to store your data, there are undoubtedly cases where avoiding a separate message queue is the best choice at least until you feel pain. That said a message queue can be quite a bit more versatile, powerful and flexible depending on your needs. With that said, let’s dive into the world of message queues.

What is a message queue?

At the simplest level, a message queue is a way for applications and discrete components to send messages between one another in order to reliably communicate. Message queues are typically (but not always) ‘brokers’ that facilitate message passing by providing a protocol or interface which other services can access. This interface connects producers which create messages and the consumers which then process them. Within the context of a web application, one common case is that the producer is a client application (i.e Rails or Sinatra) that creates messages based on interactions from the user (i.e user signing up). The consumer in that case is typically daemon processes (i.e rake tasks) that can then process the arriving messages. In other cases, various subsystems will need to communicate with each other via these messages.

Features

On top of the fundamental ability to pass messages quickly and reliably, most message queues offer additional complementary features. For instance, multiple separate queues can allow different messages to be passed to different queues. This allows the messages of different types to be consumed by different services. The consumer processing email sending requests can be on a totally different queue (and/or server) then the one that resizes uploaded images. Message delivery behavior will often vary depending on your need. Certain messages will go from one producer to a single consumer (direct) while other times the message is sent to multiple different listening consumers (fanout).

Another critical feature of message queues is robustness and reliability through persistence strategies. In order to keep reliability of the messages high, most message queues offer the ability to persist all messages to disk until they have been received and completed by the consumer(s). Even if the applications or the message queue itself happens to crash, the messages are safe and will be accessible to consumers as soon as the system is operational. In contrast, transient tasks performed synchronously in a web app or on a thread in memory will be lost if anything goes awry. This is especially relevant when dealing with deployments and updating code since restarting components no longer put your messages or tasks at risk.

In addition, message queues can give you a greater visibility into the volume of messages. At the minimum, since there’s typically a broker for messages, inspecting the tasks being processed at any given time is much easier then with synchronous processing. Similarly, handling a high volume of tasks to process is simpler since you can just horizontally scale your message consumers to handle higher loads with minimal fuss. As an added bonus, the application itself now doesn’t have to deal with these tasks, so higher volumes of tasks won’t slow down the response times of the front-end web app.

Arguably though the most powerful reason to introduce a message queue is the architectural benefits that their use can afford. In particular, when you have message queues that enable lightweight communication between any number of disparate services, the ability to separate your application into many small subsystems is much easier which will often improve your architecture in several ways including making the individual pieces easier to maintain, test, debug and scale. In addition, with a services oriented approach made possible with message queues, you can easily have multiple teams working along well-defined boundary points and even using different tools or stacks when necessary.

Considerations

All that said, message queues are not a panacea and like all tools have their downsides. Setting up and configuring message queues, especially more complicated ones can add a lot of moving parts to your application. Often in small apps, you don’t actually need to introduce that overhead early on and can instead slowly offload tasks to message queues as the volume of traffic increases over time. Also, with a traditional message queue, error handling is sometimes a very manual effort if a message or task fails and communicating with message queues adds certain complexity into your application logic. In addition, for more general queues, you often have to define your own state machine for messages. That is, a message that needs to be passed to three different services for processing requires you to manually architect a queue workflow that supports that. Nonetheless, for asynchronous processing of many types, a message queue is often the best tool for the job depending on your needs.

Still, which message queue should we use then? What options and alternatives exist? Why would we pick one over another? What are the differences?

Comparing Message Queues

Overview

The good news and the bad news is that there are a lot of message queues to choose from. In fact, there are dozens of message queues with all sorts of names, features, and different pros and cons such as sparrow, starling, kestrel, kafka, Amazon SQS, and many more that will not be discussed here at length.

The easiest place to start is to explain the emerging open standard protocols for message queues which are AMQP and STOMP. These are the most popular message queue standards around today and many of the message queues implement one or both of these protocols. In addition, there is also JMS (Java Message Service) which is widely used on-top of the JVM. For completeness, there is also MSMQ which is not covered in this post but is the defacto queue for most .NET applications.

There are certain generic message queue implementations that have emerged as well, many of which built on top of the aforementioned protocols. The most popular and well-supported general purpose message queues available today are RabbitMQ, Apache ActiveMQ and ZeroMQ.

How do these options compare? Well, fortunately all three tools are being actively developed and maintained, have a large user base, good documentation and decent client libraries across many languages. So in a way, you can’t really go wrong with any of these options. However, let’s spend a moment understanding how they compare with one another. In the end, the one to use depends on the needs of your application and the required flexibility.

ZeroMQ

The interesting thing to understand is that ZeroMQ is actually not so much a pre-packaged message queue like the others but instead acts as a framework for building message queues. ZeroMQ focuses mostly on just passing the messages very efficiently over the wire while RabbitMQ acts as a full-fledged ‘broker’ which handles persisting, filtering and monitoring messages. ZeroMQ has no broker built in which means that it does not have a central dispatcher to manage your messages and is really not a “full service” message queue.

Think of ZeroMQ as it’s own toolbox or framework for creating message queues tailored to your own needs. An example is given above for using ZeroMQ to create a basic instant messaging service. RabbitMQ on the other hand tries to be a more complete queue implementation. RabbitMQ is much more packaged and as such requires less configuration and setup overhead for typical use cases.

Of course, with RabbitMQ or ActiveMQ, the broker and persistence built in adds quite a bit of overhead but those libraries choose to sacrifice raw speed to provide a much richer feature set with less manual tinkering. ZeroMQ is an excellent solution when you want more control or just want to do it yourself. In other cases where you just want to use a queue for typical use cases and you are willing to accept the higher overhead, you should consider RabbitMQ or ActiveMQ.

RabbitMQ and ActiveMQ

Comparing RabbitMQ with ActiveMQ is closer to a head-on comparison because they are solving similar problems. However, the differences here are mostly in the details. ActiveMQ is built in Java on the JMS (Java Message Service) and is very frequently used within applications on the JVM (Java, Scala, Clojure, et al). ActiveMQ also supports STOMP which provides support for Ruby, PHP and Python. RabbitMQ is built on Erlang, powered by AMQP and is used frequently with applications within Erlang, Python, PHP, Ruby, et al.

As you might expect, the developers preferences influenced the choices they make throughout the queues. For example, the configuration for ActiveMQ is in XML and the routing of messages is handled with custom rules defined by ActiveMQ. In contrast, the configuration of RabbitMQ is through an Erlang syntax and the advanced routing and configuration follows standard AMQP specifications. The protocols (AMQP vs JMS) used by each queue have certain underlying differences as well. One key difference is that in AMQP a producer sends a message to the broker without knowing the intended distribution strategy while in JMS the producer is aware of the strategy to be used explicitly.

The key takeaway here is that for a general purpose messaging queue to handle all your messaging needs and supporting most advanced requirements, you could use any of the three options listed above. However, my preference and recommendation in most cases for a Ruby or Python web application is to select RabbitMQ on AMQP. Conversely, if you are building on the JVM stack with Scala or Java, then my recommendation would be to use ActiveMQ. In either case, if you are looking for a customizable general message queue, then you won’t be disappointed. Of course, don’t be too quick to dismiss ZeroMQ if you are looking for raw speed and are interested in a light-weight, do-it-yourself protocol for delivering messages.

Specialized Queues

Incidentally, this is not the end of the story though. While message queues are incredibly helpful here, the major ones listed were built to be generic and general purpose. That is, they don’t solve one particular use case but instead support a wide plethora of use cases requiring varying levels of configuration. These ‘industrial strength’ message queues can power chat rooms, instant messaging services, inter-service communication, or even multiplayer online games.

For the average web application though, the requirements are very different. To understand your requirements, ask yourself if you are sending messages to communicate between different services in your application or if you just want to process simple background jobs. In the latter case, we certainly don’t need the most powerful or flexible message queue nor the expensive associated setup costs.

Most popular web applications really only need a way to do background job processing and offload tasks to an asynchronous queue. These more specific and constrained requirements open up the possibility for a lighter-weight message queue that is easier to use and focused on doing one thing well.

Takeaways

In this article we have covered the features of a message queue, how they work, popular message queue libraries and how to understand if you need a general message queue or a more specialized one. Hopefully you now understand the potential benefits introducing a message queue into your system architecture but also the tradeoffs and overhead of adding it prematurely. In addition, you should be able to see how a message queue works complementary to a traditional database and how queues used appropriately can help improve the modularity, maintainability and scalability of your applications.

In the next part of this series, we will explore a specialized message queue specifically built to process background jobs called a work queue. We will explore how a work queue operates, what features they typically have, how to scale them and which are most popular today. Hope that this introduction was helpful and please let me know what topics or issues you’d like to have covered in future parts of this series.

Asynchronous Processing in Web Applications, Part 1: A Database Is Not a Queue

When hacking on web applications, you will inevitably find certain actions that are taking too long and as a result must be pulled out of the http request / response cycle. In other cases, applications will need an easy way to reliably communicate with other services in your system architecture.

The specific reasons will vary; perhaps the website has a real-time element, there’s a live chat feature, we need to resize and process images, we need to slice up and transcode video, do analysis of our logs, or perhaps just send emails at a high volume. In all the cases though, asynchronous processing becomes important to your operations.

Fortunately, there are a variety of libraries across all platforms intended to provide asynchronous processing. In this series, I want to explore this landscape, understand the solutions available, how they compare and more importantly how to pick the right one for your needs.

Asynchronous Processing

Let’s start by understanding asynchronous processing a bit better. Web applications undoubtedly have a great deal of code that executes as part of the HTTP request/response cycle. This is suitable for faster tasks that can be done within hundreds of milliseconds or less. However, any processing that would take more then a second or two will ultimately be far too slow for synchronous execution. In addition, there is often processing that needs to be scheduled in the future and/or processing that needs to affect an external service.

In these cases when we have a task that needs to execute but that is not a candidate for synchronous processing, the best course of action is to move the execution outside the request/response cycle. Specifically, we can have the synchronous web app simply notify another separate program that certain processing needs to be done at a later time.

Now, instead of the task running as a part of the actual web response, the processing runs separately so that the web application can respond quickly to the request. In order to achieve asynchronous processing, we need a way to allow multiple separate processes to pass information to one another. One naive approach might be to persist these notifications into a traditional database and then retrieve them for processing using another service.

Why not a database?

There are many good reasons why a traditional database is not well-suited for cases of asynchronous processing. Often you might be tempted to use a database because ther can be an understandable reluctance to introduce new technologies into a web stack. This leads people to try to use their RDBMS as a way to do background processing or service communication. While this can often appear to ‘get the job done’, there are a number of limitations and concerns that should not be overlooked.

There are two aspects to any asynchronous processing: the service(s) that creates processing tasks and the service(s) that consume and process these tasks accordingly. In the case of using a database as a method for this, typically there would be a database table which has records that represent notified tasks and then a flag to represent which state the task is in and whether the task is completed.

Polling Can Hurt

The first issue revolves around how to best consume and process these tasks stored within a table. With a traditional database this typically means a service that is constantly querying for new processing tasks or messages. The service may have a database poll interval of every second or every minute, but the service is constantly asking the database if there has been an update. Polling the database for processing has several downsides. You might have a short interval for polling and be hammering your database with constant queries. Alternatively, you could perhaps set a long interval in which case there will be many unnecessary processing delays. In the event of having multiple processing steps, the fastest route through your system has now been delayed to the sum of all the different polling intervals.

Polling requires very fast and frequent queries to the table to be most effective, which adds a significant load to the database even at a medium volume. Even worse, a given database table is typically prepared to be fast at adding data, updating data or querying data, but almost never all three on the same table. If you are constantly inserting, updating and querying, race conditions and deadlocks become inevitable. These ‘locks’ occur because multiple consumers will all be “fighting” with each other over the same table and with the ‘producers’ constantly adding new items. You will see load increase for the database, performance decrease and pile-ups become increasingly common as the volume scales up.

Manual Cleanup

In addition to that, clearing old jobs can be troublesome as well because at even a medium volume, there will be many tasks in the database table. At certain intervals, the old completed tasks need to be removed otherwise the table will grow quite large. Unfortunately, this has to be performed manually and deletes are even often not particularly efficient for tables especially when being removed so frequently in conjunction with updates and queries all happening at the same time.

Scaling Won’t Be Easy

In the end, while a database will appear to work at first as a simple way to send background instructions to be processed, this approach will come back to bite you. The many limitations such as constant polling, manual cleanups, deadlocks, race conditions and manual handling of failed processing should make it fairly clear that a database table is not a scalable strategy for this type of processing.

Takeaways

The takeaway here is that asynchronous processing is useful whether you need to send an sms, validate emails, approve posts, process orders, or just pass information between services. This type of background processing is simply not a task that a traditional RDBMS is best-suited to solve. While there are exceptions to this rule with PostgreSQL and its excellent notify support, there is a different class of software that was built from scratch for asynchronous processing use cases.

This type of software is called a message queue and you should strongly consider using this instead for a far more reliable, efficient and scalable way to perform asynchronous processing tasks. In the next part of this series, we will explore several different message queues in detail to understand how they work as well as how to contrast and compare the various options. Hope that this introduction was helpful and please let me know what you’d like to have covered in future parts of this series.

Continue to Part 2: Developers Need To Understand Message Queues

Hello New Friends, Hello Old Friend

I joined Miso’s engineering team last month after ten years of running my own companies and freelancing as a software developer. Several people asked me if I was worried about making the transition from “being the boss” to “having a boss.” Coming from people who know me, but who don’t know Miso, I could understand their concern. But after I met this team and got an insider’s view of some of the projects they’re (rather, we’re!) working on, any anxiety I may have had was replaced by pure excitement. I am thrilled to be a part of this very talented team and I can’t wait to see our amazing projects take flight.

Joining Miso also means that I have the opportunity to re-immerse myself in the ruby community after spending the past 18 months on a java gig. One of the chief complaints I had with ruby (and rails) was the difficulty in running multiple versions on the same machine and maintaining a sane set of gems for each rails project. While I was “gone” the ruby community solved this problem. Twice. Using your choice of RVM or rbenv plus Bundler, you can run any number of different Ruby versions mixed and matched with any number of gemsets. (I just discovered that RVM now even has a Mac GUI.) One gotcha with RVM that bit me: sudo is no longer your friend when it comes to installing gems. In fact,

sudo gem install xyz

will totally subvert RVM’s clean bucketing of your gems into gemsets, so I needed to quickly break the habit of using sudo to let RVM work its magic….

Manging rubies with RVM or rbenv and dependencies with bundler is just the tip of the iceberg. The more I dig back into ruby, the more progress I’m finding the community has made in tools to support test-driven development, mock/factory objects, Rails alternatives (buona sera, Padrino!) . . . the list goes on and on.

By joining Miso, I got to meet a whole bunch of new friends. By diving back into ruby, I reconnected with an old friend and found out that he had gotten in shape, bought a whole new wardrobe and learned kung fu.

Yet Another Pull-To-Refresh Library

Pull to refresh is by far the most popular paradigm used in refreshing feed based views on an iOS device. First introduced by Loren Brichter of Tweetie, pull to refresh has made its way into just about every touch based device out there. Here’s the latest incarnation on the PS Vita (taken from @mattgemmell):

In fact, the idea was so good that Brichter decided to legally slap his name on it. Here’s a picture to help you better understand how this technology works (taken from gorumors.com):

Thankfully, Brichter doesn’t require you to obtain a license to use it shamelessly in your own app, bless him!

Current Solutions

Now, our own Miso app also uses this familiar pattern to refresh our views. But the task of integrating a 3rd party library into our app was not as idiot proof as we had hoped. Here are some popular libraries and why they made me feel uncomfortable:

Enormego’s EGOTableViewPullRefresh
Leah Culver’s PullToRefresh

Both libraries are essentially the same, with slight variations. Both libraries employs the use of 3 absolutely crucial methods of UIScrollViewDelegate:

    - (void)scrollViewWillBeginDragging:(UIScrollView *)scrollView;
    - (void)scrollViewDidScroll:(UIScrollView *)scrollView;
    - (void)scrollViewDidEndDragging:(UIScrollView *)scrollView willDecelerate:(BOOL)decelerate;

In enormego’s implementation, it is expected that the developer chain the scrollview’s delegate methods to the EGOTableViewPullRefresh to drive the refresh cycle’s core logic. This isn’t ideal because it forces the developer to implement those delegate methods regardless of the developer’s requirements. This solution could definitely be more self-contained and less intrusive on the clients code.

In Culver’s implementation, the problem is the opposite of enormego’s. The delegate methods are literally stolen from you in order to drive the logic. If you implemented them yourself, the entire thing breaks.

One last variant is to have the library become the UIScrollView’s delegate, but also chain every single method back to the original delegate. This is fine, except it becomes extremely laborious to maintain especially if your UIScrollView is actually a UITableView (which extends the protocol UIScrollViewDelegate), or if apple adds some more delegate methods.

Both libraries combine the logic and views into one class, perhaps for the sake of simplicity. What if I didn’t want a boring rotating arrow but an animating rainbow? There’s basically no way to customize the views without modifying the source code directly.

Introducing MSPullToRefreshController

Here’s my main beef with existing libraries:

Too intrusive on my own code. I either play an essential part to making the library work, or the library will not work if I exercise some innocent freedoms.
I couldn’t provide any custom views.
I couldn’t provide any custom behaviors to the refresh cycle.

Instead of whining like I did, our very own CTO (Tim Lee) offered an actual solution to Beef #1: KVO (Key Value Observing). Specifically, to observe the contentOffset property of UIScrollViews.

        [_scrollView addObserver:self forKeyPath:@"contentOffset" options:NSKeyValueObservingOptionNew | NSKeyValueObservingOptionOld | NSKeyValueObservingOptionPrior context:NULL];

Using this trick, I was able to secretly offer pull to refresh features to any UIScrollView without having to become its delegate. With this simplification, I was able to quickly extend pull to refresh to all 4 edges of a UIScrollView, just in case you wanted to use the bottom side to load more, or if your feed was laid out in a carousel fashion.

Through the small exercise above, I saw MSPullToRefreshController as a self-contained object that handles nothing more except managing the logic involved pull to refresh cycles. It just didn’t make sense to add anything else to it, unless it enhanced the core behaviors. Translation: Beef #2′s solution doesn’t belong in this class.

To solve Beef #2 and #3 together, I decided to introduce the familiar delegate pattern to MSPullToRefreshController. It is through these methods that the developer will be able to customize behaviors and transform custom views to reflect each state in the refresh cycle.

Now I could easily write a wrapper around a MSPullToRefreshController instance and recreate any existing pull to refresh library by implementing 6 easy delegate methods! In fact, I’ve done just that in the included sample project on github!

Conclusion

Though this library is very simple, this marks my first step in to the world of open source (to reinforce our engineering culture per @nicolas). As @joshbuddy and @nesquena would say, 99% of open source projects stem from frustrations with existing solutions (or lack thereof). I hope somewhere someone will find this more suitable to his/her needs. Thanks for reading!

Why American Entrepreneurs Are Superior…

I’ve been living in San Francisco for more than 2 years. I wasn’t an entrepreneur when I moved here and I’m still not. Hopefully one day I will be, but I guess those 2 years have been more of an introduction to entrepreneurship. Discovering the tech startup world in San Francisco has been far more fulfilling than my last 5 years studying in college. Coming from France, attuning to American culture has been a long process for me, and it has taken some time for me to embrace it. Here are some takeaway lessons I’ve picked up over these last years.

American Story Telling

The simple example I have in mind is that we (as French people) don’t know how to talk about ourselves. Actually, I feel more comfortable to speak about myself in English than in French, probably because I first learned to tell my story in english. So you might ask why I didn’t learn to tell my story in French first? I believe it’s culturally and linguistically different. French is a refined and elaborate language and we really care about our look but sometimes I feel like it’s too much, and we don’t get straight to the point. For instance, at the end of a formal letter, in English we usually write “Yours faithfully” or “Sincerely“, while in French we would write “veuillez agréer, Monsieur Dupont, l’expression de mes sentiments distingués“, which translates to “Please accept, dear sir, the expression of my most humble greetings“. In the end, our desire to appear so eloquent adds many barriers and limits our effectiveness as story tellers.

Don’t get me wrong! French is a beautiful and complex language, but when the matter is business or to market a product, French doesn’t sound so great. Just check this iPhone ad taken from apple.com and apple.fr:

There’s 1 thing that I would point out: it’s “dernier cri” which means “latest fashion”, my guess is that “dernier cri” was the less worst translation they could find, that expression is way too complicated and it’s not commonly used in french, and ironically to me it’s almost old fashioned to say “dernier cri”.

American Positive Thinking

The most striking thing to me since I’ve been here is how positive people are day to day. Here I have noticed everyone gets excited for all sorts of things: trying a new restaurant, a new product launch, etc. In French, we don’t have any equivalent translation to say “I’m excited”. The closest equivalent would be “Je suis enthousiasme” (I am enthusiastic), but I would never say “Je suis excité(e)” unless you are talking to your boyfriend or girlfriend with a sexual connotation :)

Lack of French Role Models

This lack of our expression of excitement about anything could turn against us. How could you build a successful company if you can’t show how much you’re excited about it and how much you believe in it? At Miso, sometimes I feel like I am the most skeptical person because I don’t always know how to show my excitement about something. It becomes alarming if you ask me if to name any french entrepreneur role model… Certainly not Bernard Arnaud (CEO LVMH), neither Arnaud Lagardère (Groupe Lagardère). I’ve a lot of respect for them, but it’s not that kind of entrepreneur that I would admire.

Xavier Niel (CEO Free) would be the first I would name, he’s one of few willing to make a difference, willing to change how wireless market works in France, that by the way one of the things Americans should take example of. Also I’m pretty sure there’re a lot successful french entrepreneurs out there, it’s I think we don’t have enough inspiring models in France, Michel Lévy-Provencal (CEO Joshfire) or Stéphane Distinguin (CEO faberNovel) could be good candidates, it just a matter of exposure.

On other hand if you ask the same question for americans, that list can be very long. If I had to pick one, I would name Kevin Systrom (CEO Instagram), I’m still amazed how successful Instagram has been. Only a 2 year old company and less 10 employees with 25 millions users and only on iPhone! I would recommend to watch this interview to see an example of a good role model.

Conclusion

For the time being, I enjoy being a software engineer at Miso and watching Somrat and Tim as entrepreneurs. It is inspiring to me and to the rest of the team, and eventually one day I will step up to try for myself. First things first though, let’s change the way that people watch TV. #drinkthesoup

My journey though open-source and fun JSON tricks

I recently attended a Ruby on Rails speed dating event here in San Francisco. Participants there were attempting to meet companies who would hire them on fulltime. And they only had five minutes to wow you. This is all over the din of 39 other participants attempting to do the same thing and with the shouting of the organizer barking out minute-by-minute reminders. You’re gonna have to be impressive to stand out, and you will have to leave something behind that you can be remembered by.

I’ve been doing open source software for a few years now. I came to Miso through open source. I really like open source, and within the Ruby community, most of that open source development seems to occur through Github. When I did the speed dating rounds, I asked people for their Github account names (I would have accepted self-hosted alternatives). Some people were nervous to share; some people didn’t have a Github account. I understand that, because…

Open source is scary

When I first started doing open source, I was struggling with what to work on. I didn’t feel like I had any projects I could easily contribute easily to. I had recently read Stefen Goessner article on JSONPath, and that made me think, maybe I could code this in Ruby. I looked around for an existing implementation in Ruby and when I couldn’t find one, I set out to write my own.

I hacked around for half a day, got something working and submitted my request to Rubyforge to have my project hosted. It took two days to get approval, but once I had that, I thought, I was finally on my way to becoming an open source hero. I didn’t have any particular need for the Gem I had written, it was more of a fun challenge, but there was something cathartic about getting my code out there for everyone to see.

My first Ruby gem, Jsonpath

I released my first Ruby gem and I was pretty proud of myself. I could do things like:

require 'jsonpath'

object = [
  {"title" => "Sayings of the Century", "price" => 18.75},
  {"title" => "Moby Dick", "price" => 20.95}
]

JsonPath.wrap(object).path("$..price").path("$..price").to_a
# => [18.75, 20.95]

I had implemented the full Jsonpath syntax; I even supported the odd array slice syntax and filter syntax. I thought I was doing well.

Later I opened a Github account and got to put my code up there. While the experience on Rubyforge was more akin to putting your code in a gallery for the world to see, Github was much more of a living, breathing thing. Suddenly, my README was on the front page, and, could I let my old README sit there unchanged? Though at the time, I thought I was pretty clever, pretty soon I felt shame. How could I not have documented my code better, for instance.

Getting over shame

Shame shapes the software we’re willing to release and not release. @mattmight recognized this shame within academia, and released the CRAPL license to address it. This license claims to “absolve authors of shame, embarrassment and ridicule for ugly code”. Looking back on Jsonpath, I feel bad. My Ruby knowledge was weak, I didn’t even know how to package up a gem correctly, I didn’t have the sense to add any sort of documentation to my project. But getting over my lack of knowledge was useful. My gem has proven to be useful to others, and in turn, those other people have submitted pull requests. My interactions through open source have made me a better coder, and I’m grateful for it.

Speed dating and Ruby

If you’re coming to a Ruby employment event, the best place to start to wow people is via open source. Start with something you care about, even if that program is only useful to you. It doesn’t have to be beautiful, it can just be your “learning how to code” project. If you’re looking for work as a programmer, undoubtedly you’re working on some projects in your spare time to sharpen your skills. Ultimately, code is not meant to sit in a gallery; your code is art waiting to be seen by the wide world. I remember once Matz was asked as then end of one of his talks what he thought the most beautiful code was. His response was simply: “Your code.” Even if it’s not completely beautiful yet, it’s yours, and that sense of ownership and passion will come through in how you express yourself and see your own code.

What I’m not looking for is impressive code, though, I’m happy to find that as well. What I want is to know you’ve gotten over your shame. It’s not just in the academic community where shame and fear rules, but that in many programming communities, the resistance to released half-finished code or “messy” code is strong, and code stay locked up.

The only consolation is that by getting over this shame and being willing to share your code is that you will get feedback, you will meet people who will help you, you’ll learn to participate in other people’s projects.

I’m still ashamed, but that’s okay

I’ve released version 0.5.0 of this gem, but it’s not really finished. (It never really is.) When I look back on my implementation which has remained largely unchanged, I know I could do a lot better. I intend to re-write this gem at some point, but, my “good enough” implementation has been helpful to people I don’t even know. I’m not sure the shame ever really goes away. And that’s okay. Soon your shame will be joined by other emotions. Collaborating with friends on a new project. The thrill of your first pull-request. Getting to submit your own first pull request to another project. Maybe even a project you use. Or other people use.

If you’re still hesitating, that’s okay. Looking back though, I’m glad I got over my own fears and released something, even if it’s not entirely pretty or finished, at least I got started.

Adventures in Scaling, Part 3: PostgreSQL Streaming Replication

In my last post in this series, I overviewed how to get PostgreSQL setup, running and properly tuned on your system. This works well to setup a primary database for your system but depending on your needs, you may find yourself in a position where you would like to start taking advantage of the Streaming Replication and High Availability built into PostgreSQL 9.

When running a database such as PostgreSQL on a server, a single primary master database that performs all reads and writes for your applications will likely not be able to sustain your traffic forever. At a certain point, your application will generate too many concurrent reads and writes, you’ll want to be able to recover from database failure, or for whatever reason your database will need help scaling.

Hot Standby? High Availability?

On a high level, the challenge with scaling out a database is a problem of synchronization. With a single database, there is a single source of truth for your application. All records are in one place which acts as the authority for your data. However, once you begin scaling your database, you have multiple servers all responsible for your data. As data is created, updated and destroyed, this information must necessarily be propagated as quickly as possible to all other servers as well. If servers are out of sync, data can be mis-reported or outright lost. Therefore, each database must be in a consistent state with the others with little margin for error. Since no particular approach solves this problem perfectly for all needs, instead there are many strategies for scaling a database.

The approach you should take as load increases has a lot to do with the specific characteristics of your application. Is your application read-heavy? or write-heavy? If your application has many more reads by volume, then the most common strategy is to setup secondary databases that synchronize data with the primary database. This strategy of setting up read-only databases that track changes to the primary is only part of the scaling story. There will be also be cases where you have too many writes for a single box (multi-master) or you need to distribute data across many boxes (sharding), but this post will focus on solving the problem of simply scaling reads. Look for a future post on more advanced scaling and replication strategies.

Fortunately, the strategy for scaling reads is fairly straightforward. The primary database that accepts writes is called the “master” and all other databases that can only help with the reads are called “slaves”. Typically, you can setup as many slaves as you’d like to help distribute the querying load on the database. In addition, in some database setups, these slaves can also help in cases of database/server failure. In the event that the master database fails or becomes unavailable, a slave in a good position can elect to take over as the master so that your application can still accept and retrieve data without downtime. In this scenario, the slaves that can be elected to be masters are called hot standbys and this concept is a strategy contributing towards achieve high availability which means that your database is robust against failure.

Understanding Replication

To understand how to achieve high availability and how multiple PostgreSQL databases keep synchronized, one must understand the concept of “continuous archiving” and “write-ahead logs”. The key concept of high availability has to do with the database being resilient to server failure. Specifically, in the event that one database server goes down, another one should be ready to step up and assume the responsibilities. This secondary server that must be ready to step up at any time is called a “standby”.

In order to achieve this synchronization, the primary and standby servers work together. The primary server is continuously archiving the data coming in or being modified, while each standby server operates in continuous recovery mode, reading the archives from the primary. This “continuous archive” is a comprehensive log of all activity coming into the database (every table created, every row inserted, et al). This comprehensive log is called the “write-ahead log” or WAL and is essential to the replication process.

WAL is described well in the PostgreSQL user’s guide:

Write-Ahead Logging (WAL) is a standard method for ensuring data integrity. WAL’s central concept is that changes to data files (where tables and indexes reside) must be written only after those changes have been logged, that is, after log records describing the changes have been flushed to permanent storage. If we follow this procedure, we do not need to flush data pages to disk on every transaction commit, because we know that in the event of a crash we will be able to recover the database using the log: any changes that have not been applied to the data pages can be redone from the log records.

As logs are being written by the master, these log (WAL) records are then transferred to other database servers that act as standbys. Generally the best way for this process to happen is to streams WAL records to the standby as they’re generated, without waiting for the WAL file to be completely filled. This process of moving the WAL records (in small chunked segments) in this way is known as “Streaming Replication”. One important aspect to note about this process is that transferring happens asynchronously. This means that the WAL records are shipped after a transaction has been committed so there is a small delay between committing a transaction in the primary and for the changes to become visible in the standby. However, “streaming replication” has delays generally under one second unlike older transfer strategies.

Setting This Up

Great, now that we have covered the high-level concepts, let’s jump into how to setup hot standby streaming replication step-by-step. This guide assumes you have a UNIX server provisioned (Debian-based) and have already followed our guide to setup PostgreSQL on that machine.

The first thing to do to setup a standby is to provision another server (preferably same specs as the primary) and follow the steps to setup the basic PostgreSQL server again as similarly to the primary database as possible. Ideally, the standby is essentially a clone of your first server, so if you can duplicate the server, feel free to do that as a time saver. These database servers should ideally be part of the same local network and communicate through their private IPs.

Master Setup

Next, we need to setup your master such that the database can connect to your standby machines in /etc/postgresql/9.1/main/pg_hba.conf:

# /etc/postgresql/9.1/main/pg_hba.conf
host   replication      all             SLAVEIP/32              trust

Be sure to replace SLAVEIP with the correct IP address for the standby. Next, it is a good practice to create the empty WAL directories on both the master and the slaves:

master$ sudo mkdir /var/lib/postgresql/9.1/wals/
master$ sudo chown postgres /var/lib/postgresql/9.1/wals/
slave$ sudo mkdir /var/lib/postgresql/9.1/wals/
slave$ sudo chown postgres /var/lib/postgresql/9.1/wals/

You should also be sure that the master has easy keyless SSH access to the slave:

su - postgres
ssh-keygen -b 2048
ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave.host.com

Now that your master can easily access your slave, you will need to enable streaming replication and WAL logging for your database in /etc/postgresql/9.1/main/postgresql.conf:

# /etc/postgresql/9.1/main/postgresql.conf
# ...
# Listen allows slaves to connect
listen_addresses = 'MASTERIP, localhost'
wal_level = hot_standby
wal_keep_segments = 32 # is the WAL segments to store
max_wal_senders = 2 #  is the max # of slaves that can connect
archive_mode    = on
archive_command = 'rsync -a %p root@slave.host.com:/var/lib/postgresql/9.1/wals/%f </dev/null'

Now create a user that the slave can use to connect to master:

> SELECT pg_switch_xlog();
> CREATE USER replicator WITH SUPERUSER ENCRYPTED PASSWORD 'secretpassword12345';

At this point you should probably restart PostgreSQL:

sudo /etc/init.d/postgresql restart

In order for the standby to start replicating, the entire database needs to be archived and then reloaded into the standby. This process is called a “base backup”, and can be performed on the master and then transferred to the slave. Let’s create a snapshot to transfer to the slave by capturing a binary backup of the entire PostgreSQL data directory.

su - postgres
psql -c "SELECT pg_start_backup('backup', true)"
rsync -av --exclude postmaster.pid --exclude pg_xlog /var/lib/postgresql/9.1/main/ root@slave.host.com:/var/lib/postgresql/9.1/main/
psql -c "SELECT pg_stop_backup()"

This will transfer all the binary data from the PostgreSQL master to the slave. Next, time to setup the standby machine to begin replication.

Standby Setup

First, let’s shutdown the database on the standby:

sudo /etc/init.d/postgresql stop

Let’s enable the slave as a hot standby:

# /etc/postgresql/9.1/main/postgresql.conf
# ...
wal_level = hot_standby
hot_standby = on

and modify the recovery settings:

# /var/lib/postgresql/9.1/main/recovery.conf
standby_mode = 'on'
primary_conninfo = 'host=MASTERIP port=5432 user=replicator password=secretpassword1234'
trigger_file = '/tmp/pgsql.trigger'
restore_command = 'cp -f /var/lib/postgresql/9.1/wals/%f %p </dev/null'
archive_cleanup_command = 'pg_archivecleanup /var/lib/postgresql/9.1/wals/ %r'

One interesting option in the file above that is worth noting is the trigger_file parameter which tells the standby when to become the master. In the event of a machine failure and that file becomes present, the slave will be triggered and become the master. In the event that this standby is one day elected as the master, you need to ensure certain files are in place. Let’s clone the connections listed on master:

# /etc/postgresql/9.1/main/pg_hba.conf
host   replication      all             MASTERIP/32              trust
host   replication      all             SLAVEIP/32              trust

At this point, the slave should be ready to start accepting WAL records and replicating all data from the primary. Let’s restart the database:

sudo /etc/init.d/postgresql start

Verification

Check the logs to make sure that the connection was properly established on the slave (and/or master):

# /var/log/postgresql/postgresql-9.1-main.log
LOG:  entering standby mode
LOG:  streaming replication successfully connected to primary

If you see those two lines, then this means replication has started successfully. If not, follow the error messages and fix the configuration until you can restart and see these lines.

You can also check certain processes on the machines to ensure replication is established:

master$ ps -ef | grep sender
slave$ ps -ef | grep receiver

If both machines have the expected process, then that means that replication is running along happily.

Wrapping Up

At this point, replication should be established on the standby machine and you can repeat this process for any number of additional standbys. Once a standby is established, the WAL records should be continuously streamed to the machine keeping the data synchronized with the primary. In the event of a disconnection, the standby is generally resilient enough to backfill data and catch back up to the primary.

Once the standbys are in place, since they have been configured as ‘hot’ they can actually be used easily as read slaves. Simply setup your application to read from these slaves but only write to the master. If you are using Rails, one good gem to support this concept of standbys is called Octopus.

Hopefully you have found this post useful as a straightforward guide to understanding and setting up hot standby streaming replication. In a future post, we hope to cover more advanced replication strategies as well as a detailed guide from how to do a large data migration from MySQL to PostgreSQL. If you run into any problems or spot errors in this guide, please let us know so we can update the steps.

Work Efficiently With XCode

XCode often slows down the entire system to a crawl, forcing the coder to either restart the application or even the computer itself. Unfortunately, we can’t control how aggressively XCode uses system resources. Fortunately, we can learn to work faster and more efficiently by learning some of the hidden conveniences inside XCode.

Below are some tips and tricks I use in my everyday coding sessions. This post has the beginner in mind, though I hope advanced and expert XCode users may occasionally find something useful.

**Disclaimer: Examples below are done using XCode 4.2.1

Master Your Shortcuts

The goal here is to use the hot keys to speed up your work flow. Reducing the number of times you use your mouse to navigate the UI will help you stay focused on the task at hand by recapturing your attention to the cursor back to the actual lines of code.

Open new tabs using ⌘T, navigate between them using ⌘⇧{ or ⌘⇧}
Build/Run/Clean using ⌘B/⌘R/⌘K, respectively
Quickly open files using ⌘⇧O and enter file name
Go to a line using ⌘L and enter line number
Jump between .h/.m files using ^⌘↓
Navigate forwards/backwards in a single tab using ^⌘→ or ^⌘←
Go to the end or beginning of a line using ⌘→ or ⌘←
Find text using ⌘F
In the middle of typing, Tab to autocomplete, Esc to see close matches
Highlight some code and ⌘/ to comment/uncomment
Indent or dedent(?) code using ⌘] or ⌘[

This is not an exhaustive list, but these are certainly enough to get you started on training your muscle memory. If you have any other cool shortcuts you’d like to share, please leave it in the comments below.

Maximize Your Usage of Screen Real Estate

How you do this will be a little bit of personal taste. In any case, the point here is make it so everything you can see on the screen is relevant to the task at hand. For example, don’t clutter your windows with unnecessary panels that you will never use. Do you really need the debugger panel on the bottom if you are coding? Or the object panel on the right unless you’re editing a .xib?

Note that window layouts are done on a per-tab basis, and creating a new tab will clone the layout from the previous tab.

Here’s a screenshot of how I chose to layout a single tab for coding:

Let’s consider this portion of the screen (top right):

Direct your attention to 3 buttons (which are mutually exclusive) above “Editor”. From left to right they represent Standard Editor, Assistant Editor, and Version Editor. I use the Assistant Editor, this allows me to edit my implementation file in the middle while having an auxiliary editor on the right, which I use exclusively for “counter parts”, or more specifically, header files. You are free to use a drop down menu to configure what you would like to be placed there. Like so:

If your screen is too small to support two editors, consider using multiple tabs instead.

Next, consider the 3 buttons above “View”. Toggling each in turn will slide in and out a panel from the left, bottom, or right. In this case, I only want the left panel, and more specifically, I only want to see the 4th tab (⌘4) in this panel for any warnings/errors I may introduced while coding this class.

**Aside: You may have noticed the presence of line numbers, which are very useful when looking at code with another person. To turn this on, find the option under the “Text Editing” tab in Preferences (⌘,) .

Moving on, let’s take a look at a tab I use for debugging (a topic to be addressed in a future blog post):

Here’s the setup. A Standard Editor to get the maximum view of a breakpoint’s context. A left panel (5th tab) to detail the thread and stack trace. Also note that I eliminated the Toolbar at the top altogether. A Debugger panel on the bottom for, well, debugging. Note that you also want the split view for the debugger so you can see the local variables and the log at the same time.

Miscellaneous

Handling XCode Behaviors

Say you have two tabs open, one for file1 and the other for file2. You also have a breakpoint in file1, but you are on file2′s tab. Now you run the project and it hits the breakpoint in file1, the default behavior will change the tab with file2 into file1, leaving you with an annoying situation where you have two tabs that are looking at the same file. After a few more hours you’ll end up with multiple duplicates, and now you need to either close the tabs or change the duplicates back to what it should have been.

Here’s how I get around this inconvenience. XCode allows you to change its behavior at crucial times of the development process. Here we are interested to when we hit a breakpoint, which is equivalent to saying the “Run Pauses”, note that this will also happen in the case of a crash. To change this behavior, go to Preferences and mimic this:

In top-down order, the lines with blue check marks tell XCode to do the following.

Show the tab named “Debug” (it will create one if this doesn’t exist)
Show the 5th tab on the left panel (the one with stack traces)
Show the bottom panel (debugger) with the splitview
Hide the toolbar at the top
Show the Standard Editor

And this pretty much recreates my debugging layout for me. As you can see, there’s a lot of behaviors you can customize and automate, in the end, it is a matter of taste.

Alternative to NSLogs

Consider a different situation, you’ve built and ran your app. You’re noticing some weird behaviors, and you know it may involve a couple functions. To investigate these functions’ order of invocation and/or relationships, you stop the app and start filling in some NSLogs and rerun.

Don’t do this!! Instead, deploy a couple breakpoints in key places and right click on them to edit, pulling up this menu:

This menu is extremely powerful. You can conditionally enable/disable a breakpoint, ignore for some number of times, perform an action, or to automatically continue.

Using a combination of the above, you can effectively sprinkle “adhoc” nslog statements without having to stop your app and recompile the whole thing. And if you know more about gdb or lldb, you can print out values of surrounding local variables too!

Jump to Relevant Definition or Documentation

Maybe you want to see where something is defined. You might try to ⌘F on the project and look for it, but instead you can ⌘+(left click) on an interesting block of character, such as a method name, and it will either pop up a menu to ask you to be more specific, or take you to the only definition right away! You can do the same trick for Apple’s objects as well. You can do the same trick for Apple’s objects as well.

Have you ever tried to search the Documentation in Organizer? It takes an eternity. Instead, use ⌥+(left click) on something you want to search for to bring up a menu like this:

Not satisfied? Click on any of the blue text to bring you full fledged documentation window. I’ve literally typed what I wanted to search in some random place just so I can do the above trick to get to the documentation as fast as possible.

Global Breakpoints

When your application crashes, you probably go into the debugger to identify where the crash happened, then you set up some breakpoints before the offending line, and try to reproduce the exact steps. And hopefully, you’ve reproduced your steps to hit the exact same crash. What if you can insert a breakpoint dynamically right when the system throws the exception? Here’s how:

⌘6 to bring up the breakpoint navigator
Click + on the bottom left
Add Exception Breakpoint…
Done

If you’ve done it correctly, you should see:

That’s it! Next time you crash, you’ll be able start investigating right away!

Conclusion

Now that you know how to work smarter, hopefully you can spend the rest of your time on things that actually matter, such as implementing a feature or fixing a crash. While I’ve dipped into a little bit of tricks regarding debugging, it really deserves its own blog post. So next time, I’ll be giving a small tutorial on how you can use the various instrument tools to help you manage your apps memory usage and performance bottlenecks. Thanks for reading!

You’re Overthinking It

You comb Hacker News daily, marveling at the neatly packaged startup tales, uber-effective best practices, super clever engineering solutions, and lots and lots of links to websites filled with Helvetica, minimalism, and pastel colors. You’ve attended Lean Startup workshops, read Four Steps to the Epiphany, and subscribe to the Silicon Valley Product Group blog.

Honestly, it’s all very intimidating.

My product advice, from one overthinker to another overthinker – throw it all away. I mean, read the articles, enjoy the stories, and try to form your own opinions, but I wouldn’t take it too seriously.

A year into my first startup, my first major product epiphany was to never, never, ever try to build a product you couldn’t be a user for. That may be obvious, but I still read people discussing strategies for building products that they don’t use. There is no better user study, no more accurate persona than asking yourself what is good. There are probably product people out there that can do it, but, no offense, it’s probably not you, and it’s certainly not me.

There are many pros to building a product that you would use. Actually using a product (and I mean really using it), allows you to access the powers of intuition, an infinitely more valuable product tool than reason. Your intuition explains to you in a moment what it takes your reason an hour to break down. Your reason will lead you down a dozen wrong roads.

Another way of saying that is: if you think it’s cool, it’s probably cool. If you think it sucks, it probably sucks.

However, building a product for yourself doesn’t give you a free pass from user research, personas, and all the other things that product gurus tell you to do. Spend a year having the same product discussions with the same group of people, and the discussions will lose all meaning. Talking to five people outside of the company will bring you back to earth real fast.

One last thing…whatever you build, make sure it looks good and is the highest quality possible.

But wait, you say, look at eBay, Amazon, and Craiglist – they look like crap. Implement an MVP with product/market fit, and it doesn’t matter what it looks like. That’s true sometimes, but it also depends on where you are on Maslow’s hierarchy of needs. The lower on the pyramid your product is, the crappier it can look. If your product is core to helping people make money, pirate movies, or sell your useless couch, you don’t need a designer. But if you’re high on the pyramid, ugly/clunky UI makes it impossible to for people to see your vision.

If you read Steve Jobs biography, it talks about one of the three original Apple philosophies: an odd word called impute. It’s basically a philosophy around impressions. On product, they said, “If we present them in a slipshod manner, they will be perceived as slipshod”.

Speaking of the biography, I’ll wrap up with my impression of that book. It’s a story about a product genius, but it’s a story with as many missteps as triumphs. I take the moral of the story to be: forget the experts, the know-it-alls, and the doubters. Trust yourself and your vision, and go build something.

Quips: Hackathon #2 Recap

It’s been a while since our last hackathon, so we were excited to do another to end the year 2011 at Miso. In the 6 months since our last hackathon, the team at Miso has grown significantly and that has allowed us the option to split into two different teams to work out on separate ideas. I got placed on the Quips team, so that’s what I’ll be talking about today.

What is Quips?

Have you ever hung out with friends, and someone said something so hilarious you wanted to quote them on your social network of choice? That’s the moment we wanted to capture. Drawing inspiration from quote cards from Miso sideshows, we came up with an idea for an iPhone app that let’s you take a picture, slap a quote over it and share it over Twitter! Of course, what’s a photo sharing app without filters a la Instagram? We got that covered too. Our tag line? Appropriately: “Because your friends are hilarious”.

The Journey

The app took 5 of us just 3 days to complete. The team consisted of 2 shared designers (we had two teams), two engineers on iOS and one more on the backend server. Although a relatively simple app, from design and planning to execution, the whole process reminded me of those sleepless nights finishing senior design projects. The difference was that I had a lot of fun doing it, and the only thing that was on the line was my pride in being to do this in the short amount of time we had. At the end, we finished everything within 3 days, and submitted to the app store the day after. Overall it was a success and everyone in the office loved playing with the app.

Lessons Learned

Since it was the second hackathon I’ve participated in, a few mistakes were learned in the past and some news ones were made this time around. I thought it’d be helpful to share a few tips for those attempting a hackathon themselves.

Small and focused – Google wasn’t built in a day, so don’t expect that of yourselves. Avoid ideas where it involves hundreds and thousands of users for it to be worthwhile. Focus on a small idea, and make it awesome. The result will likely be more compelling.

Plan for the worst - I think it’s fair to say that most software engineers are terrible at estimating. Being one, I’m absolutely in love with the idea of building anything in the most ideal time frame. In reality, everything that can go wrong will go wrong when it comes to coding. Though we did a better job this time when it came to coming up with a reasonable design, we still ended up stripping out some parts to cross the finish line.

The devil’s in the details - It took us the first two days to have something somewhat functional, but another whole day to polish and make everything look good and work solid.

Have fun – It will be intense, but don’t let that stop you from exploring new ideas and learning new tools. We liked the Facebook/Path menu, but we didn’t know how it worked so we invested time into building it just for this. Sure it would have been faster going with UI elements we’re familiar with, but what’s the fun in that?

Quip it!

Funny story, we had to resubmit Quips to the app store due to a screenshot having the word “badass” on it. Apple likes to keep everything PG-13 apparently. The app is available in the app store. Go grab it and let us know how you like it!