When we started to plan the Evernote service in 2007, we knew that we would need to support both “thin” clients (like web browsers) and “thick” synchronizing clients on the day that we launched. This forced us us to think about remote protocols and client APIs before we built any web GUI, rather than waiting a few months to staple an API onto an existing web service.
Our application forced certain requirements onto this API, including:
- Cross-platform. When we launched in February ’08, we had production code using server-side Java, client-side Win32 (and WinMobile) C++, and client-side Obj-C Cocoa.
- Binary efficient. Clients synchronize structured Notes that may contain hundreds of embedded images totaling tens of megabytes. We wanted an API that would transmit a 15MB note using ~15MB of bandwidth.
- Forward/backward compatibility. Once someone has installed version 1.1 of our client on their laptop, we don’t want to force them to upgrade the local software every time we extend our data model.
- Native bindings. We didn’t want to write a bunch of parsing/marshaling code on every client. This is time consuming and error-prone, and tends to make #3 unlikely in practice.
- Standard-y and/or open-source. All things being equal, we didn’t want to lock our service API into a single commercial ORB product for the usual reasons.
- Not gigantic. I’d prefer not to deploy a 1MB runtime with 200 classes on every mobile client.
We spent a couple of months researching and testing various alternatives. XML-RPC or SOAP were strong in some areas (1, 5). ZeroC’s ICE was strong in others (2, 4). We also argued a bit about rolling our own little ad-hoc protocol in a fit of NIH.
A friend-of-Evernote recommended we take a look at Facebook’s recently-open Thrift framework. Facebook was using this internally for back-end servers to exchange messages with other internal servers, frequently across language boundaries (e.g. PHP to C++). Other folks seemed to use Thrift for a similar task: back-end internal server communication.
We were looking for something a bit different: a framework that could be used for server-to-server communications in our backend, but also for massive client-server synchronization over the Internet. At the time, Thrift was the best fit we could find for our requirements:
- Cross-platform. We define our data model and service operations using Thrift’s Interface Definition Language, and then run their compiler to spit out client (and server) structures and service stubs for a dozen different languages.
- Binary efficient. If we specify a ‘binary’ data field holding 1MB of data, that will marshal as 1MB on the wire.
- Forward/backward compatibility. This is where Thrift really shines. If you are careful and understand how Thrift works (big caveat, obviously), then you can add structures, fields, service methods, and function parameters without breaking existing clients. The Windows or Mac clients we released 3 years ago can still sync with Evernote today.
- Native bindings. See #1. At the time, Thrift didn’t include anything for Objective-C Cocoa, so Andrew McGeachie (our one-man Mac team) added this to the Thrift compiler.
- Standard-y/Open-source. Facebook gave Thrift to Apache, which was awfully nice of them.
- Not gigantic. The Thrift runtime libraries and generated code were really small and straightforward. I could easily read them and understand exactly what they were doing. (Since then, it’s grown a bit flabbier as various contributors’ use cases have been added and optimized, but I’d say that it’s still pretty compact compared to alternatives.)
The end result was the Evernote Service API, which allows all of our own clients (and hundreds of third-party applications) to talk to a common API using generated native code. With over three million active users, frequently using Evernote on multiple platforms, I think that the majority of computers/devices running Thrift code are Evernote clients.
Et tu?
You’re going to implement an API for your web service … should you use Thrift, too?
If your application had the exact same requirements as Evernote, then Thrift may be a good choice. If you don’t have a complex data model with big binary attachments (requirement #2), then the answer isn’t so clear.
Web services with simpler data models tend to use simpler REST protocols with ad hoc XML or JSON data marshaling. This approach means that simple operations are really simple to test and implement. If I only need to do two things with Twitter’s API, I can test them manually from the command-line with curl/wget and then hand-roll the code into my app with printf/println/regexps/etc. This means that there’s a very low bar for third party developers to get started with this type of API.
Our Thrift API imposes a higher burden on developers, who need to get all of the transport layer details done and the library dependencies into their app before they can do any testing at all. We’ve tried to help this a bit with the sample code and packaged libraries in our API distribution, but it’s still going to be more work than a simple REST scheme.
On the other hand, the low bar for these types of ad hoc data marshaling APIs tends to be offset by subsequent compatibility hassles (#3, above). Our Twitter gateway uses the third-party Twitter4J library to talk to Twitter’s REST-based API. Our gateway has broken twice in the last year or so due to Twitter’s server-side changes and Twitter4J’s incorrect assumptions about the interpretation of the ad hoc XML data structures (e.g. the maximum size of a twitter direct message ID).
A more formal IDL and native code generation may help provide better long-term client stability, so the initial Thrift overhead for developers might be justified for some services that are concerned about client stability and longevity.
