Protocol buffer kita altcon header

Type-safe Web APIs with Protocol Buffers in Swift

My name is Yusuke Kita, and I work at a Japanese company based in San Francisco named Mercari. Our product is marketplace applications (like eBay), and we’ve been internationalizing our app for a U.S. market.

I’d like to share my experience on protocol buffers, which have many benefits.

Web APIs

Many apps have network communications - such as Twitter and the GitHub client. The calls to APIs usually returns with JSON from the server. Though JSON is a standard way to serialize data, I feel it’s not Swifty enough when I write Swift code.

Consider the following HTTP request when convert JSON object to data a type.


let json = ["id": 1] 
let data = try? JSONSerialization.data( 
    withJSONObject: json, 
    options: [] 
) 

var request = URLRequest(url: url) 
request.httpBody = data

This request is to fetch user information from the server after sending a userId. The key is a string, and the ID itself is an int. Convert this to a data type, and set it to the httpBody.

With the response, convert the data type to a JSON object through URLSession.

Get more development news like this


{"user": {"name": "Yusuke"}} 
URLSession.shared.dataTask(with: request) { (data, _, _) in 
    let json = (try? JSONSerialization.jsonObject( 
        with: data!, 
        options: [] 
    )) as? [String: Any] 
    let user = json?["user"] as? [String: Any] 
    let name = user?["name"] as? String 
} 

Here, if I mistyped the ID (to a string), or had the incorrect case, it will fail. In Swift, everything is a type, and because you must use an Any to work with JSON, I believe it’s not as Swifty as it can be.

Protocol Buffers

Protocol buffers, a.k.a. Protobuf, was developed by a group in 2008. Protobuf is about serialization formatting.

Example

Reconsider the HTTP request where you convert the JSON object to a data type with Protobuf. Protobuf allows you to define your custom type.


custom type 
let userRequest = UserRequest.with { 
    $0.id = 1 
} 
let body = try? userRequest.serializedData() 

var request = URLRequest(url: url) 
request.httpBody = body

You don’t need to write a key name with the string. Protobuf also has serialization methods, serializedData(). By calling it, it will send to httpBody.

With the HTTP response, using Protobuf, you could have a UserResponse type, which I’ve defined myself. The UserResponse type has user type containing the user names property. All you have to do is to deserialize the data type to UserResponse type so that you could access the property.


UserResponse(user: User(name: "Yusuke")) 
URLSession.shared.dataTask(with: request) { (data, _, _) in 
    // custom type 
    guard let response = try? UserResponse(serializedData: data!) else { 
        // error 
        return 
    } 
    let user = response.user 
    let name = user.name 
}

Protobuf is more Swifty, because we have types.

How Does this Work?

You need to install the Protobuf compiler (the latest version is 3.3). You will then also need a plug-in for use with Swift that was officially released by Apple.

Protobuf supports both Objective-C and Swift; you can also use both at the same time!

There are three parts to implementing Protobuf:

  1. Define message types
  2. Generate code
  3. Serialize/Deserialize

Message type defines data structures in .proto files, and it’s key-value pairs based code.

Let’s take the iOSCon profile page as an example in making a .proto file. It has user ID, user name, introduction for URL, user type. Each field has a type (int, string, or enum).


talk.proto syntax = "proto3"; 

import "user.proto"; 

message Talk { 
    int32 id = 1; 
    string title = 2; 
    string desc = 3; 
    User speaker = 4; 
    repeated string tags = 5; // Array 
}

We also have talk.proto file. It has ID, title, description, the user type, and also tags. This is similar, but since I used user type, I import user.proto at the top and you have to specify which version of protocol buffers itself (the latest version is free). That’s why I write proto3 on the top.

Protobuf compiler generates code from .proto

Protobuf generates all the code from .proto file.

The basic types they support are: int, Bool, String, Dictionary. It supports other languages such as: C++, JavaScript, Ruby, Rust.

These Swift features are from the official program from Apple:

  • It’s struct so we have data structures
  • struct, not class.
  • we could define enum but RawValue is going to be int always.

Each property has default values, which means you cannot have an optional type for the property itself. If you check users name property string, you have to see also if it’s empty or not because if it’s not optional it has the empty strings as the default.

Serialize/deserialize

Serialization is already implemented Protobuf itself, and you can get JSON or text as a result. All you have to do is call seralizedData();

Binary Encoding

message Talk { 
    int32 id = 1; ← // Field number 
    string title = 2; 
    string desc = 3; 
    User speaker = 4; 
    repeated string tags = 5; 
}

Here, we have group of types. If it’s int, it’s going to be 0; if it’s string it’s going to be 2.

Consider this struct:


message Test1 { 
    int32 a = 1; 
} 
test1.a = 300 

// encoded message 
08 96 01 

08 // field number and wire type 
96 01 // value which is 300

We said 300 to everybody. A is int, which will result in 8, 96, and 1. The first two digits represent a combination of field number and wire type.

Because of the binary encoding, the size would be smaller than JSON and everything is numeric. And as a result, this gives us high network performance.

You Might Have Concerns About…

Versioning

Protobuf has free backward compatibility. Let’s say if the response has a non-field, it ignores it. Or if something is missing from the response, it’s going to default the value. You can easily add new properties from the server-side, or delete existing, unused property on the client-side.

Coexistence of protobuf & JSON

You can have both JSON and Protobuf because you can look at the content type in the HTTP header. We can set the return as JSON or Protobuf.

Let’s look at the cons

Protobuf not human-readable because it’s binary data. If you want to see row value in the console, you have to deserialize it to be a custom type.

Working with Protobuf is time-consuming at the beginning because you have to know all about Protobuf, in addition to other platforms as it’s not strictly used on iOS project.

Codable

Recently, Swift announced Codable. It allows you to do type safety in terms of serializations. It’s a protocol, and you can do the same thing with Codable as you can with Protobuf. This is officially from the Swift Foundation.

Q & A

Q: Can you use Protobuf with Ruby? Yusuke: Yes, Ruby’s supported.

About the content

This talk was delivered live in June 2017 at AltConf. The video was recorded, produced, and transcribed by Realm, and is published here with the permission of the conference organizers.

Yusuke Kita

Yusuke is an iOS developer at Mercari in San Francisco who’s been working on internationalization of the Mercari iOS app. He’s passionate about learning new technology in iOS. When not coding, you can find him cycling.