Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, communications across different timezones, etc. In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics: Workflow automation with Fastlane; code review with Danger and SwiftLint; collecting and visualizing code metrics with InfluxDB and Grafana; build time reduction; and code modularization.

Introduction

My name is Yusei Nishiyama. My talk is about building for iOS at scale. I work for a company named Cookpad. It is the largest recipe sharing service in the world with the aim of making everyday cooking fun. We cover 67 countries and 21 languages, and we have about 100 million users all around the world. There are 10 iOS developers. Four in the U.K., four in Japan, one in Lebanon and one in Colombia.

Working on an app that supports users all over the world in a distributed chain involves many challenges. I will talk about working with a large team and working with a large code base. The first part of the talk covers Fastlane, Danger and code metrics. The second part covers build time reduction and code modularization.

Working with Large Teams

If you work in a team, you may have issues that you never have when you work alone. For example, someone broke the build, and the rest of the team does not know how to fix it properly. Or perhaps someone’s vacation may prevent the team from making a new release because only one person has the distribution certificate.

Generally, you can deal with those common issues with continuous integration (CI) and automation. None of those tasks should be developer dependent.

Continuous integration should consist of several subsystems. This is an overview of our CI system: The diagram starts from the point where a developer pushes their change to GitHub. GitHub triggers a Jenkins job through a webhook. Then Jenkins executes a Fastlane script.

Fastlane is a suite of tools that make it much simpler to automate mobile developers’ daily chores. You can define your custom workflow in a text file called a Fastfile. We have seven Fastfiles which have 28 “lanes”. Fastlane is now very popular among mobile developers and you might have been familiar with some common use cases like running tests or uploading your app.

A practical example is syncing dSYMs. If you enable bitcode, your compiled bitcode and generated dSYMs are uploaded to the server, then you will need to download the generated dSYMs from the upload server, and then upload them to a crash reporting service such as Crashlytics so that crash reports can be symbolicated. This script automates the process:

version = get_version_number
build_number = get_build_number
# Download dSYM files from iTC into a temporary directory
download_dsyms(version: version, build_number: build_number)
# Upload them to Crashlytics
upload_symbols_to_crashlytics
# Delete the temporary dSYM files
clean_build_artifacts

Our server also automatically builds up and distributes the app to testers for every push to github. But there are some changes that you may not want to distribute to testers - for instance, documentation updates because it simply wastes CI resources; we distribute a better version only when a pull request is labeled as “beta distribution”.

Getting labels from github is also very easy with Fastlane. Fastlane has a github_api action with which you can call a github API, like this:

desc "Check if pr is marked as beta distribution"
private_lane :beta_distribution do
	labels.any? { |label| label == 'beta distribution' }
end
desc "Get labels of pull request"
private_lane :labels do
	pr_number = ENV['PR_NUMBER']
	repo = ENV['GITHUB_PULL_REQUEST_REPO']
	result = github_api(
		http_method: 'GET',
		path: "/repos/#{repo}/issues/#{pr_number}/labels"
	)
	result[:json].map { |label| label['name'] }
end

Those lanes in the example fetch labels from a specific pull request and check if they contain a beta distribution label.

At Cookpad, each push triggers a beta distribution to internal testers. Members of a company should be some of your best users, and their feedback is really valuable.

Unfortunately, because pushes can happen automatically, and a lot of developing features are occurring in different branches, if we developers don’t communicate with testers well, they stop giving us feedback because they can’t know what we want them to test.

One solution is to convey to testers the purpose of the build through the icon. This Fastlane script appends a badge on the icon. This makes communication with developer and tester much easier:

if is_release?
	next
elsif is_rc?
	title = 'RC'
	number = get_version_number
elsif is_beta?
	title = 'Beta'
	number = sh('git rev-parse --short=7 HEAD').strip
elsif is_pr?
	title = 'PR'
	number = ENV.fetch('PR_NUMBER')
end

badge(shield: “#{title}-#{number}-blue",
	dark: true,
	no_badge: true)

Testers can now easily identify which one is from latest candidate branch:

Code Review with Danger

Before talking about Danger, let me ask a question. Have you ever posted a nitpicky comment such as “please remove your parentheses surrounding the condition”? Something like this should be done by a machine automatically. This is where Danger comes in.

Danger runs in your CI process. It automates common code review tasks with the philosophy of leaving developers to think about harder problems. All you need to do is to define the rules that every pull request should meet in a text file called a Dangerfile. Here’s an example:

github.dismiss_out_of_range_messages({
	error: false,
	warning: true,
	message: true,
	markdown: true
})

xcode_summary.inline_mode = true

xcode_summary.ignored_files = ['Pods/**']

xcode_summary.ignored_results { |result|
	result.message.start_with? 'ld'
}

log = File.join('logs', 'test.json')
xcode_summary.report log if File.exist?(log)

swiftlint.binary_path = './Pods/SwiftLint/swiftlint'
swiftlint.lint_files inline_mode: true

One important to note is that you can execute Linter through your Dangerfile. Danger can comment on your pull request on behalf of you, and warn about rule violations. This way, you no longer need to spend your time on minor styling mistakes, like saying “please remove TODO before merging the branch” or “this line exceeds the character limit”.

Collecting Code Metrics

Our system is unique when it comes to code metrics. We collect our code metrics in a database which is called InfluxDB. InfluxDB is a data storage which specializes in time series data. It has a built-in HTTP API and a lot of client libraries in many languages. It’s very easy to integrate InfluxDB into your build system. It’s written in GO, so it’s really fast and responsive.

We have a Fastlane script which collects code coverage and posts the data to InfluxDB. Those data is read and visualized through Grafana. Grafana is a metrics dashboard and editor that supports InfluxDB. Here is an actual change in our test coverage, as visualized in Grafana: If your metrics are visualized and accessible, everyone can be aware of both the good and the bad things that are happening on your team. That’s why visualization is really important.

We also utilize InfluxDB and Grafana to measure other metrics such as the number of issues and number of pull requests. We can use that data to see how productive our team is, and how stable our app is. This helps because we can set a clear goal. We can say we have number x and we should target number y.

Working with a Large Code Base

Here is an example of an issue when working with a large code base. Suppose you need to improve your app’s registration screen to reduce the abandonment rate. Here’s what might happen:

• Implement a redesign.
• Hit the run button to check whether it’s working as you intend.
• You find a bug in the layout. You fix that and then run it again.
• Unfortunately, you find another bug which only occurs on a specific version of iOS. You fix that bug, and then you push the change to github.
• Your CI server starts building your branch and launches tests against it.
• The CI fails due to a failing test.
• You fix it, run the test locally, then push the change to github.
• Then you need to wait for CI again.

When your codebase is large, and the size of your team is big, the above example can present a serious issue in regards to building.

Reduce Build Time

There are techniques to measure and improve Swift build time. I will show you how to improve the Swift build time and gain productivity in two parts: measurement and improvement.

Measuring the Build Time

If you want to improve something, you should start from understanding the problem. The first thing you could do is enable the ShowBuildOperationDuration flag. It displays the build time in Xcode, so that you can always be aware of this.

However, it’s not enough. The build process can be broken down into multiple processes such as dependency management and compilation of each source file, and you may run Linter and Formatter in a run-script phase. You need to know how long each process takes to identify the exact bottleneck.

Fastlane does a great job here: it generates a record as an XML file with which you can get a good insight into how long each process takes. As it’s just a JUnit style XML file, Jenkins can render it as test results. In the Jenkins console, you can check which lane was executed, how long each took, and whether they were successful or not:

You can further break down the compile phase into the compilation of each function. An easy way to find a function which takes a long time to compile is to enable frontend-warn-long-function-bodies flag. If you turn the flag on, Xcode warns you of functions that have long compile times. For example, I set the threshold to 100 milliseconds, and Xcode complained because a function took 15 seconds to compile.

You can also let the compiler print out how long each function takes to compile by using the frontend-debug-time-function-bodies flag. You can play with the data just by copying and pasting it in your terminal, but that is not tooling friendly. The extension of the log file is .xcactivitylog - it is just a gzip file, so you can unzip it and read it in plain text.

With the build log, one of my colleagues developed a Danger plugin which comments directly on a function that takes too long to compile:

(in Dangerfile)

xcprofiler.thresholds = {
	warn: 50,
	fail: 500
}
xcprofiler.report 'ProductName'

It gives us a chance to consider if there’s any other way to write this function in a compiler-friendly way.

Improving Build Time

In this section, I will show you how the build time was improved with real data from a real application. This application has 40,000 lines of Swift code, so it’s a medium-sized app. CocoaPods is used as a dependency management tool; it has 38 direct dependencies to pod library and ends up installing 60 libraries.

Whole Module Optimization

To start, it took 48 minutes to test and archive on CI. The easiest but the most effective technique to fix this is to start with Whole Module Optimization. With it, the Swift compiler compiles all files in the module at the same time.

The intended benefit of Whole Module Optimization is to let the compiler optimize code more aggressively. There’s a hack with which you can tell a compiler to compile the whole module at once, but without any optimization. You can try this by setting optimization level “Whole Module Optimization” and specifying optimization level “none” in the “other” Swift flag: You can find more discussions about build time reduction using this technique. Whole Module Optimization without any optimization may sound weird, but it makes the build time really fast in most cases. With that setting, now CI takes less than 20 minutes to test and archive.

Playgrounds for Prototyping

The build time improved, but it could be better. You really do not need to build the entire app every time with Playgrounds. Playgrounds is more than an Apple environment because it can render views, and we can use Playgrounds as a powerful prototyping tool.

So Playgrounds seems to be a solution, but it’s not 100% true because it’s rare that your code doesn’t rely on any existing code. Ideally, you should have a separate module for each purpose so that you can play with them in Playground easily. In this example, I have my own UI framework which is imported from both Playground and my application project, and we can access a custom color scheme through Playground:

import UIKit
@testable import YourUIKit

let color = UIColor.myGreen

----
class ViewController: UITableViewController {
	let myView = MyView()

	override func viewDidLoad() {
		super.viewDidLoad()
		view.backgroundColor = .myColor
		view.addSubview(myView)
	}
}

Code Modularization and Microservice Architecture

Code modularization is the idea of having separate modules for each purpose. In 2014, James Lewis proposed the microservice architecture. This is an approach to develop a single application as a suite of small services. It does not completely correspond here, as the idea relates to web applications, but we can get similar benefits from it.

If you apply the idea to apps, you will get this kind of architecture: You have a core framework consisting of a design framework, a networking framework, and a persistence layer, and you can build each feature on top of those core libraries.

The benefit of code modularization is more than just keeping your project cleaner. As the default access level of Swift is internal, everything you put into your framework is private from other frameworks by default. It gives you more opportunities to consider which methods and properties should be public and which should be private.

Another benefit is that you can get a faster incremental build time. A change you make to a module doesn’t trigger a re-build of other modules. So it could improve your incremental build time dramatically.

Another thing you could get from code modularization is autonomous teams. As organizations expand, it’s hard to synchronize teams with each other. It can take months to get a stable version because it’s really hard to make sure that a change to a feature doesn’t break any other features. When there are clear boundaries between modules, people can focus only on the individual modules that they are working on without being concerned about what others are doing.

To make your app modular, put all of your developing frameworks into a single repository. The main benefit of this approach is that you don’t need to change your workflow. You can work with your frameworks as you did for a single monolithic application. The downside may be that those frameworks are not ready to be open-sourced.

You can also manage your frameworks in different repositories. One obvious downside of the approach is that it makes your workflow more complex. Imagine developing a new feature which requires changes to your main application and changes to your framework, say the networking layer. Then you may need to submit two different requests. One is to your main application, and one is to the API framework. This obviously makes your workflow more complex.

Dependency Management

Next is to consider dependency management. I will compare four possible approaches: CocoaPods, git-submodule, Carthage and gitsubtree.

CocoaPods has been a de facto standard among iOS developers and is still one of the best dependency management tools. The only thing you need to do is to define dependencies in a Podfile, and CocoaPod manages everything else. To put it another way, there are a lot of things happening under the hood, which makes things harder when you face a problem.

Strictly speaking, git-submodule is not a dependency management tool. It just checks out dependent repositories for you, and everything else is up to you. However, additional complexity is introduced in your workflow because every time you make a change to your framework, you need to update the reference in the parent repository to point to the new commit.

Carthage is a middle ground approach between CocoaPods and git-submodule. You define dependencies in a Cartfile, and Carthage downloads them for you, but how to link them is up to you. Carthage also pre-builds libraries, so Xcode doesn’t need to compile and index them, which saves a lot of time. One thing you must be careful with is ABI (application binary interface ) stability. At Swift, ABI stability is not yet available; all your previous frameworks must be built with the same version of the Swift compiler.

Git subtree is less popular than the other three. It clones the sub-project, and merges it into the parent project, so you can make changes both to the parent project and the sub-project in the same place. When you push those changes, git subtree goes through the commit, picking the changes that should go to each sub-project. With the magic of git subtree, you can do your work as if you are in a single repository even though it is actually backed by multiple repositories. The pros and the cons are mostly the same as with git-submodule, but one significant advantage is that you don’t need to change your workflow. A downside is that you need to use the slightly tricky merging strategies of git subtree, or you can wind up with a serious conflict or dirty git history. There’s a medium post that explains how a team modularized their code using git subtree. I highly recommend that you read it!.

If your app is a huge, monolithic application, then code modularization is not a simple task. Once you decide to do this, you should start from a single repository, because at the beginning of the process your frameworks are probably not very stable. Then, after a while, when you get to the point where your framework is not changed very often, you can consider separating them into multiple repositories, and use previous versions of them.

A common pitfall with code modularization is dynamic framework loading time. You can check how long dynamic library loading takes by enabling the DYLD_PRINT_STATISTICS flag. If an app takes very long to open, the system queues you up. Some people work around this problem by merging all the frameworks into a single, dynamic framework, and link that into an application, but this is not a very simple solution.

Dynamic library loading time was the only concern in moving this idea forward, and it seems that even Apple does not encourage it. So I was undecided about this approach for a while. However, my concerns have recently vanished, because Xcode 9 started supporting Swift static library targets.

Conclusion

1. Automating your daily workflows with Fastlane, CI, and Danger saves you time for code review.
2. Visualize everything you want to improve.
3. Revising project settings may improve build time.
4. Code modularization improves build index time and helps you to maintain a well-structured team and code base.

Next Up: New Features in Realm Obj-C & Swift