This talk will cover a foundational skill important to every Android developer and project, continuous integration. Building software at every change is an important principle in the delivery of high quality software. Great CI does more than just testing if software builds. Automatically running tests, checking for lint issues and code style conformance are additional pipeline steps that insure the quality of your work product. And why stop there? Distribute your app to a small group of testers and to the masses through the Play store, automatically, on every code change. How? Come to this session to learn the tools and tricks that allow continuous integration to guarantee quality in your Android work, no matter how small or large your audience might be.
This is Android Basics: Continuous Integration if you’re in this room.
I’ve been doing Android since Cupcake, and last five years, I was at a startup being acquired by a Fortune50 mammoth, helped to have five million Play Store downloads to their flagship project to product. But now I’m in stealth startup mode, I’m some combination of retired and taking the summer off. I’m not going to talk about my day job; I don’t have one at the moment. I’m not going to tell you about my stealth project; either because it’s stealthy, or because it barely exists.
A few weeks ago, fun fact: I was in the New York Times. I was interviewed by a reported in San Francisco and it ended up in the Sunday edition on a story on the front page of the Business section. I like hackathons, I’m approaching $50,000 in total winnings. I appreciate those hackathon organizers. They put together large novelty checks. Being from Colorado, I am local to Denver. I love my snow sports. I give many talks. I must have taken a time travel machine to Huyen’s keynote yesterday and took her advice to heart to get started at meetups and move on up and I’ve delivered a keynote in Paris, which is that picture on the right. I give my talk in Hong Kong, San Francisco, Silicon Valley, Texas, Boston, all over the place. And for audiences of 2000 and more, I will get into makeup.
I do run the Google Developer Group in here in Denver. If you listened to Huyen’s keynote yesterday and you were thinking about giving a talk some time in the future, whether you’re in Denver or somewhere else, it’s a great place to visit, come on out, join the meetup group. I’d love to have you as a speaker. There’s 500 people in the community. We’d love to have your bright smiling faces in front of them.
Nothing more dramatic than CI
There is nothing more exciting and dramatic in software engineering than continuous integration. And Charles Dickens predicted this in a sense way back in 1859. Charles Dickens wrote: “It was the best of times, it was the worst of times,”. That is the story with continuous integration.
Before you get CI, it is the worst of times. Once you get CI, we’re going to talk about how it is the worst of times. I was in Las Vegas giving a talk and I talked to some people at major companies who were embarrassed by their CI workflow pipeline: it’s a common story that when CI goes down, there’s one person who understands how to make it work. Sometimes it’s Jenkins inside a Docker container, using Vagrant, and nobody knows how to restart the thing, or Gradle needs to be updated on the box and nobody knows how to do that. When CI goes down, if you’ve integrated it into your company, it is a huge problem because nobody can get builds out. You can’t deliver new software to users.
- I’m going to motivate you to CI. I’m going to pitch you on why CI is a good thing, even for one of your hobby projects.
- We’re going to define CI. If you’re familiar with CI, you’ll notice that I use a superset of what CI technically is.
- We’ll go through CI workflow tools and related tools (more than 10 or 12). You’ll want to use these tools to add CI into your software development process in the future.
- We’ll laugh, cry, and learn along the way
Most startups’ philosophy
Let’s imagine we’re a startup. You’re a startup, it’s your own nighttime and weekend project. Big companies operate this way. The mantra is, in the beginning, “We’re going to write a ton of unit tests, we’re going to have Espresso Tests for everything.” We’re going to have well defined requirements before engineers ever start working on code, and then no plan survives the first brush with the enemy. Day one of the sprint: “fast, faster, we have to get those features out. Forget the test, forget the integration test. We don’t have comps, doesn’t matter, let’s ship it out.”
Get more development news like this
We’re going to do a quick improv comedy game. I am playing the role of a software engineer that has no CI. I’m at a company with no CI and my primary goal is to ship code faster.
“I shipped three new activities. I forgot to add the third one to the manifest. That’s no big deal, look at all these users downloading my app!”
“I added German language support to my app. Did I test it on smaller screens? Why would that be important?”
“You mean users can rotate the phone during network activity?”
Masters of Scale
The Masters of Scale podcast had an episode with Mark Zuckerberg. He tells this same story that at Facebook, originally, their whole mantra was ship code as fast as you can. But then they got into this feedback loop where they were having bugs, and were having to fix the bugs. The bugs fixes were creating new bugs, and everything was slowing down and grinding to a halt. He says:
“We needed a new strategy to go faster. We’re going to build the best infrastructure… An engineer can come here from any company… is going to be able to ship their product faster, test it better.” -Mark Zuckerberg
Slowing down to speed up, making sure you have the right processes in place.
Secretly, most of us are terrible
For those of you who are out there who are maybe getting started with Android or you work at a company and been there for a while and you don’t have CI. Most every major company, companies in Denver, we all have complaints. I’ve had this conversation with many people: we use Tool A (Jenkins). Is it difficult to use? It’s a nightmare for us.
Don’t stress out if you’re not perfect, it’s important to start somewhere, to have some CI, to be building something on every PR open, not necessarily every commit; to be tracking how your link progress is going. You want to know if you do a PR and suddenly double your number of link warnings or link errors. You want to be doing static code analysis on every build. You want to potentially be automating the process of user feedback and deployment.
This is not a new idea
This is idea of being efficient by analyzing how the process of creating work is not new. Everybody’s familiar with Henry Ford and the Model T and the modern assembly line, which was not necessarily the first, but it was the first place where there was an assembly line and there were conveyor belts between the different stations. Parts were going on one end, and a completed product would go out on the other end. This is a good metaphor to think about software.
You have ideas and requirements, and smart people, and on the other end, you should have software that goes out without work. You should, without many headaches, be able to put your head down and code most of the day.
Henry Ford was not always successful; he had an aviation company that crashed and burned. He was always a farmer and a watch repairman back in the day, and a car racer. But Henry Ford was not the inventor of the assembly line. I assumed he was. He’s the inventor of the modern assembly line, which is the one with conveyor belts between different parts. In fact, it goes all the way back to 1104 when the Venetian Arsenal employed 16,000 workers in a huge water based, essentially old school assembly line and gave Italy huge might in ocean faring activities because of that. It had such a big impact on the world that it is mentioned in Dante’s Inferno.
But this isn’t even as old as it gets, it gets even older. More than 1,000 years ago in China, there were assembly lines used to make weapons and farming tools.
Programmers are lazy
Programmers are lazy, and you might have a different definition of lazy than I do. There are smart computer scientists that will emphasize this point, and this is a great message:
“I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.” -Bill Gates
This is a great metaphor for continuous integration. There are things that we don’t want to do, but we know they’re good for us.
By automating and operationalizing, we can take what Henry Ford and the Venetians, making a delivery of product more efficient. We can do these boring things every time and piggyback on the good thing about being lazy as a programmer to deliver better software. Then, we can be like Michael Guerra, an Italian cyclist, who has an innovative cycling technique and we can speed past the competition and deliver better software.
Hopefully, you’re sufficiently motivated to want to do continuous integration now.
CI is not a new idea.
“Continuous integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily, leading to multiple integrations per day. Each integration is verified by an automated build, including tests, to detect integration errors as quickly as possible.” -Martin Fowler
We’re familiar with this in Android, it’s the same story as it was in 2006. Maybe the integrations are more frequent than daily, maybe we’re doing fancy stuff, e.g. taking screenshots and multiple different sizes to check out that our internationalization is working. But this is an old idea and it’s definitely applicable to Android development.
CD as in Continuous Deployment means “not to production.” This is more common, and most of the companies I work with are doing continuous deployment. Through HockeyApp for example, to beta testers, to the product managers in the company, or maybe the developers. If you’re a lone wolf, you’re deploying and you’re pushing to your phone. You can open up the app anytime, and make sure it’s still working.
CD as in Continuous Delivery, which is a newer concept, is pushing out to the Play Store. Many of the tools that we’re going to go over allow you to automate pushes to the Google Play Store, including to subset: You can do alpha/beta testing on the Google Play Store and you can push out to the smaller groups.
CI is really a superset of what CI strictly is. You can do great things once you have CI in place. Strict CI is simply building your code, making sure that it does build, and perhaps running the unit test. But then you can also do all these other things.
Static code analysis, we’re going to go through a bunch of tools. There are security static code analyzers, there are bug-finding static code analyzers, there’re code style static code analyzers. You can run unit tests.
You can run Espresso test, integration test, e.g. Robolectric. You can save artifacts and reports. This is helpful if your manager comes over and says, “Why do we have 1,000 LIN errors?”, and you go and you look back in the history and you realize that three weeks ago, Todd over in whatever department did a PR and all of a sudden the number of LIN errors went from 15 to 987, now you’ve effectively passed the buck over to Todd.
You can also save other reports from the static code tools. You do notifications, you can post messages to Slack, “The build completed successfully”, “The build didn’t complete successfully.” You can post those to Twitter, to email. There are people who use Internet Of Things light bulbs, and they have like green and red. You can get creative with how you deploy your CI.
You can hook into source code repositories. I’m lazy myself, and I don’t want to kick off these CI builds. To allow GitHub, or whatever source code repository tool you use, and maybe Bitbucket to automatically run CI for you is a lifesaver. In this wider umbrella of CI, there’s also crash reporting, and user feedback that can be automated. When we go through the companies that are providing tools for CI workflow, many of them offer that as well.
Everybody uses lint, whether you know it or not. PMD is a source coding analyzer; checkstyle is a style analyzer. You can input whatever style you want to use: you might want to use the AOSP style guide, be in the camp that hates some of the stuff in the AOSP style guide, tweak it. There is FindBugs, which finds bugs in Java programs. It’s a neat tool that does static code analysis. These are some of the things that you can add onto your CI process.
You can run Espresso tests in CI
It’s important to talk about the ability to run Espresso tests, but it didn’t feel like any of those static code analysis, and it didn’t feel like the CI workflow mega tools.
But the defacto best practice place to run Espresso tests now is Firebase Test Lab For Android, which is a mouthful. This allows you to be billed by device-minute and you can run tests in parallel or all on a single device. You can use actual real production devices that are in an enormous closet, probably out in Mountain View, California, somewhere, and you can deploy your Espresso test.
CI Tool review
Who has never done CI, or haven’t worked at a company with CI on Android in the last three years? 15 to 20% of the room has no CI.
Three-quarters of the room has used Jenkins at some point, which isn’t a surprise. Also, because it’s on HBO in Silicon Valley. If they choose to highlight it as a CI tool of choice, then why shouldn’t you? People have a love-hate relationship with Jenkins. It’s great because it does help programmers be lazy; it helps you deliver better code, but arguably, maybe it could have a better interface, and does require some ninja-level skills sometimes to ring out the performance and the extra tricks you want to run in your CI process.
About 10%. There’s probably a reason for that. The word on the street is that TravisCI is slow, particularly with Android. In other platforms, it might be great but people have struggled to wrest some performance out of TravisCI. This is another tool you can look into when you’re trying to implement continuous integration at your company.
We’re going to call that less than 5%. This is the new kid on the block. I’m going to take one of these companies later and go through some screenshots and show you what they do for you.
Less than 5% as well, which is surprising. CircleCI is popular in the Android community. There are big companies, big name developers who, in their side projects, are using CircleCI. It’s fast, it sounds it’s a nightmare to configure. I’ve seen chat messages about, “I can do this great magic on my computer, but I can’t quite figure out how to get CircleCI to do that.” They did come out with a 2.0 product that I think is in beta, that maybe it isn’t officially released yet, but there’re people who are excited about CircleCI. If you work at a company where you want to get CI in place, maybe you work with people that like bright, shiny new things, maybe CircleCI 2.0 is right for you.
This is the most surprising: 0% coverage. I went out and talked to people, this was the most loved tool of anything. I had people who said, “I used Tool X, and I hate it because Y, Z, A,” and they weren’t even prompting to say why they hate it. When people answered BuddyBuild, Matt Harris, who works for a local startup, Dispatch Health, they’ll come to you; this was his quote: “It’s a complete end to end solution, that works, 0% configuration, 100% happiness.” That’s what he said about BuddyBuild. He was smiles and psyched about the product.
Because of his ringing endorsement, we’re going to look at some screenshots from BuddyBuild. Keep in mind that this tour applies to almost all those other tools. You can do everything, most of everything that you’re going to see in BuddyBuild in these other products.
BuddyBuild is a CI tool, but it does more philosophy. There’s a superset to the definition of continuous integration. They break it into four parts:
- Continuous integration is complex and cumbersome.
- Continuous deployment is difficult to provision users.
- Feedback is inefficient and ambiguous.
- Analytics and crash reporting is incomplete and lacking context.
Continuous integration with BuddyBuild
For continuous integration, they make it easy, as we saw Matt say earlier. For continuous deployment, they make it super simple, they email APKs to your users and manage your users. For feedback, they built in feedback tools. They have a Wage Shake feature, which means that user can shake their phone and then leave feedback with screenshots and annotations and stack traces for the developer and leave good feedback. They integrate it with analytics and crash reporting as well.
This is an example; they’ve recently added a feature where you can do testing on real devices, Espresso-style testing. And this is what their portal looks like and you can see that they have live video. Anytime there’s a crash of an Espresso test, you can watch the video of what was happening at the time. You can also see the different devices it ran on, and see the Stack traces, you can download the logs and test reports.
For each test run on physical devices, BuddyBuild now provides all the logs generated during the UI test run, a video replay of UI test run on each device, a detailed call stack for any test failure during test runs. And here’s another look at their dashboard. You can see on the top that you can deploy, you can look at Google Play, you can look at your JUnit test and see how that did, you can see your logs, your details.
In summary of BuddyBuild, you can build your app, run your unit and UI test, you can deploy your apps to testers and to Google Play, collect user feedback in crash reports, and it’s built specifically for mobile developers, and it’s Matt Harris-approved.
Those of you who know Matt can be excited about BuddyBuild, but all those tools I went through are good choices. They’re all better than the alternative anti pattern, which is, “Let’s go fast, let’s go fast.” Because that’s going to send you up in a wave. All these companies, when you go to these websites, are going to have a screen that looks like this and they’re all the same.
If you’re excited by the companies on one of these particular lists, you can go with that tool. But use a tool and integrate continuous integration.
Nobody’s used it, but we have some people who’ve heard of it. Bazel came out of Google. Google has big CI problems and challenges and have an amazing system that can run probably millions of unit tests at this point. It’s called Blaze, which is an anagram of Bazel. And they recently open-sourced Bazel, open-sourced their version of Blaze. It’s great for large projects. There’s an emphasis on speed, correctness, and reproducibility, and you can write rules in macros in a language called Skylark, which is a subset of Python.
Buck brings us mostly full circle to Facebook. Buck was designed by Facebook because they had trouble building large apps for Android, with lots of lines of code. It’s a high-performance build too. I would probably put this towards the bottom of your research list. But you can reach out to the community and see if it’s a tool that makes sense for you.
To circle back to Charles Dickens, and why I think continuous integration is interesting. When you don’t have CI, we saw in the wave gif and the improv scene earlier that bad things can happen. You can ship code with errors in production, runtime errors. It can be the worst of times. When you get CI and CI is working, it can be the best of times. You feel good about your software working. When you’re working on a small team or a large team, you might be stepping on other people’s code, and it feels good to know that you’re not breaking their code or when they’re doing PRs, that they’re not breaking your code.
CI can be the worst of times because it’s a common pattern to have a team of 10 Android developers at a company, and there’s one guy, one engineer, who understands how the CI pipeline works, and when Jenkins goes down (or whatever CI tool you have goes down), and is on vacation, you’re stuck and can’t do builds for a couple weeks.
Joel Spolsky created Stack Overflow, or rather his company, Fog Creek Software, created Stack Overflow, and also created Trello. He worked at Microsoft and then started his own thing and has had two mega hits since then. He used to have one of the original blogs.
This is an article about what you should look at in a company when you’re evaluating switching jobs and what makes for a good software company. It focuses on this idea that speed is not the most important thing. We want to work in a sane environment.
He asked this series of 12 questions, “Do you use source control? “Can you make a build in one step?” And that is CI. “Do you make daily builds?” “Do you have a bug database? “Do you fix bugs before writing new code? “Do you have an up-to-date schedule?”
You can see parallels between what Joel is espousing and between what I was promoting in the beginning of the talk when I was motivating you to look into CI. It’s a anti-chaos pattern. We want regularity.
If Dante’s inferno, Henry Ford, and HBO’s Silicon Valley taught us anything: CI is good. Invest in CI, spend the time at the front so you can move faster later.
I hope today I’ve inspired you to either expand on your current CI, or to go investigate CI, either on a project that doesn’t have CI yet, or one that might have it.
Questions you might have
Q: What about Kotlin Kotlin does mostly work with CI except for the static code analyzers. You might be familiar enough with Kotlin to know that Kotlin creates byte code that is Java compatible, but we’ll say that the byte code is messy. It’s something that the static code analyzers get confused by and think is problematic. There is a static code analyzer for Kotlin but it hasn’t been fully released yet. It’s in development now, and it’s called Detekt. If you are a Kotlin shop and you want to do CI and static code analysis on your Kotlin code, you’re going to want to look into Detekt.
Q: What about cost? I am here as a hobbyist, I like to do Android programming on the side “because my day job’s in Cobalt, and it’s boring; are these tools free, what’s the deal? Go to the websites and check. In general, the rule is Jenkins is free. There are vendors, Cloudbees is a in-the-cloud way to easily manage Jenkins instances. But a common thing in smaller shops is to have a Mac Mini in the closet somewhere that’s running a Jenkins instance, and you can do that for free. But most of those other tools, the fancier ones that do that expanded version of CI and deploy to the Play Store and have sexy reporting, do cost money. And they can get expensive. The Firebase Test Lab does cost money as well. You need to use an account that charges your credit card. But the good news is that most of them have a free tier, a free platform. The rule is you can have one instance running serially, your CI work for free. Some of them offer opensource incentives. If your project’s on GitHub, and it’s opensource, you can have like the second tier for free. Maybe get four simultaneous instances. Your builds will go faster. If you’re a one-man / one-woman shop, one serialized build instance is probably good enough.
About the content
This talk was delivered live in July 2017 at 360 AnDev. The video was recorded, produced, and transcribed by Realm, and is published here with the permission of the conference organizers.