In this talk, Kazuaki Matsuo covers the testing methodology used at Cookpad, one of the most popular recipe sharing services in the world.

Introduction

I’m Kazuaki Matsuo. I work at Cookpad as a test engineer and software engineer in quality. My focus is on automation for mobile and improve the overall development process. I work with multiple languages: Swift, Ruby, Java for Android and Elixir.

Recently, I’ve been maintaining the Appium Ruby binding. If you’re unfamiliar, Appium is one of the most popular mobile test automation tools.

What Is Cookpad?

Cookpad is one of the most popular recipe sharing services in the world. Currently, we provide our services for Japan, and different countries around the world. According to similarweb.com, Cookpad is the largest site worldwide in the food category.

We have two iOS apps: one is for Japan and the other is for rest of the world.

Cookpad for iOS (Japan)

For the purpose of demonstration, the test target is the Japanese edition. The Japanese version of the app has been in development for a substantial period of time; it has around 100,000 lines of code. Our release cycle is around one month. Changes typically include many new features, and refactor to the internal logic.

The Kano model

The Kano model is a product development and customer satisfaction theory that is popular in Japan. The model has two main qualities: must-be quality, and attractive quality. Must-be quality is defined as passing basic standards and functionality, for example, in Japan, this means a crash-free app for almost all users, and no features can be degraded for a long time.

Mobile app environment has changed quickly through time. With each new version, user experiences require “quality by market”. I’ve come to think of this as diachronic quality. This word is influenced by linguistics. Diachronic means sometimes has developed and evolved through time.

Basic Strategy for Testing

To make the target testable, we should consider architecture for the apps from two aspects: internal and external. Internal is defined as the production cost side, and external is the UI/end user experience.

Check if the UI behavior is broken by rewriting and refactoring features, then implement and conduct unit tests. Developers can find bugs if their implementations break unexpectedly. UI testing includes manual checks. We’ll then be able to proceed re-engineering the app with this cycle step-by-step.

UI Tests Should Be Automated

One area that cries out for automation is UI testing. In many cases, UI tests take time and human resources because test automation for this area is difficult.

Converting manual checking into automated UI tests is important. If you’re able to increase your automated UI tests, you can automatically check most of the layers, screen transactions, etc.

You can also conduct tests for combinations of various OS versions and resolutions without additional human resources. Designing test architecture for automation is also important.

Implement the Strategy

We’ve been developing UI tests since 2014 to support re-engineering. Before, we had many manual tests and non-automated UI tests. But currently we have many UI tests, which cover around 80% screen transaction and support manual testing.

Our architecture keeps the independencies between end user scenario and the concrete implementations depends on iOS framework. Thus we can decrease maintenance costs for scenario side and iOS side.

Scenarios With Data-driven Testing

I implemented scenarios for data-driven testing style with Turnip. Turnip arose to describe tests with natural language. This scenario is close to the end-user. These are the steps and wrappers to convert the natural language to programming languages. Here, it’s done in Ruby.

step "Conduct tests with :device device" do |device|
 start_device(device)
end

def start_device(device)
  driver_start_with(desired_capabilities: des_app_caps, server_caps: des_server_caps)
  set_location
  close_initial_information(device)
end

Tips

1. Reduce dependency from the internal product

It’s important to describe test scenarios independent of internal logic from the production code. For example, this find_element must try to find out elements with xpath.

find_element :xpath, "//UIAApplication[1]/UIAWindow[1]/UIATableView[1]/UIATableCell[1]"

Xpath indicates a view hierarchy. But finding elements will easily be broken if the OS version is updated, as the pair strongly depends on the iOS frameworks logic.

2. Don’t conduct tests for all boundaries in UI

For example when testing validations for the text field, because UI tests are too slow, if you implement all boundaries there, it will be inefficient. It is better to move tests from UI level to method level and refactor there.

Checking Other Results

Test automation is more than just a pass/fail of a result, it can include different tools that provide feedback to different teams. For example, we also implement an image diff to compare the app with the design, and provide feedback to the designers. Designers can know the difference between the previous version and the new version easily.

Introducing Swift to Our App

We introduced Swift into the app a few months ago. Currently, nearly 30% of the code is already implemented in Swift for our application.

The current UI tests are ample for current development processes, but as our team becomes bigger, I will introduce alternative tools such as XEY test for Earl Gray for UI testing. This will be used in conjunction with Appium, as its responsibility and test frameworkout lifecycle is different.

Next Up: Keep on testing, all the way to your Realm apps