Tryswift tj usiyan cover

Property-Based Testing with SwiftCheck

Unit tests are a challenge to write. “Did I think of every relevant case?” is an almost impossible question to answer. Fortunately, we have the tools to help find more relevant cases with less searching. In this talk from try! Swift NYC 2016, TJ demonstrates how property-based testing using SwiftCheck helps us find edge cases and become more confident about the assumptions that our code is built upon.


Test Cases Are Difficult to Write (0:58)

Test cases are difficult to write because we have to think of everything that could go wrong, or every case that matters. Then you have to organize them in a way that when they fail, or when they succeed, we can read the tea leaves.

How many test cases do we have to write before we’ve satisfied our need to know that our code works? How can we find test cases?

Let’s Get Testing (0:58)


public struct Rational {
 public let numerator : Int
 public let denominator : Int
}

I have pretty simple type called Rational. This type has a numerator and a denominator. Another word for this is a fraction or a ratio. We would like to test this type. The first thing that we’re going to do is actually make something random.

Arbitrary


public protocol Arbitrary {
 /// The generator for this particular type.
 public static var arbitrary: SwiftCheck.Gen<Self> { get }
}

Get more development news like this

I want to just get a ratio, or an Int. I don’t care which one, because this should work for everything that conforms to Arbitrary, and then give back a generator that will create instances of our type. You can see here that our arbitrary property just returns a generator for our type. The generator returns arbitrary values.

Gen


public struct Gen<A>

It receives a random number generator and a size, and it uses both of those things to control what goes in. This random number generator is used because we can give numbers and reproduce failing test cases so we can figure out what actually went wrong. So, we just return a generator of A, (A being the type we’re interested in).

Now that we have those two things, we can create random values. We conform to Arbitrary by adding an arbitrary property, and the simplest way to actually return just a random thing is to use compose.

Conforming to Arbitrary


public static var arbitrary: SwiftCheck.Gen<Rational> {
 return Gen.compose { comp in
 return Rational(comp.generate(), comp.generate())
 }
}

Gen has a method named Compose, and into the block we pass in a composer. A composer is just a convenient way to call arbitrary on anything. Here, we call comp.generate() twice, and all we’re doing is passing in a random integer to our numerator and our denominator.

The denominator should not be zero, and we need a way to express that: suchThat. Gen has a method where we can filter.

Filtering Generated values


Int.arbitrary.suchThat { $0 != 0 } // Denominator


extension Rational : Arbitrary {
 public static var arbitrary: SwiftCheck.Gen<Rational> {
   return Gen<Rational>.compose { comp in
   let denGen = Int.arbitrary.suchThat { $0 != 0 }
   return Rational( comp.generate(), comp.generate(using: denGen)
   }
 }

What to Test? (7:44)

Properties

These are different types of properties. Round trip properties are the easiest. This is when you go from one value to another value, and then back to the original value. Suppose we have an Int, then made a String out of it by calling description and then passed that into Int; we should hopefully arrive back at the original Int.

Round Trip Properties

property("round trip to string") <-
 forAll { (i: Int) in
 return Int(i.description)! == i
}


Another example of a round trip property is if we have an array, where if you reverse it twice, we would get the original array.


// for all foo: [Int]
foo.reverse().reverse() == foo


However, there are some flaws to this strategy. One of the flaws is that if you make the same mistake twice, say in your encoder and your decoder, the test won’t catch that. If I encode in exactly the same wrong way that I decode, it won’t be a problem. If the reverse method simply returned the original array, then we would not catch that either.

Commuting Diagram Properties

This is similar to the round trip. This tests if I can arrive to the same place in two different ways. Suppose I have a base 64 encoder and Apple has a base 64 encoder. Both of those encoders should end up with the same String.


property("base64 encoding commutes with Apple's ") <- forAll { (data: Data) in
 return data.base64Encoding == data.myBase64Encoding
}

I want to test addition. Commutativity is required such that 1 + 2 is the same as 2 + 1. This makes sense, but if your program does not respect this, it will be wrong. It’s easy to implement your method in some way where that just doesn’t happen because of an oversight. We can also test with three arguments.

func testAddition() {
  property("Addition is commutative") <- forAll { (i: Rational, j: Rational) in
    return (i + j) == (j + i)
  }
}

property("Addition is commutative 2") <- forAll {
  (i: Rational, j: Rational, k: Rational) in
  let caseOne = (i + j + k)
  let caseTwo = ((i + j) + k)
  let caseThree = (i + (j + k))
  return (caseOne == caseTwo) && (caseTwo == caseThree)
}

Replay (13:47)

What happens when your test fails? Once you get the error, you catch it, and then you can have it replay that as many times as you need to figure out what the problem is.

Mixed Testing (15:00)


func testEquality() {
   let oneOverTwo = Rational(1, 2)!
   let twoOverFour = Rational(2, 4)!
   let ichiUeNi = Rational(1, 2)!
   XCTAssertEqual(oneOverTwo == twoOverFour, true)
   XCTAssertEqual(twoOverFour == ichiUeNi, true)
   XCTAssertEqual(ichiUeNi == oneOverTwo, true)
   XCTAssertEqual(oneOverTwo  twoOverFour, false)
   XCTAssertEqual(twoOverFour  ichiUeNi, false)
   XCTAssertEqual(ichiUeNi  oneOverTwo, true)
   let halfIntMax = (Int.max / 2) - 1
   property("value holds") <- forAll { (value: Rational) in
   return ((value.numerator < halfIntMax) && (value.denominator < halfIntMax)) ==> {
   let doubledSame = Rational(value.numerator * 2, value.denominator * 2)!
   return (value == doubledSame) && ((value  doubledSame) == false)
   }
 }
}

You can see here in the fourth line that I’m still using assertions. I can still use XCtest to run boring, not random tests. You can mix them up in the same methods, and you just have to use the property syntax for the properties that you actually want.

Shrinking (15:26)

Imagine you had an array with 2,000 elements and it failed, and then you got a test case back saying that it failed. Shrinking asks whether there is a way to make this array smaller and still fail.

The basic idea is that we’ve introduced this randomness, but we don’t want to just accept every case. Not every type that we create will have a reasonable shrink. Large collections you’ll want to shrink, but fortunately SwiftCheck implements arbitrary and shrink for arrays, sets, and dictionaries.

You can spend less time writing tests. You’ll still have to think through your test, and you’ll still have to spend time figuring out how to implement arbitrary and shrink, but the nice thing is it pays off. Hopefully you’ll get better tests out of this.

No one said that you have to stop writing tests the old way. You just have a new way to write tests. In my opinion, you get easier diagnosis, because as you fail in sort of these random cases, you get to see things that you didn’t think of.

You should write hard coded test cases if you find edge cases that you didn’t capture in your properties. There’s absolutely no problem in using the replay arguments or just hard coding it and saying “I want to be sure that every time I run my tests, this crazy crazy bug happens over and over”. If you’re still using Objective-C, you can use, you can use Fox, which gives us the ability to use this in Objective-C and Swift. However, you don’t get the generics. Hopefully you will all go out and download SwiftCheck or Fox.

Next Up: Keep on testing, all the way to your Realm apps

General link arrow white

About the content

This talk was delivered live in September 2016 at try! Swift NYC. The video was recorded, produced, and transcribed by Realm, and is published here with the permission of the conference organizers.

TJ Usiyan

TJ is a writer, musician, and developer interested in crafting interesting and artful work, and developer of the universal app Chordal Text and AU Additive Synthesizer. TJ is a graduate of Eugene Lang College and Berklee College of Music.

4 design patterns for a RESTless mobile integration »

close