End-To-End Testing using Cucumber, Frank & Kiwi – Part 1

A few days ago I was talking about Test Driven Development on the Cocoa Pulver Podcast. I mentioned that I like the idea behind TDD but that there are some quirks about it when it comes to testing view controllers (especially in Objective-C). I also mentioned Behavior Driven Development (BDD) and shortly explained the gist of it.

Now I don’t want to write too much about TDD and BDD here. There are a lot of other resources out there that are much more qualified to do that than I am. What I want to show today is my idea of how End-To-End testing in app development should look like. That’s a current snapshot of how I want to tackle my current pet project. I didn’t find too much information about this out there, so I started a learning process which I hope might also help some others to get better at testing their apps.

The pain of manual UI testing

Now, what probably most people do or did, and how I did it until recently is to start their project, build that app and realize later on that they should have tested their code. Suddenly there are bugs and feature requests coming in or the whole app was tailored for one customer but now there are other opportunities to sell it to other customers which need a few changes here and there. It took me a long time to come to a point where I just didn’t want to do all this hideous work of testing everything manually again and again and again… Change a small line of code here, run the app in the simulator and test all those edge cases that might be effected even if you didn’t want that. Change another line of code and do it all over again.

I did some tests with Apple’s UIAutomation before and just didn’t like it. It never worked out really well for me so I dropped it completely and never visited it again. But I wanted to automate all those annoying manual UI tests that were going on in my last project over and over again.

Cucumber comes into play

That’s when I started to look into BDD and specifically Cucumber. Cucumber is a great tool to write user interface tests by simply describing what the app should do and what you expect to happen on the UI when you interact with it. Cucumber itself is basically just a framework that gives you the tools to write the tests, but it needs the help of other tools and some scripts to actually run the tests on your app. I looked at a few tools like KIF, Calabash for iOS or Frank. Calabash turned out to be on its way to be integrated into Frank, so I stick with that. I really don’t get the name for the tool, but I think it’s the best you can get when it comes to UI testing for iOS. You should definitely go over there and check how to use it. It is of great help and extremely simple to install and integrate with your project.

Writing UI tests with Cucumber is really a joy. It needs some ground work though, and I really recommend to read The Cucumber Book to everyone who is interested in using it for their testing. Once you got familiar with how to write Scenarios (as tests are called by Cucumber) you really can describe your app in real human language without programming anything in the first place. A really great feature is that you can write the scenarios in your actual language. It doesn’t matter whether you write them in German, English, Swedish or whatever language you want, which I believe could be a great benefit for some projects, especially when the vocabulary you use is very domain specific and sometimes hard to translate. I could go on and on about how cool Cucumber is… but really: Just read the book, it’s a great introduction and gives you all the things you need to write good scenarios.

One of the biggest advantages of Cucumber in my opinion is the possibility to grow your app step by step. I sometimes have the problem that I have an idea for an app but I don’t get a grip on how I really should implement that idea or which features I want to build into it. When writing scenarios you a) have a living documentation of the features of your app and b) you will think about what your app does and how you want it to be done.

Once you have a set of scenarios there are some snippets required to actually drive your app. Those snippets are called step definitions. Each scenario in Cucumber consists of several steps and each step needs a step definition which drives and explores your app. Again, please check the documentation of Cucumber and especially Frank about how to implement the connection between your scenarios and the app. Those step definitions are written in Ruby. Only very basic Ruby is required though, so don’t be afraid! Ruby is quite fun actually – compared to JavaScript which is just downright horrible in my opinion.

The problem with these step definitions is that you have no actual app to test yet. Writing them requires a very prolific knowledge of the internals of your app or at least a very detailed idea of how you will implement a scenario. I think you can’t really drive Objective-C coding by writing the steps, like you can do in Ruby applications. But what you still can do is to build your app based on the scenarios you wrote. Scenarios in this case are just the specific use cases you want to implement. Take them and create your app using TDD – we’ll get there later. As soon as your app is running, you can implement the step definitions for your scenarios and see if your app is doing what you expect it to do. As soon as the scenarios are running, you can be sure that your code changes have no effect on the behavior of your app anymore.

But the UI isn’t everything

Having Cucumber Scenarios running is great. It gives you confidence in your code and your end product. But what about all those changes in your model, view model, data storage or business logic? Can you really be sure that they work as expected just by running the UI tests? What happens when you have to adjust your business logic to another client or add a new feature? Maybe you see a component you created for one app that you want to use in your other app now too but you need to tweak it a little bit. How can you be sure that it really is doing what it should?

That’s why I don’t want to only use BDD from the top and check my app from the outside. I also want to build my app the TDD way from the inside. Here’s the deal: I am going to test the behavior of my app from the outside using Cucumber scenarios. At the same time I’ll develop the basis of the app using TDD. That means that I will write unit tests for all my code from the bottom up to the view controllers. I won’t test them anymore as I expect them to just drive the user interface which is then tested by my Cucumber scenarios.

I used to write unit tests for view controller code, checking outlet connections etc. but this is tedious work. That code is really hard to test and the tests will be fragile. My experience is that you’re better off not to write those unit tests at all. If the user interface works as expected by your Cucumber scenarios, you can be pretty sure that the app is not going to crash because of an outlet not being set in a view controller. This is valid as long as your scenarios are complete of course. If they are not, the approach I am trying to use here, might not fit for your use case.

Since I am already in behavior driven mode using Cucumber, I want to maintain that state and write my unit tests the same way. That’s why I am not really fond of using XCTest or some other xUnit based testing framework. I use Kiwi to write unit tests, but there are other similar frameworks out there like Specta. It’s just a matter of taste which framework you like more. In my case I am doing fine with Kiwi. It has a built-in mocking framework and I really like the way the matchers are written down. Others may disagree, and that’s ok… it’s really just a matter of taste.

Having a lot of unit tests also helps debugging because you are able to nail down problems more easily. I recently fixed a few bugs in conditions I have not tested yet. I wrote new tests for that case and now I can be sure that this problem will never happen again. The problem is out of your mind and you don’t have to worry about it anymore when you go on building the app. You can concentrate on new stuff and don’t have to think about what might break when editing some code.

The workflow

Let me summarize my current workflow of implementing new features:

  1. If I have no clue at all about how the app should look like, how it should work or how I think what a specific screen should present to the user, I create a new Cucumber scenario. Here I can describe what the app should do in real language.
  2. When that scenario is complete, I go on and see what my model needs to provide to the (not yet existing) view controllers to work correctly. I scribble down some objects which might be useful and create a simple design for that.
  3. Having the basic design draft on paper I go into Xcode and start writing unit tests using Kiwi. I strictly follow the TDD paradigm here. I really write the tests before I write the code. If I start testing a new class, I even have the initial test before the class files are existing. Once a test is written, I execute it. Having that failing test, I write the code to make the test succeed. If needed I refactor.
  4. After I created the model and all the tests are in place to secure me from breaking things with future changes, I can go on and create the view controllers using my model with the goal to accomplish what I described in my Cucumber scenarios before.
  5. When the view controllers are ready and I played around a bit in the simulator I go on and run the Frank console to explore my app from the inside to be able to easily write the step definitions for my scenarios.
  6. Having all that information about my app from the Frank console, I write the ruby snippets to glue the scenarios to the app.
  7. I run Frank and let him execute all my Cucumber scenarios. Now I see whether the app behaves as expected or not. Sometimes it turns out that the scenarios are described wrong or that some step definitions need adjustments, which is ok.

The outcome of this workflow is that you have a set of tests from top to bottom. Your app is tested from the outside to do what you expect it to do. Your code is tested from the inside, so you can make changes to it and still know that everything works as before. All this together puts you into a really great position: you are confident about the quality of your app! You are confident that your changes didn’t break anything. You are able to adjust things for customers and fix bugs without having to worry to introduce new bugs. You know that your app is doing fine. From the inside and from the outside. End-to-end.

I plan to write another part on this topic to show how a project could be set up to run Cucumber, Kiwi etc. and maybe even doing continuos integration with all those tools.