The Search for Mobile App Test Automation

Mobile app UI test automation is still in its infancy.  Whereas web testing had around 20 years to bake some good tools such as Selenium, Watir, and many others, Mobile apps in the current incarnation and popularity have only existed for 6 years since iOS & Android launched in mid-2007.  Because Mobile apps are quite young there hasn’t been a lot of time for good test tools to be developed, and most that do exist are developer-centric.

Our team was doing a lot of manual app testing which left us open to missing the occasional important regression.  We started looking and evaluating automation test tools about 2.5 years ago.  Some of the most popular options we encountered along the way were Robotium for Android, Xcode’s Instruments for iOS, and KIF for iOS, all of which we tried with limited success before we landed on Appium as our framework of choice.

Our Robotium framework died within a month – before we had any test cases running on a regular basis.  Additions to our KIF test cases plateaued because they require coding ObjectiveC tied directly to our app code.  Our Instruments test cases were working but presented a serious maintenance burden.

Robotium Drawbacks:

  1. Had to compile each test case separately.  Yes, ant and pom file maintenance, Jenkins compile job creation, getting Eclipse setup properly along with the Android SDK.  It felt really heavy compared to most test case frameworks I’ve used.  I could just feel my productivity slow to a crawl. 
  2. Had to compile a library version of our app so the test case compile could link to it.  This worked for a while, but then one day although everything compiled fine, but test run-time broke.  Didn’t have the time to investigate why and it just died – real life release schedules cause complicated (read “high maintenance”) test frameworks to die.
  3. Had to code in Java and test code gets intertwined with the app class hierarchy.  If you really want quick/agile test case development/maintenance you usually use a scripting language.
  4. Only Static test cases were possible.  Test case workflow cannot interact with any other Server/DB validation during the test run.  You wind up your little test case mouse, let it run, wait for it to finish, and then you can perform some Server/DB validation in the wrapper script.  This does not support test cases where you need to validate the Server/DB between various app test actions, and possibly dynamically drive the next app test actions based on Server/DB state.  Nope – all you have is very static test cases along with post test run validation.

WindupRobot

Wind Up Robot source

Xcode’s Instruments Drawbacks:

  1. Had to code in Javascript.  Well at least we are scripting here – that was a plus over Java for Robotium, but Javascript was invented for different purposes and given a better alternative not many testers really want to use it as a test language.
  2. Had to record steps in Instruments and then copy out lines to your test script and customize.  Your test code can use accessibility labels of the UI which helps, but it still gets its fingers into the view hierarchy.  This presented a maintenance hassle with rapid design iterations when working with an agile team.

After struggling with Robotium and Instruments, the Appium experience felt like this:

Beach

Tropical Beach source

It took us 2 weeks from trying Appium (http://appium.io & https://github.com/appium/appium) to get our first project running complete with setup and wrapper scripts.  Instantly we felt the productivity improvements – the existing test case maintenance was reduced, and new test case development was definitely faster.  Additionally all the testers enjoyed writing new test cases because it isn’t drudgery.

Appium’s Advantages:

  1. Appium is modeled after the well-established web-based test framework: Selenium.  Exact fit for latest Agile & CI technologies such as Jenkins, nose, buildout.
  2. This isn’t a “record and playback” UI-oriented test framework, then again your test cases won’t be brittle like those systems tend to be.  We use the same test code to drive our app testing on iPhone as well as iPad. And in some cases, we were able to use the same test cases for Android.
  3. It uses the same language for coding iOS GUI automation as Android GUI automation.  If your iOS and Android apps had the exact same element tagging and the exact same screens/layout/workflows you could use the same test code for both.  This will probably never be a reality for most companies because iOS & Android support different features and it’s hard to keep both your iOS and Android development and releases in lock-step.  But this ability has allowed us to share some common automation libraries.
  4. It allows you to create dynamic test cases that validate any service/DB, and drive dynamic test case actions.  This allowed us to add Google Analytics validation to our app tests very rapidly.
  5. It allows you to code in Python, Ruby, Javascript, Perl, PHP, Java, C#, ObjectiveC, or Clojure, and code revision control system of your choice. Your test code is not locked up in some framework’s Test DB or their proprietary language.
  6. It allows you to use Appium.app MacOS GUI to discover element names.  No need to have the app source code or have your testers dig into the source code or repeatedly sync and recompile the app in Xcode to develop test cases.
  7. It allows you to use xpath, name, tag_name to find elements.  Using element ‘name’ search is ideal because it accesses app’s accessibility labels and makes test code less brittle to app changes, but when not everything in your app is tagged then xpath is super powerful.  All the search methods are a powerful way to access elements without knowing app class hierarchy.
  8. It allows you to drive automation on iOS/Android devices and iOS/Android Simulators without any code changes (just need to code some slight setup differences).
  9. It boasts low operational and infrastructure costs:  we are using 3 Mac Minis to run our Appium automation.
  10. It allows you to upload test cases into Appium cloud and use a provider like SauceLabs to run your tests on a large number of devices and simulators.
  11. It is open source, which means you can add your own features, and get bugs fixed faster.
  12. It can be used to automate your MobileWeb pages on the device’s browser.  Again, test code is Selenium just like what you use for DesktopWeb testing.  Appium reports that they do MobileWeb testing better because they’ve solved certain issues with using straight Selenium/Webdriver.
  13. Webviews test exactly the same as native app code.
  14. Appium sits on top of Instruments for iOS, and UIAutomator for Android so it can harness the full power of the primary automation platforms.
  15. It runs on Windows/Linux/Mac hosting OS – although Mac supports both iOS/Android, whereas Linux/Windows only support Android.

Appium Drawbacks:

  1. Appium is still fairly young.  Not every GUI gesture is natively supported, although you can execute javascript through Appium to perform missing methods.  So far we’ve only made about 10 javascript wrapper methods and it hasn’t ever blocked us.  Appium is an active Open Source project – new features/fixes every day so the future looks bright.
  2. It can only run 1 test case at a time per hosting machine.  Limitations of Instruments (for iOS) force this.  You can plug in multiple devices to the hosting machine, but can only harness one of them at a time.
  3. It is hard to get Android automation working for Android OS versions that are < 4.1  Those OS versions are supported through an additional Selendroid library, but it takes extra work and Selendroid does not support xpath search for elements which is a blocker issue for us, so we haven’t tried to go there.
  4. Test cases do take some tuning to get the timing correct so they don’t incorrectly fail.  Sometimes the server you app is talking to can delay the response and you have to write polling loops to detect when app is ready for next test step.
  5. Appium documentation is a little weak – you are probably going to need an experienced coder to lay your Appium foundation.
  6. Like most Mobile automation frameworks your test case runs within the app. This means that it is harder to test scenarios outside the app like device Notifications.
  7. Sometimes the test code needs to scroll up or down to find elements that are currently not visible.
  8. For iOS you can’t hook in an automatic Alert handler (for dialog boxes that popup) like you can in Instruments.  Instead you have to know everywhere in the test automation code when you expect an alert to come up and then overtly switch over to the handler to handle it.
  9. Testing Webviews in Android are currently difficult to code since discovering the elements is broken (Note: Webviews in iOS work just fine).  It looks like this might be fixed when using Android OS 4.4

None of Appium’s drawbacks have been a showstopper for our testing, and usually features that we’ve wanted have been added to Appium with a few months.  A side benefit also occurred once we converted over to Appium.  Before only experienced coders had the skills to code and maintain our Robotium and Instruments framework and test cases, but now with our Appium wrapper code foundation we’ve been able to get more novice coders to add test cases – and our test case count has started to explode.  Our test code is being maintained and we get many 100% Pass test runs.

After working on Appium for 6 months I have one additional thing to be thankful for this Thanksgiving.

Thanksgiving

Twice Berry Pie source

 

 

9 thoughts on “The Search for Mobile App Test Automation

  1. Excellent post! I completely agree, Appium is THE framework to start mobile automation projects with.

  2. Thank you for the thorough review of Appium with the detailed pros/cons! Appium already seemed clearly the best choice to me, but the list of cons is very helpful to know what hurdles to anticipate.
    -pcp

  3. […] This was originally written by Curtis Siemens for the Zillow Engineering blog, find the rest of the post here. […]

  4. Hey Curtis,

    I’d be interested to get your take on TouchTest (http://www.soasta.com/products/touchtest/) as all the Apium cons you mention are taking care of by TouchTest. Especially around parallel test execution, custom wait (no need for polling loops), support for ALL gestures across iOS and Android, support for Android 2.3.3 to Android 4.4 KitKat. I’d be happy to set you up with a free account if you’re interested. Let me know!

    Fred

  5. Great post! By the way, you mention that you even implemented Google Analytics validation tests, how did you manage to do that for iOS apps? As far as I am concerned it is not possible to launch Apple’s simulator with specific proxy settings, you must set the proxy settings at a system level in Mac OS right?

    • Curtis Siemens

      The way we validate Google Analytics is to wrap the GA SDK and have it log to the device console with a special pre-pended string such as something like “MyCompanyGA:”. Then in your appium tests have capabilities include “showSimulatorLog”:true and then check the appium server log file for new events that have been logged (of course filtering only for the MyCompanyGA: lines). Then I built a library that allowed us to do differences of what we saw logged vs. a benchmark of what GA events we expected to be logged. We do an action-X, and then check against GA-benchmark-X, then do another action-Y and then check against GA-benchmark-Y – this way you see exactly what action’s GA broke expectations.

  6. Really great piece of review. I was looking exactly for that kind of mix of pros/cons comparison of most popular testing tools. Now it’s time to learn some Appium ;)

  7. Great post. While researching what tool to use to test mobile apps that my company is developing, I stumbled upon this post and will likely now also move forward with Appium. I hope they fix the documentation problem really soon (that is the most concerning of disadvantages in the list above to me).

  8. Is there a way with Appium to verify that the Android app is running? Is app-activity necessary to specify?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>