Appium, when automation testing can be randomly wrong

Appium is an UI automation testing framework, helping developers to automatically test their app. This tool can be really powerful but my experience with it let me think it’s not enough accurate to be used everyday and at its full potential.

When developing an app, you can’t always be sure that the new feature you’ve just added didn’t break another one. Automation testing helps to figure that out. It’s also true for UI part: maybe a button is now disabled or hidden, maybe it was deleted or moved below an other view: it’s not available anymore.

For this reason, UI Automation testing lets developers check user flow and be sure this is not happening.

Appium seems to be a great tool for that, it’s based on Selenium and compatible with iOS, Android and web apps. Configuration is quite simple and its community is really important.

Install appium

brew install node
npm install -g appium
npm install wd
appium &
node custom-appium-test.js

Oh, and you can code your test with almost all languages (Java, PHP, Ruby, Perl, Javascript…) which is really appreciate.

Instantiate capabilities for my test:

public static $browsers = array(
    array(
        'desiredCapabilities' => array(
            'platformName' => 'iOS',
            'platformVersion' => '9.1',
            'deviceName' => 'iPhone 2'
        )
    )
);

However, I’ve got issues that let me think it’s not yet completely functional, at least for iOS.

Appium performance is hardly depending of version you’re using. A good test code can work for a version and break in next one which is frustrating. Hopefully, its community and forum are really helpful to report them and keep an updated (working) version.

It’s also depending of architecture you’re using: I started with a local server but finally moved to a remote one to monitor my tests. I quickly noticed that right test became wrong because of the request delay, I couldn’t accept UIAlertView for notification by default: I had to add more time to keep it working. This shows me how it can be limited: if you’re not 100% sure of your server side (Appium here), why trying testing the front then…

However, used with Sauce Labs platform, it becomes such a great thing: SauceLab allows you to capture screenshot, screencast and every command lines used. Your test will be run on VM and all devices are available for testing. It can also be useful with continuous integration (CI). If something went wrong you know where in the code but also in the UI.

Saucelabs appium

Working with Appium at work, we’re using it in a way to keep our back-end api compatible with older front apps. So it’s more for retro compatibility than maintaining the front apps itself.

My conclusion is that Appium is not yet mature and good enough to maintain front apps. Even if it can be cross platform, I’m pretty sure we can keep this part divided and I can be more efficient with UI Automation. However, it’s free and open source, and you don’t need big infrastructure to make it work. I also bet on his community which can push the technology higher, so I keep it in mind and wait for it.