If you have some experience developing Android apps in a production environment, you already know about the importance of testing your apps. While testing, we can do a lot of different tests, unit, integration… but for me, the ones that make me feel more secure about the codebase, are UI tests.
Traditionally in Android, UI tests are done with Espresso and require a physical or virtual device to be run, so as you can imagine, we are talking about tests that are heavy and in consequence, take a lot of time to run even if you don’t have a huge test coverage.
Some months ago, I discovered about screenshot testing, which is basically taking a screenshot of your app in a specific state that you interpret as valid and then comparing it to another screenshot taken while the test is running.
It seems too easy, right? Well, that’s because it is indeed.
Alternatives for screenshot testing in Android
In my investigation, I found that there are three top screenshot testing libraries:
Dropbox just released one but to be honest, I didn’t have time to check it out or read about it, so I’m keeping it out of this story, but keep it in your mind if any of the suggested ones doesn’t fit you.
Each one of those has its advantages and disadvantages, but in my case, I ended up using Shopify Testify.
The reason behind this decision is that there were some restrictions that complicated taking UI screenshot testing to a CI environment with Jenkins & Firebase, and I also found more documentation and support with the Shopify alternative. As far as I know, all of them work more or less in the same way, so if you start with one and want to switch teams, it doesn’t look like it would be a specially painful process.
Things to know before we start coding
Implementing a basic screenshot test is pretty easy, but there are two limitations that you need to be aware of.
First, when running screenshot tests, it’s mandatory that the device is the exact one with the same Android version always, or the emulator that you use needs to have the exact configuration every time. At the end of the day, what these tests are doing is just taking a screenshot to use as a baseline when the UI is in a state that you considered to be correct, and then running the app again, take a screenshot and compare it to the baseline. If the device has a different screen size, or the language is different, or the Android version changes, the test is going to fail, as we are comparing different images.
Second, I don’t know about the other libraries that I suggested before, but Shopify Testify is only able to take screenshots of the specified activity when the test is configured. If you navigate to another activity, the screenshot is not going to be taken correctly. I believe that this library was thought for applications that follow the Single Activity pattern where the app has one activity that acts as a container and all the UI is driven by fragments and different navigation stacks, or maybe there were some framework limitations, but I’m more inclined to the first possibility.
As you probably do with Espresso, remember to disable the animations scale in the developer settings of the device when running UI tests
Okay, once we are aware of the limitations, we can start with some code. As always, we need to start configuring the Gradle project, adding the following dependencies in our project build.gradle.
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath "com.shopify.testify:plugin:1.2.0-alpha01"
}
}And for the app module or the module that you are going to be working on this.
plugins {
id("com.shopify.testify")
}
dependencies {
androidTestImplementation "androidx.test:rules:1.4.0"
}After a Gradle sync, it’s time to write a test. For that, as you can see in the documentation, we just need the following lines.
@get:Rule var rule = ScreenshotRule(MainActivity::class.java)
@ScreenshotInstrumentation
@Test
fun default() {
rule.assertSame()
}This is going to launch the MainActivity and use that screen to take the screenshots. There are some Gradle tasks pre-built to interact with the library, so don’t forget to check them out.
In order to run the test that we just wrote, we need to record a baseline screenshot before. This screenshot is going to be stored by default in src/androidTest/assets/screenshots/<deviceName>where deviceNameis <apiLevel>-<deviceResolution>@<deviceDPI>-<locale>. For example, if we are running the test in a Pixel 3 emulator running Android 10, the screenshots are going to be store in src/androidTest/assets/screenshots/30-1080x2160@440dp-en_US.
This is why it was important to use always the same emulator configuration, because if you take the baseline screenshot and the run the test with a different emulator, the test is not even going to compare the screenshots because it’s not going to find a baseline.
To make a baseline screenshot of the above test, we can run the following command in the terminal.
./gradlew screenshotRecord -PtestClass=com.screenshotSample.MyScreenshotTest#defaultAfter the baseline screenshot is taken, you can run the test as every other test with Android Studio, and you will have your first screenshot test!
It’s also useful to know that you can configure the rule to run Espresso actions before taking the screenshot for the baseline and for the test run, so you can navigate to the part of the UI that you need to test.
Some extra features you may use
You can check the wiki in the GitHub project to learn more about what the library offers, but here I’m going to sum up the three that I used the most in my projects.
Exclude certain parts from the screenshots
This feature ignores a specific part of the UI that you choose. In the baseline screenshots you will see that even tough you exclude some views, the screenshot shows the entire UI, but when the test is run, it’s going to ignore that part and compare only the rest. This feature is useful for example for views that have a random component or for when you are working on something specifically and don’t want your tests to fail while you are at it.
Compare images
With screenshot testing, in order to know why the test is failing, you need to manually check the baseline screenshot and then the one that was taken while running the test. Sometimes is easy to see, but as you can imagine, this is not always the case. For that, the library comes with a Gradle task that relies on ImageMagick to compare the images and give you a visual representation of exactly what parts are different in your UI.
Run the test with a build variant
By default, the library is going to take the first build variant that you have set up, but if you want to take the baselines and run the tests with different build variants, you need to configure it in the Gradle module. Maybe this is something that not a lot of people uses, but in my case it was important, and it wasn’t explained in the documentation, so I had to read the source code to learn how to do this. Remember that you can do the same if you need something that is not explained in the wiki.
So, to specify a build variant, you just need to add the next lines to your Gradle module with the configuration that you need.
testify {
installAndroidTestTask ""
installTask ""
applicationPackageId ""
testPackageId ""
}Conclusions
If you’ve been coding along while reading the tutorial, you should have noticed how much faster and easier to run are this tests in comparison with Espresso. They have some limitations, sure, but I think that the speed is crucial, specially if you are running a CI system that makes you pay per use.
What do you think? Are you going to try this? Don’t forget to leave a comment if you want to add anything or ask any questions.
Featured image by David Travis on Unsplash
If you want to read more content like this and support me, don’t forget to check the rest of the bolg or subscribe here to get an email every time I publish new content.