Continuous integration can enhance automated mobile testing. Test data from multiple mobile devices running parallel tests can be consolidated to support monitoring. Jira tickets from manual testing can trigger the build process to ensure that testers will have the correct software version to do the manual testing. At QA Challenge Accepted, Burak Ergören shared his experiences from automating their mobile testing.
Continuous mobile development can benefit from test automation, by reducing manual testing efforts, minimizing human errors, and accelerating the release cycle. As the mobile test automation project grew in size and complexity, they sought ways to improve their agility and speed, Ergören mentioned. This led them to explore Continuous Integration (CI) and Continuous Delivery (CD), as Ergören explained:
So, how does CI work? Imagine you’re part of a software development team using a version control system (VCS). Developers submit their work to the VCS, and a CI tool automatically triggers actions when new code is added. These actions include building the software, running tests, and notifying the team.
They developed a CI pipeline—a series of automated steps for incorporating code changes. This pipeline handles tasks such as building the app, checking network connectivity, removing previous device installations, setting up the environment, running tests, and sending reports to the team, Ergören said.
When running tests in parallel, multiple devices are simultaneously being tested, resulting in the generation of multiple logs simultaneously. To manage these logs, they developed a centralized logging project. This project aims to consolidate complex logs into a single point for easier monitoring on a per-test basis, Ergören said. For this purpose, they selected the powerful and widely used Graylog tool to monitor the collected data.
Within their existing automation system, they maintain a mobile device farm connected to multiple servers. Given the diverse log types received from various sources, they implemented a solution to collect and transmit log data to Graylog using HTTP requests with specific configurations, facilitated by "Gelf Http Input":
At the conclusion of each test cycle, we gather data from Appium server logs, Android logcat logs, test result logs, and test output logs. We then transform this data into JSON format and transmit it to the Graylog server.
Once the data reaches Graylog, we can filter and analyze it as necessary, Ergören said. This allows them to inspect logs almost in real-time, organized by test method and test class, he added.
Access to detailed logs empowers us to make more precise assessments and offer better guidance to the relevant developers. Thanks to CI, our automation project has become more agile and responsive to changes, Ergören concluded.
InfoQ interviewed Burak Ergören about automated testing with continuous integration.
InfoQ: How do you combine manual testing with continuous integration?
Burak Ergören: We developed a project that integrates with Jira to address the issue of manual testers not being able to obtain mobile builds.
Through this project, tickets that have passed the developer review stage in Jira are automatically triggered in our CI system by our Jenkins autoBuild job before reaching the testing phase. This ensures that the appropriate build is obtained and ready for the tester to use from the relevant branch.
InfoQ: Do you have a system in place to measure and analyze the progress of your automation process?
Ergören: How long does the pipeline take? What is our total number of tests? What is our success rate? On how many devices do we run the tests?
We started our mobile metrics project to answer these questions at any time, analyze the progress of our mobile test automation project and track our metrics in Jenkins pipelines.
We developed a project using Node.js that utilizes Jenkins APIs to retrieve the necessary metric data from Jenkins.
This job processes the data and uploads it to BigQuery, and it is integrated with Grafana for monitoring purposes. This setup allows every member of the software development team to access the necessary metrics specific to automation.