Faster Web Tests with Parallel Batches in Thucydides
Introducing Thucydides
Thucydides on Github
Thucydides on Java.net
News
Presentations
Releases
Tips and Tricks
Tutorials
Web tests are as a rule much slower than other types of tests, but they can be sped up significantly by running them in parallel. However, this is often harder to implement than it sounds. The latest version of Thucydides (version 0.6.0) comes with support for running parallel test batches, making this task much easier.
Web tests make good candidates for concurrent testing, in theory at least, but the implementation can be tricky. For example, although it is easy enough to configure both JUnit and easyb to run tests in parallel, running several webdriver instances of Firefox in parallel on the same display, for example, tends to become unreliable.
The natural solution in this case is to split the web tests into smaller batches, and to run each batch on a different machine and/or on a different virtual display. When each batch has finished, the results can be retrieved and aggregated into the final test reports.
However splitting tests into batches by hand tends to be tedious and unreliable – it is easy to forget to add a new test to a batch, for example, or have unevenly-distributed batches.
The latest version of Thucydides lets you do this automatically, by splitting your test cases evenly into batches of a given size. In practice, you run a build job for each batch. You need to specify two parameters when you run each build: the total number of batches being run ( thucydides.batch.count ), and the number of the batch being run in this build ( thucydides.batch.number ).
For example, the following will divide the test cases into 3 batches (“thucydides.batch.count”), and only run the first test in each batch ( thucydides.batch.number ):
This will only work with the JUnit integration. However this feature is also now supported in easyb (as of easyb version 1.5), though using different parameters. When using the Thucydides easyb integration, you also need to provide the equivalent options for easyb:
If you have both easyb and JUnit Thucydides tests, you will need to specify both options.
This approach is easy to set up on Jenkins using a multi-configuration build. In the following screenshot, we are running a multi-configuration build to run web tests across three batches. We use a single user-defined parameter (BATCH_NUMBER) to define the batch being run, passing this parameter into the Maven build job properties we discussed above.
The most robust way to aggregate the build results from the different batches is to set up a second build job that runs after the test executions, and retrieves the build results from the batch jobs. You can use the Jenkins Copy Artifacts plugin to do this. First, ensure that the multi-configuration build archives the Thucydides reports, as shown here:
This build will then trigger another, freestyle build job. This job needs to copy the Thucydides report artifacts from the matrix build jobs into the current workspace, and then run the mvn thucydides:aggregate command to generate the Thucydides aggregate reports. The matrix build job reports need to be copied one-by-one for each batch, as the current version of the Copy Artifacts plugin does not support copying from multiple projects in the same action.
Then make sure you publish the generated HTML reports (which will be in the target/site/thucydides directory) for easy access to the test results.
This simple example shows a parallel test running 3 batches – this brought the test execution time from 9 minutes to slightly over 1 minute. Results will vary, of course, but a typical real-world set of web tests would have a larger number of batches.
Israel Santiago Dec 25 2011 Hi John, congratulations on this new feature! by the way, you got to verify the project TLB ( http://test-load-balancer.github.com/ ), seems promising to the task of parallelizing tests ! Reply
Hi John, congratulations on this new feature! by the way, you got to verify the project TLB ( http://test-load-balancer.github.com/ ), seems promising to the task of parallelizing tests !
kriskris8 May 29 2012 Hi, I am using thucydides about 8 months and I am really pleased about it. I am trying to configure jenkins to run tests in parallel batches following this post but I can’t figure out what should I put at “Build other projects”=> Projects to buid field. Is this an existing project from workspace, is a new project? Can you please help me with a suggestion? Thanks! Reply jfsmart May 30 2012 In the “Build other projects” field, you put a new build job. This build job will take care of aggregating the different batch results. Reply
I am using thucydides about 8 months and I am really pleased about it. I am trying to configure jenkins to run tests in parallel batches following this post but I can’t figure out what should I put at “Build other projects”=> Projects to buid field. Is this an existing project from workspace, is a new project? Can you please help me with a suggestion? Thanks!
jfsmart May 30 2012 In the “Build other projects” field, you put a new build job. This build job will take care of aggregating the different batch results. Reply
In the “Build other projects” field, you put a new build job. This build job will take care of aggregating the different batch results.
kriskris8 May 29 2012 Hi John, While trying to run tests in batches I have noticed that the the last build job for the 3’rd batch is not executed. I have 9 tests. I use batch.count=3 and batch.number=1; Only 2 build jobs are executed, so I get 6 tests executed, instead of 9. I have tested with 12 tests, batch count=3 and batch number1, again the 3’rd job build is not executed, but at 2’nd execution same tests as for 1’st batch are executed, so from 12 tests, 4 are executed, 2 times. No mather how many batch.count I specify, only 2 build jobs are executed. and the same tests are executed in the 2’nd build job. Reply jfsmart May 30 2012 Hi, I can’t reproduce this issue – could you raise a JIRA issue and attach a sample test case that reproduces the issue? Thanks, John. Reply
Hi John, While trying to run tests in batches I have noticed that the the last build job for the 3’rd batch is not executed. I have 9 tests. I use batch.count=3 and batch.number=1; Only 2 build jobs are executed, so I get 6 tests executed, instead of 9. I have tested with 12 tests, batch count=3 and batch number1, again the 3’rd job build is not executed, but at 2’nd execution same tests as for 1’st batch are executed, so from 12 tests, 4 are executed, 2 times. No mather how many batch.count I specify, only 2 build jobs are executed. and the same tests are executed in the 2’nd build job.
jfsmart May 30 2012 Hi, I can’t reproduce this issue – could you raise a JIRA issue and attach a sample test case that reproduces the issue? Thanks, John. Reply
I can’t reproduce this issue – could you raise a JIRA issue and attach a sample test case that reproduces the issue?
Thanks,
John.
Tatyana Dec 7 2012 Hi John, Can you please tell more about mvn thucydides:aggregate in the second project, which pom file is used? I can’t figure it out because after copying artifacts from batches I have 3 folders inside report project (BATCH_NUMBER=1, BATCH_NUMBER=2, BATCH_NUMBER=3). How it should generate one report on them? Reply
Hi John, Can you please tell more about mvn thucydides:aggregate in the second project, which pom file is used? I can’t figure it out because after copying artifacts from batches I have 3 folders inside report project (BATCH_NUMBER=1, BATCH_NUMBER=2, BATCH_NUMBER=3). How it should generate one report on them?
Татьяна Назаренко Dec 7 2012 Hi John, Can you please tell more about mvn thucydides:aggregate in the second project, which pom file is used? I can’t figure it out because after copying artifacts from batches I have 3 folders inside report project (BATCH_NUMBER=1, BATCH_NUMBER=2, BATCH_NUMBER=3). How it should generate one report on them? Reply
Faster Web Tests with Parallel Batches in Thucydides | Java 6 EE Testing | Scoop.it
Рубрика «Полезное чтиво». Выпуск 16 « XP Injection
Fill in your details below or click an icon to log in:
You are commenting using your WordPress.com account. ( Log Out / Change )
You are commenting using your Twitter account. ( Log Out / Change )
You are commenting using your Facebook account. ( Log Out / Change )
You are commenting using your Google+ account. ( Log Out / Change )
Connecting to %s
Notify me of new comments via email.
Note: HTML is allowed. Your email address will never be published.
Subscribe to comments
Search
Blogroll Official Selenium Blog Thucydides on Github Thucydides User Manual Thucydides User Manual (Single Page) Wakaleo Consulting – The Thucydides Company
Official Selenium Blog
Thucydides User Manual
Thucydides User Manual (Single Page)
Wakaleo Consulting – The Thucydides Company
Archives July 2015 (1) May 2015 (1) April 2015 (1) March 2015 (2) August 2014 (1) July 2014 (3) March 2014 (1) December 2013 (1) May 2013 (1) March 2013 (2) February 2013 (2) January 2013 (4) December 2012 (1) November 2012 (2) October 2012 (1) August 2012 (1) July 2012 (1) June 2012 (2) May 2012 (3) February 2012 (2) December 2011 (2) November 2011 (1) October 2011 (5) September 2011 (7)
July 2015 (1)
May 2015 (1)
April 2015 (1)
March 2015 (2)
August 2014 (1)
July 2014 (3)
March 2014 (1)
December 2013 (1)
May 2013 (1)
March 2013 (2)
February 2013 (2)
January 2013 (4)
December 2012 (1)
November 2012 (2)
October 2012 (1)
August 2012 (1)
July 2012 (1)
June 2012 (2)
May 2012 (3)
February 2012 (2)
December 2011 (2)
November 2011 (1)
October 2011 (5)
September 2011 (7)
ATDD batch-strategy BDD data-driven easyb example-driven firefox fluent FluentLenium HtmlTable Java jbehave jenkins jira JUnit page objects parallel testing proxy support release release tags reporting reports requirements Safari screenshots screeshots selenium spring tags testing Thucydides webelementfacade
Follow Blog via Email Enter your email address to follow this blog and receive notifications of new posts by email.
Enter your email address to follow this blog and receive notifications of new posts by email.
About This is the official blog for the Thucydides automated acceptance testing library. It contains news and updates about Thucydides releases, as well as tips, tricks and tutorials on how to use Thucydides to improve your WebDriver-based web tests, and links to related material.
Pages Introducing Thucydides Thucydides on Github Thucydides on Java.net
Create a free website or blog at WordPress.com.
The Traction Theme .
Get every new post delivered to your Inbox.
Join 3,120 other followers