Skip to content

Posts tagged ‘reporting’

31
Dec

Thucydides Release 0.9.229


There have been many releases since the last release notes were published, so this entry will summarize some of the highlights that are included in release 0.9.229 (though many came out in releases previous to that). Some of the highlights include:

  • Added support for Selenium 2.39.0
  • Testing AngularJS is made easier with support for the ng-model attribute in the Thucydides @FindBy annotation. Suppose you have this AngularJS input field:
<input ng-model="angularField" value="Model value" />

You can now find this directly using the Thucydides @FindBy:

import net.thucydides.core.annotations.findby.FindBy
 @FindBy(ngModel = "angularField")
 public WebElementFacade ngModelField;
  • Support for nested page objects – you can have page object fields inside other page objects. This makes it easier to write page objects for smaller reusable sections of the screen.
  • You don’t need to override the constructors for ScenarioSteps and PageObjects any more
  • You can provide your own webdriver instance using the ‘webdriver.provided.type’. Just implement the DriverSource interface. For example, you could implement a class like this:
package com.acme
public class MyFunkyDriverSourceImpl implements DriverSource {
    public WebDriver newDriver() {
       return new FunkyWebDriver();
    }
}

Then just run the tests with the following properties (e.g. in the thucyiddes.properties file or on the command line):

webdriver.driver = provided
 webdriver.provided.type = funky
 webdriver.provided.funky = com.acme.MyFunkyDriverSourceImpl
  • Lots of improvements to the reports, including:
    • Reports now have the option to hide the pie chart on the aggregate pages
    • Test reports now display the date and time of the report generation on each page.
    • You can use the ‘show.related.tags’ system property to display (default) or hide related tag statistics on the dashboard
    • Support for reporting on releases/versions by integrating with JIRA (there will be a full article on this feature shortly)
    • Support for integration with 3rd party test management software such as the JIRA Zephyr plugin to report on manual as well as automated tests
  • Many bug fixes

If you are still on an older version, update your dependencies today!

Advertisements
9
Mar

Thucydides Release 0.9.103 – Improved reporting, other enhancements and bug fixes


Thucydides Release 0.9.103 adds some useful new features and enhancements.

Reports distinguish between errors and failures

Test reports now distinguish between test errors and failures. The following screenshot will make this clear.

Reports distinguish between Test Errors and Failures

Reports distinguish between Test Errors and Failures

Detailed report will not show screenshots column if there are no screenshots in the test

Tests that do not have any screenshots will no longer show an empty screenshots column in the details page.

no-screenshots-column

Injecting WebElementFacades in Page objects

Instead of declaring WebElement variables in Page Objects and then calling element() or $() to wrap them in WebElementFacades, you can now declare WebElementFacade variables directly inside the Page Objects. This will make the Page Object code simpler more readable.

So, instead of writing


@DefaultUrl("http://en.wiktionary.org/wiki/Wiktionary:Main_Page")
public class DictionaryPage extends PageObject {

    @FindBy(name="search")
    private WebElement searchTerms;

    private WebElement go;    //variable name matches element id or name

    public DictionaryPage(WebDriver driver) {
        super(driver);
    }

    public void enter_keywords(String keyword) {
        element(searchTerms).type(keyword);
    }

    public void lookup_terms() {
        element(go).click();
    }

    public List getDefinitions() {
        WebElement definitionList = getDriver().findElement(By.tagName("ol"));
        List<WebElement> results = definitionList.findElements(By.tagName("li"));
        return convert(results, toStrings());
    }

    private Converter<WebElement, String> toStrings() {
        return new Converter<WebElement, String>() {
            public String convert(WebElement from) {
                return from.getText();
            }
        };
    }
}

you can write

@DefaultUrl("http://en.wiktionary.org/wiki/Wiktionary:Main_Page")
public class DictionaryPage extends PageObject {

    @FindBy(name="search")
    private WebElementFacade searchTerms;

    private WebElementFacade go;       //variable name matches element id or name

    public DictionaryPage(WebDriver driver) {
        super(driver);
    }

    public void enter_keywords(String keyword) {
        searchTerms.type(keyword);  //directly use facade
    }

    public void lookup_terms() {
        go.click();                //directly use facade
    }

    public List getDefinitions() {
        WebElement definitionList = getDriver().findElement(By.tagName("ol"));
        List<WebElement> results = definitionList.findElements(By.tagName("li"));
        return convert(results, toStrings());
    }

    private Converter<WebElement, String> toStrings() {
        return new Converter WebElement, String>() {
            public String convert(WebElement from) {
                return from.getText();
            }
        };
    }
}

Switching to another page object

A new method, switchToPage() has been added to PageObject class to make it convenient to return a new PageObject after navigation from within a method of a PageObject class. For example,

@DefaultUrl("http://mail.acme.com/login.html")
public class EmailLoginPage extends PageObject {

    ...
    public void forgotPassword() {
        ...
        forgotPassword.click();
        ForgotPasswordPage forgotPasswordPage = this.switchToPage(ForgotPasswordPage.class);
        forgotPasswordPage.open();
        ...
    }
    ...
}

Other minor enhancements and bug fixes

  • Selenium version updated to 2.31.0.
  • Added containsOnlyText and shouldContainOnlyText methods to WebElementFacade. These methods are similar to containsText/shouldContainText methods but check for exact match.
  • Modified shouldContainText method in WebElementFacade so that error message shows both the expected text, and the text found in the element.
  • Fixed a bug in the error message of the method WebElementFacade.shouldNotContainText.
  • Now you can both activate and deactivate native events for Firefox by setting the thucydides.native.events property to true or false.
  • Thucydides-141: Fixed a bug due to which when trying to open a page with slash in the end of url, browser tried to open page without slash.
6
Feb

Thucydides Release 0.9.95 – Example reporting for junit tests


Example reporting comes to Junit tests

Our latest release (0.9.95) brings the example reporting format changes (released last month for jBehave) to parametric junit tests. A few cosmetic changes have also been done for better organization of information and improved readability.

Let’s look at these changes in detail.

Tests table is reorganized to remove clutter.

As the screenshot above demonstrates, the test table on main report page reduces clutter by showing test results at a test-method level instead of a separate row for each parameter set.

Test details page is better organized for readability

Test details page is better organized for readability.

The details page will list all parameter sets in a table. By default, the parameters are names “Parameter 1”, “Parameter 2” and so on. But this can be configured with a comma separated list of column headings passed as an attribute to the @TestData annotation.


@TestData(columnNames = "Word,Definition")
public static Collection<Object[]> testData() {
  return Arrays.asList(new Object[][]{
    {"apple", "A common, round fruit"},
    {"banana", "An elongated curved fruit"}
  });
}

Further, the steps section will show a collapsed view with a row for each parameter set. The plus  icon can be clicked to expand the top level step to reveal all the steps and corresponding screenshots. Failed test examples will be highlighted in red.

Support for Firefox preferences

A new property firefox.preferences can be used to supply a semicolon separated list of Firefox configuration settings. For ex.,

-Dfirefox.preferences="browser.download.folderList=2;browser.download.manager.showWhenStarting=false;browser.download.dir=c:\downloads"

Integer and boolean values will be converted to the corresponding types in the Firefox preferences; all other values will be treated as Strings. You can set a boolean value to true by simply specifying the property name, e.g. -Dfirefox.preferences=app.update.silent.

For the lazy, here’s a complete reference to Firefox’s configuration settings – http://kb.mozillazine.org/Firefox_:_FAQs_:_About:config_Entries

Support for test count based batch strategy

Thucydides makes it possible to split the web tests into smaller batches, and to run each batch on a different machine or different virtual display. This is achieved by  specifying two parameters when you run each build: the total number of batches being run (thucydides.batch.count), and the number of the batch being run in this build (thucydides.batch.number).

For example, the following will divide the test cases into 3 batches (thucydides.batch.count), and only run the first test in each batch (thucydides.batch.number):

mvn verify -Dthucydides.batch.count=3 -Dthucydides.batch.number=1

However, this strategy divides the test cases equally among batches. This could be inefficient if some of your test cases have more tests than others. This release adds a new batching strategy based on number of tests in the test cases. The strategy is implemented with the help of a TestCountBasedBatchManager which evenly distributes test cases across batches based on number of test methods in each test case.

29
Jan

Thucydides Release 0.9.94 – More improvements to reports and some bug fixes


We released version 0.9.94 of Thucydides over the weekend.

Improved reports home page

Reports home page now shows two pie charts, one with the overall tests count and another with the test count weighted by number of steps.

Screen Shot 2013-01-30 at 6.27.24 AM

 

 

 

Screen Shot 2013-01-30 at 6.27.33 AM

 

 

The first pie chart gives an overall status of the tests as it shows how many tests have passed, failed or are still pending. The second pie-chart gives the test status weighted by number of steps which could be used to deduce the nature/criticality of test failures.

There are also some minor improvements to the requirements reporting page.

Updated Selenium version to 2.29.0

Selenium version has been updated to 2.29.0 (Selenium Release notes)

Minor enhancements and bug Fixes

  • #Thucydides-122: Added thucydides.verbose.steps property, that provides more detailed logging of WebElementFacade steps.
  • #Thucydides-125: Fixed a bug that caused reports to fail for parameterized tests with new line characters in the test data.
  • #Thucydides-130: Fixed a bug due to which PageObject.upload was not working for files on classpath.
20
Jan

Functional Test Coverage – taking BDD reporting to the next level


Conventional test reports, generated by tools such as JUnit or TestNG, naturally focus on what tests have been executed, and whether they passed or failed. While this is certainly useful from a testing perspective, these reports are far from telling the whole picture.

BDD reporting tools like Cucumber and JBehave take things a step further, introducing the concept of “pending” tests. A pending test is one that has been specified (for example, as an acceptance criteria for a user story), but which has not been implemented yet.

In BDD, we describe the expected behaviour of our application using concrete examples, that eventually form the basis of the “acceptance criteria” for the user stories we are implementing. BDD tools such as Cucumber and JBehave not only report on test results: they also report on the user stories that these tests validate.

However this reporting is still limited for large projects, where the numbers of user stories can become unwieldy. User stories are not created in isolation: rather, user stories help describe features, which support capabilities that need to be implemented to achieve the business goals of the application. So it makes sense to be able to report on test results not only at the user story level, but also at higher levels, for example in terms of features and capabilities. This makes it easier to report on not only what stories have been implemented, but also what features and capabilities remain to be done. An example of such a report is shown in Figure 1 (or see the full report here).

new-reporting

Figure 1: A test coverage report listing both tested and untested requirements.

In agile projects, it is generally considered that a user story is not complete until all of its automated acceptance tests pass. Similarly, a feature cannot be considered ready to deliver until all of the acceptance criteria for the underlying user stories have been specified and implemented. However, sensible teams shy away from trying to define all of the acceptance criteria up-front, leaving this until the “last responsible moment”, often shortly before the user story is scheduled to be implemented. For this reason, reports that relate project progress and status only in terms of test results are missing out on the big picture.

To get a more accurate idea of what features have been delivered, which ones are in progress, and what work remains to be done, we must think not in terms of test results, but in terms of the requirements as we currently understand them, matching the currently implemented tests to these requirements, but also pointing out what requirements currently have no acceptance criteria defined. And when graphs and reports illustrate how much progress has been made, the requirements with no acceptance criteria must also be part of the picture.

Requirements-level BDD reporting with Thucydides

Thucydides puts some of these concepts into practice. Thucydides reports not only on how the tests did, but also fits them into the broader picture, showing what requirements have been tested and, just as importantly, what requirements haven’t.

During the rest of this article, we will see how to report on both your requirements and your test results using a very simple directory-based approach. You can follow along with this example by cloning the Github project at https://github.com/thucydides-webtests/thucydides-simple-demo

Simple requirements in Thucydides – a directory-based approach

Thucydides can integrate with many different requirement management systems, and it is easy to write your own plugin to tailor the integration to suite your particular environment. A popular approach, for example, is to store requirements in JIRA and to use Thucydides to read the requirements hierarcy directly from the JIRA cards. However the simplest approach, which uses a directory-based approach, is probably the easiest to use to get started, and it is that approach that we will be looking at here.

Requirements can usually be organized in a hierarchial structure. By default, Thucydides uses a three-level hierarchy of requirements. At the top level, capabilities represent a high-level capacity that the application must provide to meet the application’s business goals. At the next level down, features help deliver these capabilities. To make implementation easier, a feature can be broken up into user stories, each of which in turn can contain a number of acceptance criteria.

Figure 2: JUnit test directories mirror the requirements hierarchy.

Figure 2: JUnit test directories mirror the requirements hierarchy.

Of course, you don’t have to use this structure if it doesn’t suit you. You can override the thucydides.capability.types system property to provide your own hierarchy. For example, if you wanted a hierarchy with modules,epics, and features, you would just set thucydides.capability.types to “module,epic,feature”.

When we use the default directory-based requirements strategy in Thucydides, the requirements are stored in a hierarchial directory structure that matches the requirements hierarchy. At the lowest level, a user story is represented by a JBehave *.story file, an easyb story, or a JUnit test. All of the other requirements are represented as directories (see Figure 2 for an example of such a structure).

In each requirements directory, you can optionally place a file called narrative.txt, which contains a free-text summary of the requirement. This will appear in the reports, with the first line appearing as the requirement title. A typical narrative text is illustrated in the following example:

Learn the meaning of a word
In order to learn the meaning of a word that I don't know
As an online reader
I want to be able to find out the meaning of the word

If you are implementing the acceptance criteria as JUnit tests, just place the JUnit tests in the package that matches the correspoinding requirement. You need to use the thucydides.test.root system property to specify the root package of your requirements. For the example in Figure 2, this value should be set to nz.govt.nzqa.lssu.stories.

Figure 3: The narrative.txt file appears in the reports to describe a requirement.

Figure 3: The narrative.txt file appears in the reports to describe a requirement.

If you are using JBehave, just place the *.story files in the src/test/resources/storiesdirectory, again respecting a directory structure that corresponds to your requirement hierarchy. Thenarrative.txt files also work for JBehave requirements.

Progress is measured by the total number of passing, failing or pending acceptance criteria, either for the whole project (at the top level), or within a particular requirement as you drill down the requirements hierarchy. For the purposes of reporting, a requirement with no acceptance criteria is attributed an arbitrary number of “imaginary” pending acceptance criteria. Thucydides uses 4 per requirement by default, but you can override this value using the thucydides.estimated.tests.per.requirement system property.

Figure 3: For JBehave, everything goes under src/test/resources/stories.

Figure 3: For JBehave, everything goes under src/test/resources/stories.

Conclusion

BDD is an excellent approach for communicating with, and reporting back to, stakeholders. However, for accurate acceptance test reporting on real-world projects, you need to go beyond the story level, and cater for the whole requirements hierarchy. In particular, you need to not only report on tests that have been executed, but also allow for the tests that haven’t been written yet.

Thucydides puts these concepts into practice: using a simple directory-based convention, you can easily integrate your requirements hierarcy into your acceptance tests.

17
Jan

Thucydides Release 0.9.92 – Improved Requirements reporting, screenshot blurring and bug fixes


Release 0.9.92 is now available and contains the following enhancements:

Improved requirements reporting

The Requirements reporting tab has been enhanced to show test results at a requirements level. This provides an overall view of the progress of the project by making it easier to view not only what stories have been implemented, but also features and capabilities that remain to be done. Here’s a screenshot.

Requirements Reporting tab

Requirements Reporting tab

You can read more about Thucydides’ approach to requirements reporting in this excellent blog post – Functional Test Coverage – Taking BDD Reporting To The Next Level

Blurring screenshots (Thucydides-116)

For security/privacy reasons, it may be required to blur sensitive screenshots in Thucydides reports. This is now possible by annotating the test methods or steps with the annotation @BlurScreenshots. When defined on a test, all screenshots for that test will be blurred. When defined on a step, only the screenshot for that step will be blurred. @BlurredScreenshot takes a string parameter with values LIGHT, MEDIUM or HEAVY to indicate the amount of blurring. For example,

 @Test
 @BlurScreenshots("HEAVY")
 public void looking_up_the_definition_of_pineapple_should_display_the_corresponding_article() {
     endUser.is_the_home_page();
     endUser.looks_for("pineapple");
     endUser.should_see_definition_containing_words("A thorny fruit");
 }

This feature is currently available for junit stories but we’ll extend it shortly to jbehave and easyb.

Other enhancements and bug fixes

  • Uninstantiated page objects in step libraries are now automatically instantiated.
  • Fixed a bug that caused tests to wait unnecessarily in subsequent steps after a step had failed.
  • A custom requirements tag provider, if specified, will be given preference over the default file system based requirements provider.

Keep watching this space for more updates and announcements.

4
Jan

Thucydides Release 0.9.90 – Improved reporting format for jBehave data-driven tests


Improved jBhave examples reporting

Our latest release (0.9.90) introduces a more user friendly report format for data-driven (example-driven) jBehave tests.

The following screenshots will explain the changes in detail.

All examples are displayed in a table.

All examples are displayed in a table on top.


The top level step shows example data for easy reference

The top level step shows example data for easy reference.


Error Steps are highlighted in red for easy identification.

Error Steps are highlighted in red for easy identification.


Currently, this feature is available only for jBehave tests but will soon be integrated with jUnit and easyb tests as well.

Support for iPhone and Android Selenium web drivers

Basic support for iPhone and Android web drivers is now available in Thucydides. Set the property webdriver.driver to ‘iPhone’ or ‘Android’ when running your tests.

Bug Fixes

  • Thucydides-117: Fixed a regression defect that had caused thucydides.take.screenshots to break in the previous release.

29
Feb

New features in Thucydides 0.7.10


The latest version of Thucydides (0.7.10) to be released has some great new features! Find out more below:

Flag a test as ‘pending’ or ‘ignored’ from within a test step/page object
This is useful if a test can’t be completed for reasons outside the scope of the test, which occasionally happens with acceptance tests. To do this, just call ‘Thucydides.pendingStep()’ or ‘Thucydides.ignoredStep()’, as shown here:

public String findFirstPartiallyPublishedBookingId() {
    String partiallyPublishedBookingId = null;
    try {
        partiallyPublishedBookingId = getFirstBookingIDWithStatus("Partially Published");
    } catch (NoMatchingBookingFoundError e) {
        Thucydides.pendingStep("No partially published booking available");
    }
    return partiallyPublishedBookingId;
}

Both methods take a message as an argument, which appears in the final report, on the screenshot corresponding to the step where this occured. Any steps following a pending step will be flagged as ‘ignored’. Any steps following a step marked as ‘ignored’ in this way will be executed normally.

Improved reporting
The reports in Thucydides 0.7.10 come with a brand new dashboard which provides you with a quick overview of the state of your tests, as well as many other improvements:

New, improved dashboard

Proxy support
If you need to use a proxy to access the web site you are testing, you can now do this by providing the system properties thucydides.proxy.http and thucydides.proxy.http_port. Note: This feature is currently supported for Firefox only.

Tell us what you think of the changes…

25
Dec

Faster Web Tests with Parallel Batches in Thucydides


Web tests are as a rule much slower than other types of tests, but they can be sped up significantly by running them in parallel. However, this is often harder to implement than it sounds. The latest version of Thucydides (version 0.6.0) comes with support for running parallel test batches, making this task much easier.

Web tests make good candidates for concurrent testing, in theory at least, but the implementation can be tricky. For example, although it is easy enough to configure both JUnit and easyb to run tests in parallel, running several webdriver instances of Firefox in parallel on the same display, for example, tends to become unreliable.

The natural solution in this case is to split the web tests into smaller batches, and to run each batch on a different machine and/or on a different virtual display. When each batch has finished, the results can be retrieved and aggregated into the final test reports.

However splitting tests into batches by hand tends to be tedious and unreliable – it is easy to forget to add a new test to a batch, for example, or have unevenly-distributed batches.

The latest version of Thucydides lets you do this automatically, by splitting your test cases evenly into batches of a given size. In practice, you run a build job for each batch. You need to specify two parameters when you run each build: the total number of batches being run (thucydides.batch.count), and the number of the batch being run in this build (thucydides.batch.number).

For example, the following will divide the test cases into 3 batches (“thucydides.batch.count”), and only run the first test in each batch (thucydides.batch.number):

mvn verify -Dthucydides.batch.count=3 -Dthucydides.batch.number=1

This will only work with the JUnit integration. However this feature is also now supported in easyb (as of easyb version 1.5), though using different parameters. When using the Thucydides easyb integration, you also need to provide the equivalent options for easyb:

mvn verify -Deasyb.batch.count=3 -Deasyb.batch.number=1

If you have both easyb and JUnit Thucydides tests, you will need to specify both options.

Parallel web tests on Jenkins

This approach is easy to set up on Jenkins using a multi-configuration build. In the following screenshot, we are running a multi-configuration build to run web tests across three batches. We use a single user-defined parameter (BATCH_NUMBER) to define the batch being run, passing this parameter into the Maven build job properties we discussed above.

The most robust way to aggregate the build results from the different batches is to set up a second build job that runs after the test executions, and retrieves the build results from the batch jobs. You can use the Jenkins Copy Artifacts plugin to do this. First, ensure that the multi-configuration build archives the Thucydides reports, as shown here:

This build will then trigger another, freestyle build job. This job needs to copy the Thucydides report artifacts from the matrix build jobs into the current workspace, and then run the mvn thucydides:aggregate command to generate the Thucydides aggregate reports. The matrix build job reports need to be copied one-by-one for each batch, as the current version of the Copy Artifacts plugin does not support copying from multiple projects in the same action.

Then make sure you publish the generated HTML reports (which will be in the target/site/thucydides directory) for easy access to the test results.

This simple example shows a parallel test running 3 batches – this brought the test execution time from 9 minutes to slightly over 1 minute. Results will vary, of course, but a typical real-world set of web tests would have a larger number of batches.

28
Sep

Thucydides Release 0.4.9 – improved issue tracking and JIRA integration


A new version of Thucydides – version 0.4.9 – is now available in the Maven repositories. This release provides support for the new Selenium 2.6 ExpectedConditions (see here for details), and also focuses on new features for integration with your issue tracking system. Read more »