Skip to content

Archive for

12
Mar

It ain’t just reds and greens: Automated Acceptance Testing and quaternary test outcomes


Although they seem simple enough on the surface, test outcomes are actually quite complicated beasts. Traditional unit tests, and basic TDD tests, have just two states, passing or failing, represented by red and green in the famous “RED-GREEN-REFACTOR” dicton. In Behaviour Driven Development (BDD), on the other hand, we have the additional concept of ‘pending’ tests: tests that have been specified (for example, in a Cucumber or JBehave story) but not yet implemented. When we report on test results, we need to be able to distinguish these three states, as a pending test has very different semantics to a failing test. Pending means it’s not yet done yet, but this may well be as expected, especially towards the start of a sprint. A failing test, on the other hand, needs fixing. Now.

Most BDD tools, such as Cucumber, JBehave, Concordion, easyb and so forth, report test results in terms of these three states. However, the complexity doesn’t stop here. Maintaining web tests, for example, requires ongoing effort, and can perturb the test reporting if not handed with care. For example, if a web page changes during normal development or refactoring work, the tests that use this page may break. Although good software engineering practices such as the use of Page Objects can reduce the risk of this quite a bit, and reduce the work involved in maintaining the tests when it does, it is still something that will happen regularly. And again, the semantics of a test that is broken is quite different to those of a failing test. A broken test needs maintenance work on the test suite. It may also mask an application error, but you will need to investigate to find out. A failing test means that the application is broken, and therefore needs urgent fixing.

In an attempt to address this limitation in conventional BDD reporting, Thucydides now distinguishes between test failures (triggered by an assertion error) from test errors (triggered by any other exception). When you run your automated acceptance tests using Thucydides, any error that triggers an AssertionError (or a subclass of AssertionError) will be considered a test failure. Anything else (such as the NotFoundException, when an element is not found on the page) is considered to be an error, and therefore indicative of a broken test.

In the future we may extend Thucydides further to make this concept more configurable: for example, so that users can provide exceptions that should be considered as either an error or a test failure, or even adding additional outcome states (e.g infrastructure failure, database not setup, etc.).

9
Mar

Thucydides Release 0.9.103 – Improved reporting, other enhancements and bug fixes


Thucydides Release 0.9.103 adds some useful new features and enhancements.

Reports distinguish between errors and failures

Test reports now distinguish between test errors and failures. The following screenshot will make this clear.

Reports distinguish between Test Errors and Failures

Reports distinguish between Test Errors and Failures

Detailed report will not show screenshots column if there are no screenshots in the test

Tests that do not have any screenshots will no longer show an empty screenshots column in the details page.

no-screenshots-column

Injecting WebElementFacades in Page objects

Instead of declaring WebElement variables in Page Objects and then calling element() or $() to wrap them in WebElementFacades, you can now declare WebElementFacade variables directly inside the Page Objects. This will make the Page Object code simpler more readable.

So, instead of writing


@DefaultUrl("http://en.wiktionary.org/wiki/Wiktionary:Main_Page")
public class DictionaryPage extends PageObject {

    @FindBy(name="search")
    private WebElement searchTerms;

    private WebElement go;    //variable name matches element id or name

    public DictionaryPage(WebDriver driver) {
        super(driver);
    }

    public void enter_keywords(String keyword) {
        element(searchTerms).type(keyword);
    }

    public void lookup_terms() {
        element(go).click();
    }

    public List getDefinitions() {
        WebElement definitionList = getDriver().findElement(By.tagName("ol"));
        List<WebElement> results = definitionList.findElements(By.tagName("li"));
        return convert(results, toStrings());
    }

    private Converter<WebElement, String> toStrings() {
        return new Converter<WebElement, String>() {
            public String convert(WebElement from) {
                return from.getText();
            }
        };
    }
}

you can write

@DefaultUrl("http://en.wiktionary.org/wiki/Wiktionary:Main_Page")
public class DictionaryPage extends PageObject {

    @FindBy(name="search")
    private WebElementFacade searchTerms;

    private WebElementFacade go;       //variable name matches element id or name

    public DictionaryPage(WebDriver driver) {
        super(driver);
    }

    public void enter_keywords(String keyword) {
        searchTerms.type(keyword);  //directly use facade
    }

    public void lookup_terms() {
        go.click();                //directly use facade
    }

    public List getDefinitions() {
        WebElement definitionList = getDriver().findElement(By.tagName("ol"));
        List<WebElement> results = definitionList.findElements(By.tagName("li"));
        return convert(results, toStrings());
    }

    private Converter<WebElement, String> toStrings() {
        return new Converter WebElement, String>() {
            public String convert(WebElement from) {
                return from.getText();
            }
        };
    }
}

Switching to another page object

A new method, switchToPage() has been added to PageObject class to make it convenient to return a new PageObject after navigation from within a method of a PageObject class. For example,

@DefaultUrl("http://mail.acme.com/login.html")
public class EmailLoginPage extends PageObject {

    ...
    public void forgotPassword() {
        ...
        forgotPassword.click();
        ForgotPasswordPage forgotPasswordPage = this.switchToPage(ForgotPasswordPage.class);
        forgotPasswordPage.open();
        ...
    }
    ...
}

Other minor enhancements and bug fixes

  • Selenium version updated to 2.31.0.
  • Added containsOnlyText and shouldContainOnlyText methods to WebElementFacade. These methods are similar to containsText/shouldContainText methods but check for exact match.
  • Modified shouldContainText method in WebElementFacade so that error message shows both the expected text, and the text found in the element.
  • Fixed a bug in the error message of the method WebElementFacade.shouldNotContainText.
  • Now you can both activate and deactivate native events for Firefox by setting the thucydides.native.events property to true or false.
  • Thucydides-141: Fixed a bug due to which when trying to open a page with slash in the end of url, browser tried to open page without slash.