Skip to content

July 27, 2015

8

Serenity 1.1 is out!

by jfsmart

A brand new version of Serenity is out, with bug features and some very cool new features, including fully-integrated feature coverage reporting and the ability to include both manual and automated tests in your reports!

The Serenity team is proud to announce the release of Serenity version 1.1 (the current exact version is 1.1.1, but as is custom with the Serenity project, we like to release in a regular stream of minor features and bug fixes, so check the latest release numbers on Bintray or Maven Central.)

This new release has several major new features, along with a number of bug fixes and improvements. Major new features include:

  • Smooth integration between test reporting and requirements reporting (living documentation)
  • You can now flag tests (in JUnit, Cucumber or JBehave) as manual (more on this further down)

Fully-integrated Requirements Reporting

Serenity is an automated testing and reporting library that tries to implement many key concepts from the world of Behavior Driven Development (BDD) such as Living Documentation and Feature Coverage. One of the main principles behind Serenity reporting is that the value of an automated test is directly related to the value of the feature it is testing. Automated tests are most useful when they demonstrate both that a feature works and that it is doing something valuable for the customer.

Serenity distinguishes between two distinct views of your test results. The first, the test reports, presents the results from the point of view of what tests were executed:

Test reports

These reports also give you an overview of the test results, in terms of the number of passing and failing tests:

Test reports

This representation is a classic list of test results that will be familiar to testers.

The second focuses less on what tests were executed, and more on what features were delivered. If the definition of done ifor the features you want to deliver is accurately described by the acceptance criteria (one of the cornerstones on BDD), and if you automate these acceptance criteria, then you can get a good idea of whether a feature has indeed been delivered from the results of the automated acceptance criteria.

For this to work, Serenity needs to know how your requirements are structured. A flat requirements structure is a poor representation for all but the most simple projects. Well-designed requirement structures help a reader understand what business goal each feature is helping to deliver, or what capability the feature is helping to provide. For this reason, teams often organize features by capability or in some other meaningful functional grouping.

Doing this in Serenity allows you to present a hierarchical view of the requirements, as illustrated here:

Requirements reports

The simplest way to represent a requirements structure in Serenity is to use a hierarchical directory structure, where each top level directory describes a high level capability or functional domain. You might break these directories down further into sub directories representing more detailed functional areas, or just place feature files directly in these directories:

Requirements directory structure using Cucumber

The same thing works for JUnit, except you use packages instead of directories:

Requirements directory structure using JUnit

To get the JUnit package structure to work, you need to set the serenity.test.root system property to the top level package containing your requirements:

serenity.test.root=net.thucydides.showcase.junit.features

When you organize your tests this way, Serenity will show you where each test belongs in the requirements hierarchy:

Test result showing parent requirements

These breadcrumbs let you go directly to the corresponding requirements pages, as illustrated here:

Test result showing parent requirements

Manual tests It is often useful to be able to flag certain tests as manual tests, which are not consider cost-efficient to automate, but which you would still like to see in the overall test reports.

You can mark a JUnit test as a manual one simply by using the @Manual annotation. For example, the following Serenity test class has two automated tests, and one flagged as manual:

@RunWith(SerenityRunner.class)
public class SearchByKeyword {

    @Managed
    WebDriver driver;

    @Steps
    BuyerSteps buyer;

    @Test
    public void search_for_articles_by_keyword() {
        buyer.opens_home_page();
        buyer.searches_by_keyword("wool");
        buyer.should_see_results_summary_containing("wool");
    }

    @Test
    public void search_for_articles_by_shop_name() {
        buyer.opens_home_page();
        buyer.searches_for_shop_called("docksmith");
        buyer.should_see_shop_search_result_summary_of("1 shop found for docksmith");
    }

    @Test
    @Manual
    public void should_respect_standard_look_and_feel() {}

}

In Cucumber, just use the @manual tag on a scenario:

@manual
Scenario: Should respect standard look and feel

You can add the usual Given/When/Then steps if you want some instructions about how to test this scenario in the living documentation, or leave it simply as a place-holder for the tester later on.

And in JBehave, use the @manual tag in the Meta: section of a scenario:

Scenario: Display social media links for a product
Meta:
@manual

Given I have searched for 'Docking station' in my region
When I select item 1
Then I should see social media links

In all cases, the manual test appear in the reports as a special kind of “Pending” test, indicated by a special icon:

Manual tests

Manual tests also have their on tag, making it easy to get a view of all of the manual tests in one place.

Bug fixed and enhancements

There are also a few important bug fixes and enhancements, including:

  • Fixed a thread leak that sometimes caused problems on build servers for large projects.
  • Move the caption for the screenshots to the top of the screen for better readability.
  • Added the deep.step.execution.after.failures system property. This allows you to decide whether @Step methods should simply be skipped after a previous step has failed (the default: this is faster, but it means that only top-level steps will be reported), or if the subsequent steps will be executed in “dry-run” mode (will report on nested steps as well as the top level ones)
  • Upgrades to Appium 3.1.0
  • Improved error and exception reporting for RestAssured tests.

Coming up in the not-so-distant future will be deep JIRA integration, including integration with Zephyr. Older versions of Thucydides supported this integration using the old SOAP API for JIRA and a beta version of the Zephyr API. This has now been completely rewritten using the latest REST APIs for both tools.

Advertisements
Read more from Tips and Tricks
8 Comments Post a comment
  1. Jul 27 2015

    I am getting below error when I updated pom.xml to use Serenity and Serenity-Cucumber versions as 1.1.1

    Error: The POM for net.serenity-bdd:core:jar:1.1.1 is missing, no dependency information available

    Also, I don’t see updated version of ‘Core’ in below link.
    http://mvnrepository.com/artifact/net.serenity-bdd

    Reply
  2. Nikolay
    Jul 29 2015

    Hi John,
    thanks for this Serenity update. I was trying to apply this to my project reports (i’m using only Junit framework) and got some questions:

    1) my project has following hierarchical directory structure
    +test
    ++java
    +++ThucTest
    ++++functionality
    +++++gallerypage
    ++++++GalleryPageSanity
    ++++++GalleryPageSmoke
    +++++homepage
    ++++++HomePageSmoke

    also i did following settings:

    serenity.test.root=ThucTest.functionality
    serenity.requirement.types=area,type

    However on the report main page under Related Tag (along with Areas and Types) i see

    Stories
    Gallery Page Sanity
    Gallery Page Smoke
    Home Page Smoke

    Why do I have “Stories” tag? I’m not using it anywhere

    2) Also for Types and Areas on the left there is a + sign which should expand section. However after clicking on it + changes to – and there is nothing under this section only a space. I expected something like nested test cases assigned for this tag

    Thank you in advance!

    Reply
    • Jul 30 2015

      Stories is probably the default for JUnit, I will check how to override it.
      But I would suggest that Sanity and Smoke would be better defined as tags, rather than subcategories, so that you can see all of the sanity tests and all of the smoke tests together in dedicated pages.

      Reply
      • Nikolay
        Jul 31 2015

        thank you, John. This makes sense, i’ll try to do as you suggested

  3. Aug 11 2015

    Hi,

    Can Serenity be used to create a framework using Appium for mobile web automation on android? I tried doing it but it does not create any reports? Does Serenity support android/ios automation?

    Reply
  4. Netra
    Dec 8 2015

    I have developed few test cases using Selenium, Junit & Serenity. It runs perfectly fine with default driver – Firefox. But when I tried run them in IE

    I get message saying “unsupported driver ” for each test.

    Could you please help me by give instruction to run these tests in IE.
    I tried all possible instructions at several websites, none of them worked for me

    Thank You
    Netra

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments

%d bloggers like this: