Skip to content

Posts from the ‘Tutorials’ Category

9
Apr

Customizing Cucumber Feature File organization with Serenity


Many Cucumber projects organize their feature files in a single directory. This flat structure will work for very small projects, but in real-world project features typically need to be grouped by more high-level concepts, such as capabilities or epics. Serenity BDD makes this very easy
By default, Serenity uses a two-level hierarchy based on capabilities and features. A capability is represented by a directory, and the features associated with that capability are placed inside that directory.
3039EF3C-29B0-4158-811A-AB81ED1EC42A
If you use a directory structure like this, the “Requirements” tab in the Serenity reports will display a hierarchy of capabilities and features, and aggregate test results for each capability and feature:
3E31D88F-B5DA-48EF-84A2-52B6128A59EB
We can customize the hierarchy using the serenity.requirement.types system property. If we wanted epics instead of capabilities, we could add the following line to our serenity.properties file:
serenity.requirement.types = epic, feature
This would produce epics rather than capabilities in the requirements report:
DB980BC9-365A-48E3-BB31-24CC5903C62D
Now suppose you wanted to add a “theme” level above the epics and features. You could organise the features directory like this:
0AC724A3-4EF6-42DB-81D4-890FF6E53DBE
Then you would configure the serenity.requirement.types property as shown here:
serenity.requirement.types = theme, epic, feature
This would add a “Themes” requirements level above your epics in the Requirements reports.
You can also organize your features into releases, iterations, or sprints. You do this using the version tag. For example, the following feature would be scheduled for Release-2:
@version:Release-2
Feature: Search by keyword
  In order for buyers to find what they are looking for more efficiently
  As a seller
  I want buyers to be able to search for articles by keywords

  Scenario: Search for articles by keyword
    Given I want to buy a wool scarf
    When I search for 'wool'
    Then I should see only articles related to 'wool'
(Note that Cucumber does not like spaces in tags, so you need to write “Release-2” and not “Release 2”).
You can also assign a feature to both a release and a sprint, e.g.
@version:Release-1
@version:Sprint-1.1
Feature: Add item to shopping cart
  As a buyer
  I want to be able to purchase items online
  So that I can get them faster

  Scenario: Add item to cart
    Given I have searched for 'docking station'
    And I have selected item 2
    When I add it to the cart
    Then the item should appear in the cart
    And the shipping cost should be included in the total price
This will produce a “Releases” tab in the reports, similar to the following:
E9F98098-0217-4A73-8503-653A57EB0A49
You can tell Serenity what terms your release organisation uses (e.g. releases then sprints or releases then iterations) using the serenity.release.types system property. For example, if you wanted to use versions, then iterations, you would put the following line in your serenity.properties file:
serenity.release.types = Version,Iteration
Advertisements
20
Aug

Handling work-in-progress with Thucydides and JBehave using @pending and @wip tags


Thucydides version 0.9.268 has just been released, with a few very interesting new features. Thucydides is an open source reporting library that helps you write more effective BDD-style automated acceptance criteria, and generate richer test reports, requirements reports and living documentation. In this article, we will look at some of the new ways this version lets you handle work-in-progress or pending scenarios with Thucydides and JBehave. Read more »

14
Jul

BDD Requirements Management with JBehave, Thucydides and JIRA – Part 2


Thucydides is an open source library designed to make practicing Behaviour Driven Development easier. Thucydides plays nicely with BDD tools such as JBehave, or even more traditional tools like JUnit, to make writing automated acceptance tests easier, and to provide richer and more useful living documentation. In this series of articles, we look at the tight one and two-way integration that Thucydides offers with JIRA. The first article discussed basic one-way integration with JIRA. In this article, we will looking at taking that integration further. We will see how to insert links to the Thucydides reports into JIRA, how to update the state of JIRA issues based on the Thucydides test outcomes, and how to report on JIRA versions and releases in the Thucydides reports.

The rest of this article assumes you have some familiarily with Thucydides. For a tutorial introduction to Thucydides, check out the Thucydides Documentation or this article for a quick introduction.

The simplest form of two-way integration between Thucydides and JIRA is to get Thucydides to insert a comment containing links to the Thucydides test reports for each related issue card. To get this to work, you need to tell Thucydides where the reports live. One way to do this is to add a property calledthucydides.public.url to your thucydides.properties file with the address of the thucydides reports.

thucydides.public.url=http://buildserver.myorg.com/latest/thucydides/report

This will tell Thucydides that you not only want links from the Thucydides reports to JIRA, but you also want to include links in the JIRA cards back to the corresponding Thucydides reports. When this property is defined, Thucydides will add a comment like the following to any issues associated with the executed tests:

images/jira-thucydides-comment.png

The thucydides.public.url will typically point to a local web server where you deploy your reports, or to a path within your CI server. For example you could publish the Thucydides reports on Jenkins using theJenkins HTML Publisher Plugin, and then add a line like the following to your thucydides.properties file:

thucydides.public.url=http://jenkins.myorg.com/job/myproject-acceptance-tests/Thucydides_Report/

If you do not want Thucydides to update the JIRA issues for a particular run (e.g. when running your tests locally), you can also set thucydides.skip.jira.updates to true, e.g.

thucydides.skip.jira.updates=true

This will simply write the relevant issue numbers to the log rather than trying to connect to JIRA.

Updating JIRA issue states

You can also configure the plugin to update the status of JIRA issues. This is deactivated by default: to use this option, you need to set the thucydides.jira.workflow.active option to true, e.g.

thucydides.jira.workflow.active=true

The default configuration will work with the default JIRA workflow: open or in progress issues associated with successful tests will be resolved, and closed or resolved issues associated with failing tests will be reopened. If you are using a customized workflow, or want to modify the way the transitions work, you can write your own workflow configuration. Workflow configuration uses a simple Groovy DSL. The following is an example of the configuration file used for the default workflow:

    when 'Open', {
        'success' should: 'Resolve Issue'
    }

    when 'Reopened', {
        'success' should: 'Resolve Issue'
    }

    when 'Resolved', {
        'failure' should: 'Reopen Issue'
    }

    when 'In Progress', {
        'success' should: ['Stop Progress','Resolve Issue']
    }

    when 'Closed', {
        'failure' should: 'Reopen Issue'
    }

You can write your own configuration file and place it on the classpath of your test project (e.g. in theresources directory). Then you can override the default configuration by using thethucydides.jira.workflow property, e.g.

thucydides.jira.workflow=my-workflow.groovy

Alternatively, you can simply create a file called jira-workflow.groovy and place it somewhere on your classpath (e.g. in the src/test/resources directory). Thucydides will then use this workflow. In both these cases, you don’t need to explicitly set the thucydides.jira.workflow.active property.

Release management

In JIRA, you can organize your project releases into versions, as illustrated here:

images/jira-versions.png

You can and assign cards to one or more versions using the Fix Version/s field:

images/jira-fix-versions.png

By default, Thucydides will read version details from the Releases in JIRA. Test outcomes will be associated with a particular version using the “Fixed versions” field. The Releases tab gives you a run-down of the different planned versions, and how well they have been tested so far:

images/releases-tab.png

JIRA uses a flat version structure – you can’t have for example releases that are made up of a number of sprints. Thucydides lets you organize these in a hierarchical structure based on a simple naming convention. By default, Thucydides uses “release” as the highest level release, and either “iteration” or “sprint” as the second level. For example, suppose you have the the following list of versions in JIRA – Release 1 – Iteration 1.1 – Iteration 1.2 – Release 2 – Release 3

This will produce Release reports for Release 1, Release 2, and Release 3, with Iteration 1.2 and Iteration 1.2 appearing underneath Release 1. The reports will contain the list of requirements and test outcomes associated with each release. You can drill down into any of the releases to see details about that particular release

images/releases.png

You can also customize the names of the types of release usinge the thucydides.release.typesproperty, e.g.

thucydides.release.types=milestone, release, version

Conclusion

Thucydides has powerful one and two-way integration with JIRA. In these articles, we have seen how you can incoporate links from Thucydides to JIRA, from JIRA to Thucyides, and even update the status of issues in JIRA based on the test results. And, if you are managing your versions in JIRA, you can also report on how well each version has been tested, and what remains to be tested before the next release.

Want to learn more? Be sure to check out the Thucydides web site, the Thucydides Blog, or join theThucydides Google Users Group to join the discussion with other Thucydides users.

Wakaleo Consulting, the company behind Thucydides, also runs regular courses in Australia, London and Europe on related topics such as Agile Requirements GatheringBehaviour Driven DevelopmentTest Driven Development, and Automated Acceptance Testing.

10
Jul

BDD Requirements Management with JBehave, Thucydides and JIRA – Part 1


Thucydides is an open source library designed to make practicing Behaviour Driven Development easier. Thucydides plays nicely with BDD tools such as JBehave, or even more traditional tools like JUnit, to make writing automated acceptance tests easier, and to provide richer and more useful living documentation. In a series of two articles, we will look at the tight one and two-way integration that Thucydides offers with JIRA.

The rest of this article assumes you have some familiarily with Thucydides. For a tutorial introduction to Thucydides, check out the Thucydides Documentation or this article for a quick introduction.

Getting started with Thucydides/JIRA integration

Atlassian JIRA

JIRA is a popular issue tracking system that is also often used for Agile project and requirements management. Many teams using JIRA store their requirements electronically in the form of story cards and epics in JIRA

Suppose we are implementing a Frequent Flyer application for an airline. The idea is that travellers will earn points when they fly with our airline, based on the distance they fly. Travellers start out with a “Bronze” status, and can earn a better status by flying more frequently. Travellers with a higher frequent flyer status benefit from advantages such as lounge access, prioritized boarding, and so on. One of the story cards for this feature might look like the following:

images/jira-story.png

This story contains a description following one of the frequently-used formats for user story descriptions (“as a..I want..so that”). It also contains a custom “Acceptance Criteria” field, where we can write down a brief outline of the “definition of done” for this story.

These stories can be grouped into epics, and placed into sprints for project planning, as illustrated in the JIRA Agile board shown here:

images/jira-agile.png

As illustrated in the story card, each of these stories has a set of acceptance criteria, which we can build into more detailed scenarios, based on concrete examples. We can then automate these scenarios using a BDD tool like JBehave.

The story in Figure 1 describes how many points members need to earn to be awarded each status level. A JBehave scenario for the story card illustrated earlier might look like this:

Frequent Flyer status is calculated based on points

Meta:
@issue FH-17

Scenario: New members should start out as Bronze members
Given Jill Smith is not a Frequent Flyer member
When she registers on the Frequent Flyer program
Then she should have a status of Bronze

Scenario: Members should get status updates based on status points earned
Given a member has a status of <initialStatus>
And he has <initialStatusPoints> status points
When he earns <extraPoints> extra status points
Then he should have a status of <finalStatus>
Examples:
| initialStatus | initialStatusPoints | extraPoints | finalStatus | notes                    |
| Bronze        | 0                   | 300         | Silver      | 300 points for Silver    |
| Silver        | 0                   | 700         | Gold        | 700 points for Gold      |
| Gold          | 0                   | 1500        | Platinum    | 1500 points for Platinum |

Thucydides lets you associate JBehave stories or JUnit tests with a JIRA card using the @issue meta tag (illustrated above), or the equivalent @Issue annotation in JUnit. At the most basic level, this will generate links back to the corresponding JIRA cards in your test reports, as illustrated here:

images/jira-test-report.png

For this to work, Thucydides needs to know where your JIRA server. The simplest way to do this is to define the following properties in a file called thucydides.properties in your project root directory:

jira.url=https://myserver.atlassian.net
jira.project=FH
jira.username=jirauser
jira.password=t0psecret

You can also set these properties up in your Maven pom.xml file or pass them in as system properties.

Thucydides also supports two-way integration with JIRA. You can also get Thucydides to update the JIRA issue with a comment pointing to the corresponding test result.

Feature Coverage

But test results only report part of the picture. If you are using JIRA to store your stories and epics, you can use these to keep track of progress. But how do you know what automated acceptance tests have been implemented for your stories and epics, and, equally importantly, how do you know which stories or epics have no automated acceptance tests? In agile terms, a story cannot be declared “done” until the automated acceptance tests pass. Furthermore, we need to be confident not only that the tests exist, but they test the right requirements, and that they test them sufficiently well.

We call this idea of measuring the number (and quality) of the acceptance tests for each of the features we want to build “feature coverage”. Thucydides can provide feature coverage reporting in addition to the more conventional test results. If you are using JIRA, you will need to add thucydides-jira-requirements-provider to the dependencies section of your pom.xml file:

        <dependencies>
            ...
            <dependency>
                <groupId>net.thucydides.plugins.jira</groupId>
                <artifactId>thucydides-jira-requirements-provider</artifactId>
                <version>0.9.262</version>
            </dependency>
        </dependencies>

(The actual version number might be different for you – always take a look at Maven Central to know what the latest version is).

You will also need to add this dependency to the Thucydides reporting plugin configuration:

        <build>
            ...
            <plugins>
                ...
                <plugin>
                    <groupId>net.thucydides.maven.plugins</groupId>
                    <artifactId>maven-thucydides-plugin</artifactId>
                    <version>0.9.262</version>
                    <executions>
                        <execution>
                            <id>thucydides-reports</id>
                            <phase>post-integration-test</phase>
                            <goals>
                                <goal>aggregate</goal>
                            </goals>
                        </execution>
                    </executions>
                    <dependencies>
                        <dependency>
                            <groupId>net.thucydides.plugins.jira</groupId>
                            <artifactId>thucydides-jira-requirements-provider</artifactId>
                            <version>0.9.262</version>
                        </dependency>
                    </dependencies>
                </plugin>
            </plugins>
        </build>

Now, when you run the tests, Thucydides will query JIRA to determine the epics and stories that you have defined, and list them in the Requirements page. This page gives you an overview of how many requirements (epics and stories) have passing tests (green), how many have failing (red) or broken (orange) tests, and how many have no tests at all (blue):

images/requirements-view.png

If you click on an epic, you can see the stories defined for the epic, including an indicator (in the “Coverage” column) of how well each story has been tested.

images/epic-details.png

From here, you may want to drill down into the details about a given story, including what acceptance tests have been defined for this story, and whether they ran successfully:

images/story-report.png

Both JIRA and the JIRA-Thucydides integration are quite flexible. We saw earlier that we had configured a custom “Acceptance Criteria” field in our JIRA stories. We have displayed this custom field in the report shown above by including it in the thucydides.properties file, like this:

jira.custom.field.1=Acceptance Criteria

Thuydides reads the narrative text appearing in this report (“As a frequent flyer…”) from the Descriptionfield of the corresponding JIRA card. We can override this behavior and get Thucydides to read this value from a different custom field using the jira.custom.narrative.field property. For example, some teams use a custom field called “User Story” to store the narrative text, instead of the Description field. We could get Thucydides to use this field as follows:

jira.custom.narrative.field=User Story

Conclusion

Thucydides has rich and flexible one and two-way integration with JIRA. Not only can you link back to JIRA story cards from your acceptance test reports and display information about stories from JIRA in the test reports, you can also read the requirements structure from JIRA, and report on which features have been tested, and which have not.

In the next article in this series, we will learn how to insert links to the Thucydides reports into the JIRA issues, and how to actively update the state of the JIRA cards based on the outcomes of your tests.

Want to learn more? Be sure to check out the Thucydides web site, the Thucydides Blog, or join theThucydides Google Users Group to join the discussion with other Thucydides users.

Wakaleo Consulting, the company behind Thucydides, also runs regular courses in Australia, London and Europe on related topics such as Agile Requirements GatheringBehaviour Driven DevelopmentTest Driven Development, and Automated Acceptance Testing.

20
Jan

Functional Test Coverage – taking BDD reporting to the next level


Conventional test reports, generated by tools such as JUnit or TestNG, naturally focus on what tests have been executed, and whether they passed or failed. While this is certainly useful from a testing perspective, these reports are far from telling the whole picture.

BDD reporting tools like Cucumber and JBehave take things a step further, introducing the concept of “pending” tests. A pending test is one that has been specified (for example, as an acceptance criteria for a user story), but which has not been implemented yet.

In BDD, we describe the expected behaviour of our application using concrete examples, that eventually form the basis of the “acceptance criteria” for the user stories we are implementing. BDD tools such as Cucumber and JBehave not only report on test results: they also report on the user stories that these tests validate.

However this reporting is still limited for large projects, where the numbers of user stories can become unwieldy. User stories are not created in isolation: rather, user stories help describe features, which support capabilities that need to be implemented to achieve the business goals of the application. So it makes sense to be able to report on test results not only at the user story level, but also at higher levels, for example in terms of features and capabilities. This makes it easier to report on not only what stories have been implemented, but also what features and capabilities remain to be done. An example of such a report is shown in Figure 1 (or see the full report here).

new-reporting

Figure 1: A test coverage report listing both tested and untested requirements.

In agile projects, it is generally considered that a user story is not complete until all of its automated acceptance tests pass. Similarly, a feature cannot be considered ready to deliver until all of the acceptance criteria for the underlying user stories have been specified and implemented. However, sensible teams shy away from trying to define all of the acceptance criteria up-front, leaving this until the “last responsible moment”, often shortly before the user story is scheduled to be implemented. For this reason, reports that relate project progress and status only in terms of test results are missing out on the big picture.

To get a more accurate idea of what features have been delivered, which ones are in progress, and what work remains to be done, we must think not in terms of test results, but in terms of the requirements as we currently understand them, matching the currently implemented tests to these requirements, but also pointing out what requirements currently have no acceptance criteria defined. And when graphs and reports illustrate how much progress has been made, the requirements with no acceptance criteria must also be part of the picture.

Requirements-level BDD reporting with Thucydides

Thucydides puts some of these concepts into practice. Thucydides reports not only on how the tests did, but also fits them into the broader picture, showing what requirements have been tested and, just as importantly, what requirements haven’t.

During the rest of this article, we will see how to report on both your requirements and your test results using a very simple directory-based approach. You can follow along with this example by cloning the Github project at https://github.com/thucydides-webtests/thucydides-simple-demo

Simple requirements in Thucydides – a directory-based approach

Thucydides can integrate with many different requirement management systems, and it is easy to write your own plugin to tailor the integration to suite your particular environment. A popular approach, for example, is to store requirements in JIRA and to use Thucydides to read the requirements hierarcy directly from the JIRA cards. However the simplest approach, which uses a directory-based approach, is probably the easiest to use to get started, and it is that approach that we will be looking at here.

Requirements can usually be organized in a hierarchial structure. By default, Thucydides uses a three-level hierarchy of requirements. At the top level, capabilities represent a high-level capacity that the application must provide to meet the application’s business goals. At the next level down, features help deliver these capabilities. To make implementation easier, a feature can be broken up into user stories, each of which in turn can contain a number of acceptance criteria.

Figure 2: JUnit test directories mirror the requirements hierarchy.

Figure 2: JUnit test directories mirror the requirements hierarchy.

Of course, you don’t have to use this structure if it doesn’t suit you. You can override the thucydides.capability.types system property to provide your own hierarchy. For example, if you wanted a hierarchy with modules,epics, and features, you would just set thucydides.capability.types to “module,epic,feature”.

When we use the default directory-based requirements strategy in Thucydides, the requirements are stored in a hierarchial directory structure that matches the requirements hierarchy. At the lowest level, a user story is represented by a JBehave *.story file, an easyb story, or a JUnit test. All of the other requirements are represented as directories (see Figure 2 for an example of such a structure).

In each requirements directory, you can optionally place a file called narrative.txt, which contains a free-text summary of the requirement. This will appear in the reports, with the first line appearing as the requirement title. A typical narrative text is illustrated in the following example:

Learn the meaning of a word
In order to learn the meaning of a word that I don't know
As an online reader
I want to be able to find out the meaning of the word

If you are implementing the acceptance criteria as JUnit tests, just place the JUnit tests in the package that matches the correspoinding requirement. You need to use the thucydides.test.root system property to specify the root package of your requirements. For the example in Figure 2, this value should be set to nz.govt.nzqa.lssu.stories.

Figure 3: The narrative.txt file appears in the reports to describe a requirement.

Figure 3: The narrative.txt file appears in the reports to describe a requirement.

If you are using JBehave, just place the *.story files in the src/test/resources/storiesdirectory, again respecting a directory structure that corresponds to your requirement hierarchy. Thenarrative.txt files also work for JBehave requirements.

Progress is measured by the total number of passing, failing or pending acceptance criteria, either for the whole project (at the top level), or within a particular requirement as you drill down the requirements hierarchy. For the purposes of reporting, a requirement with no acceptance criteria is attributed an arbitrary number of “imaginary” pending acceptance criteria. Thucydides uses 4 per requirement by default, but you can override this value using the thucydides.estimated.tests.per.requirement system property.

Figure 3: For JBehave, everything goes under src/test/resources/stories.

Figure 3: For JBehave, everything goes under src/test/resources/stories.

Conclusion

BDD is an excellent approach for communicating with, and reporting back to, stakeholders. However, for accurate acceptance test reporting on real-world projects, you need to go beyond the story level, and cater for the whole requirements hierarchy. In particular, you need to not only report on tests that have been executed, but also allow for the tests that haven’t been written yet.

Thucydides puts these concepts into practice: using a simple directory-based convention, you can easily integrate your requirements hierarcy into your acceptance tests.

23
Sep

Introducing Thucydides


What is Thucydides

Thucydides is a tool that lets you use WebDriver-based unit or BDD tests to write more flexible and more reusable WebDriver-based tests, and also to generate documentation about your acceptance tests, including a narrative description of test, along with the corresponding screen shots, and also high-level summaries and aggregations of the test results. Read more »