Fakultas Ilmu Komputer UI

Skip to content
Snippets Groups Projects
Daya Adianto's avatar
Daya Adianto authored
Create tutorial instructions about BDD

Closes #10

See merge request !8
57e1bd25

Sitodo (PMPL Variant)

pipeline status coverage report

A basic todo app project for teaching basic Web programming, Git workflows, and CI/CD. Heavily inspired by the running example in "Test-Driven Development with Python" book by Harry Percival.

Note: The project has been customised from the upstream in order to be used in PMPL (SQA) course at the Faculty of Computer Science Universitas Indonesia.

Table of Contents

  1. Getting Started
    1. Setting Up Development Environment
    2. Build & Run
    3. Running Example
  2. Tutorial: Writing Behavior-Driven Development (BDD) Test Suite
    1. Executable Specifications
    2. Glue Code
    3. Execute Test Suite & Reporting
  3. Tasks
    1. Mandatory Tasks
    2. Additional Tasks
  4. License

Getting Started

Setting Up Development Environment

The following tools need to be installed in order to build and run the project:

Ensure java, javac, and mvn commands can be invoked from inside the shell:

$ java --version
$ javac --version
$ mvn --version

The output should display the versions of java, javac, and mvn respectively.

We recommend IntelliJ IDEA Community Edition as the IDE for developing the project. Other IDE or text editors, such as Eclipse and Visual Studio Code, might work. However, we may not be able to help troubleshoot any IDE-related issues. In addition, we include IntelliJ-specific run configurations in the codebase that will add shortcuts for running the test suites and coverage reporting from within IntelliJ.

Build & Run

To run the entire test suite, execute:

mvn test

To run a select test suite, e.g. unit or functional test, add -Dgroups parameter. For example, to run only the unit test suite, execute mvn test -Dgroups=unit. Similarly, to run only the functional test suite, execute mvn test -Dgroups=e2e.

To build an executable Spring Boot application, execute:

mvn package -DskipTests

The -DskipTests option lets package task to build the app into executable JAR file without running all test suites. If the option was omitted, then all test suites will run, thus increasing the duration of the building process, especially the functional test suite that runs much longer than the unit test suite.

The JAR file will be generated at ./target directory. To run it, execute:

java -jar sitodo.jar

You can customise the configuration by providing an application.properties file in the same directory as the executable JAR file. See the built-in configuration in the source code.

Additionally, you can also set the configuration during runtime using environment variables. Following Spring Boot convention, properties are named in all uppercase and dot separators are replaced with underscores. For instance, spring.datasource.url becomes SPRING_DATASOURCE_URL when configured using an environment variable. See the example in the GitLab CI/CD configuration, specifically in the job for running tests.

Running Example

See the running example based on the upstream's main branch at Fly.io.

Tutorial: Writing Behavior-Driven Development (BDD) Test Suite

Previously, the lecture session in the class has explained the motivation and the basics of BDD. This tutorial will cover the practical aspects by outlining the basic steps of writing BDD test suite. We will see the process of writing the executable specifications and the "glue" code, followed by running the test suite and generating the test report. Finally, we will do an exercise to write new executable specifications and the corresponding "glue" code, while maintaining the correctness and the code coverage.

Executable Specifications

The executable specifications in BDD follow a structured outline using Gherkin syntax. In this example, we follow the Given-When-Then format. Each section serves a specific purpose:

  • Given is used to set up the pre-conditions or the initial context for the test scenario
  • When is used to define the event or action that we want to simulate
  • Then is used to verify the expected outcome of the performed action

For example, look at the following executable specification for "Add new todo" feature:

Feature: Add items into the todo list

  Scenario: Add single item into the list
    Given Alice is looking at the TODO list
    When she adds "Touch grass" to the list
    Then she sees "Touch grass" as an item in the TODO list

  Scenario: Add multiple items into the list
    Given Alice is looking at the TODO list
    When she adds "Buy bread" to the list
    And she adds "Buy candy" to the list
    Then she sees "Buy bread" as an item in the TODO list
    And she sees "Buy candy" as an item in the TODO list

The executable specification above describes two possible user flows when they are adding items to the todo list. The first user flow is to add a single item into the list, and the second user flow is to add multiple items into the list.

One of the advantages of following this format is that the test scenario can be easily written by non-technical roles. They can focus on writing the test scenario without worrying about the implementation details.

Eventually, there needs some way to "glue" the executable specifications into the actual test code. That is, the executable specifications that is written in Gherkin syntax drives the test procedures in the test code. To achieve that, we will write the "glue" code and apply two patterns relevant to our need: Screenplay & Page Object Model (POM).

Note: There also exists the equivalent Gherkin syntax in other languages such as Bahasa Indonesia. For example, Given, When, and Then are equals to Jika, Ketika, and Maka, respectively. See the Localization package on Cucumber website for more information. It may be useful if you want to write the executable specifications in a natural language other than English.

Glue Code

Remember that while the executable specifications can be written by non-technical roles, the corresponding "glue" code needs to be written by developers or technical QA personnel. We will explore how we can bridge the gap between the executable specifications and the automated test procedures.

First, we need to decide at what level the test procedures shall be done. BDD can be used to drive the unit testing procedures. That is, the test script in the unit test suite is written to satisfy the executable specifications. However, recall that the unit tests, by design, exercise the code at the unit level. There might be cases where interaction between units (e.g., shared states or interdependency between modules) are not covered, which are something that can be done if we run a test suite against the actual deployed and running system. Therefore, we will use BDD at the level of functional (or, end-to-end) test in this project.

Before we can begin in writing the "glue" code in BDD, let us see an example of a functional test written using Selenium and Selenide:

/**
 * Simplified, non-refactored version of a test case in AddTodoItemTest.java functional test suite.
 */
@Test
@DisplayName("A user can create a single todo item")
void example_addTodo_singleItem() {
    // Open the initial page
    open("/");

    // Create a new item
    WebElement inputField = $(By.tagName("input"));
    inputField.sendKeys("Touch grass", Keys.ENTER);

    // See the list for the newly inserted item
    $(By.tagName("tbody")).findAll(By.tagName("tr"))
                          .shouldHave(CollectionCondition.size(expectedItems.size()))
                          .should(CollectionCondition.allMatch("Valid row", (row) -> isRowValid(List.of("Tounch grass"), row)));

    // The page can be accessed at the new, unique URL
    String currentUrl = webdriver().driver().url();
    assertTrue(currentUrl.matches(".+/list/\\d+$"), "The URL was: " + currentUrl);
}

private boolean isRowValid(List<String> expectedItems, WebElement row) {
    List<WebElement> columns = row.findElements(By.tagName("td"));

    assertEquals(DEFAULT_NUMBER_OF_COLUMNS, columns.size(),
                 "There were " + columns.size() + " columns in a row");

    String id = columns.get(0).getText();
    String title = columns.get(1).getText();

    return Pattern.matches("\\d+", id) && expectedItems.contains(title);
}

The code example above simulates the actions of a user when adding a todo item. The user opens the web application, then proceed to create a todo item by filling the expected input field. After submitting the new todo item, the user expects it to be stored and visible in the list. As you may have noticed, the test code is highly technical and unlikely to be written by someone without the technical expertise, such as the customer or business analyst.

After looking at the example above, we can see why the "glue" code is needed when writing BDD test suite. We need to write the "glue" code to simulate the actions done by the user, which are described as part of the executable specifications.

To begin writing the "glue" code, we need to create the abstraction of the UI elements and user actions as defined in the executable specifications. Let us start with creating the UI elements abstraction by following the Page Object Model (POM) pattern. See the following page object example that represents the todo list page:

/**
 * Page object model of the main todo list page,
 * specifically the significant elements of the todo list page.
 *
 * Reference:
 * https://serenity-bdd.github.io/docs/screenplay/screenplay_webdriver#using-the-target-class
 */
public class TodoListPage extends PageObject {

    public static Target ITEM_NAME_FIELD = Target
        .the("item name field")
        .located(By.id("id_new_item"));

    public static Target ENTER_BUTTON = Target
        .the("enter button")
        .located(By.tagName("button"));

    public static Target ITEMS_LIST = Target
        .the("item list")
        .located(By.xpath("//tbody/tr/td[contains(@class, 'todo-item-title')]"));
}

The TodoListPage class above represents the todo list page as a page object. It only contains the references to UI elements that are significant in the context of running the test procedures. For example, the existing BDD test suite currently only requires simulating user interaction with three UI elements: the input text field, the submit button, and the table that contains the todo items.

Now, we will see how we can abstract the user actions by using Screenplay pattern. Let us start with creating the abstraction for the action of "adding a todo item":

/**
 * A custom interaction class to fill a text input field.
 *
 * Reference: https://serenity-bdd.github.io/docs/screenplay/screenplay_webdriver#interacting-with-elements
 */
public class AddAnItem {

    public static Performable withName(String itemName) {
        return Task
            .where("{0} adds an item with name " + itemName,
                Enter.theValue(itemName) // Uses Serenity's `Enter` interaction class
                    .into(TodoListPage.ITEM_NAME_FIELD)
            )
            .then(Task.where("{0} clicks the enter button",
                Click.on(TodoListPage.ENTER_BUTTON) // Uses Serenity's `CLick` interaction class
            ));
    }
}

The AddAnItem class above represents the action of adding a todo item performed by an "actor" (representation of user). The action is modeled as a set of Java statements and written in fluent-style to enhance readability. In addition, it utilises the interaction classes provided by the Serenity BDD test framework, which under the hood are actually Selenium code.

Once we have the infrastructure set up, i.e. the abstraction of page and user actions, we can begin write the "glue" code. The "glue" code comprises step definitions that will correspond to Gherkin statements found in the executable specifications. The following code example is the step definitions for "add new todo item" feature:

public class AddItemStepDefinitions {

    @Given("{actor} is looking at the TODO list")
    public void actor_is_looking_at_her_todo_list(Actor actor) {
        actor.wasAbleTo(NavigateTo.theTodoListPage());
    }

    @When("{actor} adds {string} to the list")
    public void she_adds_to_the_list(Actor actor, String itemName) {
        actor.attemptsTo(AddAnItem.withName(itemName));
    }

    @Then("{actor} sees {string} as an item in the TODO list")
    public void she_sees_as_an_item_in_the_todo_list(Actor actor, String expectedItemName) {
        List<String> todoItems = TodoListPage.ITEMS_LIST.resolveAllFor(actor)
            .textContents();

        actor.attemptsTo(
            Ensure.that(todoItems)
                .contains(expectedItemName)
        );
    }
}

The AddItemStepDefinitions class above is the "glue" code for the executable specifications of "add new todo item" feature. Notice that each Gherkin statement, i.e. statements that start with Given, When, and Then, is mapped to a corresponding method. The Serenity BDD test framework utilises pattern matching to pair the Gherkin statement with the corresponding method during test suite execution.

Now that we have created the "glue" code, it is time to run the BDD test suite.

Execute Test Suite & Reporting

Before we run the BDD test suite, let us see how it is configured first at CucumberTestSuite.java:

package com.example.sitodo.bdd;

import io.cucumber.junit.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;
import org.junit.runner.RunWith;

/**
 * The test runner class that runs the executable scenarios,
 * i.e. the feature files written in Gherkin format.
 */
@RunWith(CucumberWithSerenity.class) // Runs the test suite using the test runner provided by Serenity
@CucumberOptions(
    plugin = {"pretty"},
    features = "src/test/resources/features"
)
public class CucumberTestSuite { }

The CucumberTestSuite class represents the BDD test suite. It specifies the test runner, i.e. CucumberWithSerenity class, and the options for the test runner such as the location of the test scenarios.

The options for the test runner provided by Serenity can also be provided via a configuration file named serenity.conf. The content of the configuration file is as follows:

serenity {
    take.screenshots = FOR_FAILURES
}

headless.mode = false

webdriver {
  autodownload = true
  base.url = "http://localhost:8080"
  driver = firefox
  capabilities {
    browserName = "firefox"
    acceptInsecureCerts = true
  }
}

The options in the configuration file are self-explanatory. More information about the configuration can be found in the Serenity documentation.

In the case of the BDD suite, the test runner expects the web application has been running at specified URL, i.e. the value set in base.url. The web application will not automatically run when we execute the BDD test suite. Therefore, we have to build and deploy the web application prior to running the tests.

Before running the BDD test suite, run the web application in a shell:

mvn spring-boot:run

Then, run the BDD test suite in a different shell:

mvn -P bdd verify

The BDD test suite can also be run directly in IntelliJ by running CucumberTestSuite run configuration. Before executing the BDD test suite, IntelliJ will automatically run the web application.

To see the test report generated by the test runner, open the HTML document located at ./target/site/serenity/index.html. The report will show information such as the total number of features and their test status.

Note: To see how the test suites are configured and run by Maven build system, you can peruse the build configuration at pom.xml. The configuration utilises several plugins such as Maven Surefire and Maven Failsafe to run the test runners. Additionally, to ensure that the BDD test suite can be run independently, the configuration use a custom Maven profile to isolate the BDD test suite from the other test suites.

Tasks

Your tasks are to write new executable specifications for two existing features that yet to be covered by the BDD test suite. The two features are: "see motivation message" and "mark a todo item." At minimum, you need to write two new test scenarios for both features. The test scenarios can be as simple as covering the "positive" and "negative" case that may occur on both features, or to cover other possible outputs.

Mandatory Tasks

  • Fork this project into your own namespace (GitLab account)
  • Set up the development environment for the project locally
  • Run the project locally and try the existing features
    • Try to add new todo items and mark their completion status
    • See how the motivation message is updated based on the number of completed todo items
  • Run the test suites locally, ensure all test passes
  • Write a new executable specification for "see motivation" feature that contains at least two test scenarios
  • Write a new executable specification for "mark a todo item" feature that contains at least two test scenarios
  • Write the required "glue" code to support the new executable specifications
  • Maintain code coverage greater than or equal to 97%
  • Summarise the whole process and write them in a summary.md
    • Write an analysis of what you did and what you learned from this exercise and please include the BDD test report

Note: You are allowed to modify the production code (i.e. the src/main directory) to make your test code can obtain the reference to the UI elements in the page. For example, you can add a new HTML attribute, CSS class, or CSS id to an element in the page to make it easier to be located by selector methods.

Additional Tasks

  • Implement the error handling when posting an empty todo item by following TDD approach in BDD (also known as: outside-in TDD)
    • Start by writing the executable specification that describes the error handling
    • Then, implement the error handling by following the TDD cycle (i.e. red-green-refactor phases)
    • Ensure the error handling works
    • Finally, demonstrate that both the unit and BDD test suites run successfully
  • Increase the code coverage to 100%
  • (Not Graded) Deploy the app to a cloud platform and run the BDD test suite by targeting the deployed app

License

This project is licensed under the terms of the MIT license.