Cucumber Reports

What are Cucumber Reports?

Cucumber Reports are output files generated by Cucumber to document the results of test execution. They provide insights into which scenarios passed, failed, or were skipped, along with details like step execution times, error messages, and tags. Cucumber supports multiple report formats, such as HTML, JSON, and JUnit XML, making it easy to integrate with CI/CD pipelines or share with stakeholders.

Why Use Cucumber Reports?

• Test Analysis: Quickly identify passed, failed, or pending scenarios.
• Debugging: View error messages and stack traces for failed steps.
• Team Communication: Share clear, visual reports with developers, testers, and business stakeholders.
• CI/CD Integration: Use reports in tools like Jenkins, GitHub Actions, or GitLab CI for automated testing.
• Documentation: Serve as a record of test execution for auditing or compliance.

Common Cucumber Report Formats

Cucumber supports several report formats via plugins. The most commonly used are:

1. Pretty: Outputs readable text to the console (enabled by default).
2. HTML: Generates a user-friendly, web-based report viewable in a browser.
3. JSON: Produces a machine-readable report for integration with other tools.
4. JUnit XML: Creates a report compatible with CI tools like Jenkins.
5. Usage: Shows step definition usage and execution times (useful for optimization).

You can generate multiple report formats simultaneously by configuring plugins in your TestRunner class or CLI command.

Generating Cucumber Reports

Let’s create a feature file, run tests, and generate HTML and JSON reports to demonstrate how Cucumber Reports work. Assume you have a Cucumber project set up with Java and Maven, as described in the Installation and Setup post.

Example: Feature File

Create a file named login.feature in src/test/resources/features:

Feature: User Login
  As a user, I want to log in to the application so that I can access my account.

  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

  @regression
  Scenario: Failed login with invalid credentials
    Given the user is on the login page
    When the user enters "user1" and "wrongpass"
    Then the user should see an error message

Step Definitions

Create LoginSteps.java in src/test/java/steps:

package steps;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.When;
import io.cucumber.java.en.Then;

public class LoginSteps {
    @Given("the user is on the login page")
    public void userIsOnLoginPage() {
        System.out.println("Navigating to login page");
    }

    @When("the user enters {string} and {string}")
    public void userEntersCredentials(String username, String password) {
        System.out.println("Entering username: " + username + ", password: " + password);
        // Simulate a failure for the second scenario
        if (password.equals("wrongpass")) {
            throw new RuntimeException("Invalid credentials");
        }
    }

    @Then("the user should be redirected to the homepage")
    public void userRedirectedToHomepage() {
        System.out.println("Verifying redirection to homepage");
    }

    @Then("the user should see an error message")
    public void userSeesErrorMessage() {
        System.out.println("Verifying error message");
    }
}

Explanation:

• The second scenario is designed to fail by throwing an exception when the password is "wrongpass".
• This allows us to see both passed and failed results in the reports.

TestRunner Configuration

Create TestRunner.java in src/test/java/runner:

package runner;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    plugin = {
        "pretty",
        "html:target/cucumber-reports/cucumber.html",
        "json:target/cucumber-reports/cucumber.json"
    },
    monochrome = true
)
public class TestRunner {
}

Explanation:

• plugin: Specifies the report formats:
  • pretty: Console output.
  • html:target/cucumber-reports/cucumber.html: HTML report saved to the target directory.
  • json:target/cucumber-reports/cucumber.json: JSON report saved to the target directory.
• monochrome = true: Makes console output more readable by removing color codes.

Run the Tests

Run the tests using Maven:

mvn test

Console Output (Pretty Format)

The pretty plugin outputs to the console:

Feature: User Login
  As a user, I want to log in to the application so that I can access my account.

  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

  @regression
  Scenario: Failed login with invalid credentials
    Given the user is on the login page
    When the user enters "user1" and "wrongpass"
      java.lang.RuntimeException: Invalid credentials
      ...
    Then the user should see an error message

2 Scenarios (1 passed, 1 failed)
6 Steps (4 passed, 1 failed, 1 skipped)
0m0.245s

HTML Report

After running the tests, open target/cucumber-reports/cucumber.html in a browser. The HTML report includes:

• A summary of passed, failed, and skipped scenarios.
• Details for each feature and scenario, including tags.
• Step-by-step results with error messages for failures.
• Visual indicators (green for passed, red for failed).

Sample View:

• Feature: User Login
  • Scenario: Successful login with valid credentials → Passed ✅
  • Scenario: Failed login with invalid credentials → Failed ❌ (with stack trace for “Invalid credentials”)

JSON Report

The JSON report at target/cucumber-reports/cucumber.json contains structured data, useful for parsing or integration. Example snippet:

[
  {
    "uri": "file:src/test/resources/features/login.feature",
    "elements": [
      {
        "name": "Successful login with valid credentials",
        "tags": ["@smoke"],
        "steps": [
          {"name": "the user is on the login page", "result": {"status": "passed"}},
          {"name": "the user enters \"user1\" and \"pass123\"", "result": {"status": "passed"}},
          {"name": "the user should be redirected to the homepage", "result": {"status": "passed"}}
        ]
      },
      {
        "name": "Failed login with invalid credentials",
        "tags": ["@regression"],
        "steps": [
          {"name": "the user is on the login page", "result": {"status": "passed"}},
          {"name": "the user enters \"user1\" and \"wrongpass\"", "result": {"status": "failed", "error_message": "java.lang.RuntimeException: Invalid credentials..."}},
          {"name": "the user should see an error message", "result": {"status": "skipped"}}
        ]
      }
    ]
  }
]

Generating Reports with the Cucumber CLI

You can also generate reports using the Cucumber CLI instead of TestRunner. Run:

mvn cucumber:test -Dcucumber.features=src/test/resources/features -Dcucumber.glue=steps -Dcucumber.plugin=pretty,html:target/cucumber-reports/cucumber.html,json:target/cucumber-reports/cucumber.json

This produces the same reports as the TestRunner configuration.

Advanced Reporting: Filtering by Tags

You can generate reports for specific tagged scenarios to focus on subsets of tests.

Example: Run Smoke Tests Only

Update TestRunner.java to run only @smoke scenarios:

@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    tags = "@smoke",
    plugin = {
        "pretty",
        "html:target/cucumber-reports/smoke.html",
        "json:target/cucumber-reports/smoke.json"
    },
    monochrome = true
)

Run the tests:

mvn test

Output:

Feature: User Login
  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

1 Scenarios (1 passed)
3 Steps (3 passed)
0m0.123s

The HTML report (target/cucumber-reports/smoke.html) will only show the @smoke scenario results.

Integrating Reports with CI/CD

To integrate Cucumber Reports with CI/CD tools like Jenkins:

1. Configure your TestRunner or CLI to generate JUnit XML reports:

   plugin = {"junit:target/cucumber-reports/cucumber-junit.xml"}
   

2. In Jenkins, add a post-build step to publish JUnit reports, pointing to target/cucumber-reports/cucumber-junit.xml.
3. Archive the HTML report as an artifact for team access:

   plugin = {"html:target/cucumber-reports/cucumber.html"}
   

This allows you to view test results and trends in your CI dashboard.

Best Practices for Cucumber Reports

1. Use Multiple Formats: Generate HTML for human-readable reports and JSON/JUnit for CI integration.
2. Organize Report Files: Save reports in a dedicated directory (e.g., target/cucumber-reports) with meaningful names.
3. Enable Monochrome: Use monochrome = true for clearer console output in CI logs.
4. Review Failures: Check HTML or JSON reports for stack traces to debug failed scenarios.
5. Archive Reports: Store reports in your CI system or version control for historical analysis.
6. Keep Reports Focused: Use tags to generate separate reports for different test suites (e.g., smoke, regression).

Troubleshooting Report Issues

• Missing Reports: Ensure the plugin option is correctly specified in TestRunner or CLI.
• Incorrect Paths: Verify the output paths (e.g., target/cucumber-reports) are writable and correctly formatted.
• Empty Reports: Check that scenarios ran successfully; undefined steps or syntax errors may prevent report generation.
• CI Integration Errors: Ensure JUnit XML reports are compatible with your CI tool’s parser.
• Large Reports: For large test suites, consider filtering by tags to reduce report size.

Tips for Beginners

• Start with HTML Reports: Open HTML reports in a browser to understand test results visually.
• Check Console Output: Use the pretty plugin to see results in real-time during development.
• Experiment with Tags: Generate reports for specific tags to focus on critical tests.
• Use in Team Reviews: Share HTML reports with your team to discuss test outcomes.