Debugging Cucumber Tests

← Previous

Why Debugging is Important

Debugging Cucumber tests is essential for several reasons:

Identify Failures: Pinpoint the exact cause of test failures, whether due to script errors, application bugs, or environmental issues.
Improve Test Quality: Refine test scripts to make them more robust and accurate.
Save Time: Quickly resolve issues to maintain productivity in test development.
Enhance Collaboration: Share clear error details with developers and stakeholders to align on fixes and requirements.

Common issues that require debugging include undefined steps, element not found errors in Selenium, incorrect API responses, or timing issues in dynamic applications.

Common Debugging Techniques

Below are the most effective techniques for debugging Cucumber tests, with examples to illustrate their use. These methods are applicable to various test types (web, API, mobile) and are designed to be accessible to beginners.

1. Using Dry-Run to Check for Undefined Steps

The dry-run option in Cucumber allows you to verify if all steps in your feature files have corresponding step definitions without executing the tests. This is useful for catching undefined or missing steps early in development.

How to Use Dry-Run

In TestRunner: Set dryRun = true in your TestRunner class.
In CLI: Use the --dry-run flag with Maven.

Example Feature File (login.feature):

Feature: User Login
  As a user, I want to log in to the application so that I can access my account.

  Scenario: Successful login
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the dashboard

TestRunner (src/test/java/runner/TestRunner.java):

package runner;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    dryRun = true,
    plugin = {"pretty"},
    monochrome = true
)
public class TestRunner {
}

Run Command:

mvn test

Output (if all steps are defined):

Feature: User Login
  Scenario: Successful login
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the dashboard

1 Scenarios (1 undefined)
3 Steps (3 undefined)
0m0.000s

If steps are undefined, Cucumber will provide snippets for missing step definitions, which you can copy into your step definition file.

Why Use It?

Identifies missing step definitions quickly.
Prevents runtime errors due to undefined steps.

2. Enabling Monochrome Output

The monochrome option makes console output cleaner by removing color codes, which is especially helpful when reviewing logs in CI/CD systems or terminals.

How to Enable

In TestRunner, set monochrome = true.
In CLI, use --monochrome.

Example TestRunner:

@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    plugin = {"pretty", "html:reports/cucumber.html"},
    monochrome = true
)

Run Command:

mvn test

Output (clean, readable format):

Feature: User Login
  Scenario: Successful login
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the dashboard

Why Use It?

Improves readability of console logs, especially in CI/CD pipelines.
Makes it easier to copy logs for debugging.

3. Adding Print Statements or Logging

Adding print statements or using a logging framework (e.g., Log4j, SLF4J) in step definitions helps track execution flow and inspect variable values.

Example Step Definitions (src/test/java/steps/LoginSteps.java):

package steps;

import io.cucumber.java.en.*;

public class LoginSteps {
    @Given("the user is on the login page")
    public void userIsOnLoginPage() {
        System.out.println("Navigating to login page");
    }

    @When("the user enters {string} and {string}")
    public void userEntersCredentials(String username, String password) {
        System.out.println("Username: " + username + ", Password: " + password);
        // Simulate failure
        if (password.equals("wrongpass")) {
            throw new RuntimeException("Invalid credentials");
        }
    }

    @Then("the user should be redirected to the dashboard")
    public void userRedirectedToDashboard() {
        System.out.println("Verifying dashboard redirection");
    }
}

Output (for a failing scenario):

Navigating to login page
Username: user1, Password: wrongpass
java.lang.RuntimeException: Invalid credentials

Using Log4j (optional):
Add Log4j dependency to pom.xml:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.23.1</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.23.1</version>
</dependency>

Create log4j2.properties in src/test/resources:

appenders = console
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %m%n
rootLogger.level = info
rootLogger.appenderRefs = stdout
rootLogger.appenderRef.stdout.ref = STDOUT

Update LoginSteps.java:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class LoginSteps {
    private static final Logger logger = LogManager.getLogger(LoginSteps.class);

    @When("the user enters {string} and {string}")
    public void userEntersCredentials(String username, String password) {
        logger.info("Username: {}, Password: {}", username, password);
        if (password.equals("wrongpass")) {
            throw new RuntimeException("Invalid credentials");
        }
    }
}

Why Use It?

Tracks execution flow and variable values.
Logging frameworks provide structured, timestamped logs.

4. Using IDE Debugging Tools

Modern IDEs like IntelliJ IDEA and Eclipse allow you to set breakpoints in step definitions to pause execution and inspect variables, call stacks, and application state.

Steps in IntelliJ IDEA

Open LoginSteps.java.
Click the left margin next to a line (e.g., inside userEntersCredentials) to set a breakpoint.
Right-click TestRunner.java and select Debug.
When execution pauses, inspect variables and step through code using the debugger.

Debug Configuration:

In IntelliJ, go to Run > Edit Configurations.
Add a new JUnit configuration.
Set Test kind to Class, select TestRunner.
Save and debug.

Example:
Set a breakpoint at:

logger.info("Username: {}, Password: {}", username, password);

Run in debug mode to inspect username and password values.

Why Use It?

Provides detailed insight into code execution.
Ideal for complex logic or unexpected failures.

5. Interpreting Cucumber Reports

Cucumber generates detailed reports (HTML, JSON, JUnit) that highlight failed steps and provide error messages.

Example Feature File with Failure:

Feature: User Login
  @regression
  Scenario: Failed login
    Given the user is on the login page
    When the user enters "user1" and "wrongpass"
    Then the user should be redirected to the dashboard

Run Tests:

mvn test

HTML Report (reports/cucumber.html):

Shows the failed scenario with the error: java.lang.RuntimeException: Invalid credentials.
Highlights the failing step and skips subsequent steps.

JSON Report (reports/cucumber.json):

[
  {
    "elements": [
      {
        "name": "Failed login",
        "steps": [
          {"name": "the user enters \"user1\" and \"wrongpass\"", "result": {"status": "failed", "error_message": "java.lang.RuntimeException: Invalid credentials"}}
        ]
      }
    ]
  }
]

Why Use It?

Provides a clear view of which steps failed and why.
Useful for sharing with the team.

6. Debugging Common Scenarios

Different test types require specific debugging approaches. Below are tips for common scenarios:

Web Tests (Selenium)

Element Not Found: Use browser developer tools (e.g., Chrome DevTools) to verify locators (ID, CSS, XPath). Example:
```
driver.findElement(By.id("user-name"));
```
Check if the ID exists using DevTools.

Timing Issues: Use explicit waits:

import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
wait.until(ExpectedConditions.elementToBeClickable(By.id("login-button")));

API Tests (REST-assured)

Incorrect Response: Log request and response:

import io.restassured.RestAssured;

RestAssured.given().log().all().get().then().log().all();

Authentication Errors: Verify tokens or credentials in the request.

Mobile Tests (Appium)

Element Not Found: Use Appium Inspector to find correct locators.
Device Issues: Check device logs via adb logcat or Appium logs.

Why Use It?

Tailors debugging to specific test types.
Leverages tools like DevTools and Appium Inspector.

7. Using Scenario Hooks for Debugging

Scenario hooks (@Before, @After) can add debugging capabilities, such as capturing screenshots on failure.

Example: Screenshot on Failure:

import io.cucumber.java.After;
import io.cucumber.java.Scenario;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;

@After
public void tearDown(Scenario scenario) {
    if (scenario.isFailed() && driver != null) {
        byte[] screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
        scenario.attach(screenshot, "image/png", "failure-screenshot");
    }
    if (driver != null) {
        driver.quit();
    }
}

Why Use It?

Captures visual evidence of failures in HTML reports.
Helps identify UI issues in web or mobile tests.

8. Using the @debug Tag

You can use a custom @debug tag to enable debugging for specific scenarios, such as adding extra logging or pausing execution.

Example:

@debug
Scenario: Debug login
  Given the user is on the login page
  When the user enters "user1" and "pass123"
  Then the user should be redirected to the dashboard

Step Definition:

@When("the user enters {string} and {string}")
public void userEntersCredentials(String username, String password) {
    logger.info("Debug mode: Entering username: {}, Password: {}", username, password);
    // Add breakpoint or extra logging
}

Run with @debug:

mvn test -Dcucumber.filter.tags="@debug"

Why Use It?

Isolates debugging to specific scenarios.
Useful during development or troubleshooting.

Best Practices for Debugging

Start with Dry-Run: Always check for undefined steps before running tests.
Use Clear Logging: Add meaningful log messages to track execution.
Leverage IDEs: Use breakpoints for complex issues.
Review Reports: Analyze HTML reports for failure details.
Collaborate: Share logs and screenshots with the team to clarify issues.
Test Incrementally: Debug one scenario at a time to isolate problems.

Troubleshooting Debugging Issues

Undefined Steps: Run with dryRun = true to generate step definition snippets.
Unclear Errors: Enable monochrome output and add detailed logging.
Flaky Tests: Use explicit waits for web/mobile tests or log API responses.
Report Issues: Ensure plugin settings in TestRunner are correct (e.g., html:reports/cucumber.html).
IDE Debugger Not Working: Verify the debug configuration points to TestRunner.

Tips for Beginners

Start Simple: Use print statements and dry-run to learn debugging.
Check Reports: Open HTML reports to understand failures visually.
Practice with IDEs: Experiment with breakpoints in IntelliJ or Eclipse.
Use Public Apps: Test with sites like [saucedemo.com] for practice.

← Previous