Overview

AiExplanationExtension is a JUnit 6 lifecycle extension that automatically integrates AI-powered test analysis into your test suite. When a test fails (or succeeds/skips, if configured), the extension:

1. Collects rich test context — test name, scenario, failure message, stack trace, logs, screenshots, page source, Xray metadata, and custom data.
2. Assembles a structured prompt tailored to the test type (UI, API, Cucumber, or Default).
3. Sends the prompt to an LLM (OpenAI or AWS Bedrock) and receives a natural-language analysis.
4. Stores the LLM response in VeriSoft's StoreManager — which is then automatically picked up by CustomReportPortalExtension and emitted as a 🤖 AI explanation: log entry directly into the Report Portal test item.
5. Redacts sensitive data (API keys, Bearer tokens, passwords) from all logged output before anything is written to logs or reports.

Report Portal integration is automatic

AiExplanationExtension

before

CustomReportPortalExtension

CustomReportPortalExtension

ERROR

1. Registration

Register both extensions on your test class or base class. Order matters — AiExplanationExtension must come first so its response is available when CustomReportPortalExtension finishes the test item:

@ExtendWith({
    AiExplanationExtension.class,         // must be first
    CustomReportPortalExtension.class
})
public class MyTest { ... }

2. Configuration

Configuration is resolved with the following precedence (highest wins):

Runtime Override  >  Environment Variables  >  ai-config.properties

Create src/test/resources/ai-config.properties:

# ─────────────────────────────────────────────
# REQUIRED — Feature flag
# ─────────────────────────────────────────────
llm.enabled=true

# ─────────────────────────────────────────────
# REQUIRED — LLM Provider & Model
# ─────────────────────────────────────────────
llm.provider=OPENAI          # OPENAI | BEDROCK
llm.model=gpt-4.1

# ─────────────────────────────────────────────
# REQUIRED (one of the two, depending on provider)
# ─────────────────────────────────────────────
llm.openai.api-key=sk-...    # For OPENAI (prefer env var OPENAI_API_KEY)
llm.bedrock.region=eu-west-1 # For BEDROCK

# ─────────────────────────────────────────────
# OPTIONAL — LLM Behaviour
# ─────────────────────────────────────────────
llm.timeout-ms=30000         # Default: 30000
llm.max-retries=1            # Default: 1
llm.response.language=Hebrew # Default: LLM default (English)
llm.response.role=QA         # QA | DEVELOPER | PRODUCT_MANAGER | TEAM_LEAD

# ─────────────────────────────────────────────
# OPTIONAL — Context Collection Triggers
# ─────────────────────────────────────────────
context.collect.on.failure=true    # Default: true  ← ONLY this is true by default
context.collect.before=false       # Default: false
context.collect.on.success=false   # Default: false
context.collect.on.skip=false      # Default: false
context.export.before=false        # Default: false

# OPTIONAL — Output
context.output.directory=target/ai-context  # Default: target/ai-context

Equivalent Environment Variables

Runtime Override (Programmatic)

For tests that need per-class config (e.g. a different model):

AiExplanationConfig override = AiExplanationConfig.builder()
    .provider(Provider.OPENAI)
    .model("gpt-4o")
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .responseLanguage("Hebrew")
    .responseRole("QA")
    .collectOnFailure(true)
    .collectOnSuccess(true)
    .build();

AiExplanationConfigResolver.getInstance().setRuntimeOverride(override);

3. Marking Test Type (for Prompt Specialisation)

The extension selects a specialised prompt builder based on test type. Annotate your test class or method with @AiTestType:

import co.verisoft.fw.llm.prompt.AiTestType;
import co.verisoft.fw.llm.prompt.TestType;

@AiTestType(TestType.API)      // UI | API | CUCUMBER | DEFAULT
@ExtendWith({
    AiExplanationExtension.class,
    CustomReportPortalExtension.class
})
public class MyApiTests { ... }

Resolution order: method annotation → class annotation → superclass → DEFAULT.

4. How the AI Response Reaches Reports

The flow is fully automatic — no extra code is needed in your tests:

Test fails
    └─► AiExplanationExtension.handleTestExecutionException()
            ├─ Collects context (logs, stack trace, screenshots, Xray…)
            ├─ Sends prompt to LLM (OpenAI / Bedrock)
            ├─ Redacts sensitive data from response
            ├─ Stores response → StoreManager [ai.llm.response]
            └─ Re-throws the original exception

    └─► CustomReportPortalExtension.afterTestExecution()
            ├─ Checks that AiExplanationExtension is registered
            ├─ Reads response from StoreManager [ai.llm.response]
            └─ Emits "🤖 AI explanation:\n{response}" to Report Portal
               at ERROR log level on the failing test item

AiExplanationExtension

before

CustomReportPortalExtension

@ExtendWith

5. Supported LLM Providers

OpenAI

llm.provider=OPENAI
llm.model=gpt-4.1
llm.openai.api-key=sk-...

• Targets https://api.openai.com/v1/chat/completions by default.
• Custom baseUrl can be set via runtime override (for OpenAI-compatible proxies).
• Retries automatically on HTTP 429 / 5xx with exponential back-off.

AWS Bedrock

llm.provider=BEDROCK
llm.model=<model-id-or-ARN>
llm.bedrock.region=eu-west-1

• Uses standard AWS credential chain (env AWS_REGION, IAM role, etc.).

6. Logback Configuration — Required Format

The extension uses MDC key executionId to correlate log entries per test. Your FILE-INFO appender must include %X{executionId:-NA} in the pattern:

<appender name="FILE-INFO" class="ch.qos.logback.core.FileAppender">
    <file>target/logs/logs-info.log</file>
    <append>true</append>
    <immediateFlush>true</immediateFlush>
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>INFO</level>
    </filter>
    <encoder>
        <pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} [%thread] [exec:%X{executionId:-NA}] %-5level %logger{35} - %msg%n</pattern>
    </encoder>
</appender>

The [exec:%X{executionId:-NA}] token will print the unique UUID for the test run, or NA if no test is active. This makes it trivial to grep all log lines belonging to a single failing test.

7. Complete Example

import co.verisoft.fw.CustomReportPortalExtension;
import co.verisoft.fw.extensions.jupiter.AiExplanationExtension;
import co.verisoft.fw.llm.prompt.AiTestType;
import co.verisoft.fw.llm.prompt.TestType;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

import static org.junit.jupiter.api.Assertions.assertEquals;

// AiExplanationExtension MUST be declared before CustomReportPortalExtension
@ExtendWith({
    AiExplanationExtension.class,        // must be first — populates the store
    CustomReportPortalExtension.class    // reads store and emits to Report Portal
})
@AiTestType(TestType.API)               // Selects API-specialised prompt builder
public class PaymentApiTest {

    @Test
    void paymentShouldReturn200() {
        // When the test fails:
        //  1. AiExplanationExtension collects context + calls LLM
        //  2. The AI analysis is stored automatically in StoreManager
        //  3. CustomReportPortalExtension picks it up and emits it
        //     as "🤖 AI explanation:" on the Report Portal test item
        int actualStatusCode = 401;
        assertEquals(200, actualStatusCode,
                "Expected payment API to return 200 OK but got " + actualStatusCode);
    }
}

With src/test/resources/logback-test.xml — the FILE-INFO appender must include %X{executionId:-NA}:

<configuration debug="true">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>WARN</level>
        </filter>
        <encoder>
            <pattern>%date{HH:mm:ss.SSS} [%thread] %highlight(%-5level) %cyan(%logger{15}) - %msg %n</pattern>
        </encoder>
    </appender>

    <appender name="FILE-DEBUG" class="ch.qos.logback.core.FileAppender">
        <file>target/logs/logs-debug.log</file>
        <append>true</append>
        <immediateFlush>true</immediateFlush>
        <encoder>
            <pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- FILE-INFO: executionId token is REQUIRED for AiExplanationExtension log correlation -->
    <appender name="FILE-INFO" class="ch.qos.logback.core.FileAppender">
        <file>target/logs/logs-info.log</file>
        <append>true</append>
        <immediateFlush>true</immediateFlush>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <encoder>
            <pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} [%thread] [exec:%X{executionId:-NA}] %-5level %logger{35} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="net.jvw" level="DEBUG"/>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="FILE-DEBUG"/>
        <appender-ref ref="FILE-INFO"/>
    </root>
</configuration>

[exec:%X{executionId:-NA}]

NA

grep

With src/test/resources/ai-config.properties:

llm.enabled=true
llm.provider=OPENAI
llm.model=gpt-4.1
# API key read from env: OPENAI_API_KEY

llm.response.language=Hebrew
llm.response.role=QA
llm.timeout-ms=45000
llm.max-retries=2

context.collect.on.failure=true
context.collect.on.success=false
context.collect.before=false
context.collect.on.skip=false
context.output.directory=target/ai-context