Agentic Analysis

info

This feature is experimental. We are actively working on it, so its behavior might change without notice. It is only active when it has been enabled for your environment and when AI Assisted Mode is enabled for the project.

When a test run finishes with failed tests, Orangebeard can analyze those failures for you. This page explains how that works, how to read the explanation that comes with an analysis, and what you can configure.

The analysis never replaces your own judgment. An analysis that a user made or confirmed is never overwritten by an automatic one.

How a failure gets its analysis

Analyzing every failed test with an AI agent is slow and expensive, and most failures in a run are not new. Orangebeard therefore works in steps and only goes to the next step when the previous one could not decide.

1. Grouping. Failures in the run that fail for the same reason are grouped. The group is analyzed once, and the result is applied to every test in the group.
2. Memory. Orangebeard checks whether it has seen this failure before. A verdict that a user gave, or that the agent gave earlier, is reused, including its description. If the failure has no match in the new memory, the regular Auto Analyzer search for similar earlier failures is still used, exactly as before. Matches found this way that were analyzed by a user are remembered as user verdicts from then on.
3. Quick classification. A fast AI model looks at the failure and answers a few fixed questions about it. The classification is derived from those answers by fixed rules, not by the model.
4. Deep investigation. Failures that the quick classification could not decide, and product bugs that Orangebeard has not seen before, are handed to the analysis agent. The agent investigates with the same tools the AI assistant has: logs, attachments, test history, earlier analyses and, when configured, your MCP servers. It then writes the explanation described below.

Failures that passed when the test was retried are never taken from memory and their result is never remembered, because a failure that sometimes passes should not hide a failure that never does.

Where does an analysis come from?

Source	Meaning
User	A user analyzed this failure. This always wins.
Agent	The analysis agent investigated a failure like this one earlier.
Classifier	The quick classification. Only stored for later reuse when its confidence is high.

Defect types

The agent uses the same defect types as you do:

• Product bug: the product under test does not behave as it should.
• Automation issue: the test itself is wrong, outdated, flaky, or its data or environment setup was not right.
• System issue: an environment, infrastructure or dependency problem, outside the product and the test.
• Follow-up error: the failure is caused by another failure in the same run.
• To investigate: the analysis could not decide. The explanation tells you what was found and what is missing.

The eight questions (D1 to D8)

To reach a conclusion the agent answers eight questions about every failure it investigates. Internally these are numbered D1 to D8, and you may see these codes in explanations. Each question is answered with Yes, No or Unknown. Unknown is a valid answer: it means there was not enough evidence, and the agent is not allowed to guess.

Code	Question	What the agent looks at
D1	Was the request or interaction of the test valid, and the one it meant to make?	The request or UI action in the logs and attachments, the test code
D2	Were the preconditions of the test actually met (login, seed data, feature flags, fixtures, the right environment)?	Setup logs, earlier steps, fixture output
D3	Was the error raised by the system under test itself?	Stack traces and service names in the error, earlier confirmed analyses, run and test attributes, linked work items
D4	Is the observed behavior a contract violation (a 5xx error, an unhandled exception, a schema mismatch, a crash)?	Status code, response body, the API specification
D5	Is there a sign of an infrastructure or dependency problem (connection refused, DNS or TLS errors, 502/503/504, a restarting container, exhausted resources)?	Logs, health checks, deployment tools
D6	Did other tests in this run use the same operation, and did they pass or fail?	The rest of the test run
D7	Did the test, its fixtures or the product change recently?	Commits and merge requests around the failing component, when a source control tool is connected
D8	Does the failure depend on timing, order or data (passes on retry, passes in isolation, relies on a value created elsewhere)?	Retry results, test history, test metadata

From answers to a defect type

The agent does not pick a defect type directly. It walks through the following rules in order, and the first rule that applies decides. A rule that depends on a question answered Unknown does not apply.

1. D5 is Yes and the failing component is a dependency → System issue.
2. D1 is No or D2 is No → Automation issue. The test broke before the product had a fair chance.
3. D4 is Yes and D3 is Yes → Product bug.
4. D3 is Yes, D4 is No, and the product returned a valid but unexpected value → the agent compares this with the specification. If the specification supports the test, it is a Product bug. If it supports the product, the expectation of the test is out of date and it is an Automation issue. Without a specification: To investigate.
5. D8 is Yes and the product reported no error → Automation issue.
6. Otherwise → To investigate.

Two things follow from these rules:

• A conclusion needs evidence. If the key question is Unknown, the result is To investigate, not a best guess.
• The order matters. A dependency outage (D5) is reported as a system issue even when the test also shows an error.

Confidence and the runner-up

Every analysis has a confidence between 0 and 100%. It reflects how much of the evidence was found, not how certain the agent sounds. Many Unknown answers, or evidence that points in two directions, lower it. Treat a low confidence as an invitation to take a look yourself.

The explanation also names the second most likely defect type and the one piece of evidence that would have selected it instead. If the agent cannot name such evidence, the two options are not separated by the evidence and the result is To investigate.

Example

The test posted an order with a valid payload and the seed data was present (D1, D2: Yes). The pricing service responded with a 500 and a NullPointerException in its own stack trace (D3, D4: Yes). No connection or infrastructure errors were found (D5: No). → Product bug, confidence 90%. Second most likely: Automation issue; it would apply if the order payload had been rejected by validation, which was not seen.

In current explanations the agent describes these findings in plain words. If an explanation does contain codes, read them with the table above, for instance D6: Unknown means that no other test in the run used the same operation, so the run gave no extra information.

Quick classification

The quick classification in step 3 uses a smaller set of fixed questions: where the error originated (product, test, external or unknown), whether there is an infrastructure signal, whether the product broke its own contract, and whether the test's own interaction was invalid. It can also recognize that a failure is explained by another failing group in the same run, which becomes a Follow-up error. The same principle applies: a rule never applies on an Unknown answer, and everything it cannot settle is passed on to the deep investigation as To investigate.

The confidence of a classification is high only when all questions were answered and an earlier analyzed look-alike agrees with the result. Only high-confidence classifications are stored for reuse.

Reading the result in Orangebeard

• The defect type, and the description with the explanation, are filled in on the failed tests. The auto analyzed flag marks that the analysis was not made by a user.
• Tests that failed for the same reason receive the same defect type and description.
• If you change the analysis of a test, your analysis is kept. It becomes a verdict that is reused for the same failure in future runs.
• Tests that stay on To investigate with an explanation were investigated, but the evidence was not enough. The explanation lists what was found and what is missing.

Token usage

The deep investigation uses the language model that is configured for your organization (see AI Model), so it counts towards your daily token budget.

Privacy and data handling

• The quick classification only runs on models hosted in the EU (NL) and uses a limited part of the failure log.
• Everything in logs and attachments is treated as data. Instructions found inside test output are ignored.
• Analysis texts and logs are never written to service logs.
• The agent only reads the project when AI Assisted Mode is enabled for it, see Project settings.

Who owns a failure

A test failure is not always about the product that you test. To decide whether a failure belongs to the product, the test, or something else, the agent looks for evidence about where the failing component sits, in this order:

1. The failure itself and the rest of the run: error messages, stack traces and service names.
2. Run and test attributes. An attribute can point to extra information, for example a Jira work item or a GitLab merge request for the change that is being tested. The agent may follow such a pointer to learn what changed.
3. Earlier analyses of the same failure that a user confirmed.
4. Connected tools. When your organization has registered MCP servers, the agent can query them too, for example for work items, merge requests or documentation.

A combination of these is common. If the origin of the failing component cannot be established, the confidence is capped and the explanation says so. Adding an attribute to your runs that links to the work item under test is the most effective way to improve the quality of the analysis.