RobotFramework

Installation

Install the pip package:
pip install orangebeard-robotframework

The source code can be found on GitHub: RobotFramework Listener.

Configuration

To configure the Orangebeard RobotFramework Listener, create a json file (called orangebeard.json) in the working directory or up in the directory structure:

{
  "endpoint": "https://app.orangebeard.io/{ORGANIZATION}",
  "token": "{LISTENER_ACCESS_TOKEN}",
  "project": "{ORANGEBEARD_PROJECT}",
  "testset": "RobotFramework Tests",
  "description": "A run from robot framework",
  "attributes": [
    {
      "key": "Key 1",
      "value": "Some value"
    },
    {
      "value": "Tag value"
    }
  ],
  "referenceUrl": "https://link.to.my/build-server"
}

While the orangebeard.json file must exist, you can provide or override any parameter using environment or command-line variables. It is good practice to place static configuration like endpoint and project in the JSON file, and provide secrets (like the token) and dynamic values via environment or command-line variables.

Configuration is applied in the following order of precedence, with later steps overriding earlier ones:

1. orangebeard.json file
2. Environment variables
3. Robot Framework command-line variables

Using environment variables:

export ORANGEBEARD_ENDPOINT=https://app.orangebeard.io/{ORGANIZATION}
export ORANGEBEARD_TOKEN="{LISTENER_ACCESS_TOKEN}"
export ORANGEBEARD_PROJECT="{ORANGEBEARD_PROJECT}"
export ORANGEBEARD_TESTSET="RobotFramework Tests"
export ORANGEBEARD_DESCRIPTION="A run from robot framework"
export ORANGEBEARD_ATTRIBUTES="key1:some value;Tag value"
export ORANGEBEARD_REFERENCE_URL=https://link.to.my/build-server

Using command-line variables:

robot --listener orangebeard_robotframework.listener \
--variable orangebeard_endpoint:"https://app.orangebeard.io/{ORGANIZATION}" \
--variable orangebeard_accesstoken:"{LISTENER_ACCESS_TOKEN}"  \
--variable orangebeard_project:"{ORANGEBEARD_PROJECT}" \
--variable orangebeard_testset:"Testset name" \
--variable orangebeard_loglevel:"INFO" \
--variable orangebeard_reference_url:"https://orangebeard.io/myref" \
./tests

note

Appending Attributes: Attributes are always appended. If you define attributes in the JSON file, environment variables, and on the command line, all of them will be combined and sent with the test run.
Re-using a Test Run: There is also a special orangebeard_testrun variable, which is used to attach to an already existing test run. This is primarily used for parallel testing scenarios, as shown in the pabot section below.

Usage

Run your tests with the listener registered:

robot --listener orangebeard_robotframework.listener ./tests

Log Level Configuration

You can control the level of logs sent to Orangebeard. Valid options are DEBUG, INFO (default), WARN, and ERROR.
The DEBUG level is special, as it also includes TRACE level logs from Robot Framework.

info

The log level can only be configured as a Robot Framework command-line variable. It cannot be set via orangebeard.json or environment variables.

Example:
--variable orangebeard_loglevel:"DEBUG"

Parallel test reporting with pabot

When running tests in parallel with pabot, you must manage the Orangebeard test run manually. If you don't, each parallel pabot process will create its own separate test run in Orangebeard.

The correct approach is to start a single run before pabot executes, and finish it after pabot is done. This can be achieved using the orangebeard-cli utility, which is included with the orangebeard-python-client.

Example shell script:

#!/bin/bash
testrunUuid=$(orangebeard-cli -x start)

pabot --listener orangebeard_robotframework.listener  \
      --variable orangebeard_testrun:"$testrunUuid" .
      
orangebeard-cli -x finish -id $testrunUuid

This script first starts a new test run using orangebeard-cli and captures its unique ID. This ID is then passed to pabot via the orangebeard_testrun variable, ensuring all parallel threads report to the same run. Finally, orangebeard-cli is used to finish the run after all tests have completed.

For more CLI configuration options, see the Python Client README or run orangebeard-cli -h.