Introduction
Orangebeard is an intelligent software solution that helps you manage and improve your automated testing through intelligent analyses, clear insights and selecting and running the right tests for each build.
Main features
Fully detailed test results and logging
Automated defect classification
Workspace, project and test set level quality insights
All test results in one place, in real-time
Contextual test set optimization
For more information, visit our website!
Collecting data
It all starts with data. That’s why Orangebeard aims to collect all relevant information on your test runs.
Orangebeard provides a listener interface to connect your test tools- and frameworks, as well as tool-specific
integrations to send your test runs, -suites and -cases’ output to Orangebeard.
You can choose to connect to Orangebeard using one of our tool-specific connectors or roll your own using one of our open source API clients.
Regardless of where you store and run your tests from, Orangebeard handles data from any origin, such as your developer laptop, or a Cloud based CI service. And everything in between. Besides test run output, you can also provide Orangebeard with version information of your system under test, so Orangebeard knows what changes are being tested.
Data model & Terminology
Orangebeard aims to provide a uniform representation of test results, regardless of the test tool used to generate them.
For that, we aim to present all test results in the same data model scheme:
In Orangebeard, the first thing to define is a project. A project is a logical name that will contain one or more test sets that can be executed. The execution of a test set, becomes a test run.
When a test run is reported to Orangebeard, it consists of a structure of suites that can be nested. A suite is like a folder that can contain test cases.
Test cases can contain logs, but can also be structured more by reporting steps to hold the log items. These steps can also be nested.
Projects can be combined into a workspace to provide a higher level overview of the state of your testing efforts.
Structure summary
- Workspace
- Project
- Test Set
- Test Run
- Suite
- Test
- Step
- Log
Projects can be combined into workspaces. A workspace provides a broader overview of your testing efforts. Workspaces can be managed by administrators from the Admin panel.
When a user has access to at least 1 project in a workspace, the entire workspace will be visible to the user. This
includes the overview cards of projects of which the user is not a member.
If the user is a member of a project then the project overview and test results of that project are clickable. Clicking
the card's header will navigate to the project overview.
A project is a collection of test sets that can be executed. A project is also the main construct used to allow users access to certain test results and features. A user can be made a member of any project, allowing the user to view and analyze test results.
Test sets and SUT Components are unique within a project and can be managed through the Project management pages.
A test set is a collection of tests. A test set is created whenever a test run is started with a test set name that is not known a a test set yet. The test run will then be executed as the first run of the named set. Any subsequent runs with the same test set name will be registered as runs of that set.
Test sets can be configured by project managers from the project management page. A display name can be configured and test cases and sut components can be managed.
A test run is a test set execution. Orangebeard tracks test cases within a test run and their history.
A test suite is a 'folder' containing test cases. Within a test run, there is at least 1 (top-level) suite containing the tests. Suites can contain other suites, tests or both.
A test is the representation of a test case that is executed and reported to Orangebeard. A test can contain (nested) test steps, logs or both.
A step can be used to structure reporting. It is a structure item to group logs into logical groups. Steps can be infinitely nested, if that helps you structure your reporting.
A log item is a piece of logging that is relevant to the test report. Logs have a log level (Debug, Info, Warning or Error) and a Format (Plain text, HTML or Markdown).