Test Runner API (V2)

Note

This API is in public preview.

Use this API to run regression tests for solutions and flow. It leverages the previous output of flows referred to as ground truth (also known as golden output) to the test output.

To authorize your request, the Test Runner API requires an Authorization header with the value Bearer XYZ, where XYZ is your access token. See API authorization.

In this document, URL_BASE refers to the root URL of your Instabase instance, such as https://www.instabase.com.

import requests

url_base = "https://www.instabase.com"
test_runner_api_url = url_base + '/api/v2/testrunner'

Execute test suites

Method Syntax
POST URL_BASE/api/v2/testrunner/ibtests/run

Description

Use this API to run a test that’s defined by an .ibtest file.

Request body

The request body is a JSON object containing details about different tests to execute. Each test is a dictionary containing below parameters:

Parameter Type Description Values
ibtest_path string The path of the source .ibtest file. Path of the .ibtest file containing the test definitions.
result_path string The path where the result summary is stored Path where the generated result is created as .json
name string Name of test Ibtest file name

Sample request body:

{
  "tests": [
    {
      "ibtest_path": "user/my-repo/fs/Instabase Drive/files/TestRepo/ClassifyExtractSolution.ibtest",
      "result_path": "user/my-repo/fs/Instabase Drive/files/TestRepo/results/run_1/ClassifyExtractSolution.ibtest.json",
      "name": "ClassifyExtractSolution.ibtest"
    },
    {
      "ibtest_path": "user/my-repo/fs/Instabase Drive/files/TestRepo/ExtractionTest.ibtest",
      "result_path": "user/my-repo/fs/Instabase Drive/files/TestRepo/results/run_1/ExtractionTest.ibtest.json",
      "name": "ExtractionTest.ibtest"
    }
  ]
}

Test specification

Details about the format and specification of the .ibtest file can be found in Test Runner documentation.

Response schema

Key Description
status Status of request, "OK" for successful response.
job_id Job ID of the created test runner job

You can check the status of the test runner job with the test execution status API. The type of the job is async.

Get test execution status

Method Syntax
GET /api/v2/testrunner/status?job_id=<job_id>

Description

Gets the execution status of a test runner job.

Request parameters

Parameter Type Description Values
job_id string job_id of the Test Runner job Job ID of the Test Runner job to get the status for.

Response schema

Key Type Description Value
status string Status of the request OK, ERROR
state string Job state. PENDING, DONE, COMPLETE
msg string Job status message
is_waiting_for_resources string Indicates if the underlying test flow is waiting for scheduler resources false, true
job_id string The unique identifier for the job.
results/metadata string Additional result metadata, this field is empty.
results/result_metadata string JSON string that contains summary and detailed result information for all the tests.
cur_status string JSON string with job id and flow progress.

result_metadata is a JSON string that contains summary and detailed results for all tests in below format:

{
            "name": "ExtractionTest.ibtest",
            "output_summary_dict": {
                "Summary": {
                    "Total Tests": 1,
                    "Successful Tests": 1,
                    "Failed Tests": 0,
                    "Test Duration": 6.1975,
                    "Test Responses": {
                        "New Test Case": {
                            "job_id": "c755d90d-dd9e-4f99-94b8-8a1a6dd69457",
                            "output_folder": "user/my-repo/fs/Instabase Drive/files/TestRepo/output/ExtractFlow"
                        }
                    }
                },
                "Detailed_Information": {
                    "Passed Tests": [
                        "Extraction Test"
                    ],
                    "Failed Tests": [],
                    "Field Level Details for all Tests": [
                        {
                            "Test Name": "Extraction Test",
                            "Errors": null
                        }
                    ]
                },
                "Description": {
                    "Extraction Test": "Test 1"
                }
            },
            "err": null,
            "is_passed": true
        },
        {
            "name": "ClassifyExtractSolution.ibtest",
            "output_summary_dict": {
                "Summary": {
                    "Total Tests": 1,
                    "Successful Tests": 0,
                    "Failed Tests": 1,
                    "Test Duration": 16.4632,
                    "Test Responses": {
                        "ClassifyExtractTest": {
                            "job_id": "f27e47ce-4631-4c7d-899d-a6a2f5b7841e",
                            "output_folder": "user/my-repo/fs/Instabase Drive/files/TestRepo/output/ClassifyFlow"
                        }
                    }
                },
                "Detailed_Information": {
                    "Passed Tests": [],
                    "Failed Tests": [
                        "ClassifyExtractTest"
                    ],
                    "Field Level Details for all Tests": [
                        {
                            "Test Name": "ClassifyExtractTest",
                            "Errors": [
                                {
                                    "Comparison Errors": ["Error Message"]
                                }
                            ]
                        }
                    ]
                },
                "Description": {
                    "ClassifyExtractTest": "Tests a simple multiple document classification extraction flow"
                }
            },
            "err": null,
            "is_passed": false
        }
    ]