Test Runner

The Test Runner app uses test specifications to run regression tests for flows and apps on the Instabase platform. Test Runner leverages the previous output of flows that were successfully run and compares the representative ground truth (also known as the golden output) with the current test output. Test Runner runs the comparison operation using the test specification and creates a comparison report.

Usage guide

Getting started

  1. Open the Test Runner app (All apps > Test Runner).
  2. Click Open tests folder.
  3. Select the folder that contains your test specification files (.ibtest). You can also right-click the folder in the file explorer and select Open with > Test Runner.

    Note: If you haven’t created any test specifications files, select any empty directory.

The test library page opens and displays a list of all .ibtest files in the selected directory.

Note: .ibtest files may be nested inside other folders in the opened directory.

Creating and managing test suites

To create a new test suite and add test cases:

  1. On the test library tab, click + New suite.

  2. Name your test suite.

  3. Click Create test.

  4. Select the test suite in the list.

  5. Click Add new test.

  6. Configure the new test case.

    Note: Click + Add section to add pre-run hooks, post-run hooks, or comparators to your test case.

  7. Click Save.

  8. Add additional test cases as needed.

To view and edit a test suite:

  1. On the test library tab, select the test suite to view.

  2. From the test suite view, you can add, edit, delete, and view test cases in the test suite. See the test specification section for more details.

To permanently delete a test suite from the file system:

  1. On the test library tab, locate the test suite to delete.

  2. Click the delete (trash can) icon.

  3. Click Confirm.

Running test suites and viewing results

  1. On the test library tab, select the checkbox next to the test suite to run. You can select multiple test suites to run at the same time.

    Note: To select all test suites, select the checkbox at the top of the test suite list. You can also run an individual test suite by selecting the test suite on the test library tab and clicking Run suite from the test suite view.

  2. Click Run test. The selected test suites run and Test Runner navigates to the execution history tab. From the execution history tab, you can view the results of your test. Test results save in a new folder named results in the opened directory.

You can also view past run results, test statistics, and errors in the execution history tab.

Using the legacy app:

While the updated Test Runner UI improves the process of creating and maintaining regression test suites, the legacy version is still available. Click View in legacy app to access the legacy version of Test Runner.

Supported APIs

Test type API Description
Flow run_flow_async single Flow async
Flow run_flows_async multi Flows async
Flow Binary run_binary_async single Flow Binary async
Metaflow run_metaflow_async non-binary Metaflow async
Metaflow Binary run_binary_async binary Metaflow async
IBSolution run_solution run solution async

Test specification

The JSON-formatted .ibtest file includes a version and one or more test specifications. Enclose each test specification in square brackets.

For the test to succeed, test specification settings that alter the output of the Flow must match the settings of the Flow that produced the golden outputs.

Settings that do not alter the output of the Flow can be different, like enable tracing.

run_flow_async example test specification

{
  "version" : "1",
  "tests" : [
    {
        "name":"run-flow-test",
        "api":"run_flow_async",
        "payload": {
          "ibflow_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows/sample-3.ibflow",
          "input_folder": "input",
          "compile_and_run_as_binary": true,
          "delete_out_dir":true,
          "output_has_run_id":false,
          "skip_steps":[],
          "log_to_timeline": false
        },
        "on_done_config":
        {
          "failure_expected": false
        },
        "comparison_list": [
          {
            "comparator": "ibocr_comparator",
            "params": {
              "test_output" : [
              "s4_merge_files/out.ibocr"
            ],
              "golden_output": [
              "prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/out-10-02-2020/sample-3/out/s4_merge_files/out.ibocr"
            ],
            "compare_config": {
                    "check_refined_phrases": true,
                    "compare_fields": ["annotation_data.separator"]
                }
            }
          }
        ]
    }
  ]
}

run_binary_async example test specification

{
 "version" : "1",
 "tests" : [
   {
       "name":"run-bin-test",
       "api":"run_binary_async",
       "payload": {
         "binary_path": "/prai/my-repo/fs/Instabase Drive/flow-experiment/build/bin/exercise1_Amazon/exercise1_Amazon-2020-06-08-16:20:26.ibflowbin",
         "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/input amazon",
         "settings": {
           "delete_out_dir": true,
           "disable_step_timeout": true,
           "flow_batch_size": "10",
           "log_to_timeline": true,
           "notification_emails": [],
           "notification_emails_string": "",
           "on_done_webhook_url": "",
           "output_has_run_id": true,
           "runtime_config": {},
           "use_flow_batch_mode": true
         }
       },
       "comparison_list": [
         {
           "comparator": "ibocr_comparator",
           "params": {
             "test_output" : [
             "s4_merge_files/out.ibocr"
             ],
             "golden_output": [
               "prai/my-repo/fs/Instabase Drive/flow-experiment/build/bin/golden-binary/s4_merge_files/out.ibocr"
             ]
           }
         }
       ]
   }
 ]
}

run_flows_async example test specification

{
 "version" : "1",
 "tests" : [
   {
       "name":"run-flows-custom-posthook",
       "owner": "user@instabase.com",
       "api":"run_flows_async",
       "payload": {
         "flow_root_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows/",
         "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/input",
         "compile_and_run_as_binary": true,
         "delete_out_dir":true,
         "log_to_timeline":false,
         "output_has_run_id":false,
         "post_flow_fn": "none"
       },
       "pre_run_hook": {
           "type": "custom",
           "params": {
               "abs_script_path": "prai/my-repo/fs/Instabase Drive/golden-tests/Custom/flow_features/test_delete_output.py",
               "method_name": "log_time_test_runner"
           }
       },
       "comparison_list": [
         {
           "comparator": "ibocr_comparator_easy",
           "params": {
             "compare_files" : [
                 "sample-3/out/s4_merge_files/out.ibocr",
                 "sample-4/out/s4_merge_files/out.ibocr"
             ],
             "golden_output_root": "/prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/out-10-02-2020"
           }
         }
       ]
   }
 ]
}

run_metaflow_async example test specification

 {
  "version" : "1",
  "tests" : [
    {
        "name":"metaflow-test",
        "owner": "user@instabase.com",
        "api":"run_metaflow_async",
        "payload": {
            "metaflow_file_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows2/workflows63d181f1-1d1e-475d-ab29-8d9a4c3471f8.ibmetaflow",
            "classifier_file_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/classifier/classifier_custom.ibclassifier",
            "compile_and_run_as_binary": true,
            "delete_out_dir": false,
            "enable_ibdoc": false,
            "flow_root_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows2/",
            "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/metaflow-input",
            "log_to_timeline": false,
            "output_has_run_id": false,
            "post_flow_fn": "none",
            "skip_process_files": false
        },
        "comparison_list": [
          {
            "comparator": "ibocr_comparator",
            "params": {
              "test_output" : [
              "exercise1_Amazon/out/s4_merge_files/out.ibocr",
              "exercise2_Uber/out/s4_merge_files/out.ibocr",
              "exercise3_Food/out/s4_merge_files/out.ibocr"
            ],
              "golden_output": [
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise1_Amazon/out/s4_merge_files/out.ibocr",
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise2_Uber/out/s4_merge_files/out.ibocr",
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise3_Food/out/s4_merge_files/out.ibocr"
            ],
            "compare_config": {
                    "check_refined_phrases": true,
                    "compare_fields": ["annotation_data.separator"]
                }
            }
          }
        ]
    }
  ]
}

run_solution example test specification

{
    "version": "1",
    "tests": [
        {
            "name": "simple_app",
            "owner": "dummy@instabase.com",
            "api": "run_solution",
            "payload": {
                "solution_path": "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/simple_app.ibsolution",
                "input_folder": "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/inputs",
                "settings": {
                    "delete_out_dir": false,
                    "disable_step_timeout": false,
                    "flow_batch_size": "2",
                    "log_to_timeline": true,
                    "notification_emails": [],
                    "notification_emails_string": "",
                    "on_done_webhook_url": "",
                    "output_has_run_id": true,
                    "runtime_config": {
                        "sample_runtime_key": "sample_runtime_value"
                    },
                    "use_flow_batch_mode": true
                }
            },
            "comparison_list": [
                {
                    "comparator": "file_comparator",
                    "params": {
                        "test_output": [
                            "s1_apply_udf/input_1.txt.text",
                            "s1_apply_udf/input_2.txt.text"
                        ],
                        "golden_output": [
                            "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/out/s1_apply_udf/input_1.txt.text",
                            "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/out/s1_apply_udf/input_2.txt.text"
                        ]
                    }
                }
            ]
        }
    ]
}

Test specification field reference

The test specifications fields depend on the API called and the Flow you are testing. Most fields are described below.

  • version: the version of Test Runner. Default: 1.

  • tests: the test category for one or more named tests

  • name: self-describing name of a test. For example, "run-flow-test".

  • api: a supported API name: run_flow_async, run_flows_async, run_binary_async, run_metaflow_async

  • payload: category that describes the Flow specifications.

  • ibflow_path: path to the Flow or root directory for flows.

  • input_folder: by convention, this folder is "input"

  • compile_and_run_as_binary:

    • true - to compile and run the Flow as Binary.
    • false - to run Binary version of Flow.

  • delete_out_dir: (optional) Delete existing content in the output folder before running the Flow. If not set, default is false.

  • output_has_run_id: (optional) Write output to a directory with a unique, timestamped run_id. If not set, default is false.

  • skip_steps: (optional) A comma-separated list of integers that represent consecutive steps to skip while running a Flow. The steps must be the same as the ground truth (golden output) Flow. For example, to skip 2 consecutive steps at the beginning of the Flow with five steps, "skip_steps": [1,2]. To skip three consecutive steps at the end of the same Flow, "skip_steps": [3,4,5].

  • on_done_config: category that describes the on done configurations

  • failure_expected: true or false, true if flow failure is expected and has to be ignored.

  • post_flow_fn: none or a function name

  • skip_process_files: true or false

  • pre_run_hook: a function that just runs before the flow. Create a Python file with a fixed format as shown in Example: pre-hook function. Call this function by specifying a JSON object with the following fields:

    • type: should be the string "custom"
    • params: A JSON object specifying settings for the customer pre-hook:
      • abs_script_path: absolute path to the scripts directory
      • method_name: registered function name

  • comparison_list: A list of JSON objects specifying different comparisons to run for the test. Each object must have the following format:

    • comparator: type of comparator

      • "file_comparator": compares two JSON, .csv, .txt, or .xls files.
      • "file_comparator_easy": compares two JSON, .csv, .txt, and .xls files when the output directory is the same for all ground truth outputs.
      • "ibocr_comparator": compares two .ibocr files based on the fields specified under compare_fields.
      • "ibocr_comparator_easy": compares two .ibocr files when the output directory is the same for all ground truths (golden outputs).
      • "custom_comparator": use Custom comparison logic

    • params: A JSON object of specifications for the given comparator, which can have the following fields:

      • test_output: a list of all the comparison (new) output files relative to the root folder.
      • golden_output: a list of all ground truth (golden output) files with absolute paths.
      • compare_config: section of optional key-value pairs to configure comparisons.
      • check_refined_phrases (bool): if True, compare refined phrases for the output and ground truth set. By default this is set to True.
      • skip_hidden_phrases (bool): if True, skip comparing refined phrases for hidden fields (fields that start with __). By default this is set to False.
      • check_extracted_tables (bool): if True, the output file will include a section “Table Accuracies By Field” that displays the accuracy for each table in a field’s extracted tables list. By default this is set to False.
      • compare_fields (list[str]): a list of fields in the IBOCR to compare against. Fields must be from the following list:
        • 'text' - the text of an IBOCR record.
        • 'annotation_data.separators' - the record’s annotation data separators.
        • 'classification_payload' - the record’s classification payload, which includes information like the class label, score, and page ranges.

Custom comparison logic

Custom comparators use a registered custom comparator that lets you write your own logic for test. For example, write a custom comparator that checks if a file exists or counts the number of .ibocr files in a folder.

To register a custom comparator:

  1. Create a file with Python script that holds the custom comparator logic.

  2. Register the function with these three arguments: {file_client, logged_in_user, params}

  3. Define results.

    On success return: [True, None] Otherwise, return: [False, error]

Custom comparator field reference

  • params: specifications for custom comparator
  • script_path: absolute path of the file that holds the Python code for custom comparator
  • test_method: the function to execute in your script
  • additional_params: (optional) define other parameters of any name to pass some more information to your comparator logic, these parameters can be consumed by the test_method
from typing import Tuple, Text
from instabase.storage.fileservice import FileService
from instabase.test_runner_utils.common import ComparisonParams
from instabase.content.filehandle import ibfile


def is_rbi_pdf_present(file_client, logged_in_user, params):
  # type: (FileService.Iface, ComparisonParams) -> Tuple[bool, Text]
  file_path = params.comparison_params['additional_params']['output_file_path']
  if ibfile.exists(file_path)
    return True, None
  return False, 'RBI.pdf File not found'

Define pre-hook function

A pre-hook function is a Python script that runs before the Flow.

For Test Runner, register the pre-hook function using a specific structure.

  • Create a class that inherits PreHookBase class
  • Provide an implementation for the ‘execute’ method of this class
  • Register a function that calls this execute method internally
  • Give the name of this method as a parameter of pre-hook in the test specification configuration as a part of a test.
  • The execute method must have the return type as Tuple[bool, Text].
    • If the core logic inside execute succeeds, then return [True, None]
    • If the logic is unsuccessful, return [False, error_message]

Example pre-hook function

This example shows the required format of the Python script for a pre-hook function.

from instabase.test_runner_utils.common import PreHookParams, PreHookBase, PreHookContext

class ClassName(PreHookBase):
  def __init__(self, context):
    super(ClassName, self).__init__(context)


  def execute(self):
    # type: () -> Tuple[bool, Text]

    # utilities to write pre hook function
    file_client = self.context.file_client
    logged_in_user = self.context.logged_in_user

    # your business logic for pre_run_hook here
    # return [True, None] if succeeded otherwise return [False, error_message]

def function_name(file_client, logged_in_user):
  context = PreHookContext(file_client, logged_in_user)
  prehook_base = ClassName(context)
  return prehook_base.execute()

To register a prehook, add the type and parameters to the test specification:

"pre_run_hook": {
            "type": "custom",
            "params": {
                "abs_script_path": "absolute path to the python script",
                "method_name": "function_name"
            }
        }

To open the Test Runner app

From the All Apps view:

  1. Open the workspace that contains the Flow you want to test.
  2. In the left navigation dock, click All apps, and then select Test Runner.

From the file browser:

  1. Open the workspace that contains the Flow you want to test.
  2. In the file browser, click the three dots next to the tests folder and then select Open With > Test Runner.

To create a test

Select File > New Test. Then, name your test. By convention, the test name is the name of the Flow.

View results

After you write and save the test specification, click Tools > Run to run the test.

The selected test specification .ibtest is run.

The Flow defined in the test is run, then the list of defined validations is run against the results of the Flow. The Flow executes as usual and generates output to the output folder that is specified for the Flow.

Note: The Flow always writes to a unique folder with a run ID.

If the test runs and succeeds, a success message along with the statistics of the test is presented in the UI with a link to the results directory. In general, the result of a test is written to: <current_test_dir>/out/default/<timestamp>/<run_id>/.

The output directory contains a test results file with extension .ibtestresult. This JSON-formatted results file contains the overall result of the test with details of the individual validations.

A single Flow successful test result looks like:

{
  "Summary": {
    "Total Tests": 1,
    "Successful Tests": 1,
    "Failed Tests": 0
  },
  "Detailed_Information": {
    "Passed Tests": [
      "test_invoice"
    ],
    "Failed Tests": [],
    "Field Level Details for all Tests": [
      {
        "Test Name": "test_invoice",
        "Fields Accuracy": {
          "Individual Field Accuracy": {
            "Country": 100.0,
            "DOC_TYPE": 100.0,
            "PersonNames": 100.0,
            "__FILENAME": 100.0,
            "__entities": 100.0,
            "__nlp_persons": 100.0,
            "__person_names": 100.0,
            "locations": 100.0
          },
          "Aggregate Field Accuracy": 100.0
        }
      }
    ]
  }
}

A test is marked as passed if all the validations passed.

An unsuccessful single Flow Binary test result looks like:

{
  "Summary": {
    "Total Tests": 1,
    "Successful Tests": 0,
    "Failed Tests": 1
  },
  "Detailed_Information": {
    "Passed Tests": [],
    "Failed Tests": [
      "post-metaflow-with-udf"
    ],
    "Field Level Details for all Tests": [
      {
        "Test Name": "post-metaflow-with-udf",
        "Statistics": [
          {
            "Individual Field Accuracy": {
              "__FILENAME": 100.0,
              "__helper_for_items": 100.0,
              "items": 100.0,
              "payment_type": 100.0,
              "total_paid": 100.0,
              "paystub_type": 100.0,
              "total": 100.0,
              "__helper_for_payment_type": 100.0
            },
            "Aggregate Field Accuracy": 100.0,
            "Field Mismatch Information": []
          },
          {
            "Successful File Comparisons": [
              {
                "Base File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/class_output_folders_new.json",
                "Comparison File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/class_output_folders_new.json"
              }
            ],
            "Failed File Comparisons": [
              {
                "Base File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/root_output_folder.txt",
                "Comparison File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/root_output_folder.txt"
              }
            ]
          }
        ],
        "Errors": [
          {
            "Comparison Errors": [
              "Files not equal:\nGolden path: ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/root_output_folder.txt\nOutput path: ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/root_output_folder.txt\nOutput content:\nib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out\nGolden content:\nib_eng_dne/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out"
            ]
          }
        ]
      }
    ]
  }
}

If the test does not pass, a Flow Binary is not generated.