Upgrade API

Instabase supports performing platform upgrades by API. This article outlines the required and optional Upgrade API endpoints used to perform a platform upgrade.

Before upgrading

Before upgrading, you should:

• Review the release notes for the release. The deployment guide section of the release notes lists steps you must complete before and after upgrading.

• Obtain the release bundle for the release, which is typically provided by your technical account manager and called installation.zip. The bundle includes the base configurations for the release, and, optionally, any default patches or custom network policies.

  Info

  The release version of your new base configurations file must be greater than the release version of your current base configurations. You cannot downgrade to a previous release through Deployment Manager.

• Create a test plan for how to assess the effect of the upgrade on your deployment. Examples of activities to perform as part of testing include running a flow, clicking through the UI, training a model, or running any tests set up in the Test Runner app.

• (Optional) If network policies are enabled on your cluster, apply the latest Deployment Manager network policies file (control-plane-np.yml) included in your release bundle:

  Note

  While optional, this step should be done before applying the latest Deployment Manager configuration file.

  1. Unzip the installation.zip file for the new release.

  2. On the command line, navigate to the unzipped installation folder.

  3. Apply the new Deployment Manager network policies file contained within by running the following command: kubectl -n $IB_NS apply -f control-plane/control-plane-np.yml, where $IB_NS is your Instabase namespace.

• Ensure you’re running an up-to-date and compatible version of Deployment Manager. Your Deployment Manager version, which can be found on the Version tab of Deployment Manager, must match the release version you are upgrading to.

  The best way to ensure compatibility is to update Deployment Manager by applying the latest Deployment Manager configuration file (control-plane.yml) included in your release bundle:

  1. Unzip the installation.zip file for the new release.

  2. On the command line, navigate to the unzipped installation folder.

  3. Apply the new Deployment Manager configuration file contained within by running the following command: kubectl -n $IB_NS apply -f control-plane/control-plane.yml, where $IB_NS is your Instabase namespace.

  Warning

  If your current control-plane.yml file is modified, such as including the INSTABASE_BACKEND_DB_PARAMS variable to support manual database setup, before applying the new release’s control-plane.yml file, copy any modifications over from the previous file to the new one.

• If upgrading from release 22.08 or earlier to release 23.04 or later, before starting the upgrade process, you must change the default value of the ENABLE_CONTROL_PLANE_UPGRADES_ROLLBACK variable to false. If you don’t, the upgrade will fail. See the deployment guide section of the target release’s release notes for instructions.

Overview of required and optional upgrade steps

To complete an API-based upgrade, it’s recommended to complete the following API calls in this order. Calls are required unless noted as optional.

1. (Optional, recommended) Perform preflight checks

2. Create upgrade

   • (Optional) Get upgrade

3. (Optional, recommended) Validate upgrade

4. (Optional, recommended) Diff upgrade

   • (Optional) Summarize upgrade diff

5. Start upgrade

6. (Optional, recommended) Get upgrade progress

7. (Optional, recommended) Stream upgrade events or Get upgrade events

8. Complete upgrade

In addition to the calls used to perform an upgrade, there are several optional endpoints you can call to manage the progression of your upgrade, including pausing or halting an upgrade. While optional, these endpoints might become required if an upgrade error occurs, such as a failed stage or failed upgrade.

• Pause upgrade

• Retry upgrade

• Continue upgrade

• Roll back upgrade

• Abandon upgrade

Conventions and authentication

In this document api_root defines where to route API requests for your Instabase instance and includes the stem of the API path.

import json, requests

api_root = 'https://www.instabase.com/api/v1/control_plane'

In this document api_token defines the Deployment Manager token. All API requests must include your valid Deployment Manager token to authenticate the request. The Deployment Manager API token is created in the cluster with a default, changeable value when the Deployment Manager YAML is applied.

To fetch the Deployment Manager API token, run the following command, where $IB_NS is your Instabase namespace:

export TOKEN=$(kubectl -n $IB_NS get \
secret/control-plane-metadata \
--template={{.data.api_token}} | base64 -d)

You can then reference the token in your calls using $TOKEN in place of api_token.

Preflight checks

Description

Use this API to run preflight checks. These checks are optional but recommended, as they help verify that your infrastructure is set up correctly and is accessible. The following checks are available:

• Role-based access control (RBAC) check: The rbac check reports back any inconsistencies between permissions the Deployment Manager service account requires and permissions it has been granted. A success message indicates the Kubernetes role resource associated with the Deployment Manager service account matches the permissions Deployment Manager requires. A failure message shows the expected configuration alongside the configuration found in the cluster.
• Service accounts check: The service_accounts check confirms the service account used by the Instabase observability monitoring services exists in the Kubernetes cluster.

You must call this endpoint for each preflight check, changing the {id} value in the URL path for each call.

Request parameters

The following parameters are required, unless marked as optional:

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/installer/preflight/checks/rbac", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If the request can be performed and the check meets all requirements:

{
   "statusCode":200,
   "data":{
      "note":"RBAC properly configured.",
      "ok":true
   },
   "error":null
}

If the request can be performed but the check does not meet all requirements:

{
    "statusCode":200,
    "data": {
        "note":"Monitoring service account not in namespace.",
        "ok":false
    },
    "error":null
}

If the check cannot be performed:

{
    "statusCode":500,
    "data": {
        "note": "",
        "ok": false
    },
    "error":"http conn serve: http conn serve: roles.rbac.authorization.k8s.io \"control-plane-role\" not found"
}

Create upgrade

Description

Use this API to create an upgrade resource. The upgrade resource includes all new configurations to use in the upgrade and is what is referenced in subsequent upgrade-related API calls. For example, when making the Validate upgrade call, the validation runs on the upgrade resource last created through the Create upgrade endpoint.

In addition to uploading new configurations, this call is used to set a resourcing_size value, where you can optionally update the Instabase-suggested replica count by defining a preset cluster size. If you define a new cluster size, ensure the selected option meets but does not exceed your Kubernetes cluster size. After upgrading, you can manually adjust the Instabase cluster size configuration using patches to more closely match your Kubernetes cluster size. If you are unsure of your cluster sizing, connect with your technical account manager for guidance or do not define a resourcing_size value to maintain your current settings.

Request parameters

The following parameters are required, unless marked as optional:

Defining deprecated configs to keep

Your deployment might include configs not found by default within the uploaded ZIP files, such as a control-plane-db-backend-params config if manually defining database parameters. Configs not included in the uploaded files are deprecated and not included in the upgrade resource unless explicitly listed in the deprecated_configs_to_keep array. To keep an existing config which would otherwise be deprecated by the upgrade, add its name to the array and include it in the Create upgrade call.

The following list includes common Instabase installer-generated configs which might be part of your configuration but not included in the upgrade resource by default:

'deprecated_configs_to_keep': ['control-plane-db-backend-params', 
  'aws-keys-storage', 
  'db-tls-metadata', 
  'google-private-key', 
  'google-cloud-key', 
  'aws-config'
  ]

Before completing the Create upgrade call, check the Deployment Manager UI to confirm if your installation includes the listed configs or other custom configs that must be preserved. If any of the listed configs are not present in your configuration, remove them from the deprecated_configs_to_keep list. If any configs are missing, add them to the list and include them in the call.

Response schema

All keys are returned in the response by default, unless marked as optional.

Examples

Request

Example (Python):

import requests
files = {
  'base_configs': open({base_config_zip_file_path}, 'rb'),
  'default_patches': open({patches_zip_file_path}, 'rb'),
  'network_policies': open({network_policies_file_path}, 'rb')
}
data = {
  'deprecated_configs_to_keep': [
      'control-plane-db-backend-params',
  ],
  'resourcing_size': ''
}
headers = { 'Token': api_token }
try:
  resp = requests.post(
      f"api_root/product/upgrade/upload",
      headers=headers,
      files=files,
      data=data)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
    "statusCode": 200,
    "data": null,
    "error": null
}

If an upgrade resource (that has not been abandoned) already exists:

{
   "statusCode":500,
   "data":null,
   "error":"..."
}

Get upgrade

Description

Use this API to retrieve the upgrade resource most recently uploaded through the Create upgrade call. This optional API can be used to review the upgrade resource.

Request parameters

No request parameters are accepted.

Response schema

{
   "upgrade":{
      "configs":[
         {
            "content":"<content-of-config>",
            "fileName":"<name-of-file>"
         },
      ],
      "patches":[
         {
            "content":"<content-of-patch>",
            "fileName":"<name-of-file>"
         }
      ],
      "deprecatedConfigsToKeep":[
         "control-plane-db-backend-params"
      ],
      "manifest":{
         "images":{
            "<name-of-image>":"<image-identifier>",
         },
         "releaseVersion":"<release-version>"
      },
      "creatorUsername":"<username>",
      "resourcingSize":"<resourcing-size>"
   }
}

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.get(f"api_root/product/upgrade", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
   "upgrade":{
      "configs":[
         {
            "content":"apiVersion: batch/v1\nkind: Job\nmetadata:\n...",
            "fileName":"dbcheckpoint.yml"
         },
      ],
      "patches":[
         {
            "content":"# target: webapp-ingress\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy...",
            "fileName":"net-pol-patch.yml"
         }
      ],
      "deprecatedConfigsToKeep":[
         "control-plane-db-backend-params"
      ],
      "manifest":{
         "images":{
            "instabase-artifacts":"instabase-artifacts:23.04.3-a5e1192912",
            "onprem-dbupdate":"onprem-dbupdate:23.04.3-a5e1192912",
            "solution-diagnostics-artifacts":"solution-diagnostics-artifacts:23.04.3-a5e1192912"
         },
         "releaseVersion":"23.04.3"
      },
      "creatorUsername":"Admin",
      "resourcingSize":"None"
   }
}

Validate upgrade

Description

Use this API to validate the current upgrade resource created using the Create upgrade API. Validation verifies that necessary files have been uploaded to the upgrade resource and attempts a dry run of applying the upgrade. If any issues arise, validation errors are returned.

Request parameters

No request parameters are accepted.

Response schema

The response contains a list of validation errors, each listing the name of the affected config and the error which was found. The structure is of the form:

{
"validationErrors": [
    {
        "configName": "<name-of-the-config>",
        "error": "<description-of-the-problem>"
    },
]
}

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
    resp = requests.get(f"api_root/product/upgrade/validation", headers=headers)
    print(resp.content)
except requests.exceptions.RequestException as err:
    print(f"Error: {err}")

Response

If no validation errors are found:

{"validationErrors":[]}

If validation errors are found:

{
   "validationErrors":[
      {
         "configName":"Database Migration",
         "error":"Could not find job-dbmigrate in base configs."
      }
   ]
}

Diff upgrade

Description

Use this API to generate a diff of your current configurations against the upgrade resource created with the Create upgrade call. The diff shows all configuration changes that will occur when the new upgrade resource is applied, including specific line-by-line changes to altered configs. For a summarized version of the diff, use the Summarize upgrade diff call.

Request parameters

No request parameters are accepted.

Response schema

The response contains a map of changed configs and new configs, each keyed by config name.

The structure is:

{
  "diffs": {
    "control-plane-db-backend-params":{
      "added":[
        {
          "lineNumber":<line-number>,
          "data":"<line-content>"
        }
      ],
      "deleted":[
        {
          "lineNumber":<line-number>,
          "data":"<line-content>"
        }
      ]
    },
  },
  "configs": {
    "<name-of-new-config>": "<content-of-new-config>",
  },
  "includedObjectNames":[
    "<name-of-object-included-post-upgrade>",
  ],
  "deletedObjectNames":[
    "name-of-object-removed-after-upgrade",
  ],
  "deprecationOverriddenObjectNames":[
     "name-of-object-preserved-post-upgrade"
  ]
}

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.get(f"api_root/product/upgrade/diff", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
   "diffs":{
      "control-plane-db-backend-params":{
         "added":[
            {
               "lineNumber":2,
               "data":"B8ZL97ziZxUeeybP0233hATEPFSsYg0RH3jOOoKnlgNZhifiP5GNkAMkjOZCBPfKFzNPePP..."
            }
         ],
         "deleted":[
            {
               "lineNumber":1,
               "data":"hkd0A29FGd0GCS7gFa+0RF1KkQpmR3mqUTM46+KqS9BlxJayEi/y8goVYe5y5BDMERhaghh..."
            }
         ]
      },
      "job-dbmigrate":{
         "added":[
            
         ],
         "deleted":[
            
         ]
      },
   },
   "configs":{
      "control-plane-db-backend-params":"B8ZL97ziZxUeeybP0233hATEPFSsYg0RH3jOOoKnlgNZhifiP5GNk...",
      "job-dbmigrate":"apiVersion: batch/v1\\nkind: Job\\nmetadata:\\n  name: job-dbmigrate\\n  namespace: instabase\\n...",
   },
   "includedObjectNames":[
      "control-plane-db-backend-params",
      "job-dbmigrate"
   ],
   "deletedObjectNames":[
      "webapp-ingress"
   ],
   "deprecationOverriddenObjectNames":[
      "control-plane-db-backend-params"
   ]
}

Summarize upgrade diff

Description

Use this API to generate the summary of the diff of current upgrade. Unlike the Diff upgrade response, the summary does not include line-by-line changes to altered configs.

Request parameters

No request parameters are accepted.

Response schema

The structure is:

{
   "includedObjectNames":[
      "<name-of-config-which-will-be-active-post-upgrade>"
   ],
   "deletedObjectNames":[
      "<name-of-config-which-will-be-deleted-post-upgrade>"
   ],
   "deprecationOverriddenObjectNames":[
      "<name-of-config-which-would-have-been-deleted>"
   ]
}

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.get(f"api_root/product/upgrade/diff/summary", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
   "includedObjectNames":[
      "control-plane-db-backend-params",
      "job-dbcheckpoint",
      "job-dbmigrate",
      "job-dbrollback"
   ],
   "deletedObjectNames":[
      
   ],
   "deprecationOverriddenObjectNames":[
      "control-plane-db-backend-params"
   ]
}

Start upgrade

Description

Use this API to start the current upgrade, using the latest upgrade resource created with the Create upgrade call.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/start", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful, an empty response is returned.

If unsuccessful:

{
   "code":13,
   "message":"StartUpgrade failed: updateState failed: Start failed: already started",
   "details":[
      
   ]
}

Get upgrade progress

Description

Use this API to check the progress and current stage of the upgrade. While this call is optional, it is helpful to determine the upgrade’s progress before completing a subsequent step in the upgrade process. For example, before calling the Complete upgrade endpoint, the upgrade progress must be at the CONFIRM_COMPLETION stage.

The upgrade stages typically progress in the following order (value returned under the current_stage_name field):

• CONFIRM_START
• TURN_DOWN
• MIGRATE_DB
• WRITE_NEW_CONFIGS
• UNDO_TURN_DOWN
• RUN_ACTIONS
• CHECK_HEALTH
• CONFIRM_COMPLETION
• COMPLETED

However, if a rollback is requested, the stage can revert to a previous one. And, if a rollback or abandon upgrade request is made, the stage can become ROLLEDBACK or ABANDONED.

Request parameters

No request parameters are accepted.

Response schema

The response contains the status of which stage of the upgrade process the upgrade is currently in and any errors, if present.

The structure of the response is:

{
    "upgradeProgress": {
      "started": <boolean>,
      "rollback_requested": <boolean>,
      "paused": <boolean>,
      "failed_with_error": {
          "stage": "<stage-at-which-failure-occurred>",
          "message": "<description-of-error>",
          "timestamp_ms": <milliseconds-since-unit-epoch>,
      },
      "current_stage_name": "<name-of-current-stage>",
    }
}

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.get(f"api_root/product/upgrade/progress", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
   "upgradeProgress":{
      "validated":false,
      "started":true,
      "rollbackRequested":false,
      "paused":false,
      "failedWithError":null,
      "currentStageName":"ABANDONED"
   }
}

If no upgrade has been created:

{
   "code":5,
}

Stream upgrade events

Description

Use this API to stream events. Events show logs for each stage and the status of each stage.

Request parameters

No request parameters are accepted.

Response schema

The response contains a stream of events. The structure of each response in the stream is:

{
   "events": [
      {
         "name": <number-of-the-stage>,
         "status": <number-of-the-status>,
         "logs": [
             {
                "msg": "<log-message>",
                "timestamp_ms": <milliseconds-since-unit-epoch>,
                "logLevel": <number-of-thelog-level>,
             }
          ],
      },
    ]
}

In the response, each stage name (name), stage status (status), and log level (logLevel) has a corresponding number. When reading the response, use the associated number to understand the stage name, stage status, or log level values.

The possible stage name values and associated meanings are:

• 0: CONFIRM_START
• 1: TURN_DOWN
• 2: MIGRATE_DB
• 3: WRITE_NEW_CONFIGS
• 4: UNDO_TURN_DOWN
• 5: RUN_ACTIONS
• 6: CHECK_HEALTH
• 7:CONFIRM_COMPLETION
• 8: COMPLETED
• 9: ROLLEDBACK
• 10: ABANDONED

The possible stage status values and associated meanings are:

• 1: UNSTARTED
• 2: UNDERWAY
• 3: COMPLETED
• 4: ROLLEDBACK
• 5: FAILED

The possible logLevel values and associated meanings are:

• 1: INFO
• 2: ERROR
• 3: WARNING
• 4: SUCCESS

Examples

Request

Example (Python):

import asyncio
from websockets.sync.client import connect
headers = { 'Token': api_token }
with connect(endpoint, additional_headers=headers) as websocket:
  while True:
    message = websocket.recv()
    print(f"Received: {message}")

Response

{
   "events":[
      {
         "stage":{
            "name":1,
            "status":2,
            "logs":[
               {
                  "msg":"Scaling down Deployments",
                  "timestamp_ms":1686245366195,
                  "logLevel":1
               }
            ]
         }
      }
   ]
}

Get upgrade events

Description

This API is functionally identical to the Stream upgrade events API, but does not require use of a websocket.

Request parameters

No request parameters are accepted.

Response schema

Identical to the Stream upgrade events response schema. See that API for details.

Examples

Request

import requests
headers = { 'Token': token }
try:
  resp = requests.get(f"{api_root}/product/upgrade/events/stages", headers=headers)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Complete upgrade

Description

When the upgrade reaches the CONFIRM_COMPLETION stage (see Get upgrade progress), use this API to complete the upgrade.

Before confirming completion:

• Test the effect of the upgrade on your environment according to your test plan.
• Check the upgrade event logs using the Stream upgrade events endpoint.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/complete", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Pause upgrade

Description

Use this API to pause the current upgrade. When a pause is requested, the upgrade process does not necessarily instantly stop. If upgrade tasks are underway when the request to pause is received, those tasks can continue until a checkpoint is reached, such as completing the stage or encountering an error.

To resume a paused upgrade, make a request to retry using the Retry upgrade endpoint.

To check whether an upgrade stage is still underway or has been paused, use the Stream upgrade events endpoint.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/pause", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Retry upgrade

Description

Use this API to resume or retry the current upgrade. If an upgrade was paused without any errors, the Retry upgrade call resumes the upgrade and the upgrade proceeds with any remaining stages. If the upgrade is in a failed state, the current, failed stage is retried.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/retry", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Continue upgrade

Description

Use this API to force continue the upgrade even if the current stage has stalled or resulted in an error. Before force continuing an upgrade, first retry the failed stage. If the retry fails, inspect all event logs ensure the source of error is known and acceptable before continuing.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/continue", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Roll back upgrade

Description

Use this API to roll back the current upgrade. Rolling back an upgrade attempts to return the deployment to the state it was in before the upgrade started. This process does not account for any patches or configs added during the upgrade.

Request parameters

No request parameters are accepted.

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': api_token }
try:
  resp = requests.post(f"api_root/product/upgrade/rollback", headers=headers)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Abandon upgrade

Description

Use this API to discard the current upgrade. This API is used in two scenarios:

• To discard a created upgrade resource, when the upgrade process has not yet started. Abandoning an upgrade after completing a Create upgrade call but before completing a Start upgrade call has no effect on your deployment.
• Abandoning an upgrade in progress after a rollback fails. Abandoning an in-progress upgrade should be considered an emergency action.

Request parameters

Response schema

The response has no fields.

Examples

Request

Example (Python):

import requests
headers = { 'Token': token }
data = {
  'checkUnstarted': True,
}
try:
  resp = requests.post(f"{api_root}/product/upgrade/abandon", headers=headers, data=data)
  print(resp.content)
except requests.exceptions.RequestException as err:
  print(f"Error: {err}")

Response

If successful:

{
}

If unsuccessful as a result of making an Abandon upgrade request when an upgrade has started and checkUnstarted is set to True:

{
   "code": 9,
}