Backup & Recovery#

The following guide describes the process of performing backup and restore operations.

Note

Please contact your Intel® Geti™ account representative or technical support personnel if you have any questions.

Warning

The restore procedure is not intended for uninstall / install but for restoring the projects data in case of data corruption because of some critical error.

On-Prem Server Backup & Recovery#

Below, you will find the process of performing backup/restore operations for Intel Geti On-premises servers.

We are proposing two strategies for backing up On-Prem Server before major upgrade to new OS version or new Intel Geti version.

  • Strategy 1: Backup the Entire Server Disk

  • Strategy 2: Backup Intel Geti Data Folder

Strategy 1: Backup the Entire Server Disk#

The customer can use any preferred Linux tools for performing backup/restore operations.

Linux backup/restore tools examples:

  • Terminal tools: dd, rsync

  • UI based tools: Gnome disk or Time shift

Note

Backup snapshots should be saved on an external disk.

Note

It is better to avoid using the server during the backup/restore operations. We recommend booting to the machine using ubuntu live session for doing the backup/restore operations.

Warning

We encourage the customers to implement a disaster recovery plan for Intel® Geti™ servers. Customers can implement this easily by taking a regular snapshot from the whole disk, the customer can take daily, or weekly snapshot based on their preferred policy, also they should consider taking the snapshot before the major changes like an OS update or Intel® Geti™ platform update.

Strategy 2: Backup Intel® Geti™ Data Folder#

Please connect an external SSD to your server with enough space before scheduling a backup of disk image.

Ensure that there is no training or inference running on the platform.

Then, on the server, open a terminal and execute the following steps.

Step 1: Run the commands below to extract database usernames and passwords:#

user@1233:~$ sudo kubectl -n impt get secret impt-mongodb --output json
user@1233:~$ sudo kubectl -n impt-jobs-production get secret impt-mongodb --output json

The result should look like the following:

{
   "apiVersion": "v1",
   "data": {
       "dataset-ie-mongodb-password": "UGRFbFltd0JBNER0",
       "dataset-ie-mongodb-username": "VFpPUXZW",
       "director-mongodb-password": "dG1WdGttNVBTS002",
       "director-mongodb-username": "WXRMWUtE",
       "gateway-mongodb-password": "RGM5R0pnVjducHpR",
       "gateway-mongodb-username": "RWlNWEVS",
       "inference-job-mongodb-password": "cXVLNjRzdGdZZzdk",
       "inference-job-mongodb-username": "WVN0UldC",
       "inference-operator-mongodb-password": "NHY4R3V6RnpQTEpz",
       "inference-operator-mongodb-username": "eEV5amxh",
       "inference-server-mongodb-password": "NHVCckx6VUZja08w",
       "inference-server-mongodb-username": "bERSaVdD",
       "jobs-mongodb-password": "ZERqdVJyb01qNUpB",
       "jobs-mongodb-username": "S3VITm9J",
       "jobs-ms-mongodb-password": "SXNxbUk1cUdrZTFC",
       "jobs-ms-mongodb-username": "Umd2cG1M",
       "jobs-scheduler-mongodb-password": "SkRlZXh0NzVXTVVz",
       "jobs-scheduler-mongodb-username": "YWpPT3J3",
       "mongodb-password": "OVBDQktWODJIYWZ4",
       "mongodb-username": "aFd3Qm15",
       "project-ie-mongodb-password": "QlRpcU95ZXEwTkFX",
       "project-ie-mongodb-username": "bU1RelpM",
       "resource-mongodb-password": "OXUzUk1PNE9WQ2Y4",
       "resource-mongodb-username": "YktJQVpE",
       "spice-db-mongodb-password": "eEhlMkNvd1dvMHds",
       "spice-db-mongodb-username": "YWJkU2Fl",
       "training-operator-mongodb-password": "OVFRTEhhSWs2cGky",
       "training-operator-mongodb-username": "Y1lyYlhB",
       "workload-configuration-mongodb-password": "M01ScHhvR0t0ZElL",
       "workload-configuration-mongodb-username": "Y0tTa3VR"
   }
}

user@1233:~$ sudo kubectl -n impt get secret impt-postgresql --output json
user@1233:~$ sudo kubectl -n flyte get secret impt-postgresql --output json

The result should look like the following:

{
   "apiVersion": "v1",
   "data": {
       "accountservice-postgresql-db": "YWNjb3VudHNlcnZpY2U=",
       "accountservice-postgresql-password": "dUl6aTdzU2wyYUl2",
       "accountservice-postgresql-username": "aWRma3pv",
       "creditsystem-postgresql-db": "Y3JlZGl0c3lzdGVt",
       "creditsystem-postgresql-password": "Y3JlZGl0c3lzdGVtX3Bhc3N3b3Jk",
       "creditsystem-postgresql-username": "Y3JlZGl0c3lzdGVt",
       "flyte-postgresql-db": "Zmx5dGU=",
       "flyte-postgresql-password": "V3V1MnNvZ3J0bGd6",
       "flyte-postgresql-username": "dHh2YnNl",
       "postgresql-db": "c3BpY2VkYg==",
       "postgresql-password": "UFdyV2NvZUlEZ0s2",
       "postgresql-username": "ZGdKRUpXcmVKdnla",
       "spicedb-postgresql-password": "YzQzQTljamE2V1Uw",
       "spicedb-postgresql-username": "dGdkcXp3"
   }
}

user@1233:~$ sudo kubectl -n impt get secret impt-spice-db --output json
user@1233:~$ sudo kubectl -n impt-jobs-production get secret impt-spice-db --output json

The result should look like the following:

{
   "apiVersion": "v1",
   "data": {
       "SPICEDB_GRPC_PRESHARED_KEY": "elpTdDhjU2hrbHBN"
   }
}

user@1233:~$ sudo kubectl -n impt get secret impt-ldap-service-user --output json

The result should look like the following:

{
   "apiVersion": "v1",
   "data": {
       "config_password": "NEVqNVF3eGRBcU4y",
       "config_username": "WUdRUlJ5",
       "password": "NVdnb3FuaFRWb1Jw",
       "readonly_password": "ODZ3ckNjczBWZFU5",
       "readonly_username": "VVBCeFJN",
       "username": "UHRZcVNX"
   }
}

user@1233:~$ sudo kubectl -n impt get secret impt-seaweed-fs --output json
user@1233:~$ sudo kubectl -n impt-jobs-production get secret impt-seaweed-fs --output json

The result should look like the following:

{
   "apiVersion": "v1",
   "data": {
       "accountservice_access_key": "OFJWZmI1b25wUk44N0Z3N2JST2E=",
       "accountservice_presigned_url_access_key": "dkxzMVM0b1pXc2xKSkFyRTNTaUU=",
       "accountservice_presigned_url_secret_key": "blpTc0sySVd0MmpaWVVXbk1hR2YyMEVTNnZpcFpLZGxwVkJDQlRiSA==",
       "accountservice_secret_key": "b0xJc2llWnZWWlJBTTFPOEpoSGFEb011Qk1RWkdUbW5nbGlBTUx2Qg==",
       "admin_access_key": "ekZLU1dhbVUyVkRDN3hZS3pZeHg=",
       "admin_secret_key": "TkRmWk0xODB5NXk4SHFpakhOYUl2NDVnRzlIbzNNa1Bxb3RISXBwRA==",
       "auto_train_controller_access_key": "b2J3bDkzQ0FLV0dJdkxlOFNQb1E=",
       "auto_train_controller_secret_key": "TGhyM0VqOXVCM25rTXdaNlh3R3pBYkxXQmI2TTBkbzhMc3lxWHNiZw==",
       "dataset_ie_access_key": "enBGams1OHFnSDdiNE8zcFhHNlQ=",
       "dataset_ie_s3_presigned_url_access_key": "UkFqc2NMalRFVk9UTVloUjBZNDY=",
       "dataset_ie_s3_presigned_url_secret_key": "MkZXTlRtdGpJdkprODBxU1Rqbzg1MmpaYmhJRlkyTUJMY0loMmZHVA==",
       "dataset_ie_secret_key": "c0NIMFczTjF1eW81QnBqYlA2ODlSVjNzaUNYb0hvZEtsbHFSSEtPVQ==",
       "director_access_key": "Z0hOVXFQODhzRVY3SUE1U3FyaXI=",
       "director_s3_presigned_url_access_key": "SXdFMGxsazdGMm9Sb0o2QzF5REg=",
       "director_s3_presigned_url_secret_key": "c0FRMTZvclJhc2JHMjZvaEkzMFcyOHRLWFZyUmtaUmVrUzdvQkdqNA==",
       "director_secret_key": "SWp2SjYzZjFRRlJJdDBFZ2ZoZXowVHY3OHIxaGRtS21MRmxOYnVBag==",
       "flyte_access_key": "Znc0TU1GZ2xuMmNmUW1vcmdOclM=",
       "flyte_secret_key": "Qm81RnI4SDBlWDg1dDVPN2dodkY2WmNNRnQ5cUNEUHRBd2RLWDQ0dw==",
       "flyte_workflows_access_key": "bVJnMXpYcTdGZkU5N0xYbVdTQlE=",
       "flyte_workflows_s3_presigned_url_access_key": "bEJ3a0tBdGZ1Q0VxRFVVOE1hcTY=",
       "flyte_workflows_s3_presigned_url_secret_key": "VGtyVkJiV0ZSWmt5ellXcTB0MHlXMkViOE5YRmNNUnVoZmh5aGVlYg==",
       "flyte_workflows_secret_key": "WjNmSXdmSkVibXp0aVhCYVBhUEltR1RpbkNiaVFabUZBZTVkQ1NSSg==",
       "gateway_access_key": "bFlhcGRTNnRWUHBEd2FvSzRHaUE=",
       "gateway_secret_key": "ZkkyQm16Q3dJNlZTTFBRblJZZVQ0UEJNTjVVWHJNSUpWcVB6TTcxVg==",
       "inference_gateway_access_key": "MWxMSVVLQXhqaFlkQWdwNFRERnI=",
       "inference_gateway_s3_presigned_url_access_key": "SGt1YkVrVFpkY3F6U3N5N3IyRDE=",
       "inference_gateway_s3_presigned_url_secret_key": "UGdmbVd4TXZkVU9uUnFYQ2NQcnYxMDFIajJLd0xTU1dWRU52dzlHeg==",
       "inference_gateway_secret_key": "NGJXQ2ZxY01nOHoyOURTcUo2MnZaNktQSXFScm5ZQ3lVamZEeDlDbQ==",
       "inference_operator_access_key": "bGRHZ3FjQzRDR3BwQ2ZRelNHanQ=",
       "inference_operator_s3_presigned_url_access_key": "Z2ViZXpjWU9DczhGN3Q4bEoxOHE=",
       "inference_operator_s3_presigned_url_secret_key": "ektWV3ZMRmRzcWphd0hSbm53UHFKRGFYYTN5ZUdGN005Zjcyd0VjZg==",
       "inference_operator_secret_key": "R2ZVYmx0cnVHZVd2amJRTFlTQ1o0WGNDN3NJUUw1dVQ2MUVsVUlvMg==",
       "jobs_ms_access_key": "TFBFQzU0UFdxd0RNUVFLWURHTGM=",
       "jobs_ms_s3_presigned_url_access_key": "ZGlDd2l3dFZXSDFvZ040enRZRDE=",
       "jobs_ms_s3_presigned_url_secret_key": "ZG80czJuSkE3bEJOdU45NkY4NW9kTEpBbGJyRE1CbGw5NFFoalVKUg==",
       "jobs_ms_secret_key": "cVplM01ZSG9FT20wOWE4aU9vcHA2UUJFdzU4clhQamRxSWFERDN5Vw==",
       "jobs_scheduler_access_key": "MGJBMTdrbWJGOHZaTFlibk1URUk=",
       "jobs_scheduler_secret_key": "RzFKcDJKRzdVMTRWWlBuN1NoNG1MYmh5WHBCbmJuWTJGdlJwZjdZaA==",
       "loki_access_key": "bFZ5N3BxT0U5bmpFT0RJNlZ1dzI=",
       "loki_secret_key": "THBCU3djMzhleDNUUFNrU3VRN0p4ZXhKVThCTUY1a3dtMVdMNU9kdw==",
       "media_ms_access_key": "RkVpZUxBZFJ2bEJHU0JybzRNQ04=",
       "media_ms_s3_presigned_url_access_key": "ZUZCS2N2MFhnVDA5WTdMVDc0N1M=",
       "media_ms_s3_presigned_url_secret_key": "RnVEak5EQ3BxdXpRR2RObU1MRnpSZVJ4WDFtalB6S2NTdExuUWlxOA==",
       "media_ms_secret_key": "dWpvbkduaE9ldWVpOU9Xd1Bqb1pHdVRWWHdCTU5ENm55VkV1eGlCdQ==",
       "mimir_access_key": "TGRCTmJpTENtNEIydUtjRXp1Wmk=",
       "mimir_secret_key": "ZFdrTHpJa3dlTEsya0JpWU9QdEhpUGtSem1YcjRmcWNKSG9naXhjZg==",
       "modelmesh_access_key": "VTRWb1ZCdWNRdXZIVUtGV3E3ZVI=",
       "modelmesh_secret_key": "VHZiT2pCdEwzZnhmWUJmcUhKY2o5V2JOcmRoSlVNOFYyZFhVb0RMeg==",
       "project_ie_access_key": "QThjc0pBb3RPYVZ3RUluUlpXekY=",
       "project_ie_s3_presigned_url_access_key": "bjVOeGJrZTZXZWJicmRSUVlaaTM=",
       "project_ie_s3_presigned_url_secret_key": "cVlDRzJsbFpjdnRMNzg1ZHE2UnJLTjlLdjBoWFZrc0xCVDJYelNJaQ==",
       "project_ie_secret_key": "d3JaTzdvd2JpbDB5M1V1TjZIRDU3V1Zod1V0M1FpRkZ2VDRVRDI1Mw==",
       "resource_access_key": "cDRSTVRUamtINVRuS29tZkw1aXI=",
       "resource_s3_presigned_url_access_key": "dDN5eFdqM0hZUW8zZVNCTnBLWTc=",
       "resource_s3_presigned_url_secret_key": "cGRZUHBWN1BhWWdBYXdheUNZbElEU2hHczZDZEJiYjBjbDlvU3RnNw==",
       "resource_secret_key": "blFua3FEYkZPTEVTVmlrNFZORGRnaEcwd3NYYmdjVEJheE04RG9yeQ==",
       "tempo_access_key": "UE1tNkRiRnV0MHhCN3VUMUw5RUY=",
       "tempo_secret_key": "WW13OHJuekIxTkJRTW1wY2NnbFhzUmNjTlllSkh6czlKTjAwbFlrVw==",
       "training_operator_access_key": "WWlONElXMldEWEMwSDV2c3VvRkU=",
       "training_operator_s3_presigned_url_access_key": "elE3dkRDMTM0TTgydUhCWmg1QTY=",
       "training_operator_s3_presigned_url_secret_key": "T3hveE1KUGdhUHRjaGgxM1A2QUszZ2Z4YnlqSnJpcmlieHlBYnpPSg==",
       "training_operator_secret_key": "WWtROFdlNGFjN1pXeDJSSjJ3bVEzenBJQTg5YmFjRTI5SWhFa01jOQ==",
       "weed.json": "<redacted>",
       "workload_support_proxy_access_key": "ZWZQRmlaM1owQjc1YzRoTkRBS2o=",
       "workload_support_proxy_s3_presigned_url_access_key": "QjRSM0VaNVVUUnNrU2hHajlwZXE=",
       "workload_support_proxy_s3_presigned_url_secret_key": "T0hqeUNvOWNPdEVTaEhMYjNjV3lYYVkwamJicGtacUdiejhUcUM2OQ==",
       "workload_support_proxy_secret_key": "T05MYVRtY0JJalZJekpPZDBCN0RtOE5xNnY2b3dKNkJLdzhlSkdYUA=="
   }
}

Store this information for the later use, for example on an external drive.

Step 2. Run the following command on the on-prem machine to stop components producing data:#

sudo kubectl scale deployment/impt-project-ie deployment/impt-director deployment/impt-resource deployment/impt-dataset-ie deployment/impt-training-operator deployment/impt-inference-operator deployment/impt-media-ms deployment/impt-account-service deployment/impt-openldap -n impt --replicas=0
sudo kubectl scale deployment/flytepropeller deployment/flyteconsole deployment/flyteadmin deployment/flyte-pod-webhook deployment/flytescheduler deployment/datacatalog -n flyte --replicas=0

Wait until the following deployments are stopped: impt-project-ie, impt-director, impt-resource, impt-dataset-ie, impt-training-operator, impt-inference-operator, impt-media-ms, impt-account-service, impt-openldap in impt ns. Check that those deployments are stopped too: flytepropeller, flyteconsole, flyteadmin, flyte-pod-webhook, flytescheduler, datacatalog.

You can achieve this by executing the following command and verifying that all listed deployments display a 0/0 value in the READY column.

sudo kubectl get deployments -n impt
sudo kubectl get deployments -n flyte

sudo kubectl scale statefulset/impt-mongodb statefulset/impt-seaweed-fs statefulset/impt-postgresql -n impt --replicas=0

Wait until the following statefulsets are stopped: impt-mongodb impt-seaweed-fs impt-postgresql.

You can achieve this by executing the following command and verifying that all listed statefulsets display a 0/0 value in the READY column.

sudo kubectl get statefulset -n impt

Step 3. Run the following command to copy data to external SSD drive (this will take a while!):#

Note

We have to use rsync as it preserves the whole directory and sub-directory permission on each file and folder to copied folder.

sudo rsync -avb --include='/seaweedfs/***' \
   --include='/mongodb/***' \
   --include='/postgresql-data/***' \
   --include='/openldap-data/***' \
   --include='/openldap-config/***' \
   --exclude='*' /<data_folder>/ destination_username@ip-address:/<SSD_drive>

<data_folder> - folder where platform’s data are stored - chosen during installation

Step 4. Once copying is done, start the components stopped in the 2nd step:#

sudo kubectl scale deployment/impt-project-ie deployment/impt-director deployment/impt-resource deployment/impt-dataset-ie deployment/impt-training-operator deployment/impt-inference-operator deployment/impt-media-ms deployment/impt-account-service deployment/impt-openldap -n impt --replicas=1
sudo kubectl scale deployment/flytepropeller deployment/flyteconsole deployment/flyteadmin deployment/flyte-pod-webhook deployment/flytescheduler deployment/datacatalog -n flyte --replicas=1
sudo kubectl scale statefulset/impt-mongodb statefulset/impt-seaweed-fs statefulset/impt-postgresql -n impt --replicas=1

After checking if all those components are running (this time they should have the 1/1 value in the READY column) - you can start using the platform again.

Recovery from Backed Up Data#

To recover from back up, open a terminal and execute the following steps:

Step 1: Stopping Components#

Run the following command on the on-prem machine to stop components producing data:

sudo kubectl scale deployment/impt-project-ie deployment/impt-director deployment/impt-resource deployment/impt-dataset-ie deployment/impt-training-operator deployment/impt-inference-operator deployment/impt-media-ms deployment/impt-account-service deployment/impt-openldap -n impt --replicas=0
sudo kubectl scale deployment/flytepropeller deployment/flyteconsole deployment/flyteadmin deployment/flyte-pod-webhook deployment/flytescheduler deployment/datacatalog -n flyte --replicas=0

Wait until the following deployments are stopped:

  • impt-project-ie

  • impt-director

  • impt-resource

  • impt-dataset-ie

  • impt-training-operator

  • impt-inference-operator

  • impt-media-ms

  • impt-account-service

  • impt-openldap

  • flytepropeller

  • flyteconsole

  • flyteadmin

  • flyte-pod-webhook

  • flytescheduler,

  • datacatalog.

To verify, execute the command:

sudo kubectl get deployments -n impt
sudo kubectl get deployments -n flyte

Make sure all listed deployments display a 0/0 value in the READY column.

Run the following command on the on-prem machine to stop components databases:

sudo kubectl scale statefulset/impt-mongodb statefulset/impt-seaweed-fs statefulset/impt-postgresql -n impt --replicas=0

Wait until the following db’s are stopped:

  • impt-mongodb

  • impt-postgresql

  • impt-seaweed-fs.

To verify, execute the command:

sudo kubectl get statefulsets -n impt

Make sure all listed statefulsets display a 0/0 value in the READY column.

Step 2: Verifying Data Folders#

Ensure that folders /<data_folder>/seaweedfs/, /<data_folder>/postgresql-data/, /<data_folder>/openldap-data/, /<data_folder>/openldap-config/ and /<data_folder>/mongodb/ are empty. If not, please remove or move their content.

Step 3: Restoring Data#

Run the command to restore the data from the external SSD drive:

Note

We have to use rsync as it preserves the whole directory and sub-directory permission on each file and folder to the copied folder.

Warning

This process can take some time.

sudo rsync -avb --include='/seaweedfs/***' \
   --include='/mongodb/***' \
   --include='/postgresql-data/***' \
   --include='/openldap-data/***' \
   --include='/openldap-config/***' \
   --exclude='*' destination_username@ip-address:/<SSD_drive>/datasets_current \
   /<data_folder>/

Step 4: Editing MongoDB Configuration#

Run the command to edit the MongoDB configuration:

sudo kubectl -n impt edit secret impt-mongodb --output json
sudo kubectl -n impt-jobs-production edit secret impt-mongodb --output json

This will open the default text editor with configuration information in JSON format. Replace the MongoDB database username and password with the previously extracted ones (see Step 1 in Section 2).

{
   "apiVersion": "v1",
   "data": {
      "dataset-ie-mongodb-password": "UGRFbFltd0JBNER0",
      "dataset-ie-mongodb-username": "VFpPUXZW",
      "director-mongodb-password": "dG1WdGttNVBTS002",
      "director-mongodb-username": "WXRMWUtE",
      "gateway-mongodb-password": "RGM5R0pnVjducHpR",
      "gateway-mongodb-username": "RWlNWEVS",
      "inference-job-mongodb-password": "cXVLNjRzdGdZZzdk",
      "inference-job-mongodb-username": "WVN0UldC",
      "inference-operator-mongodb-password": "NHY4R3V6RnpQTEpz",
      "inference-operator-mongodb-username": "eEV5amxh",
      "inference-server-mongodb-password": "NHVCckx6VUZja08w",
      "inference-server-mongodb-username": "bERSaVdD",
      "jobs-mongodb-password": "ZERqdVJyb01qNUpB",
      "jobs-mongodb-username": "S3VITm9J",
      "jobs-ms-mongodb-password": "SXNxbUk1cUdrZTFC",
      "jobs-ms-mongodb-username": "Umd2cG1M",
      "jobs-scheduler-mongodb-password": "SkRlZXh0NzVXTVVz",
      "jobs-scheduler-mongodb-username": "YWpPT3J3",
      "mongodb-password": "OVBDQktWODJIYWZ4",
      "mongodb-username": "aFd3Qm15",
      "project-ie-mongodb-password": "QlRpcU95ZXEwTkFX",
      "project-ie-mongodb-username": "bU1RelpM",
      "resource-mongodb-password": "OXUzUk1PNE9WQ2Y4",
      "resource-mongodb-username": "YktJQVpE",
      "spice-db-mongodb-password": "eEhlMkNvd1dvMHds",
      "spice-db-mongodb-username": "YWJkU2Fl",
      "training-operator-mongodb-password": "OVFRTEhhSWs2cGky",
      "training-operator-mongodb-username": "Y1lyYlhB",
      "workload-configuration-mongodb-password": "M01ScHhvR0t0ZElL",
      "workload-configuration-mongodb-username": "Y0tTa3VR"
   }
}

Step 5: Editing PostgreSQL Configuration#

Run the command to edit the PostgreSQL configuration:

sudo kubectl -n impt edit secret impt-postgresql --output json
sudo kubectl -n flyte edit secret impt-postgresql --output json

This will open the default text editor with configuration information in JSON format. Replace the PostgreSQL database username and password with the previously extracted ones (see Step 1 in Section 2).

{
   "apiVersion": "v1",
   "data": {
       "accountservice-postgresql-db": "YWNjb3VudHNlcnZpY2U=",
       "accountservice-postgresql-password": "dUl6aTdzU2wyYUl2",
       "accountservice-postgresql-username": "aWRma3pv",
       "creditsystem-postgresql-db": "Y3JlZGl0c3lzdGVt",
       "creditsystem-postgresql-password": "Y3JlZGl0c3lzdGVtX3Bhc3N3b3Jk",
       "creditsystem-postgresql-username": "Y3JlZGl0c3lzdGVt",
       "flyte-postgresql-db": "Zmx5dGU=",
       "flyte-postgresql-password": "V3V1MnNvZ3J0bGd6",
       "flyte-postgresql-username": "dHh2YnNl",
       "postgresql-db": "c3BpY2VkYg==",
       "postgresql-password": "UFdyV2NvZUlEZ0s2",
       "postgresql-username": "ZGdKRUpXcmVKdnla",
       "spicedb-postgresql-password": "YzQzQTljamE2V1Uw",
       "spicedb-postgresql-username": "dGdkcXp3"
   }
}

Step 6: Editing SpiceDB Configuration#

Run the command to edit the SpiceDB configuration:

sudo kubectl -n impt edit secret impt-spice-db --output json
sudo kubectl -n impt-jobs-production edit secret impt-spice-db --output json

This will open the default text editor with configuration information in JSON format. Replace the SpiceDB database grpc key with the previously extracted ones (see Step 1 in Section 2).

{
   "apiVersion": "v1",
   "data": {
       "SPICEDB_GRPC_PRESHARED_KEY": "elpTdDhjU2hrbHBN"
   }
}

Step 7: Editing OpenLDAP Configuration#

Run the command to edit the OpenLDAP configuration:

sudo kubectl -n impt edit secret impt-ldap-service-user --output json

This will open the default text editor with configuration information in JSON format. Replace the OpenLDAP database username and password with the previously extracted ones (see Step 1 in Section 2).

{
   "apiVersion": "v1",
   "data": {
       "config_password": "NEVqNVF3eGRBcU4y",
       "config_username": "WUdRUlJ5",
       "password": "NVdnb3FuaFRWb1Jw",
       "readonly_password": "ODZ3ckNjczBWZFU5",
       "readonly_username": "VVBCeFJN",
       "username": "UHRZcVNX"
   }
}

Step 8: Editing SeaweedFS Configuration#

Run the command to edit the SeaweedFS configuration:

sudo kubectl -n impt edit secret impt-seaweed-fs --output json
sudo kubectl -n impt-jobs-production edit secret impt-seaweed-fs --output json

This will open the default text editor with configuration information in JSON format. Replace the SeaweedFS database username and password with the previously extracted ones (see Step 1 in Section 2).

{
   "apiVersion": "v1",
   "data": {
       "accountservice_access_key": "OFJWZmI1b25wUk44N0Z3N2JST2E=",
       "accountservice_presigned_url_access_key": "dkxzMVM0b1pXc2xKSkFyRTNTaUU=",
       "accountservice_presigned_url_secret_key": "blpTc0sySVd0MmpaWVVXbk1hR2YyMEVTNnZpcFpLZGxwVkJDQlRiSA==",
       "accountservice_secret_key": "b0xJc2llWnZWWlJBTTFPOEpoSGFEb011Qk1RWkdUbW5nbGlBTUx2Qg==",
       "admin_access_key": "ekZLU1dhbVUyVkRDN3hZS3pZeHg=",
       "admin_secret_key": "TkRmWk0xODB5NXk4SHFpakhOYUl2NDVnRzlIbzNNa1Bxb3RISXBwRA==",
       "auto_train_controller_access_key": "b2J3bDkzQ0FLV0dJdkxlOFNQb1E=",
       "auto_train_controller_secret_key": "TGhyM0VqOXVCM25rTXdaNlh3R3pBYkxXQmI2TTBkbzhMc3lxWHNiZw==",
       "dataset_ie_access_key": "enBGams1OHFnSDdiNE8zcFhHNlQ=",
       "dataset_ie_s3_presigned_url_access_key": "UkFqc2NMalRFVk9UTVloUjBZNDY=",
       "dataset_ie_s3_presigned_url_secret_key": "MkZXTlRtdGpJdkprODBxU1Rqbzg1MmpaYmhJRlkyTUJMY0loMmZHVA==",
       "dataset_ie_secret_key": "c0NIMFczTjF1eW81QnBqYlA2ODlSVjNzaUNYb0hvZEtsbHFSSEtPVQ==",
       "director_access_key": "Z0hOVXFQODhzRVY3SUE1U3FyaXI=",
       "director_s3_presigned_url_access_key": "SXdFMGxsazdGMm9Sb0o2QzF5REg=",
       "director_s3_presigned_url_secret_key": "c0FRMTZvclJhc2JHMjZvaEkzMFcyOHRLWFZyUmtaUmVrUzdvQkdqNA==",
       "director_secret_key": "SWp2SjYzZjFRRlJJdDBFZ2ZoZXowVHY3OHIxaGRtS21MRmxOYnVBag==",
       "flyte_access_key": "Znc0TU1GZ2xuMmNmUW1vcmdOclM=",
       "flyte_secret_key": "Qm81RnI4SDBlWDg1dDVPN2dodkY2WmNNRnQ5cUNEUHRBd2RLWDQ0dw==",
       "flyte_workflows_access_key": "bVJnMXpYcTdGZkU5N0xYbVdTQlE=",
       "flyte_workflows_s3_presigned_url_access_key": "bEJ3a0tBdGZ1Q0VxRFVVOE1hcTY=",
       "flyte_workflows_s3_presigned_url_secret_key": "VGtyVkJiV0ZSWmt5ellXcTB0MHlXMkViOE5YRmNNUnVoZmh5aGVlYg==",
       "flyte_workflows_secret_key": "WjNmSXdmSkVibXp0aVhCYVBhUEltR1RpbkNiaVFabUZBZTVkQ1NSSg==",
       "gateway_access_key": "bFlhcGRTNnRWUHBEd2FvSzRHaUE=",
       "gateway_secret_key": "ZkkyQm16Q3dJNlZTTFBRblJZZVQ0UEJNTjVVWHJNSUpWcVB6TTcxVg==",
       "inference_gateway_access_key": "MWxMSVVLQXhqaFlkQWdwNFRERnI=",
       "inference_gateway_s3_presigned_url_access_key": "SGt1YkVrVFpkY3F6U3N5N3IyRDE=",
       "inference_gateway_s3_presigned_url_secret_key": "UGdmbVd4TXZkVU9uUnFYQ2NQcnYxMDFIajJLd0xTU1dWRU52dzlHeg==",
       "inference_gateway_secret_key": "NGJXQ2ZxY01nOHoyOURTcUo2MnZaNktQSXFScm5ZQ3lVamZEeDlDbQ==",
       "inference_operator_access_key": "bGRHZ3FjQzRDR3BwQ2ZRelNHanQ=",
       "inference_operator_s3_presigned_url_access_key": "Z2ViZXpjWU9DczhGN3Q4bEoxOHE=",
       "inference_operator_s3_presigned_url_secret_key": "ektWV3ZMRmRzcWphd0hSbm53UHFKRGFYYTN5ZUdGN005Zjcyd0VjZg==",
       "inference_operator_secret_key": "R2ZVYmx0cnVHZVd2amJRTFlTQ1o0WGNDN3NJUUw1dVQ2MUVsVUlvMg==",
       "jobs_ms_access_key": "TFBFQzU0UFdxd0RNUVFLWURHTGM=",
       "jobs_ms_s3_presigned_url_access_key": "ZGlDd2l3dFZXSDFvZ040enRZRDE=",
       "jobs_ms_s3_presigned_url_secret_key": "ZG80czJuSkE3bEJOdU45NkY4NW9kTEpBbGJyRE1CbGw5NFFoalVKUg==",
       "jobs_ms_secret_key": "cVplM01ZSG9FT20wOWE4aU9vcHA2UUJFdzU4clhQamRxSWFERDN5Vw==",
       "jobs_scheduler_access_key": "MGJBMTdrbWJGOHZaTFlibk1URUk=",
       "jobs_scheduler_secret_key": "RzFKcDJKRzdVMTRWWlBuN1NoNG1MYmh5WHBCbmJuWTJGdlJwZjdZaA==",
       "loki_access_key": "bFZ5N3BxT0U5bmpFT0RJNlZ1dzI=",
       "loki_secret_key": "THBCU3djMzhleDNUUFNrU3VRN0p4ZXhKVThCTUY1a3dtMVdMNU9kdw==",
       "media_ms_access_key": "RkVpZUxBZFJ2bEJHU0JybzRNQ04=",
       "media_ms_s3_presigned_url_access_key": "ZUZCS2N2MFhnVDA5WTdMVDc0N1M=",
       "media_ms_s3_presigned_url_secret_key": "RnVEak5EQ3BxdXpRR2RObU1MRnpSZVJ4WDFtalB6S2NTdExuUWlxOA==",
       "media_ms_secret_key": "dWpvbkduaE9ldWVpOU9Xd1Bqb1pHdVRWWHdCTU5ENm55VkV1eGlCdQ==",
       "mimir_access_key": "TGRCTmJpTENtNEIydUtjRXp1Wmk=",
       "mimir_secret_key": "ZFdrTHpJa3dlTEsya0JpWU9QdEhpUGtSem1YcjRmcWNKSG9naXhjZg==",
       "modelmesh_access_key": "VTRWb1ZCdWNRdXZIVUtGV3E3ZVI=",
       "modelmesh_secret_key": "VHZiT2pCdEwzZnhmWUJmcUhKY2o5V2JOcmRoSlVNOFYyZFhVb0RMeg==",
       "project_ie_access_key": "QThjc0pBb3RPYVZ3RUluUlpXekY=",
       "project_ie_s3_presigned_url_access_key": "bjVOeGJrZTZXZWJicmRSUVlaaTM=",
       "project_ie_s3_presigned_url_secret_key": "cVlDRzJsbFpjdnRMNzg1ZHE2UnJLTjlLdjBoWFZrc0xCVDJYelNJaQ==",
       "project_ie_secret_key": "d3JaTzdvd2JpbDB5M1V1TjZIRDU3V1Zod1V0M1FpRkZ2VDRVRDI1Mw==",
       "resource_access_key": "cDRSTVRUamtINVRuS29tZkw1aXI=",
       "resource_s3_presigned_url_access_key": "dDN5eFdqM0hZUW8zZVNCTnBLWTc=",
       "resource_s3_presigned_url_secret_key": "cGRZUHBWN1BhWWdBYXdheUNZbElEU2hHczZDZEJiYjBjbDlvU3RnNw==",
       "resource_secret_key": "blFua3FEYkZPTEVTVmlrNFZORGRnaEcwd3NYYmdjVEJheE04RG9yeQ==",
       "tempo_access_key": "UE1tNkRiRnV0MHhCN3VUMUw5RUY=",
       "tempo_secret_key": "WW13OHJuekIxTkJRTW1wY2NnbFhzUmNjTlllSkh6czlKTjAwbFlrVw==",
       "training_operator_access_key": "WWlONElXMldEWEMwSDV2c3VvRkU=",
       "training_operator_s3_presigned_url_access_key": "elE3dkRDMTM0TTgydUhCWmg1QTY=",
       "training_operator_s3_presigned_url_secret_key": "T3hveE1KUGdhUHRjaGgxM1A2QUszZ2Z4YnlqSnJpcmlieHlBYnpPSg==",
       "training_operator_secret_key": "WWtROFdlNGFjN1pXeDJSSjJ3bVEzenBJQTg5YmFjRTI5SWhFa01jOQ==",
       "weed.json": "<redacted>",
       "workload_support_proxy_access_key": "ZWZQRmlaM1owQjc1YzRoTkRBS2o=",
       "workload_support_proxy_s3_presigned_url_access_key": "QjRSM0VaNVVUUnNrU2hHajlwZXE=",
       "workload_support_proxy_s3_presigned_url_secret_key": "T0hqeUNvOWNPdEVTaEhMYjNjV3lYYVkwamJicGtacUdiejhUcUM2OQ==",
       "workload_support_proxy_secret_key": "T05MYVRtY0JJalZJekpPZDBCN0RtOE5xNnY2b3dKNkJLdzhlSkdYUA=="
   }
}

Step 9: Update Flyte configuration#

Run the command to edit the Flyte configuration:

sudo kubectl -n flyte edit cm flyte-admin-base-config --output yaml
sudo kubectl -n flyte edit cm datacatalog-config --output yaml

Search for following entries: username and password, then take flyte-postgresql-username and flyte-postgresql-password (from impt-postgresql secret) decode them with base64 and paste cleartext value. In case of access_key_id and secret_key, then take flyte_access_key and flyte_secret_key (from impt-seaweed-fs secret) decode them with base64 and paste cleartext value.

sudo kubectl -n flyte edit cm flyte-propeller-config --output yaml

Search for access_key_id, secret_key, FLYTE_AWS_ACCESS_KEY_ID and FLYTE_AWS_SECRET_ACCESS_KEY, then take flyte_access_key and flyte_secret_key (from impt-seaweed-fs secret) decode them with base64 and paste cleartext value.

sudo kubectl -n flyte edit cm flyte-scheduler-config --output yaml

Search for following entries: username and password, then take flyte-postgresql-username and flyte-postgresql-password (from impt-postgresql secret) decode them with base64 and paste cleartext value.

Step 10: Restarting Components#

Once secrets are updated, restart services spicedb and dex:

sudo kubectl -n impt rollout restart daemonset impt-spice-db
sudo kubectl -n impt rollout restart deployment impt-dex

Then start the databases stopped in the 1st step:

sudo kubectl scale statefulset/impt-mongodb statefulset/impt-seaweed-fs statefulset/impt-postgresql -n impt --replicas=1

To verify, execute the command:

sudo kubectl get statefulsets -n impt

Make sure all listed statefulsets display a 1/1 value in the READY column.

Finally start the components stopped in the 1st step:

sudo kubectl scale deployment/impt-project-ie deployment/impt-director deployment/impt-resource deployment/impt-dataset-ie deployment/impt-training-operator deployment/impt-inference-operator deployment/impt-media-ms deployment/impt-account-service deployment/impt-openldap -n impt --replicas=1
sudo kubectl scale deployment/flytepropeller deployment/flyteconsole deployment/flyteadmin deployment/flyte-pod-webhook deployment/flytescheduler deployment/datacatalog -n flyte --replicas=1

To verify, execute the command:

sudo kubectl get deployments -n impt
sudo kubectl get deployments -n flyte

Make sure all listed statefulsets display a 1/1 value in the READY column.

After updating the SpiceDB secret and deployment, all dependent applications need to be restarted.

sudo kubectl -n impt rollout restart deployment impt-gateway impt-jobs-ms impt-jobs-scheduler impt-workload-support-proxy

Step 11: Verifying Installation#

Open Intel Geti UI and ensure everything is functioning correctly.