inQuiry - Add New Cloud Storage - Implementation Changes & Testing

Background

This

• As of now inQuiry supports cloud storage operations with Azure, Aws, Google Storage.

• inQuiry has below flink jobs which interact with cloud storage for upload/download operation:

  • async-questionset-publish

  • questionset-republish

• inquiry-api-service transforms cloud related metadata (e.g: downloadUrl, appIcon, etc) but doesn't interact with cloud storage.

  • e.g: the service convert cloud specific path (absolute path) to cloud neutral path (relative path) and vice versa.

• In order to add support for any other cloud storage (e.g: OCI) under inQuiry components, below steps need to be followed:

1. async-questionset-publish & questionset-republish flink job:

• Both jobs uses cloud-storage-sdk for cloud storage operations. So first the sdk need a code change to have support for new cloud storage provider (e.g: OCI).

• Link to cloud-storage-sdk git repo: https://github.com/project-sunbird/sunbird-cloud-storage-sdk/tree/scala-2.12-with-latest

• Once the cloud-storage-sdk new version is published in maven central, we need to update the version in jobs-core module https://github.com/Sunbird-inQuiry/data-pipeline/blob/58d52be036712eff2e90a3842a1885d36d4473a7/jobs-core/pom.xml#L84

• Override value for below variables under private devops repo (file path: ansible/inventory/<env_name>/Knowledge-Platform/secrets.yml) for new storage account:
  cloud_service_provider
cloud_public_storage_accountname
cloud_public_storage_secret
cloud_public_storage_endpoint
cloud_storage_content_bucketname

• Configuration File Reference:
  https://github.com/Sunbird-inQuiry/data-pipeline/blob/58d52be036712eff2e90a3842a1885d36d4473a7/kubernetes/helm_charts/datapipeline_jobs/values.j2#L140

cloudstorage_replace_absolute_path
cloudstorage_relative_path_prefix
cloudstorage_base_path
valid_cloudstorage_base_urls

• Configuration File Reference:
  https://github.com/Sunbird-inQuiry/data-pipeline/blob/58d52be036712eff2e90a3842a1885d36d4473a7/kubernetes/helm_charts/datapipeline_jobs/values.j2#L148

• Build & Deploy both Job

• Test Question/QuestionSet Publish Workflow.

  • Question/QuestionSet should be published successfully.

  • metadata having cloud storage file reference should be accessible.

    • e.g: downloadUrl pointing to a file, should be downloadable.

2. inquiry-api-service:

• The Service need only configuration change to maintain relative path in database while write operation and return the absolute path for cloud related metadata while read operation.

• e.g:

  • If appIcon has been given as absolute url (https://sunbirddevbbpublic.blob.core.windows.net/sunbird-content-staging/questionset/do_213687607599996928135/artifact/download-1.thumb.jpg) in Question/QuestionSet Create API, the service should store it as relative path (CLOUD_STORAGE_BASE_PATH/questionset/do_213687607599996928135/artifact/download-1.thumb.jpg) and the read api should return back the absolute url.

  • In above example, base url of storage account & bucket name got replaced with a string value "CLOUD_STORAGE_BASE_PATH" configured in cloudstorage_relative_path_prefix variable.

• Override value for below variables under private devops repo (file path: ansible/inventory/<env_name>/Core/secrets.yml) for new storage account:

cloud_storage_content_bucketname
cloudstorage_replace_absolute_path
cloudstorage_relative_path_prefix
cloudstorage_base_path
valid_cloudstorage_base_urls

Configuration File Reference:
https://github.com/project-sunbird/sunbird-devops/blob/391343efc67403beceed15c437d7de8f55053527/ansible/roles/stack-sunbird/templates/assessment-service_application.conf#L424

• After Configuration Change, Deploy the service.

• Test Question Create & Read API with some metadata having cloud path (e.g: appIcon)

Note:

• cloudstorage_relative_path_prefix variable should have same value across the infra for all inQuiry components and Knowlg content service, search-indxer flink job.