Manage Lear - Add New CSP Storage - Implementation Changes & Testing

Context

This document details on how to enable any new CSP provider for Manage Learn capabilities which is part of Sunbird Ed

Manage Learn capabilities currently support Azure, AWS, OCI and GCP
ML Core Service interacts with cloud storage for upload/download operation, and all other services(Survey, Projects and Reports) as well as mobile app and portal use these APIs for their needs:
- Get Signed URL (To upload assets to cloud)
- Get Downloadable URL
In Mongo DB where all the transactions of projects, and observations are stored, only the relative path of the evidences of assets uploaded by users are captured. Bucket and CSP details are provided via config to the ML Core service.
In order to add support for any other cloud storage (e.g: Digital Ocean), below steps need to be followed:

ml-core-service:

Git Repos:

https://github.com/project-sunbird/knowledge-platform

https://github.com/project-sunbird/sunbird-learning-platform

Latest branch: release-5.2.0

The Service need only configuration change to maintain relative path in database while write operation and return the absolute path for cloud related metadata while read operation.
e.g:
- If appIcon has been given as absolute url (https://sunbirddevbbpublic.blob.core.windows.net/sunbird-content-staging/content/do_213687607599996928135/artifact/download-1.thumb.jpg) in Content/Collection Create API, the service should store it as relative path (CLOUD_STORAGE_BASE_PATH/content/do_213687607599996928135/artifact/download-1.thumb.jpg) and the read api should return back the absolute url.
- In above example, base url of storage account & bucket name got replaced with a string value "CONTENT_STORAGE_BASE_PATH" configured in cloudstorage_relative_path_prefix_content variable.
Override value for below variables under private devops repo (file path: ansible/inventory/<env_name>/Core/common.yml) for new storage account:

cloud_storage_content_bucketname
cloudstorage_replace_absolute_path
cloudstorage_relative_path_prefix
cloudstorage_base_path
valid_cloudstorage_base_urls

Configuration File Reference:
https://github.com/project-sunbird/sunbird-devops/blob/b61a35fad0362ea7eb0bb688ff0bc12ffc811571/ansible/roles/stack-sunbird/templates/content-service_application.conf#L484

After Configuration Change, Deploy the service.
Test Content Create & Read API with some metadata having cloud path (e.g: appIcon)

The key integration points present at the below files

Knowledge-Platform repo:

StorageService.scala → Integration points of collecting to org.sunbird.cloud.storage library. This SunbirdCloudStage SDK has to support for new CSP providers.

Sunbird cloud-stage-sdk version upgrade location
https://github.com/project-sunbird/knowledge-platform/blob/release-5.2.0/platform-modules/mimetype-manager/pom.xml

 <dependency>
      <groupId>org.sunbird</groupId>
      <artifactId>cloud-store-sdk</artifactId>
      <version>1.4.3</version>
  </dependency>

flink jobs:

Git Repo:

https://github.com/project-sunbird/knowledge-platform-jobs

Both jobs uses cloud-storage-sdk for cloud storage operations. So first the sdk need a code change to have support for new cloud storage provider (e.g: OCI).
Link to cloud-storage-sdk git repo: https://github.com/project-sunbird/sunbird-cloud-storage-sdk/tree/scala-2.12-with-latest
Once the cloud-storage-sdk new version is published in maven central, we need to update the version in jobs-core module https://github.com/project-sunbird/knowledge-platform-jobs/blob/e576a50834529c9984ca1cb5012f8ca0c59a5a29/jobs-core/pom.xml#L84
Override value for below variables under private devops repo (file path: ansible/inventory/<env_name>/Knowledge-Platform/common.yml) for new storage account:
cloud_service_provider
cloud_public_storage_accountname
cloud_public_storage_secret
cloud_public_storage_endpoint
cloud_storage_content_bucketname and cloud_storage_dial_bucketname
Configuration File Reference:
https://github.com/project-sunbird/sunbird-learning-platform/blob/59a59270b5419153b902b3d68165a8b5539f872e/kubernetes/helm_charts/datapipeline_jobs/values.j2#L731

cloudstorage_replace_absolute_path
cloudstorage_relative_path_prefix
cloudstorage_base_path
valid_cloudstorage_base_urls

Configuration File Reference:
https://github.com/project-sunbird/sunbird-learning-platform/blob/59a59270b5419153b902b3d68165a8b5539f872e/kubernetes/helm_charts/datapipeline_jobs/values.j2#L736
Build & Deploy both Job
Test Content/Collection Publish Workflow.
- Content/Collection should be published successfully.
- metadata having cloud storage file reference should be accessible.
  - e.g: downloadUrl pointing to a file, should be downloadable.