...
DB Changes
We will add one column cloud_storage_data in cassandra table → bulk_upload_process,which will store the data on upload. It will be a json format data, in following format.
...
Above data will be stored in encrypted format within database, so one cannot decipher the data at all.
Data Definitions
Structural/Code Changes/AlgorithmAlgorithm for approach 1
- Before marking the status of bulkupload to completed, we will check the type - if it is Org or User, it will continue the following process.
Note: Currently only User/Org will use this functionality, hence we will be enabling it only for this 2 functionalities.
- Create a temporary CSV file, with name <type>_<processid>.csv.
- Read the configuration for user-friendly names from system settings.
- Read the first row from results (success/failure) and map the columns to user friendly names to write the first row.
- Add one more column Result to mark the results, as Success or Failure
- Now start adding the result rows, from success results first. mark the last column as Success in each case.
- Now start adding the result rows from failure results, mark the last column as Failure in each case
- Invoke the utility, which uploads the file into cloud storage based on : storageType, container, objectId, with pre-configured timeout.
- Expected environment variable to connect to the cloud-storage. Currently only azure will be supported, which requires two environment variables AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY
- In our case it will be currently
- storageType: "azure",
- container: "org" or "user",
- objectId: "user_101010" or "org_101110"
- Once successful, store the generated URL into our db column.
- Delete the temporary file.
- Mark the status of upload to Completed.
Algorithm for approach 2
- Only step after upload changes, rest are same as approach 1:
- Once successful, encrypt the data used for uploading in above step as described in DB Changes section and store the data into cloud_storage_data column.
Public API exposed
POST /v1/bulkupload/getsignedurl
Input :{ "processId" : "processId"}
Output: "{"result": "singedUrl"}