...
This API will import the dimension csv and upload it in to the combined_input folder in cloud if there are no errors. Then adapter will use the same files to breakdown the combined input into multiple input files. Later those files will be used by the Nifi processor to process and ingest the data into the database.
Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-releasedev/spec.yaml
Provide the valid input details for the Parameters shown below.
...
This API will import the event csv and upload it in to the combined_input folder in cloud if there are no errors. Then adapter will use the same files to breakdown the combined input into multiple input files. Later those files will be used by the Nifi processor to process and ingest the data into the database.
...
Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-releasedev/spec.yaml
Provide the valid input details for the Parameters shown below.
...
After successful execution of the csv import api we get the response and we can see the file status indicating if the file is uploaded or not using GET file status API. If the file is successfully uploaded we will get the response as uploaded and if there are any errors it will send us the response indicating there was an error in the file.
Ingestion of data using API
The data can also be ingested using API’s developed for event,dimension and dataset.
Ingestion of Events using API
Step 1: Open the specified request & add the details
API Endpoint: <domain_name>/ingestion/event
HTTP Method: POST
...
This API will be used to write events into the csv file and upload it in to the combined_input folder in cloud if there are no errors. Then adapter will use the same files to breakdown the combined input into multiple input files. Later those files will then be used by the Nifi processor to ingest the data into the database. The API can be used to add individual events into csv.
Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml
Provide the valid input details for the Parameters shown below. The request body should conform to the schema stored in the database for the particular event name.
...
Step 3: Click on the send button for the request and if the request is successful the user should see a response message. Please refer to the below screenshot.
After successful execution of the event api we get the response and the data sent in the request body will be written to the csv file. If there are any errors it will be written to the separate csv file and valid data will be written to the other csv file.
...
Ingestion of Dimensions using API
Step 1: Open the specified request & add the details
API Endpoint: <domain_name>/ingestion/dimension
HTTP Method: POST
...
This API will be used to write dimensions into the csv file and upload it in to the combined_input folder in cloud if there are no errors. Then adapter will use the same files to breakdown the combined input into multiple input files. Later those files will then be used by the Nifi processor to ingest the data into the database. This API can be used to add individual dimensions into csv.
Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml . Provide the valid input details for the Parameters shown below. The request body should conform to the schema stored in the database for the particular dimension name.
...
Step 3: Click on the send button for the request and if the request is successful the user should see a response message. Please refer to the below screenshot.
After successful execution of the dimension api we get the response and the data sent in the request body will be written to the csv file. If there are any errors it will be written to the separate csv file and valid data will be written to the other csv file.
...
Things to take care of while ingesting Data / Debugging:
...
Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml: https://github.com/dhanushSunbird-2397cQube/spec-ms/blob/dev/spec.yaml . Provide the valid input details for the Parameters shown below.
...
Step 3: Click on the send button for the request and if the request is successful the user should see a response message as “FIle “File uploaded successfully”. The files will be uploaded to the emission folder created in the respective cloud storage.
...