Upserting Dimensions and Events
Dimensions can be upserted by modifying the ingestion API (/ingestion/dimension) to directly add/update the dimension into the database.
Need Clarity
To upsert events we are not storing the events anywhere in the database, So how do we achieve the upsert functionality for the events?
2. Making Index as optional
As seen in the above diagram, unique constraints are created for the columns “school_id” and “school_name”. The column “school_id” can be unique but “school_name” alone cannot be unique because there may be more than one school with the same name as shown in the below example. Ex: school_id, school-name
1 , “Kendriya Vidyalaya”
2 , “Kendriya Vidyalaya”
As we have created an index for the “school_name” it is expecting to have unique values and there is an issue while data ingestion.
Example:
PK,Index
String,string,
school_id,school_name
Approach to resolve the above issue:
To overcome the above scenario we can have composite unique constraints for the combinations of school_id and school_name. To specify the composite unique constraint remove the first row and add another row at the last in the dimension grammar
Existing Approach:
PK,INDEX
String,string, → Data Types
School_id,school_name → Columns
Suggested Approach:
String,string, → Data Types
school_id,school_name→ Columns
“school_id”, “school_id, school_name”→ Unique Constraints
As seen in the Suggested Approach we have another row at the end for unique constraints, so in the given example we need two unique constraints which is school_id and combination of school_id and school_name as composite keys.