Step-by-Step Guide to Adding a New Program in cQube


  1. Install or update to the latest cQube version: Ensure you have the most recent version of cQube installed to access all the latest features and functionalities. You can find the step by step guides as below 


2. Create event and dimension schema: Design the event and dimension schema for your program. This schema defines the structure and relationships between different data elements, such as event attributes and dimensions.

  • Event Schema: The event schema defines the structure of the events to be ingested into the system. It includes information such as the event name, its attributes (e.g. event timestamp, user ID, etc.), and data types.

  • Dimension Schema: The dimension schema defines the dimensions, or attributes, that will be used to group and filter events. It includes information such as the dimension name, its attributes (e.g. dimension value, category, etc.), and data types.


3. Create a Config.json

Config.json provides directive to processing engine on how to process the grammar and data files, where they are stored, etc. it should be placed in /ingest/<state code>/ 

globals : global settings

dimensions : It includes, file format, location and namespace

programs : program specific settings like namespace, description, input location, output location, dimensions which should be considered etc.


Only programs mentioned under config.json will be considered for ingestion.



4. Upload the event and dimension grammar spec:

  • Convert the Event and Dimension Grammar to JSON format as given in the below example

Example: if there is a event with two columns

    school_id and school_name 


Below would be the JSON object


{

     "program": "school-attendance",

     "input": {

         "type": "object",

         "properties": {

            "school_id": {

               "type": "string",

               "shouldNotNull": true

             },

            "school_name": {

                 "type": "string",

                "shouldNotNull": true

             }

}

}


  • Dimension grammar : Use the /spec/dimension API to upload the dimension grammar

  • Place the dimension grammar file as <xyz>-dimension.grammar.csv in processing-ms/impl/c-qube/ingest/<state_code>/dimensions/ folder as seen below or as per the config.json settings


Contents of the district-dimension.grammar.csv



  • Event grammar can be ingested in 2 ways:  

    1. Though API Method

Event spec API: spec/event


  1. Through YARN CLI

YARN CLI INGEST

This command is used to ingest Event and Dimension Spec and Dimension Data based on the configuration we have


  • Create a <new-pgogram> folder in processing-ms/impl/c-qube/ingest/<state_code>/programs


  • Place the event grammar file as <xyz>-event.grammar.csv in processing-ms/impl/c-qube/ingest/<state_code>/programs/<new-program> folder as seen below (example for diksha program)

This would be based on default config.json or else it should be as per config.json



Contents of the district-dimension.grammar.csv




5. Data Ingestion: This process may involve uploading the CSV file or connecting to the database directly. Following links will provide more details on the process.


6. Add event data on a regular frequency: To keep your program up-to-date, continuously add new event data to the platform. Determine an appropriate frequency for updating the data, such as daily, weekly, or monthly. Ensure that the new data is compatible with the existing event and dimension schema.