Introduction: This guide will help you add a new program in cQube by following the steps outlined below. Ensure that you have the latest Step-by-Step Guide to Adding a New Program in cQube
Install or update to the latest cQube version: Ensure you have the most recent version of cQube installed to access all the latest features and functionalities.
...
Install or Update cQube: Before you start adding a new program, make sure you have the most recent version of cQube installed. If you need assistance with the installation or upgrading process, refer to the following guides:
Installation Guide: Provides step-by-step instructions for installing cQube.
Upgradation Guide: Offers detailed steps for upgrading your cQube installation to the latest version.
Create Event and Dimension SchemaYou can find the step by step guides as below
2. Create event and dimension schema: Design the event and dimension schema for your program. This schema defines the structure and relationships between different data elements, such as event attributes and dimensions.
Event Schema: The event schema defines the structure of the events to be ingested into the system. It includes information such as the event name, its attributes (e.g.
...
event timestamp, user ID, etc.), and data types.
Dimension Schema: The dimension schema defines the dimensions, or attributes, that will be used to group and filter events. It includes information such as the dimension name, its attributes (e.g.
...
dimension value, category, etc.), and data types.
3. Create a Config.json
...
Config.json provides
...
directive to
...
processing engine on how to process the grammar and data files, where they are stored, etc.
...
it should be placed in /ingest/
...
Define the following settings in the Config.json file:
...
globals: Global settings for the cQube processing engine.
...
<state code>/
globals : global settings
dimensions : It includes, file format, location
...
and namespace
...
programs :
...
program specific settings like namespace, description, input location, output location,
...
Place Event and Dimension Grammar Files: Once you've generated Event & Dimension grammar files based on the Event & Dimension Schema, place these files in the appropriate folders within the cQube directory structure.
...
dimensions which should be considered etc.
Only programs mentioned under config.json will be considered for ingestion.
...
4. Upload the event and dimension grammar spec:
Convert the Event and Dimension Grammar to JSON format as given in the below example
Example: if there is a event with two columns
school_id and school_name
Below would be the JSON object
{
"program": "school-attendance",
"input": {
"type": "object",
"properties": {
"school_id": {
"type": "string",
"shouldNotNull": true
},
"school_name": {
"type": "string",
"shouldNotNull": true
}
}
}
Dimension grammar : Use the /spec/dimension API to upload the dimension grammar
Place the dimension grammar file as <xyz>-dimension.grammar.csv in
...
processing-ms/impl/c-qube/ingest/<state_code>/dimensions/ folder
...
as seen below or as
...
per the
...
config.json settings
...
Contents of the district-dimension.
...
...
grammar.csv
Event grammar can be ingested in 2 ways:
Though API Method
Event spec API: spec/event
...
Through YARN CLI
YARN CLI INGEST
This command is used to ingest Event and Dimension Spec and Dimension Data based on the configuration we have
Create a <new-pgogram> folder in processing-ms/impl/c-qube/ingest/<state_code>/programs
...
Place the event grammar file as <xyz>-event.grammar.csv in
...
...
Ingest Event and Dimension Schema: Import the event and dimension schema into the cQube platform, as specified in step 3.
Through API Method: Use the Event Spec API (spec/event) to import the schema.
Through YARN CLI: Use the YARN CLI INGEST command to ingest Event and Dimension Spec and Dimension Data based on the configuration in Config.json.
...
Data Ingestion: Ingest data into the cQube platform using one of the following methods:
Uploading through CSV: Upload data by providing a CSV file containing the necessary information. For detailed instructions, refer to the Uploading through CSV Guide.
Connecting to Database: Connect directly to a database to ingest data. For more information on how to establish this connection, consult the Connecting to Database Guide.
...
processing-ms/impl/c-qube/ingest/<state_code>/programs/<new-program> folder as seen below (example for diksha program)
This would be based on default config.json or else it should be as per config.json
Contents of the district-dimension.grammar.csv
...
5. Data Ingestion: This process may involve uploading the CSV file or connecting to the database directly. Following links will provide more details on the process.
6. Add event data on a regular frequency: To keep your program up-to-date, continuously add new event data to the platform. Determine an appropriate frequency for updating the data, such as daily, weekly, or monthly. Ensure that the new data is compatible with the existing event and dimension schema.