Executive Summary

• cQube is a domain-agnostic, packaged solution for enabling scheme monitoring for public programs

• cQube offers packaging, deployment ease, and infra optimization, compared to off-the-shelf tech

• Core proposition is its ability to provide out-of-the-box usability leveraging domain-specific configurations

• The configurations layer will equip cQube with pre-defined indicators, insights, and suggested actions (nudges)

• cQube for education (Ed) will accelerate adoption of VSK as envisioned under MoE’s program

• The Ed configurations, along with programmatic SOPs will constitute a VSK Playbook as an additional public good

cQube Vision and Value Prop

cQube is envisioned as a ready-to-use/pre-packaged, configurable, and extendable DPG solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sector.

There are several challenges that government program owners face while leveraging technology for monitoring public programs

• Program administrators face challenges in defining actionable insights and relevant indicators for tracking

• Data ingestion, processing & visualization available as disaggregated products, not packaged into a solution

• Governments are unable to hire expert technical staff (architects, engineers) to smartly leverage a combination of available products together for end-to-end monitoring

• Governments have budget and digital infrastructure (server) limitations

cQube offers two core differential value propositions for monitoring programs

• A domain-agnostic product packaging of ingestion, processing and visualization layer in a manner that is:

  • Easy to deploy with minimal engineering staff

  • Lite deployment (minimal infrastructure requirements optimized to program needs)

  • Built for scale (to be scale tested with ~500k users)

  • Leverages reference schemas and spec-compliant API based implementation

  • Brownfield ready with modular components interoperable with existing systems

• Domain-specific configuration layer allowing out-of-the-box contextualization of indicators, insights, role based nudges and actions

• We aim to initiate cQube Ed and other sector based configurations can also be made available going forward

cQube architecture

The current architecture of cQube has certain challenges which affect the adopter experience

• Installation & CSV-based ingestion processes affecting adoption by state deployers

• Fixed set of chart types disallowing state admins to configure the charts that they wish to see

• Minimal interoperability due to non-exposure of data that gets stored as part of cQube

cQube architecture has been revised to enable flexibility in configuration and adoption by states

• Spec-based architecture has been created for cQube Ed v5.0 with API-first design consideration

• There are 5 blocks: Ingestion > Processing > Storage > Visualisation > Insight & Action Adapter

• All the blocks can be used individually and as an end-to-end solution by the state adopters

cQube Ed (for India)

VSK was initiated by MoE as a program to improve monitoring of schemes / programs in education

• Objective to effectively collect, monitor, correlate and analyze data to take timely decisions

• Envisioned as state-level systems to monitor schemes / programs and build accountability in field-staff

• Overall goal to drive a big leap in learning outcomes through meaningful data based actions

Today, system actors face challenges in monitoring delivery of schemes / programs in education in India

cQube layered with education-specific configurations (cQube Ed) can accelerate state VSK adoption

• A VSK Playbook will define common indicators, roles, insights and suggested actions relevant across states

• The Playbook will serve as an independent public good, enabling VSK compliance independent of tech.

• cQube Ed will provide an end-to-end solution to enable VSK and come with:

VSK Specs Open

Playbook Reference Schema APIs Schemas to capture data, insights, actions NDEAR compliant APIs for enabling interoperability

cQube Ed Software   Open

Data Ingestion and Processing Ingestion of input data from existing (DIKSHA, SARAL, ODK.) or new systems Ingestion of data from new data sources through configurable schemas Processing ingested data into aggregated indicators Data Output (out of box plugins for Visualization, Insights, Nudges) Mobile app with personalized role based dashboards and nudges Out-of-box visualizations with dashboarding tools Precomputed insights downloadable weekly as reports for key role actors Ability to add new programs to generate insights and visualizations Admin Capabilities Role mapping to indicators for visibility and accountability Weekly report templates and dashboard configurations Configure and automate in-app nudges (as a plugin)

cQube Ed Configuration Layer

The cQube Ed configurations layer will allow cQube to be VSK ready for states to begin ‘using’

• The configurations layer will allow:

  • List of pre-configured programs and indicators to monitor national and state level programs

  • Mapping of users and roles (e.g., school principal, block officer) to indicators for visibility

  • Generation of insights (e.g., benchmarks) on top of indicators

  • Mapping suggested actions (e.g., nudging rules) for users against generated insights

The cQube Ed configurations layer will leverage VSK Playbook for out-of-box configurations

• VSK Playbook will serve as a public good for enabling monitoring of ed focused schemes / programs

• VSK playbook to be created by Samagra (Q3) in discussion with states, housed within MoE

• The Playbook can be evolved and contributed to by state entities, other ed ecosystem partners and MoE

• cQube Ed has multiple deployments, further enhancement needed to be VSK Playbook ready

• Today, cQube is being leveraged by MoE (NVSK) and Jharkhand, with more states expected

• Overall, goal is to enable all states to have VSK playbook implemented before the next academic year

• Certain enhancements are needed to make it VSK playbook ready

• There are some functionalities & use cases that cQube Ed (out-of-box) will not support

• cQube Ed will continue storing aggregated data , user-level indicators & nudges not possible

E.g., viewing admin officer wise school monitoring visit compliance in a given month

• For predefined indicators, cQube Ed will not ingest data in any schema other than the one specified

• cQube Ed will allow limited operations to be performed on the datasets exposed (SQL layer)

• States will be able to adopt cQube Ed based on the evolution of their own brownfield systems

• States that have existing data input, processing and visualization solutions

  • Can configure their existing systems based on VSK playbook

  • Or can connect their data input to cQube and get processing, visualization, nudges out of the box

  • Or can connect their data input and visualization to cQube and get processing, nudges out of the box

• States that don’t have existing data input, processing and visualization solutions

  • Can setup cQube through a SI and get data processing, visualization and nudges out of the box

  • Can leverage NDEAR reference data input tools - SARAL, Shiksha

• The high level architecture for cQube Ed has been finalized

• Spec-based architecture has been created for cQube Ed v5.0 with API-first design consideration

• There are 5 blocks: Ingestion > Processing > Storage > Visualisation > Insight & Action Adapter

• All the blocks can be used individually and as an end-to-end solution by the state adopters

Problem Statement

Governments usually collect a large amount of administrative data (especially in the education context), mostly for routine reporting and compliance purposes. Many countries have recently begun finding more advanced and useful ways of leveraging these data sets to allocate resources better, measure success, and improve government-administered programs' efficiency. However, a lot of state governments in the country still operate on anecdotal rather than data-backed decision making due to some key challenges:

1. Data Quality: Data sets are useful when they are accurate and reliable. When data points are inaccurate, missing, or poorly defined, the information is less useful. In most of the state governments, the data collected is isolated and unstructured.

2. Data Accessibility: Outdated technology can make it difficult for governments to extract their data in a usable format. And when contractors control computer systems, states may not even own their data or lack the technical expertise to use it.

3. Data Usage: Data, if available, should be used and leveraged to improve performance. Even if some states have structured and accurate data, there are either no dashboards to analyze the data or the dashboards are very difficult to interpret. If some information can be interpreted from the available data, there is always a gap between information and action by the user.

4. Data Sharing: Data sets are most valuable when shared across programs or agencies and combined with other data sets to get new insights. However, many agencies are not competent to maintain interoperable datasets. They also face uncertainty about how to comply with privacy laws or share data while keeping it secure, limiting the insights gleaned from the data.

5. Expert Staffing. To use data effectively, governments need staff members to understand the technical skills to manage and analyze data. These members are in short supply leading to issues related to data usage.

These challenges become a hindrance for the state governments in implementing effective review and monitoring processes, which in result also affect the government-administered programs' efficiency.

cQube Ed

cQube is envisioned as a ready-to-use/pre-packaged, configurable, and extendable solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors

cQube Ed is a pre-packaged solution for education with 120+ pre-defined actionable indicators & insights like attendance, assessment etc. It acts as an accelerator for states in fast tracking their VSK journeys.

cQube can be extended to other domains as well, similar to education through a domain specific configuration layer.

Design Principles:

cQube is based on the following design principles:

1. Solution: cQube is neither a tool nor a platform. It is a ready-to-use/pre-packaged, configurable, and extendable solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors

2. Education-specific: A pre-packaged solution of cQube for education, cQube Ed, comes with a set of predefined actionable indicators and insights which are specific to education. For eg: the metrics could be related to attendance, enrolment, assessments etc. The schema for data ingestion will also be defined for edu-specific indicators.

3. Based in Indian Context: cQube is based in the Indian context, implying that the jurisdictions and hierarchies will be defined accordingly. For eg: The hierarchy for cQube Ed will be as follows - State > District > Block > Cluster > School > Class..

Use Cases

The major use cases envisaged to be unlocked through cQube Ed are as follows:

Decentralized Observability:

Configurability:

• Indicator: Total Number of unique students who are using tablets daily

• Type of Chart to be created: Table

• Dashboard where chart should be included: New Dashboard to be created ‘e-Adhigam’

• Dashboard should contain filters like Year & Month

1. Terminologies

1. Adopter - Leadership / Decision Makers who plan to and leverage cQube Ed

2. Domain - Area of interest and expertise that we are working on. Eg: Health, Education, etc.

3. Input Sources - Any input data sources that adopters use. Eg: Database (MIS, SQL, NoSQL) and Google Sheets / Excel / CSV, adopter applications

4. Management Information System (MIS) - A data store that houses all the raw data of adopter organizations to generate events

5. Ingestion - A system where all the data from input sources reach as the first step

6. Adapter - Generates events from MIS / any other input source and pushes them to cQube Ed through an API

7. Dataset - High-level data which is computed by aggregating events. It is a data representation of the indicator. Datasets are persistent within cQube Ed. A dataset is created for at least one indicator. This has been explained in detail with examples in this section

8. Indicator - A visual representation of a dataset(s). Eg: District-wise average attendance %.

9. Event - A data structure that records an occurrence at a particular time for an entity (eg: school, etc). It is a combination of simple data types (eg: integer, varchar, etc.). An event should always contain a column/set of columns that helps you calculate the Indicator. A table with a timestamp doesn’t necessarily mean that it is an event; it should contribute to either aggregation or filtering of the dataset. This has been explained in detail with examples in this section. Additional details are in this section.

10. Allowed Data Types - SQL compliant data types found and supported across most RDBMS implementations.

    1. Numeric data types such as int, tinyint, bigint, float, real, etc.

    2. Date and Time data types such as Date, Time, Datetime, etc.

    3. Character and String data types such as char, varchar, text, etc.

    4. Unicode character string data types, for example nchar, nvarchar, ntext, etc.

11. PII - Personal Identifiable Information

12. Data Processing - Processing involves:

    1. Transformation of Events to data that updates datasets - Transformation happens through a transformer: f(eventDetails, eventSchema, datasetSchema, dimesionConfig) = [array of columns]

    2. Updating datasets

Processing in cQube Ed is done on the event(s) being emitted.

1. Dimension - Dimensions describe events. [image] This has been explained in detail with examples in this section

2. Transformer - Operation / Function being performed on an event & dimension to process them into a dataset. This has been explained in detail with examples in this section

3. Visualization - A functional block that focussed on rendering data

4. Charts - A single sheet of information in the form of a table, graph, or diagram

5. Dashboard - A collection of charts

6. Dashboard Organizer - A WYSIWYG editor that allows the placement of charts

7. Insight - An actionable comprehension of certain data and visualizations

8. Action Adapters - A type of written communication to disseminate information in order to encourage or persuade someone to do something in a certain way

9. Plugins - An external or internal component that adds functionality to vanilla cQube Ed. Since cQube Ed is an API first design and the APIs are exposed, to add additional functionality, a plugin needs to be created.

2. Use Cases [1]

Detailed user stories for each use case have been linked here.

3. Design Considerations

1. Low Tech/Minimal Coding: Leverage existing open source tools and generalization architecture which allow the adopter to meet their needs through configurations with no or minimum coding (need to balance it with ease of deployment and manage needs).

2. API First: APIs should be built for all functionality and blocks for the independent evolution of the solution. Exposing and documenting internal structures would allow for a pluggable and contributable system all around. Evolution would be critical.

3. Progressive Enhancement: The design should always allow for changes of modules in a way that there are tiers to improvement in experience.

4. Modularity: Each individual block in cQube Ed should be modular and allow the adopter to pick and choose relevant components. Blocks should be small and solve a problem in an end-to-end manner to ensure usefulness.

5. Integrations: Adopters with higher capacity and intent should have the option to enable higher-order complex capabilities as relevant to them. The higher-order capabilities need not come as part of the solution but can be enabled as an integration. Integrations can be enabled using clear boundaries between blocks and enabling communication between them through APIs over commonly shared specifications of data transfer and storage.

6. Ease of Deployment/Management: A lot of educational bodies have a weak capacity to design, implement, deploy and operate solutions. To achieve scale -

   1. Single VM Deployment - It is important to enable a pre-packaged out-of-the-box reference solution that is easy to deploy and manage on commodity hardware with min specified requirement.

   2. Cloud-agnostic deployment: On-premise deployment on VMs, and deployment on any hyper-scale using cloud-native alternatives should be possible.

7. Scalable

   1. Ingestion: cQube Ed will ingest aggregates that will be pushed as events, the frequency of which is selected by the state admin. This could lead to a lot of very small events. The ingestion should be expandable to allow for faster ingestion and to keep everything real-time.

   2. SLA Driven: An SLA of 30 mins between ingestion and insight generation. Delays in business operations and decision-making cause the government to miss opportunities and expose them to risk.

   3. Performance: In Spite of the hardware limitations, cQube Ed should never compromise on this aspect and should clearly notify the user of what can and cannot be done. Exhaustive benchmarking suites to ensure stable performance at a specified hardware level

8. Data Privacy: cQube Ed will not store data at a user level so no PII is part of the system. Only aggregated data and insights will be stored as part of cQube Ed.

9. Accessibility: The solution should be accessible on mobile devices along with desktops.

10. Data Security: To be added

4. Technical Architecture Diagram

5. Specifications (WIP)

The links for Charts and Dashboards will be added at a later stage.

6. Feature Comparison

Note: Embedding charts from the external app in the ref viz app is not allowed - cQube Ed will enable creation of charts using an external WYSIWYG editor based on dashlet spec to enable creating visualizations.

Following table differentiates between the core of cQube Ed and the plugins that will be contributed by the adopter community.

7. How to enable an indicator as part of v5.0?

cQube Ed v5.0:

cQube Ed v5.0 is a fundamental shift from how the data is currently (as part of v4.0) ingested into cQube.

cQube v4.0 had the following ways of data ingestion:

1. Either aggregated data as CSV uploaded (zipped), or

2. Emission APIOpentelemetry

v5.0 will ingest event(s) in a format that will update the dataset(s) in some shape or form. A dataset can be derived from one or more specified events, independently (i.e. no dependency between events to update the dataset). It may additionally contain dimensions and derived values from other datasets during the mapping process.

An example is shown below.

There is a top-down approach that will be followed from an adopter perspective:

The state admin first defines the Indicators that need to be visualized on the main dashboard. For example, District-wise Average Student Attendance % with dropdown filters of Year & Month. The following is a template of how this Indicator will be visualized on a dashboard:

From the Indicators, datasets are defined. There will be at least one dataset for an Indicator being visualized. For example, for the Indicator mentioned above, the dataset will have the following columns:

In order to create this dataset, data will need to be ingested from the state databases. The state will send the data to cQube Ed as per the defined event spec. For example, the event in this case will be attendance. Following will be the schema for attendance, the format in which the state will share their data (adapters can be leveraged by the states to convert their MIS / database data into this format):

There will be requirements of some dimensions as well, to be able to produce the dataset mentioned above. For example, in this case, school will be the dimension for the ‘attendance’ event.

Next step is transformation (processing) performed to convert these events and dimensions into a dataset. For the above example,

This processing will result in the following dataset:

The dataset will then result in the following visualization on the dashboard:

Let’s take another example.

Suppose there are multiple Indicators that need to be visualized on a dashboard, for example -

• State-wide Average Attendance %

• District-wise Average Attendance %

• Block-wise Average Attendance %

• School-wise Average Attendance %

There will be a separate dataset available for each of these Indicators. For example, the respective datasets for the Indicators would be:

• State_attendance

• District_attendance

• Block_attendance

• School_attendance

The following attendance event will affect each of the datasets, that will then flow into the Indicator visualization on the dashboard.

In this way, cQube Ed v5.0 will:

• Allow for a federated data source (for example - MIS) using Adapters (push-based).

• Provide the ability to use existing viz or create Viz using a UI-based dashboard organizer OR to connect dataset (using SQL) to existing BI tools.

• Provide configurable Role Based Access - allow enforced and set as a preference

8. Blocks in the future versions of cQube Ed:

Ingestion

• All the datasets are ingested into ‘Emission Data Storage’

• Data is ingested as aggregated data in a CSV

• Student-level data for assessment and attendance is stored

Proposed Implementation -

• Data is emitted/pulled through input sources.

• The adapter then converts emitted data to an event(s), in order to make it compliant to the event spec. An adapter can also ingest datasets directly and push them to cQube Ed (using the dataset spec). An adapter is a custom logic that sits inside the state data center and monitors it for new events.
  

• Events can also be sourced using an SDK that becomes part of the state applications.

• To ensure backpressure handling, no loss of input events, and throughput, the API will ensure that events are but with a smaller event bus (and not a resource intensive solution like Kafka) with offsets and message retention as first-class citizens.

• Since the events are not processed/validated in sync with the API, the admin for input sources should be notified in case of any issues.

• Events are first-class citizens in cQube Ed and hence can be directly ingested without further changes. Since the API is expected to handle a large number of events, they can either be aggregated at the adapter level or at the processing level.

• The event bus ensures that:

  • The events are only being stored until they are processed. The SLA for processing has been kept at 4 hours. (Back pressure management)

  • Since cQube Ed will become domain agnostic, the events for each domain will be created upfront as part of the domain solution in a similar manner as shown in the example above. Events are created using an event spec and pushed upstream to the adapter to enforce them at runtime.

  • Helps aggregate similar events and thus helps optimizations.

• Any amount (< 1k events/second) of data can be ingested real-time in the form of events for aggregate entities - e.g. Classroom attendance at that moment. This can be uploaded in batches as well.

• The state can fix a frequency for data updates in the ingestion spec which will run by truncating and inserting all data

• Count of NIPUN students according to the latest spot assessment taken

• Count of students who have been consistently scoring <x% on assessments

• Count of mentors who have met their target visits, and who haven’t

Where should AAES be maintained? Who should maintain it?

Option A (Outside of cQube Ed, state to maintain) - Recommendation

• This would put the onus on the state to store the data in a certain format to visualize certain Indicators

• State will only have to manage the server space to maintain this database

• State would need to monitor their servers regularly to ensure that the events are being sent so that the respective visualizations don’t fail

Option B (As part of cQube Ed, product team to maintain)

• This would put the onus on the product to manage the required server space, custom to each state’s requirements

• Hardware costs would increase for the state in order to adopt cQube Ed, as cost for AAES will also get included

• There will be less dependency on the state to manage this event store

Processing

Processing is a two-step process

1. Transformation of Events to data that updates datasets - Transformation happens through a transformer.
   f(eventDetails, eventSchema, datasetSchema, dimesionConfig) = [array of columns]

2. Updating datasets -> Datasets = And UPSERT to the data store.

Processing is done on an event(s) being emitted by the event bus.

As explained earlier in the sections, here is an example of a transformer.

Datasets

• All the processed data is stored in ‘Output Data Storage’ which is query-able

• Datasets are created on S3

Proposed Implementation -

• Datasets will be maintained on SQL compatible data store.

• The datasets will be JSON-compliant, hence enabling visualization using existing chart types on cQube

• The datasets will also be SQL-compliant, hence enabling

  • External system to build features faster over cQube Ed (Ecosystem play)

  • Existing SQL-based charting tools, data processing tools, etc.

• Datasets will be of two types:

  • Auto Datasets - generated automatically based on the domain spec.

  • Custom Datasets - allows for an extension on the existing ones if the spec doesn’t solve for it; can be generated at runtime by using well-defined SQL constructs [1], [2]; custom dataset creation is not part of the pipeline and managed separately for ease of use. Custom datasets can also be created by the states from the auto datasets. (as part of v6.0)

Visualizations/Insights

• Generates fixed types of visualizations based on pre-defined metrics and data sources

• Logged-in users can view the visualization on the browser over the internet and metrics can also be downloaded for taking action or for further analysis

Proposed Implementation -

• Visualization will be generated using current cQube charts using the json output of the datasets, to ensure backward compatibility for current cQube users

• Some external visualization tools will also be provided to create charts using the SQL output of the datasets for new cQube users

• Over time, a visualization toolkit will also be built which will be divided into three parts: Charts, Components, and Dashboard Organizer

• Charts - This will get subsumed in the dashlets spec. All charts should conform to a Chart Spec that includes three things - Chart Layout (UI Details & Expected Data format), Query to fetch the data, and data transformation from queried data to a chart-specific format. (actionable??).

• Components - Non-chart components that become part of the dashboard like images, markdowns, decision-making, etc. These should be embeddable like charts.

• Dashboard Organizer - Allows you to position charts and components. Dashboards will always be live with the timestamp for the last updated data and will be updated every 4 hours.

• Dashlet spec will be leveraged to define the charts

• In the short term the states can either use current cQube charts or connect any visualization tool as a plugin and leverage it for creation of visualizations

• In the long term, a chart configurator tool will be integrated within cQube for creation of charts & dashboard in a WYSIWYG manner. The output produced using this tool will be a dashlet.

• A chart renderer tool will be integrated within cQube Ed to render dashlets

• Recommendation to build this chart configurator + renderer tool is to embed an open-source tool like Superset, which will allow us to focus more on what is missing (lesser effort) than building it from scratch

• Now, the charts created through this tool can be visualized in the following ways:

  • On the chart configurator + renderer tool itself

  • On the current cQube Ed visualization layer

  • On any other tools like chart.js, d3js etc.

Actions

• Rule-based Action generation: Some actions will be generated based on some rules applied to the visualizations. For example, if assessment performance is less than x %, then show y pedagogical recommendations to the teacher.

• Anomaly detection: Anomalies will be detected in the data, based on which visualization has been created. For example, assessment scores submitted by a teacher.

• Correlation and Causation on factors: Correlation and Causation can be established on a metric and factors affecting it to generate relations. For example, does attendance affect the assessment performance of students in a classroom?

• Auto-generated / Templated narrative generation: Text-based narratives can be created based on the insights generated to ease decision-making for the users. For example, the narrative can tell the district officer to conduct a review of officers of x blocks on delivery of textbooks as there is a delay in the same in these blocks.

• Rule-based Nudges: Nudges can be sent through UCI based on some set rules on the visualizations created on cQube Ed. For example, an admin can nudge a district officer to take reviews on student assessments as the overall district performance has been going down. These will be enabled by plugins that a state can create to be able to implement this functionality.

• Simulator: A simulator can be created for interaction with the data to make decisions. For example, a district officer can simulate improvement in assessment performance if textbooks are delivered to schools by x date. (Here is an example of how this will work.)

Admin Dashboard

• State will have a config-driven admin dashboard for ingestion of events, dimensions, and datasets

• This dashboard will be further extended for visualization configs such as: 2.3.4, 2,3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9.

• The implementation of the admin console can be done using form schema and react-admin.

9. Installation

1. The state deployer will be able to install cQube Ed by running a single command with minimal specs. That will also include a one-time setup for ingestion, processing, and visualization blocks.

2. The state deployer then selects the domain during cQube Ed installation.

3. The state deployer is then also able to install helper modules like adapter & Anonymized and Aggregated event store for certain use cases if required.

cQube Ed will be based on a single-tenant system. Each adopter (tenant) will have their own instance (we'll always keep separate hardwares for the cQube Ed instance). The detailed architecture of how this will be done is still being laid out and will be added to this document, once finalized.

10. Software Requirements

There will be minor changes in the current software requirements than the ones being followed - https://cqube.sunbird.org/use/software-requirements. The idea is to continue building with the existing constraints on hardware but with the ability to scale if needed horizontally.

Change in the current requirements:

For ingestion, Typescript & Nestjs (Framework of express.js) will be used.

11. Other important elements:

Authentication

Access Control

1. Hierarchy based (Parent <> Child, both in organization hierarchy or Geography)

2. Role-based

3. Attribute-based?

4. DEPA?

Domain Specification

Deployment

1. Networking - Existing network architectures can be reused without changes.

2. Automated Deployment

   1. Using Ansible scripts on Jenkins on physical machines

   2. Using Ansible scripts on Jenkins on k8s

Monitoring Tools

1. Health Checks - for all services, databases, and pipelines

2. Status Pages

3. OpenTelemetry Based Distributed Debugging

Scalability

1. Event Bus

   1. Rabbitmq

   2. Kafka

2. Storage - Horizontally

Security Implications:

• The doc builds over and above what is already shared here.

• There are security concerns over access of data as a SQL source - this will be handled through row/column level granular permissions.

• PII Management - No PII is stored as part of cQube Ed in the original form.

New Use Case Creation

1. Workflow

   1. Create a domain spec

      1. …

      2. …

   2. Redeploy

2. Development Workflow

   1. `docker-compose -f docker-compose.dev.yaml -d`. This will start all the services and continue to monitor the changes done to the code