inQuiry BB has multiple components like assessment-service, flink jobs spread across multiple GitHub repositories. The objective is to decouple the inQuiry specific code from KP (Knowledge-Plaform) Git Repos and bring under inQuiry BB space in GitHub.

Current Code Structure:

github.com/project-sunbird/knowledge-platform

/knowledge-platform/assessment-api (assessment-service specific code)

/assessment-api/ assessment-service (play application)

/assessment-api/ assessment-actors (actor implementation)

/assessment-api/ qs-hierarchy-manager (hierarchy-manager)

/knowledge-platform/ontology-engine (assessment-service uses entire ontology-engine)

/ontology-engine/graph-common

/ontology-engine/graph-core_2.11

/ontology-engine/graph-dac-api

/ontology-engine/graph-engine_2.11

/ontology-engine/parseq

/knowledge-platform/platform-core (assessment-service uses entire platform-core directly or via ontology-engine)

/platform-core/actor-core

/ontology-engine/schema-validator

/ontology-engine/platform-telemetry

/ontology-engine/platform-common

/ontology-engine/cassandra-connector

/ontology-engine/kafka-client

/knowledge-platform/platform-modules (assessment-service uses only import-manager module)

/platform-modules/import-manager

/platform-modules/mimetype-manager

/platform-modules/url-manager

github.com/project-sunbird/knowledge-platform-jobs

/knowledge-platform-jobs/jobs-core (common module for all flink jobs)

/knowledge-platform-jobs/auto-creator-v2 (inQuiry BB job)

/knowledge-platform-jobs/publish-pipeline

/publish-pipeline/publish-core (common module for other publish job)

/publish-pipeline/questionset-publish (inQuiry BB job)

Dependency with Knowlg BB:

Currently all common modules are part of knowlg BB and tightly coupled with knowlg/inQuiry services/jobs.

Open Questions:

Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?
How load testing will be performed for inQuiry components? Do we get common load test infra?
Will inQuiry have its own cloud credentials or a common one? quml backend job use it.
Currently all assessment api’s are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the api roles to Question/QuestionSet Role (e.g: Question Create)?
Note: this will have impact on SunbirdEd/Diksha/etc)
What will happen to service telemetry (e.g: info/error logs) dashboard such as graylog? Do we need it? If yes, what all services/components need to be installed?

Proposed inQuiry BB Changes:

Proposed Git Repos:

Sunbird-inQuiry/quml-service (for assessment-service )
/quml-service/assessment-service
/quml-service/assessment-actors
/quml-service/qs-hierarchy-manager
/quml-service/build (build script)
/quml-service/schema (question & questionset schema)
/quml-service/definition (sunbird supported primary category & its definition script for quml)
/quml-service/tests (API Functional Test Script/Code)
Sunbird-inQuiry/quml-jobs (for backend jobs)
/quml-jobs/questionset-publish (for Question/QuestionSet Publish)
/quml-jobs/auto-creator-v2 (for import of Question/QuestionSet)
Sunbird-inQuiry/quml-editor (for questionset editor)
Sunbird-inQuiry/quml-player (for quml player)
Sunbird-inQuiry/devops (devops script for provisining, deployements)
Sunbird-inQuiry/devops-cred or devops-private (private repo)
Sunbird-inQuiry/quml-portal (dummy portal for experiencing inQuiry feature)

quml-service:

configuration change:

Cassandra keyspace configuration will be updated as per new keyspace format.
New Keyspace Format: [env_name]_[bb_name]_[keyspace_name] . e.g: dev_inquiry_hierarchy_store
Kafka topic confiuration will be updated as per new format. New format: [env_name]_[bb_name]_[topic_name]
if any other dependent services (e.g: taxonomy-service) will be deployed to inQuiry infra, that will also have similar inQuiry specific configuration changes.

schema/script change:

cassandra database script will be modified to have building block name in keyspaces.
kafka topic creation script will be modified to have building block name.

code changes:

all common module listed in Current Code Strucure section will be pushed to either a common repo or different repo for each module - Need to decide by considering maven central movement for each component?
For resolving common module dependency, we have below options:
- option 1:
  - Checkout all dependent repos along with main repo and do a sequential build (dependent module first. e.g: platform-core then graph-engine) and then build the main repo.
  - A custom jenkins build script with parameterised branch will be written for the same. so that inQuiry can point to a specific branch/tag for dependent component.
  - We need to have similar build script for local checkout and development.
- option 2:
  - Add all dependent modules as submodule.
  - Modify the jenkins build script to checkout the main repo along with all submodules.
  - For all submodules, a specific branch (e.g: 4.9.0) will be configured.
For service level code movement, we have below option:
- option 1:
  - only assessment-service related code will be committed manually in the proposed structure because we need to take the code out from knowledge-platform/assessment-api folder.
- option 2:
  - the entire knowledge-platform repo can be moved and then we can delete the unused code and modify current service folder structure.
graph-engine module need to be enhanced to either use a db connection or api call for fetching category definition. the default option would be db call, unless it is configured for api call. - for now, we can deploy taxonomy-service under inQuiry BB and use the db call. this enhancement can be taken up in next release.

Note:
1. inQuiry will build & deploy specific release version of other services (e.g: taxonomy-service, search-service, etc) and upgrade these services as and when required.

quml-jobs:

Two Jobs (questionset-publish & auto-creator-v2) will be moved to inQuiry github repo.
For above two jobs, two dependent component jobs-core and publish-cre are required.
These two common component should be present in separate repo and should be pulled during build of inQuiry specific jobs.
Three other common jobs (search-indexer, audit-event-generator, audit-history-indexer) can be build and deployed under inQuiry infra from Knowlg BB.
inQuiry will maintain only build & deployment script and BB Specific configuration for other jobs. codebase will remain under Knowlg BB.

Databases/Streams/Tools:

Neo4j:

inQuiry will have its own neo4j graph database.
No data migration will be done.
A Post installation script will be created to create necessary objects in graph db.

Cassandra:

inQuiry will share the cassandra db with Knowlg BB but will have its own keyspaces having building block names.
No data from SunbirdEd will be migrated.

Redis:

inQuiry will share the redis db with Knowlg BB but will have its own db number specific to building block.
No data from SunbirdEd will be migrated.

Elasticsearch:

inQuiry BB will share the ES db with Knowlg BB but will have its own db index specific to building block.
search indexer and other jobs will be configured to write data to inQuiry specific ES index.
search-service will be configured to inQuiry specific ES index to consume the data.
No data from SunbirdEd will be migrated.

Kafka:

inQuiry will share the kafka server with Knowlg BB but will have its own topic name specific to building block.
inQuiry will not install the data product which performs kafka topic backup.

Jenkins:

inQuiry will have its own Jenkins Server.

K8s Cluster:

inQuiry will use shared K8s Cluster but will have custom namespace.

[Design]-Codebase Movement to Sunbird inQuiry BB