Page Comparison

Background

inQuiry BB has multiple components like assessment-service, questionset-editor, player and Flink jobs spread across multiple GitHub repositories. The objective is to decouple the inQuiry specific code from KP (Knowledge-Platform) Git Repos and bring it under the inQuiry BB space in GitHub.

Current Code Structure:

Repo: github.com/project-sunbird/knowledge-platform

/knowledge-platform/assessment-api (assessment-service specific code)

/assessment-api/ assessment-service (play application)

/assessment-api/ assessment-actors (actor implementation)

/assessment-api/ qs-hierarchy-manager (hierarchy-manager)

/knowledge-platform/ontology-engine (assessment-service uses entire ontology-engine)

/ontology-engine/graph-common

/ontology-engine/graph-core_2.11

/ontology-engine/graph-dac-api

/ontology-engine/graph-engine_2.11

/ontology-engine/parseq

/knowledge-platform/platform-core (assessment-service uses entire platform-core directly or via ontology-engine)

/platform-core/actor-core

/ontology-engine/schema-validator

/ontology-engine/platform-telemetry

/ontology-engine/platform-common

/ontology-engine/platform-common

/ontology-engine/cassandra-connector

/ontology-engine/kafka-client

/knowledge-platform/platform-modules (assessment-service uses only import-manager module)

/platform-modules/import-manager

/platform-modules/mimetype-manager

/platform-modules/url-manager

Repo: github.com/project-sunbird/knowledge-platform-jobs

/knowledge-platform-jobs/jobs-core (common module for all flink jobs)

/knowledge-platform-jobs/auto-creator-v2 (inQuiry BB job)

/knowledge-platform-jobs/publish-pipeline

/publish-pipeline/publish-core (common module for other publish job)

/publish-pipeline/questionset-publish (inQuiry BB job)

Dependency with Knowlg BB:

1. Currently, all common modules are part of knowlg BB and are tightly coupled with knowlg/inQuiry services/jobs.

2. inQuiry uses below api’s

• category definition api’s, framework api’s - taxonomy-service

• search api - search-service

• asset api’s, content api’s, channel api - content-service

Open Questions:

1. Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?

2. How load testing will be performed for inQuiry components? Do we get common load test infra?

3. Currently, all assessment APIs are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the API roles to Question/QuestionSet Role (e.g: Question Create)?
   Note: this will have an impact on SunbirdEd/Diksha/etc)

4. What will happen to service telemetry (e.g: info/error logs) dashboard such as Graylog? Do we need it? If yes, what all services/components need to be installed?

5. Some basic functionalities around any objects (e.g: audit history, audit-event ) are currently with Knowlg. Should inQuiry also have these?

...

Proposed Git Repos:

quml-service:

configuration change:

• Cassandra's keyspace configuration will be updated as per the new keyspace format.

• New Keyspace Format: [env_name]_[bb_name]_[keyspace_name] . e.g: dev_inquiry_hierarchy_store

• Kafka topic configuration will be updated as per the new format. New format: [env_name]_[bb_name]_[topic_name]

schema/script change:

• Cassandra database script will be modified to have a building block name in keyspaces.

• Kafka topic creation script will be modified to have a building block name.

code changes:

• all common modules listed in the Current Code Structure section will be pushed to either a common repo or different repo for each module - Need to decide by considering maven central movement for each component?

• For resolving common module dependency, we have the below options:

  • option 1:

    • Check out all dependent repos along with the main repo and do a sequential build (dependent module first. e.g: platform-core then graph-engine) and then build the main repo.

    • A custom Jenkins build script with the parameterised branch will be written for the same. so that inQuiry can point to a specific branch/tag for the dependent component.

    • We need to have a similar build script for local checkout and development.

  • option 2:

    • Add all dependent modules as submodules.

    • Modify the Jenkins build script to checkout the main repo along with all submodules.

    • For all submodules, a specific branch (e.g: 4.9.0) will be configured.

• For service level code movement, we have the below option:

  • option 1:

    • only assessment-service related code will be committed manually in the proposed structure because we need to take the code out from knowledge-platform/assessment-api folder.

  • option 2:

    • the entire knowledge-platform repo can be moved and then we can delete the unused code and modify the current service folder structure.

• the graph-engine module needs to be enhanced to either use a DB connection or API call for fetching category definition. the default option would be DB call unless it is configured for API call. - for now, we can deploy taxonomy-service under inQuiry BB and use the DB call or can directly point to Knowlg keyspace. this enhancement can be taken up in the next release.

Note:
1. inQuiry will always make Knowlg api calls for dependent services. inQuiry won’t deploy any of them.

2. inQuiry will deploy search-indexer job, if graph db is going to be different for assessment-service.

quml-jobs:

• Two Jobs (questionset-publish & auto-creator-v2) will be moved to the inQuiry Github repo.

• For the above two jobs, two dependent components jobs-core and publish-core are required.

• These two common components should be present in the separate repo and should be pulled during the build of inQuiry specific jobs.

• Three other common jobs (search-indexer, audit-event-generator, audit-history-indexer) can be build and deployed under inQuiry infra from Knowlg BB.

• the inQuiry will maintain only the build & deployment script and BB Specific configuration for other jobs. the codebase will remain under Knowlg BB.

quml-editor:

Option 1:

• inQuiry will copy the collection code and make changes on top of it.

• quml-editor code will not have any collection & content related changes.

Option 2:

• A base editor needs to be designed & implement

• Both quml-editor & collection editor will use this base editor.

quml-player:

• quml-player code is already independent. So just we need to move from project-sunbird to inQuiry Github repo.

Deployment Structure:

• inQuiry is dependent on the below services and jobs from Knowlg BB:

  • search-service

  • taxonomy-service

  • content-service

  • search-indexer flink job

• inQuiry will install all dependent services/jobs in its own infra.

inQuiry & Knowlg using Same DB:

...

inQuiry portal will use Knowlg channel, category definition, asset & search APIs.

...

Micro Services

...

assessment-service:

• assessment-service will be deployed with its own configuration.

• assessment-service will be configured to make db calls for category definitions.

...

flink jobs

...

• questionset-publish:

  • It will be installed with inQuiry specific configuration.

• auto-creator-v2:

  • the job makes 2 api call (get hierarchy & add to hierarchy).

  • the api calls will be made using internal content-service url and end points (url & endpoint will be a configuration)

• search-indexer:

  • No need to install search-indexer job for inQuiry, as graph db is same.

...

DB’s

...

• Neo4j:

  • inQuiry doesn't need to install db extension, if graph db is same.

• Logstash:

  • No need to install for inQuiry

• Kafka:

  • assessment-service will use its own kafka topic for publishing question/questionset.

• Cassandra:

  • inQuiry will have its own keyspace names.

• Redis:

  • inQuiry will use db number assigned to BB.

• ES:

  • inQuiry will use Knowlg search-indexer job and will write data to Knowlg Index (e.g: compositesearch)

  • inQuiry will use Knowlg search-service and consume data

inQuiry & Knowlg using Different DB:

...

inQuiry portal will use Knowlg channel, category definition, asset & search api's.

...

Micro Services

...

assessment-service:

• assessment-service will be configured to make db calls to get the category definition as taxonomy-service will be installed.

...

flink jobs

...

• questionset-publish:

  • It will be installed with inQuiry specific configuration.

• auto-creator-v2:

  • the job makes 2 api call (get hierarchy & add to hierarchy).

  • the api calls will be made using internal content-service url and end points (url & endpoint will be a configuration)

• search-indexer:

  • inQuiry will install search-indexer job with inQuiry specific configuration.

  • The job will write data to inQuiry Index (e.g: inquiry_searchindex)

...

DB’s

...

Neo4j:

• inQuiry need to install db extension, because graph db server is different.

...

Logstash:

• Need to install for inQuiry

...

Kafka:

• a inQuiry specific topic need to create for search-indexer job also.

• assessment-service will use its own kafka topic for publishing question/questionset.

...

Cassandra:

• inQuiry will have its own keyspace names.

...

Redis:

• inQuiry will use db number assigned to BB.

ES:

...

inQuiry will use its own search-indexer job and will write data to own Index (e.g: inquiry_searchindex)

...

Components:

• inQuiry has below components:

• Databases and other tools required for inQuiry:

  • Neo4j

  • Apache Cassandra

  • Redis

  • Elasticsearch

  • Apache Kafka

  • Logstash

  • neo4j-db-extension

• All inQuiry components will have build & deployment scripts/ config in their respective repo.

Open Question:

1. We need same provisioning scripts across multiple repos. Where do we hold provisioning scripts?

2. For private devops repo, as per devops team, we can have a common one with BB specific folder or a BB Specific repo itself. Should inQuiry have its own private repo?

Dependency:

• inQuiry BB is dependent on Knowlg BB for below components in order to provide inQuiry experience:

• inQuiry needs to deploy all above Knowlg components in its own infra.

• inQuiry won't maintain any codebase for all above components. codebase and ownership will be with Knowlg BB.

• inQuiry will have build and deployment scripts for all dependencies. ???

  • devops team recommended that not to have any Knowlg scripts under inQuiry BB but have the copy of the jenkins jobs required for inQuiry. So all scripts and configuration will be still with Knowlg BB.

  • Generalise the dependent service/jobs config under Knowlg BB and inject the inQuiry specific value through private devops repo.

Goals:

1. Code Movement to inQuiry Github Repo

2. Configuration Changes for inQuiry components

3. Build and Deployment Script Movement - will be done under the guidance of the devops team.

4. Deployment of inQuiry components and dependent Knowlg components

Code Movement to inQuiry Github

• For Code Movement of assessment-service, questionset-publish, & questionset-editor, inQuiry has dependency on Knowlg BB.

• A custom build script will be written to checkout  specific branch/tag for dependent components and build them in sequential manner (e.g: platform-core first then ontology-engine).

Configuration Changes for inQuiry components:

• Cassandra keyspaces will have a building block name. Format is: [ENV]_[BB-NAME]_[KEYSPACE_NAME]

• Kafka Topics will also have a building block name. BB name will be appended after ENV name.

• All components configuration will be moved to their respective git repo.

Deployment of inQuiry components and dependent Knowlg components

• inQuiry is dependent on Knowlg BB. The dependent components are listed above under the dependency section. So for deployments of inQuiry components, we have below possible scenarios:

1. Deploy inQuiry in the same server where Knowlg BB components are deployed.

2. Deploy inQuiry in a different server than where Knowlg BB components are deployed.

3. Deploy inQuiry where Knowlg BB components are not deployed.

Deploy inQuiry in the same server where Knowlg BB components are deployed.

• Provisioning is required only for cassandra db and kafka.

• Only inQuiry components need to be deployed using inQuiry Jenkins.

• inQuiry micro-service (assessment-service) and Asynchronous Job (questionset-publish) can be deployed to use same databases and other tools (graph db, cassandra, redis, kafka)

• assessment-service needs to be configured to use Knowlg BB keyspace for primary category definition.

• Below diagram represents the deployment view:

...

Deploy inQuiry in a different server than where Knowlg BB components are deployed.

• As of now, inQuiry components need Knowlg BB components in the same server because of data dependency. So having Knowlg components in different servers won’t work.

• Eventually, inQuiry will be enhanced to support this. - Which release, should we target this??

• Below table represents why inQuiry need to share databases with Knowlg

Deploy inQuiry where Knowlg BB components are not deployed.

• All Databases & other tools (e.g: kafka, logstash) need to be provisioned first.

• inQuiry components need to be deployed along with dependent Knowlg BB components.

• All components including dependent Knolwg components can be deployed using inQuiry Jenkins itself.

...

Open Questions:

• Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?

  • Devops response: No. Only for Co-Create BB, it will be installed separately. For Other BB, shared one will be used as of now because k8s cluster is same and the service is going to be common one. BB specific db servers/services can be attached for monitoring purpose.

• How load testing will be performed for inQuiry components? Do we get common load test infra?

  • Devops Response: Not Yet Decided. Most likely Every BB will have their own load test infra.

• Currently, all assessment APIs are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the API roles to Question/QuestionSet Role (e.g: Question Create)?
  Note: this will have an impact on SunbirdEd/Diksha/etc)

• What will happen to service telemetry (e.g: info/error logs) dashboard such as Graylog? Do we need it? If yes, what all services/components need to be installed?

  • Devops Response: For telemetry testing, we need to use sunbird-ed infra and components. So BB specific deployments are not planned as of now.

• Some basic functionalities around any objects (e.g: audit history, audit-event ) are currently with Knowlg. Should inQuiry also have these?

  • As of now, these jobs will not be installed in inQuiry BBbecause of below reasons:

    • inQuiry doesn’t have required infra to process audit events.

    • audit history api is still not migrated to Knowlg BB Services.