[Design]-inQuiry Building Block Setup

Background

inQuiry BB has multiple components like assessment-service, questionset-editor, player and Flink jobs spread across multiple GitHub repositories. The objective is to decouple the inQuiry specific code from KP (Knowledge-Platform) Git Repos and bring it under the inQuiry BB space in GitHub.

Components:

inQuiry has below components:

Component	Proposed Github Repo	Description
assessment-service	Sunbird-inQuiry/api-services	Microservice for question & questionset api's
questionset-publish	Sunbird-inQuiry/data-pipeline	Asynchronous job for publishing question & questionset
player	Sunbird-inQuiry/player	quml player to render Question & QuestionSet
editor	Sunbird-inQuiry/editor	Editor for creating Question & QuestionSet
portal	Sunbird-inQuiry/portal	A dummy portal to experience inQuiry Capabilities.

Databases and other tools required for inQuiry:
- Neo4j
- Apache Cassandra
- Redis
- Elasticsearch
- Apache Kafka
- Logstash
- neo4j-db-extension
All inQuiry components will have build & deployment scripts/ config in their respective repo.

Open Question:

We need same provisioning scripts across multiple repos. Where do we hold provisioning scripts?
For private devops repo, as per devops team, we can have a common one with BB specific folder or a BB Specific repo itself. Should inQuiry have its own private repo?

Dependency:

inQuiry BB is dependent on Knowlg BB for below components in order to provide inQuiry experience:

Component	Component Type	Dependency
taxonomy-service	Microservice	Required for creating inQuiry specific category & its definition, framework and its related objects.
search-service	Microservice	Required to search existing Question/QuestionSet objects
content-service	Microservice	Required for asset api's as inQuiry editor needs them. Once Knowlg BB Provides asset api’s as a separate service, inQuiry will stop using content-service
search-indexer	Flink Job	This flink job is required to sync objects from graph DB to Elasticsearch
learning-service	api service (VM Based)	Question & QuestionSet can have framework data (e.g: board, medium, subject). System uses redis cache for validation of these attributes. Population of redis cache data is part of framework publish api. Once Knowlg BB migrate the framework publish api to taxonomy-service. inQuiry stop using learning-service. API migration is expected in JAS release

inQuiry needs to deploy all above Knowlg components in its own infra.
inQuiry won't maintain any codebase for all above components. codebase and ownership will be with Knowlg BB.
inQuiry will have build and deployment scripts for all dependencies. ???
- devops team recommended that not to have any Knowlg scripts under inQuiry BB but have the copy of the jenkins jobs required for inQuiry. So all scripts and configuration will be still with Knowlg BB.
- Generalise the dependent service/jobs config under Knowlg BB and inject the inQuiry specific value through private devops repo.

Goals:

Code Movement to inQuiry Github Repo
Configuration Changes for inQuiry components
Build and Deployment Script Movement - will be done under the guidance of the devops team.
Deployment of inQuiry components and dependent Knowlg components

Code Movement to inQuiry Github

For Code Movement of assessment-service, questionset-publish, & questionset-editor, inQuiry has dependency on Knowlg BB.

Component	Knowlg Dependency	Dependency Resolution Proposal
assessment-service	The micro-service is dependent on below modules: platform-core with all its submodules ontology-engine with all its submodules. platform-modules - only import-manager submodules are required.	For now, inQuiry will use these components directly from Knowlg BB repo for building inQuiry components. Once Knowlg BB makes it available as a maven dependency, inQuiry will add them and the build script will be modified.
questionset-publish	The flink job is dependent on below modules: jobs-core publish-core	inQuiry will take both modules from Knowlg BB repo while building the job. Once SB-Obsrv is the owner for jobs-core module. Once it is available as a maven dependency, inQuiry will stop using jobs-core from Knowlg. Once publish-core is available as maven dependency, inQuiry will start using it and do the build script changes.
editor	Code of collection editor and questionset editor is tightly coupled. Need to break down the code as a base-editor, collection editor & questionset editor	inQuiry will use the codebase directly from Knowlg BB until we have a base editor. For now, inQuiry won’t maintain any codebase but will hold the ownership for questionset editor related code.. Once Knowlg BB will have a base editor, inQuiry will have its own codebase for questionset editor.
player	No dependency	Codebase will move as is to inQuiry Github

A custom build script will be written to checkout specific branch/tag for dependent components and build them in sequential manner (e.g: platform-core first then ontology-engine).

Configuration Changes for inQuiry components:

Cassandra keyspaces will have a building block name. Format is: [ENV]_[BB-NAME]_[KEYSPACE_NAME]
Kafka Topics will also have a building block name. BB name will be appended after ENV name.
All components configuration will be moved to their respective git repo.

Deployment of inQuiry components and dependent Knowlg components

inQuiry is dependent on Knowlg BB. The dependent components are listed above under the dependency section. So for deployments of inQuiry components, we have below possible scenarios:

Deploy inQuiry in the same server where Knowlg BB components are deployed.
Deploy inQuiry in a different server than where Knowlg BB components are deployed.
Deploy inQuiry where Knowlg BB components are not deployed.

Deploy inQuiry in the same server where Knowlg BB components are deployed.

Provisioning is required only for cassandra db and kafka.
Only inQuiry components need to be deployed using inQuiry Jenkins.
inQuiry micro-service (assessment-service) and Asynchronous Job (questionset-publish) can be deployed to use same databases and other tools (graph db, cassandra, redis, kafka)
assessment-service needs to be configured to use Knowlg BB keyspace for primary category definition.
Below diagram represents the deployment view:

Deploy inQuiry in a different server than where Knowlg BB components are deployed.

As of now, inQuiry components need Knowlg BB components in the same server because of data dependency. So having Knowlg components in different servers won’t work.
Eventually, inQuiry will be enhanced to support this. - Which release, should we target this??
Below table represents why inQuiry need to share databases with Knowlg

Object Type	Description	Database Dependency	Resolution
ObjectCategoryDefinition	inQuiry micro-service uses ObjectCategoryDefinition data to validate Question & QuestionSet object.	No graph db dependency. As of now cassandra db is required for fetching definition.	Code changes can be done to make api calls instead of db calls. this behaviour can be controlled through service level configuration to make db call or api call.
License	Currently Question & QuestionSets are not using License but it can be configured as edge prop under object level config Service needs all License values in redis cache (edge_license).	Currently these cache record population is part of vm setup (manual cache entry), which should be enhanced to use the available data in the platform. No dependency on any other db for license	We can use composite search api to populate the available license data to redis cache instead of fetching from graph db.
Framework	Question & QuestionSet objects are having framework related configuration to validate. System Fetch all framework master categories from graph db and populate orgFramework & targetFramework related data to validate the node object. System/Service Validates Framework Categories (e.g: board, medium, subject, etc) using redis cache. key: cat_[FRAMEWORK_NAME][TERM_NAME] population of category cache is dependent on framework publish api, which again make use of graph db.	graph db hard dependency is there.	We can have the code changes to make use of taxnomoy api (may need to introduce new category list api) or search api to get all master categories. In case of cache doesn't have framework term record, We can fetch the a particular framework on demand using taxonomy api and populate the category cache. Another solution for category cache could be get all list of frameworks and then filter their terms and populate category cache using taxonomy api as part of service initialization.

Deploy inQuiry where Knowlg BB components are not deployed.

All Databases & other tools (e.g: kafka, logstash) need to be provisioned first.
inQuiry components need to be deployed along with dependent Knowlg BB components.
All components including dependent Knolwg components can be deployed using inQuiry Jenkins itself.

Open Questions:

Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?
- Devops response: No. Only for Co-Create BB, it will be installed separately. For Other BB, shared one will be used as of now because k8s cluster is same and the service is going to be common one. BB specific db servers/services can be attached for monitoring purpose.
How load testing will be performed for inQuiry components? Do we get common load test infra?
- Devops Response: Not Yet Decided. Most likely Every BB will have their own load test infra.
Currently, all assessment APIs are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the API roles to Question/QuestionSet Role (e.g: Question Create)?
Note: this will have an impact on SunbirdEd/Diksha/etc)
What will happen to service telemetry (e.g: info/error logs) dashboard such as Graylog? Do we need it? If yes, what all services/components need to be installed?
- Devops Response: For telemetry testing, we need to use sunbird-ed infra and components. So BB specific deployments are not planned as of now.
Some basic functionalities around any objects (e.g: audit history, audit-event ) are currently with Knowlg. Should inQuiry also have these?

Current Code Structure:

Repo: github.com/project-sunbird/knowledge-platform

/knowledge-platform/assessment-api (assessment-service specific code)

/assessment-api/ assessment-service (play application)

/assessment-api/ assessment-actors (actor implementation)

/assessment-api/ qs-hierarchy-manager (hierarchy-manager)

/knowledge-platform/ontology-engine (assessment-service uses entire ontology-engine)

/ontology-engine/graph-common

/ontology-engine/graph-core_2.11

/ontology-engine/graph-dac-api

/ontology-engine/graph-engine_2.11

/ontology-engine/parseq

/knowledge-platform/platform-core (assessment-service uses entire platform-core directly or via ontology-engine)

/platform-core/actor-core

/ontology-engine/schema-validator

/ontology-engine/platform-telemetry

/ontology-engine/platform-common

/ontology-engine/cassandra-connector

/ontology-engine/kafka-client

/knowledge-platform/platform-modules (assessment-service uses only import-manager module)

/platform-modules/import-manager

/platform-modules/mimetype-manager

/platform-modules/url-manager

Repo: github.com/project-sunbird/knowledge-platform-jobs

/knowledge-platform-jobs/jobs-core (common module for all flink jobs)

/knowledge-platform-jobs/auto-creator-v2 (inQuiry BB job)

/knowledge-platform-jobs/publish-pipeline

/publish-pipeline/publish-core (common module for other publish job)

/publish-pipeline/questionset-publish (inQuiry BB job)