Versions Compared

Old Version 3

changes.mady.by.user Gauraw Kumar

Saved on

compared with

New Version Current

changes.mady.by.user Gauraw Kumar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Background

inQuiry BB has multiple components like assessment-service, flink , questionset-editor, player and Flink jobs spread across multiple GitHub repositories. The objective is to decouple the inQuiry specific code from KP (Knowledge-PlaformPlatform) Git Repos and bring it under the inQuiry BB space in GitHub.

Current Code Structure:

github.com/project-sunbird/knowledge-platform

/knowledge-platform/assessment-api (assessment-service specific code)

...

Components:

  • inQuiry has below components:

Component

Proposed Github Repo

Description

assessment-service

Sunbird-inQuiry/assessment-service

...

/assessment-api/ assessment-actors (actor implementation)

/assessment-api/ qs-hierarchy-manager (hierarchy-manager)

/knowledge-platform/ontology-engine (assessment-service uses entire ontology-engine)

/ontology-engine/graph-common

/ontology-engine/graph-core_2.11

/ontology-engine/graph-dac-api

/ontology-engine/graph-engine_2.11

/ontology-engine/parseq

/knowledge-platform/platform-core (assessment-service uses entire platform-core directly or via ontology-engine)

/platform-core/actor-core

/ontology-engine/schema-validator

/ontology-engine/platform-telemetry

/ontology-engine/platform-common

/ontology-engine/platform-common

/ontology-engine/cassandra-connector

/ontology-engine/kafka-client

/knowledge-platform/platform-modules (assessment-service uses only import-manager module)

/platform-modules/import-manager

/platform-modules/mimetype-manager

/platform-modules/url-manager

github.com/project-sunbird/knowledge-platform-jobs

/knowledge-platform-jobs/jobs-core (common module for all flink jobs)

/knowledge-platform-jobs/auto-creator-v2 (inQuiry BB job)

/knowledge-platform-jobs/publish-pipeline

/publish-pipeline/publish-core (common module for other publish job)

/publish-pipeline/questionset-publish (inQuiry BB job)

Dependency with Knowlg BB:

  1. Currently all common modules are part of knowlg BB and tightly coupled with knowlg/inQuiry services/jobs.

Open Questions:

  1. Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?

  2. How load testing will be performed for inQuiry components? Do we get common load test infra?

  3. Will inQuiry have its own cloud credentials or a common one?

  4. Currently all assessment api’s are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the api roles to Question/QuestionSet Role (e.g: Question Create)?
    Note: this will have impact on SunbirdEd/Diksha/etc)

...

Proposed Git Repos:

  1. Sunbird-inQuiry/quml-service (for assessment-service )
    /quml-service/assessment-service
    /quml-service/assessment-actors
    /quml-service/qs-hierarchy-manager
    /quml-service/build (build script)
    /quml-service/schema (question & questionset schema)
    /quml-service/definition (sunbird supported primary category & its definition script for quml)
    /quml-service/tests (API Functional Test Script/Code)

  2. Sunbird-inQuiry/quml-jobs (for backend jobs)
    /quml-jobs/questionset-publish (for Question/QuestionSet Publish)
    /quml-jobs/auto-creator-v2 (for import of Question/QuestionSet)

  3. Sunbird-inQuiry/quml-editor (for questionset editor)

  4. Sunbird-inQuiry/quml-player (for quml player)

  5. Sunbird-inQuiry/devops (devops script for provisining, deployements)

  6. Sunbird-inQuiry/devops-cred or devops-private (private repo)

  7. Sunbird-inQuiry/quml-portal (dummy portal for experiencing inQuiry feature)

quml-service:

configuration change:

  • Cassandra keyspace configuration will be updated as per new keyspace format.

  • New Keyspace Format: [env_name]_[bb_name]_[keyspace_name] . e.g: dev_inquiry_hierarchy_store

  • Kafka topic confiuration will be updated as per new format. New format: [env_name]_[bb_name]_[topic_name]

  • if any other dependent services (e.g: taxonomy-service) will be deployed to inQuiry infra, that will also have similar inQuiry specific configuration changes.

schema/script change:

  • cassandra database script will be modified to have building block name in keyspaces.

  • kafka topic creation script will be modified to have building block name.

code changes:

...

all common module listed in Current Code Strucure section will be pushed to either a common repo or different repo for each module - Need to decide by considering maven central movement for each component?

...

For resolving common module dependency, we have below options:

  • option 1:

    • Checkout all dependent repos along with main repo and do a sequential build (dependent module first. e.g: platform-core then graph-engine) and then build the main repo.

    • A custom jenkins build script with parameterised branch will be written for the same. so that inQuiry can point to a specific branch/tag for dependent component.

    • We need to have similar build script for local checkout and development.

  • option 2:

    • Add all dependent modules as submodule.

    • Modify the jenkins build script to checkout the main repo along with all submodules.

    • For all submodules, a specific branch (e.g: 4.9.0) will be configured.

...

For service level code movement, we have below option:

  • option 1:

    • only assessment-service related code will be committed manually in the proposed structure because we need to take the code out from knowledge-platform/assessment-api folder.

  • option 2:

    • the entire knowledge-platform repo can be moved and then we can delete the unused code and modify current service folder structure.

...

Microservice for question & questionset api's

questionset-publish

Sunbird-inQuiry/data-pipeline

Asynchronous job for publishing question & questionset

player

Sunbird-inQuiry/player

quml player to render Question & QuestionSet

editor

Sunbird-inQuiry/editor

Editor for creating Question & QuestionSet

portal

Sunbird-inQuiry/portal

A dummy portal to experience inQuiry Capabilities.

  • Databases and other tools required for inQuiry:

    • Neo4j

    • Apache Cassandra

    • Redis

    • Elasticsearch

    • Apache Kafka

    • Logstash

    • neo4j-db-extension

  • All inQuiry components will have build & deployment scripts/ config in their respective repo.

Open Question:

  1. We need same provisioning scripts across multiple repos. Where do we hold provisioning scripts?

  2. For private devops repo, as per devops team, we can have a common one with BB specific folder or a BB Specific repo itself. Should inQuiry have its own private repo?

Dependency:

  • inQuiry BB is dependent on Knowlg BB for below components in order to provide inQuiry experience:

Component 

Component Type

Dependency

taxonomy-service

Microservice

  • Required for creating inQuiry specific category & its definition, framework and its related objects.

search-service

Microservice

  • Required to search existing Question/QuestionSet objects

content-service

Microservice

  • Required for asset api's as inQuiry editor needs them.

  • Once Knowlg BB Provides asset api’s as a separate service, inQuiry will stop using content-service

search-indexer

Flink Job

  • This flink job is required to sync objects from graph DB to Elasticsearch

learning-service

api service (VM Based)

  • Question & QuestionSet can have framework data (e.g: board, medium, subject).

  • System uses redis cache for validation of these attributes.

  • Population of redis cache data is part of framework publish api.

  • Once Knowlg BB migrate the framework publish api to taxonomy-service. inQuiry stop using learning-service.

  • API migration is expected in JAS release

  • inQuiry needs to deploy all above Knowlg components in its own infra.

  • inQuiry won't maintain any codebase for all above components. codebase and ownership will be with Knowlg BB.

  • inQuiry will have build and deployment scripts for all dependencies. ???

    • devops team recommended that not to have any Knowlg scripts under inQuiry BB but have the copy of the jenkins jobs required for inQuiry. So all scripts and configuration will be still with Knowlg BB.

    • Generalise the dependent service/jobs config under Knowlg BB and inject the inQuiry specific value through private devops repo.

Goals:

  1. Code Movement to inQuiry Github Repo

  2. Configuration Changes for inQuiry components

  3. Build and Deployment Script Movement - will be done under the guidance of the devops team.

  4. Deployment of inQuiry components and dependent Knowlg components

Code Movement to inQuiry Github

  • For Code Movement of assessment-service, questionset-publish, & questionset-editor, inQuiry has dependency on Knowlg BB.

Component

Knowlg Dependency

Dependency Resolution Proposal

assessment-service

The micro-service is dependent on below modules:

  1. platform-core with all its submodules

  2. ontology-engine with all its submodules.

  3. platform-modules - only import-manager submodules are required.

  • For now, inQuiry will use these components directly from Knowlg BB repo for building inQuiry components. 

  • Once Knowlg BB makes it available as a maven dependency, inQuiry will add them and the build script will be modified.  

questionset-publish

The flink job is dependent on below modules:

  1. jobs-core

  2. publish-core

  • inQuiry will take both modules from Knowlg BB repo while building the job.

  • Once SB-Obsrv is the owner for jobs-core module. Once it is available as a maven dependency, inQuiry will stop using jobs-core from Knowlg.

  • Once publish-core is available as maven dependency, inQuiry will start using it and do the build script changes.

editor

Code of collection editor and questionset editor is tightly coupled. Need to break down the code as a base-editor, collection editor & questionset editor

  • inQuiry will use the codebase directly from Knowlg BB until we have a base editor.

  • For now, inQuiry won’t maintain any codebase but will hold the ownership for questionset editor related code..

  • Once Knowlg BB will  have a base editor, inQuiry will have its own codebase for questionset editor.

player

No dependency

Codebase will move as is to inQuiry Github

  • A custom build script will be written to checkout  specific branch/tag for dependent components and build them in sequential manner (e.g: platform-core first then ontology-engine).

Configuration Changes for inQuiry components:

  • Cassandra keyspaces will have a building block name. Format is: [ENV]_[BB-NAME]_[KEYSPACE_NAME]

  • Kafka Topics will also have a building block name. BB name will be appended after ENV name.

  • All components configuration will be moved to their respective git repo.

Deployment of inQuiry components and dependent Knowlg components

  • inQuiry is dependent on Knowlg BB. The dependent components are listed above under the dependency section. So for deployments of inQuiry components, we have below possible scenarios:

  1. Deploy inQuiry in the same server where Knowlg BB components are deployed.

  2. Deploy inQuiry in a different server than where Knowlg BB components are deployed.

  3. Deploy inQuiry where Knowlg BB components are not deployed.

Deploy inQuiry in the same server where Knowlg BB components are deployed.

  • Provisioning is required only for cassandra db and kafka.

  • Only inQuiry components need to be deployed using inQuiry Jenkins.

  • inQuiry micro-service (assessment-service) and Asynchronous Job (questionset-publish) can be deployed to use same databases and other tools (graph db, cassandra, redis, kafka)

  • assessment-service needs to be configured to use Knowlg BB keyspace for primary category definition.

  • Below diagram represents the deployment view:

...

Deploy inQuiry in a different server than where Knowlg BB components are deployed.

  • As of now, inQuiry components need Knowlg BB components in the same server because of data dependency. So having Knowlg components in different servers won’t work.

  • Eventually, inQuiry will be enhanced to support this. - Which release, should we target this??

  • Below table represents why inQuiry need to share databases with Knowlg

Object Type

Description

Database Dependency

Resolution

ObjectCategoryDefinition

  • inQuiry micro-service uses ObjectCategoryDefinition data to validate Question & QuestionSet object.

  • No graph db dependency.

  • As of now cassandra db is required for fetching definition.

  • Code changes can be done to make api calls instead of db calls.

  • this behaviour can be controlled through service level configuration to make db call or api call.

License

  • Currently Question & QuestionSets are not using License but it can be configured as edge prop under object level config

  • Service needs all License values in redis cache (edge_license).

  • Currently these cache record population is part of vm setup (manual cache entry), which should be enhanced to use the available data in the platform.

  • No dependency on any other db for license

  • We can use composite search api to populate the available license data to redis cache instead of fetching from graph db.

Framework

  • Question & QuestionSet objects are having framework related configuration to validate.

  • System Fetch all framework master categories from graph db and populate orgFramework & targetFramework related data to validate the node object.

  • System/Service Validates Framework Categories (e.g: board, medium, subject, etc) using redis cache. key: cat_[FRAMEWORK_NAME][TERM_NAME]

  • population of category cache is dependent on framework publish api, which again make use of graph db.

  • graph db hard dependency is there.

  • We can have the code changes to make use of taxnomoy api (may need to introduce new category list api) or search api to get all master categories.

  • In case of cache doesn't have framework term record, We can fetch the a particular framework on demand using taxonomy api and populate the category cache.

  • Another solution for category cache could be get all list of frameworks and then filter their terms and populate category cache using taxonomy api as part of service initialization.

Deploy inQuiry where Knowlg BB components are not deployed.

  • All Databases & other tools (e.g: kafka, logstash) need to be provisioned first.

  • inQuiry components need to be deployed along with dependent Knowlg BB components.

  • All components including dependent Knolwg components can be deployed using inQuiry Jenkins itself.

...

Open Questions:

  • Is separate infra monitoring (e.g: monitoring service, grafana dashboard) will be installed for inQuiry?

    • Devops response: No. Only for Co-Create BB, it will be installed separately. For Other BB, shared one will be used as of now because k8s cluster is same and the service is going to be common one. BB specific db servers/services can be attached for monitoring purpose.

  • How load testing will be performed for inQuiry components? Do we get common load test infra?

    • Devops Response: Not Yet Decided. Most likely Every BB will have their own load test infra.

  • Currently, all assessment APIs are onboarded with Content Role (e.g: Content Create or Content Update). Should we change the API roles to Question/QuestionSet Role (e.g: Question Create)?
    Note: this will have an impact on SunbirdEd/Diksha/etc)

  • What will happen to service telemetry (e.g: info/error logs) dashboard such as Graylog? Do we need it? If yes, what all services/components need to be installed?

    • Devops Response: For telemetry testing, we need to use sunbird-ed infra and components. So BB specific deployments are not planned as of now.

  • Some basic functionalities around any objects (e.g: audit history, audit-event ) are currently with Knowlg. Should inQuiry also have these?

    • As of now, these jobs will not be installed in inQuiry BBbecause of below reasons:

      • inQuiry doesn’t have required infra to process audit events.

      • audit history api is still not migrated to Knowlg BB Services.