QuML Migration Notes - From V1.0 TO V1.1

Owned by Gauraw Kumar
Last updated: Jan 16, 2024
5 min read

Problem Statement:

  • inQuiry released v2 api’s in release-6.0.0 which works only with QuML version 1.1 and above. So If you need to use v2 api, the existing data having data as per QuML version 1.0 need to be migrated first.

Solution:

Migration Tools:

sync-tool:

  • It is an existing tool in Knowlg BB. Enhancement done for QuML Migration.

  • A new command migrateQuml introduced for QuML migration requirement.

  • This tool is used to generate migration event (kafka event) based on various criteria.

  • Currently QuML 1.0 format nodes are not having metadata like qumlVersion & schemaVersion. So all nodes will be fetched which will not have qumlVersion & schemaVersion property.

  • It is recommended to generate event for Image Node (e.g: Object Type QuestionImage or QuestionSetImage) separately.

  • migrateQuml command can take below parameters:

Parameter

Description

Mandatory/Optional

Parameter

Description

Mandatory/Optional

--objectType

  • Object Type Can be passed. e.g: Question or QuestionSet

  • Multiple ObjectType can be passed. e.g: --objectType Question,QuestionImage

Mandatory

--status

  • More than one status can be passed. e.g: --status Draft,Review

  • It is recommended to migrate one status at a time. so that tracking would be easy

  • If any status is not passed, tool will consider all status available in the database

Optional

--limit

  • an integer value (multiple of 50) can be passed to control number of events to be generated at a time.

  • Generally this parameter should be used, if data volume is huge.

  • If no limit is passed, then the tool will generate events for all available objects.

Optional

--delay

  • time value in millisecond can be given to create gap between event generation for each batch (default batch size is 50)

Optional

Sample Commands:

Command

Descriotion

Command

Descriotion

migrateQuml --objectType Question --status Draft,Review --delay 2000

Migrate All Question object which are in Draft or Review status

migrateQuml --objectType Question --status Live --delay 2000

Migrate All Question object which are in Live status

migrateQuml --objectType QuestionImage --status Draft,Review --delay 2000

Migrate All QuestionImage object which are in Draft or Review status

migrateQuml --objectType QuestionSet --status Draft,Review --delay 2000

Migrate All QuestionSet object which are in status Draft or Review

migrateQuml --objectType QuestionSet --status Live --delay 2000

Migrate All QuestionSet object which are in Live status

migrateQuml --objectType QuestionSetImage --status Draft,Review --delay 2000

Migrate All QuestionSetImage object which are in Draft or Review status

 

quml-migrator:

  • quml-migrator is a flink job available in inQuiry data-pipeline repo.

  • This flink job will process migration event generated by sync tool (migrateQuml command)

  • This flink job will perform all migration activity and stamp few additional metadata as below:

    • migrationVersion - a value which will indicate the status of migration. For more details, Please refer to Migration Version Table.

    • migrationError - In case of any error during migration

    • qumlVersion - value 1.1 will be stamped, only if migration success.

    • schemaVersion - value 1.1 will be stamped, only if migration success

  • If the object/node status is Live, this job will send an event for re-publish post migration activity.

  • The re-publish operation require another flink job questionset-republish to be deployed.

  • Performance of quml-migrator flink job can be optimised using below configuration parameter:

    task { consumer.parallelism = {{ quml_migrator_consumer_parallelism }} parallelism = {{ quml_migrator_task_parallelism }} router.parallelism = {{ quml_migrator_router_parallelism }} question_migration.parallelism = {{ question_migration_parallelism }}, questionset_migration.parallelism = {{ questionset_migration_parallelism }} }

  • By default, all parameters are set to 1.

questionset-republish:

  • questionset-republish is an existing flink job in inQuiry data-pipeline repo which is enhanced for re-publish activity post QuML migration.

  • This job will not consider any image node and only publish the current version (live one).

Migration Sequence:

Step-1: All Questions including visibility Parent/Default should be migrated first (all status).

  • We can further split Question Migration based on different status, if the data volume is large.

Step-2: Migrate All QuestionSet having status other than Live.

Step-3: All QuestionSet having Live status, should be migrated.

Migration Steps:

Step-1: Execute below queries in neo4j graph database and save the output for later reference.

# Get All Question Object Count, Which need to be migrated Match(n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["Question", "QuestionImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.qumlVersion is null and n.schemaVersion is null return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count; # Get All QuestionSet Object Count, Which need to be migrated Match(n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["QuestionSet", "QuestionSetImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.qumlVersion is null and n.schemaVersion is null return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count;

Step-2: Deploy Sync-tool and run below command to migrate the data.

  • Migration Sequence has to be maintained for successful migration.

  • You can execute any command from below table as per requirement (based on data).

  • ## migrateQuml --objectType Question --status Draft,Review,Retired,Failed --delay 2000 -> Migrate all Question with status Draft,Review,Retired,Failed ## migrateQuml --objectType QuestionImage --status Draft,Review,Retired,Failed --delay 2000 -> Migrate all QuestionImage with status Draft,Review,Retired,Failed ## migrateQuml --objectType Question --status Live --delay 2000 -> Migrate all Question with Live status. ## migrateQuml --objectType QuestionSet --status Draft,Review,Retired,Failed --delay 2000 -> Migrate all QuestionSet with status Draft,Review,Retired,Failed ## migrateQuml --objectType QuestionSetImage --status Draft,Review,Retired,Failed --delay 2000 -> Migrate all QuestionSetImage with status Draft,Review,Retired,Failed ## migrateQuml --objectType QuestionSet --status Live --delay 2000 -> Migrate all QuestionSet with status Live.

Step-3: Verify the migration status.

Migration Version Description:

  • We have different values for migrationVersion metadata which indicates different stages of QuML migration. Below table covers all possible values of migrationVersion indicating succes / failure states.

migrationVersion

Description

migrationVersion

Description

3.0

Migration successful, if Asset status is other than Live

3.1

Migration and republish both successful, if Asset status is Live

2.1

Migration failed irrespective of status of the Asset.

2.2

Republish failed. Applicable only on Live status Asset

2.3

Migration skipped because of data validation issue

Migration Verification:

  • Execute below queries in neo4j graph database and compare the result with pre-migration query (query executed in Step 1) result.

  • If the count is Zero (0) for above query, Then Migration Process Completed Successfully. If not, then please check if quml-migrator job is still processing data or having any other issue.

  • Once the count is Zero, you can execute below queries in neo4j graph database to see exact status of the migration.

  • Output of above queries can be analysed based on migrationVersion and further action can be defined.