Versions Compared

Old Version 1

changes.mady.by.user Gauraw Kumar

Saved on

compared with

New Version 2

changes.mady.by.user Gauraw Kumar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Migration Tools:

sync-tool:

  • It is an existing tool in Knowlg BB. Enhancement done for QuML Migration.

  • A new command migrateQuml introduced for QuML migration requirement.

  • This tool is used to generate migration event (kafka event) based on various criteria.

  • Currently QuML 1.0 format nodes are not having metadata like qumlVersion & schemaVersion. So all nodes will be fetched which will not have qumlVersion & schemaVersion property.

  • It is recommended to generate event for Image Node (e.g: Object Type QuestionImage or QuestionSetImage) separately.

  • migrateQuml command can take below parameters:

Parameter

Description

Mandatory/Optional

--objectType

  • Object Type Can be passed. e.g: Question or QuestionSet

  • Multiple ObjectType can be passed. e.g: --objectType Question,QuestionImage

Mandatory

--status

  • More than one status can be passed. e.g: --status Draft,Review

  • It is recommended to migrate one status at a time. so that tracking would be easy

  • If any status is not passed, tool will consider all status available in the database

Optional

--limit

  • an integer value (multiple of 50) can be passed to control number of events to be generated at a time.

  • Generally this parameter should be used, if data volume is huge.

  • If no limit is passed, then the tool will generate events for all available objects.

Optional

--delay

  • time value in millisecond can be given to create gap between event generation for each batch (default batch size is 50)

Optional

...

  • quml-migrator is a flink job available in inQuiry data-pipeline repo.

  • This flink job will process migration event generated by sync tool (migrateQuml command)

  • This flink job will perform all migration activity and stamp few additional metadata as below:

    • migrationVersion - a value which will indicate the status of migration. For more details, Please refer to Migration Version Table.

    • migrationError - In case of any error during migration

    • qumlVersion - value 1.1 will be stamped, only if migration success.

    • schemaVersion - value 1.1 will be stamped, only if migration success

  • If the object/node status is Live, this job will send an event for re-publish post migration activity.

  • The re-publish operation require another flink job questionset-republish to be deployed.

  • Performance of quml-migrator flink job can be optimised using below configuration parameter:

    Code Block
    task {
      consumer.parallelism = {{ quml_migrator_consumer_parallelism }}
      parallelism = {{ quml_migrator_task_parallelism }}
      router.parallelism = {{ quml_migrator_router_parallelism }}
      question_migration.parallelism = {{ question_migration_parallelism }},
      questionset_migration.parallelism = {{ questionset_migration_parallelism }}
    }

  • By default, all parameters are set to 1.

...

  • questionset-republish is an existing flink job in inQuiry data-pipeline repo which is enhanced for re-publish activity post QuML migration.

  • This job will not consider any image node and only publish the current version (live one).

...

  • Execute below queries in neo4j graph database and compare the result with pre-migration query (query executed in Step 1) result.

    Code Block
    # Get All Question Object Count, Which need to be migrated
Match(n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["Question", "QuestionImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.qumlVersion is null and n.schemaVersion is null return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count;

# Get All QuestionSet Object Count, Which need to be migrated
Match(n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["QuestionSet", "QuestionSetImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.qumlVersion is null and n.schemaVersion is null return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count;

  • If the count is Zero (0) for above query, Then Migration Process Completed Successfully. If not, then please check if quml-migrator job is still processing data or having any other issue.

  • Once the count is Zero, you can execute below queries in neo4j graph database to see exact status of the migration.

  • Code Block
    Match (n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["Question", "QuestionImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.migrationVersion>2.0 return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count,n.migrationVersion as MigrationVersion,n.migrationError as MigrationError;

Match (n:domain) where n.IL_FUNC_OBJECT_TYPE IN ["QuestionSet", "QuestionSetImage"] and n.IL_SYS_NODE_TYPE="DATA_NODE" and n.migrationVersion>2.0 return n.IL_FUNC_OBJECT_TYPE as ObjectType,n.status as Status,count(n) as Count,n.migrationVersion as MigrationVersion,n.migrationError as MigrationError;

  • Output of above queries can be analysed based on migrationVersion and further action can be defined.