Data migration problem and approach

Owned by Sajesh Kayyath
Last updated: Jun 08, 2023
3 min read

Problem Statement

With the introduction of breaking changes like multi lingual capability and QuML compliance, there was a need to support backward compatibility.

The approach that was taken was to do the following

  • Reading to do on the fly (on-demand) transformation of question and questionSet in QuML 1.0 format to QuML 1.1 format

  • Updating to do transformation of QuML 1.0 question and questionSet to QuML 1.1 and store the changes permanently.

With the pursued approach, there is a limitation encountered with the QuestionSets that contains questions which are of global visibility (i.e that can be added to any questionSets)

This is the case with crowd sourced questions which is created through CoKreat and brought into InQuiry (ED) using question import features.

Possibilities of QuestionSet is as below,

Possibility 1

QuestionSet - Q1 (Parent Visibility) - Q2 (Parent Visibility)

Visibility Parent means that the question can be used only within that QuestionSet and cannot be used outside

Possibility 2

QuestionSet - Q1 (Parent Visibility) - Q2 (Default Visibility)

Visibility Default means that the question can be used in any QuestionSet

On-demand migration and problem

When the creator edits a questionSet, we can do an auto migration to QuML 1.1.

This is straightforward when all the Questions in the QuestionSet is of visibility Parent (as explained in the Possibility 1)

When the QuestionSet contains Questions in mixed visibility (as explained in Possibility 2), we cannot auto migrate all the questions. As the visibility Default question could be in use in some other QuestionSet which is yet to be migrated.

Solutions to overcome this

The below table looks at some of the possible solutions to over come this,

<<Draft>>

Migrated Question will not be playable in the Old Player

Newly created Questions will not be playable in the Old Player

Solution Description

Pros

Cons

Solution Description

Pros

Cons

On demand migration - WITH end user intervention

  • Only QuestionSet that is updated will be migrated

    • Other QuestionSet will NOT be migrated and consumption from Old Player will not effected

  • End user is made aware of the migration and there will be a call to action (Migrate button)

  • We can’t do migration of the Questions that are public (Visibility Default)

    • Questions that cannot be migrated will need to be removed (Visibility Default) from the QuestionSet. Again needs user action.

On demand migration - WITHOUT end user intervention

  • End user is NOT aware of the migration and there will be NO call to action (Migrate button)

  • All the Questions (Visibility Default OR Parent) will be migrated

  • Need a deep copy of the Question that is Visibility Default and create it as Visibility Parent

  • If there was one Default Question now there will be multiple Questions each specific to a QuestionSet

Question

  • In the context of adopters using inQuiry what is the volume of questions that is Default Visibility and how many QuestionSets uses it? [Diksha / VDN / iGot / UPSMF]

Static Migration

  • No need for on-demand / on the fly data transformation

  • Only one time of migrating the old Question / QuestionSet and any new will be created in QuML 1.1 format

  • Consumption of old LIVE question and QuestionSet will be stopped immediately after the migration

Question

  • What is the amount of Question / QuestionSet in production system of adopters that will be affected.

  • What is the impact to adopters?

Static Migration - With data versioning

  • No need for on-demand / on the fly data transformation

  • Only one time of migrating the old Question / QuestionSet and any new will be created in QuML 1.1 format

  • Migrated questions will be playable in old player as the Question is now versioned

  • A copy of the old version data needs to be stored in the cloud when the migration happens

    • If an adopter uses 6.0.0 release without following the Static migration procedure, then the QuestionSet will not be playable

    • Needs extensive changes to the Old and New APIs to consider the version parameter for reading purpose

No Migration

  • New Question / QuestionSet will be playable in New Player

  • Old Question / QuestionSet will be playable in Old and New Player

  • Old QuestionSets cannot be modified

  • Old QuestionSets cannot be updated and sent for review or publish

  • New Questions will work only with New QuestionSet

  • Migrated Question will not be playable in the Old Player

Solutions picked & conclusion

<<TODO>>