Publish Pipeline Changes

Since the question sets are an integral part of the interactive video, it is required that all the question sets are published before the content is published. This document talks about various approaches for achieving this publishing, and also changes to the ECAR generation to enable offline consumption of interactive videos.

This feature reuses the Question Sets, and these question sets are prompted to learners at the pre-decided interception points. So, it is necessary that the question sets used in a given video are published before the video can be published. Below are 2 approaches to achieve this

Option 1 - Single publish for question sets & video

In this option, the publishing of question sets will be deferred. The question sets and video content will be published in a single publish API call.

However, the suppression will need to happen based on a condition, and not a blanket change for all question sets.

visibility flag - question sets marked as visibility=Parent will not be published
category - question sets from selected categories will not be published

The publish job for the interactive video will need to be enhanced to first publish the question sets and then publish the video

Handling Publishing Failure

In this case, even if any one question set or the video itself fails to publish, the content will not publish (i.e. move to Failed state) and the error should be available in the content.

Other Remarks

The mobile team has seen challenges with manifest JSON beyond 2MB. So the combined manifest for the content + question sets will need to be lesser than 2MB. One way to achieve this is to put a limit on the number of questions that can be added to an interactive content.

Option 2 - Chained publishing with new coordinator job

Instead of a single publish call for question sets and content, the publishing will happen sequentially using a chain of publish events pushed to the publish topic.

A chain of published event will processed by a new job which will act as a coordinator to call further jobs depending on object type in published chain array.

Behaviour of new job -

New job will read and validate the published chain input event from topic. If the event is valid, it will read publish chain array and internally generate events and publish on respective topic of questionset-publish or content-publish depending on type of event.
The publishing of the first question set will happen first, followed by the subsequent question set, and finally the content. When the publish API for interactive video is called, all the question sets and the video is moved to Processing state. This way, in case of a failure, videos stuck in Processing state can be looked up, and the failure analysed.
In the event published to questionset-publish or content-publish job, publishChain field will be passed along with metadata to notify that this event belongs to publish chain.
Changes in existing events of the questionset-publish and content-publish job -

“publishChain” field in metadata which will contain entire publish chain event (string format)

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "content-publish",
    "type": "System"
  },
  "context": {
    "channel": "Sunbird",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "3.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_11341490059629363211",
    "ver": "1637561126014"
  },
  "edata": {
    "publish_type": "public",
    "metadata": {
      "identifier": "do_11341490059629363211",
      "mimeType": "video/mp4",
      "objectType": "Content",
      "lastPublishedBy": "",
      "pkgVersion": 2,
      "publishChain": "{\n  \"eid\": \"BE_JOB_REQUEST\",\n  \"ets\": 1634812181709,\n  \"mid\": \"LP.1634812181709.88b311c2-a733-4846-b2a2-31d9dceda436\",\n  \"actor\": {\n    \"id\": \"interactivevideo-publish\",\n    \"type\": \"System\"\n  },\n  \"context\": {\n    \"pdata\": {\n      \"ver\": \"1.0\",\n      \"id\": \"org.sunbird.platform\"\n    },\n    \"channel\": \"0133866213417451520\",\n    \"env\": \"dev\"\n  },\n  \"object\": {\n    \"ver\": \"1634810156009\",\n    \"id\": \"do_213392364484722688135\"\n  },\n  \"edata\": {\n    \"publish_type\": \"public\",\n    \"metadata\": {},\n    \"publishchain\": [\n      {\n        \"identifier\": \"do identifier of questionset 1\",\n        \"mimeType\": \"application\/vnd.sunbird.questionset\",\n        \"lastPublishedBy\": \"2ca3f787-379f-42d4-9cc3-6bc03a2d4cca\",\n        \"pkgVersion\": null,\n        \"objectType\": \"QuestionSet\",\n        \"state\": \"Processing\",\n        \"publishErr\": \"\",\n        \"order\": 1\n      },\n      {\n        \"identifier\": \"do identifier of questionset 2\",\n        \"mimeType\": \"application\/vnd.sunbird.questionset\",\n        \"lastPublishedBy\": \"2ca3f787-379f-42d4-9cc3-6bc03a2d4cca\",\n        \"pkgVersion\": null,\n        \"objectType\": \"QuestionSet\",\n        \"state\": \"Processing\",\n        \"publishErr\": \"\",\n        \"order\": 2\n      },\n      {\n        \"identifier\": \"do_213392364484722688135\",\n        \"mimeType\": \"video\/mp4\",\n        \"lastPublishedBy\": \"2ca3f787-379f-42d4-9cc3-6bc03a2d4cca\",\n        \"pkgVersion\": null,\n        \"objectType\": \"Content\",\n        \"state\": \"Processing\",\n        \"publishErr\": \"\",\n        \"order\": 3\n      }\n    ],\n    \"action\": \"publishchain\",\n    \"iteration\": 1,\n    \"contentType\": \"Resource\"\n  }\n}\n\n"
    },
    "action": "publish",
    "iteration": 1
  }
}

Changes to existing publish jobs -

Event structures of questionset-publish and content-publish will have new field “publishChain“ which identifies if the event is from chained publish job.
Change the state of question sets and video.
Send event back to topic of coordinator job with updated state of the jobs so that coordinator job can pick up the next object in publish chain.

Publish event structure -

The publish event structure will be enhanced to add the details of the chain i.e. the sequence and the IDs of the content. As the chain progresses, each publish event will have the current state of the chain i.e. which are the preceding successful publishes, and which are the ones up next. Proposed changes to the publish event are below

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1634812181709,
  "mid": "LP.1634812181709.88b311c2-a733-4846-b2a2-31d9dceda436",
  "actor": {
    "id": "interactivevideo-publish",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.sunbird.platform"
    },
    "channel": "0133866213417451520",
    "env": "dev"
  },
  "object": {
    "ver": "1634810156009",
    "id": "do_213392364484722688135"
  },
  "edata": {
    "publish_type": "public",
    "metadata": {},
    "publishchain": [
      {
        "identifier": "do identifier of questionset 1",
        "mimeType": "application/vnd.sunbird.questionset",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "QuestionSet",
        "state": "Processing",
        "publishErr": "",
        "order": 1
      },
      {
        "identifier": "do identifier of questionset 2",
        "mimeType": "application/vnd.sunbird.questionset",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "QuestionSet",
        "state": "Processing",
        "publishErr": "",
        "order": 2
      },
      {
        "identifier": "do_213392364484722688135",
        "mimeType": "video/mp4",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "Content",
        "state": "Processing",
        "publishErr": "",
        "order": 3
      }
    ],
    "action": "publishchain",
    "iteration": 1,
    "contentType": "Resource"
  }
}

As the publish progresses, the publishchain gets updated as below

    "publishchain": [
      {
        "identifier": "do identifier of questionset 1",
        "mimeType": "application/vnd.sunbird.questionset",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "QuestionSet",
        "state": "Live",
        "publishErr": "",
        "order": 1
      },
      {
        "identifier": "do identifier of questionset 2",
        "mimeType": "application/vnd.sunbird.questionset",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "QuestionSet",
        "state": "Live",
        "publishErr": "",
        "order": 2
      },
      {
        "identifier": "do id of video",
        "mimeType": "video/mp4",
        "lastPublishedBy": "2ca3f787-379f-42d4-9cc3-6bc03a2d4cca",
        "pkgVersion": null,
        "objectType": "Content",
        "state": "Processing",
        "publishErr": "",
        "order": 3
      }
    ]

There is no change to the ECAR generation, each publish event will work as today, to generate an ECAR for each question set and the content.

Option 3 - Chained publishing without new coordinator job

Changes to Publish Jobs

Both publish jobs (content & question set) will be enhanced to handle both types of events

publish
publishchain

If the event is publish, whatever today happens will continue to happen
If the event is publishchain the job will do it's work and produce another publishchain event with the chain updated

The publish chain event will be same as mentioned in option 2.

Handling Publishing Failure

There is no change, since if a question set or content fails to publish the publish error is already updated in the content. However, the UI side would need to be enhanced to give visibility of the failure to the creator.

The UI can provide a way to view a list of content in Processing state. The UI will also allow to view the publishing state of each of the Question Sets contained within the content, this will allow troubleshooting in case the Content is stuck in Processing state due to one of the Question Sets failing to publish.

EDIT on 25/11 - Removed the sections related to impact on app, player & ECAR. Since there is a separate design doc discussing the ECAR structure and the player changes here - ECAR Structure for Interactive Videos