Introduction

This wiki explain explains the current design and implementation of tracking progress and score monitoring collections. The of content and collections, the challenges we have at scale and the proposed design to handle them.

Background & Problem Statement

The sunbird platform supports monitoring of collection tracking progress and monitoringscore computation. It uses the below APIs to capture the content tracking progress data, generates progress and score metrics and provide the summary.

...

We have a single API (Content State Update) to capture all the tracking progress information. So, it has a complex logic to identify the given input is for content progress or assessment submission and etc,.

...

content_status = {
  "<content_id>": <status>
}

For ex: content_status = {
  "do_1234": 2, // Completed
  "do_1235": 2, // Completed
  "do_1236": 2, // Completed
  "do_1237": 1, // In Progress
  "do_1238": 0  // Not started
}

Key Design Problems:

1. Single API to capture all the tracking progress and score data.

2. Read after write of consumption data and basic summary.

3. Data is written and fetched from multiple tables leading to consistency issues between API and Reporting Jobs

4. The low level tables (user_content_consumption & user_assessments) grow at an exponential rate when we start to track monitor everything

5. Archiving old data is not possible as the API’s read data from low level tables

Design

To be able to handle the above design problems, we have analyzed how similar products (like netflix) track monitor everything what a user does (or views) and that too at scale. Based on the analysis we have broken down the APIs into more granular APIs with a single DB update so that each API can be scaled independently. In addition we have designed the APIs as a general purpose service (similar to asset service/graph engine) on which various use-cases can be mapped.

...

1. The update APIs can update the content status in the low level table and update the content status map in redis (using hmset). Concurrent requests would not be a problem

2. The read API can read the content status directly from redis. Meanwhile the content_status is updated in a frozen map datatype field via the activity aggregator job.

3. The job serializes by user requests and reads the status from low level table before computing the overall content_status. This would ensure consistency between API and reporting

Content Consumption APIs

...

Assessment Consumption APIs

...

Viewer Service

Viewing Service collects the “content view updates” and generate events to process and provide summary to the users.

...

The computed summary will be available from API interface to download and view.

Summary Computation Jobs - Flink:

The Flink jobs are used to read and compute the summary of a collection consumption progress when the user view ends. It also computes the score for the current view and best score using all the previous views.

...

Once the view ends, the progress and score will be updated asynchronously by the flink jobs.

Viewer Service - Database Design

user_content_consumption (
    userid text,
    collectionid text, // currently labelled as courseid
    contextid text, // currently labelled as batchid
    contentid text,
    last_access_time timestamp,
    last_completed_time timestamp,
    last_updated_time timestamp,
    progressdetails json,
    status int,
    PRIMARY KEY (userid, collectionid, contextid, contentid)
)

assessment_aggregator (
    user_id text,
    collection_id text, // currently labelled as courseid
    context_id text, // currently labelled as contextid
    content_id text,
    attempt_id text,
    created_on timestamp,
    grand_total text,
    last_attempted_on timestamp,
    questions list<frozen<question>>,
    total_max_score double,
    total_score double,
    updated_on timestamp,
    PRIMARY KEY ((user_id, collection_id), context_id, content_id, attempt_id)
)

user_activity_agg (
    activity_type text,
    activity_id text,
    user_id text,
    context_id text,
    agg map<text, int>,
    content_status frozen<map<text,int>>,
    agg_last_updated map<text, timestamp>,
    PRIMARY KEY ((activity_type, activity_id, user_id), context_id)
)

...

With Viewer-Service, we will support tracking individual content consumption alsoprogress monitoring at individual content level as well (i.e.: all types of content can be monitored for progress). Below details explain how the data will be stored for a content consumption in different scenarios.

...

{
  userid: "<userid>",
  contentid: "<contentid>"
}

INSERT into ucc(userid, collectionid, contextid, contentid, status) 
values('<userid>','<contentid>','<contentid>','<contentid>','<status>')

{
  userid: "<userid>",
  contentid: "<contentid>"
}

from ucc where userid='<userid>' and collectionid='<contentid>' 
and contextid='<contentid>' and contentid='<contentid>'

{
  userid: "<userid>",
  collectionid: "<collectionid>",
  contentid: "<contentid>"
}

INSERT into ucc(userid, collectionid, contextid, contentid, status) 
values('<userid>','<collectionid>','<collectionid>','<contentid>','<status>')

{
  userid: "<userid>",
  collection: "<collectionid>"
}

from ucc where userid='<userid>' and collectionid='<collectionid>' 
and contextid='<collectionid>'

{
  userid: "<userid>",
  collectionid: "<collectionid>",
  contextid: "<contextid>", // batchId, programId, programBatchId
  contentid: "<contentid>"
}

INSERT into ucc(userid, collectionid, contextid, contentid, status) 
values('<userid>','<collectionid>','<contextid>','<contentid>','<status>')

{
  userid: "<userid>",
  collectionid: "<collectionid>",
  contextid: "<contextid>" // batchId, programId, programBatchId
}

from ucc where userid='<userid>' and collectionid='<collectionid>' 
and contextid='<contextid>'

Config Modes

While the viewer-service as an infra component provides flexibility to configure any tracking progress monitoring use-case, there is a need for preconfigured “modes” enabled at an instance level to be able to make better use of the service and also to make any solution or usecase simpler to implement and understand.

Following are the modes one can configure the viewer service at an instance level

Strict Context Mode

Any content or collection consumption is strictly tracked monitored at the context it was consumed. This is the default mode setup currently in SunbirdEd

Entry in DB

...

API responses for various other actions done by the user:

Pros

1. User experience is straight forward and the clients do know the context in which the user is opening the content

2. No extra actions or need for complex business rules either or client or server

Cons

Carry Forward Mode

Any content or collection consumption is tracked at its level only and is carry forwarded into any context. This is the expected mode of SunbirdCB. In this mode there are two sub-modes as well:

Full Carry Forward Mode

Entry in DB

...

user_id

...

collection_id

...

context_id

...

content_id

...

status

...

Rahul

...

single-digit-addition

...

single-digit-addition

...

single-digit-addition

...

2

API responses for various other actions done by the user:

...

Action

...

Progress

...

Rahul opens the “single-digit-addition” content after going into the course “class-1-maths” when batch-1 is active

...

Complete

...

Rahul opens the “single-digit-addition” content after searching for it

...

Complete

...

batch-1 has expired and Rahul has rejoined batch-2 and opened the content “single-digit-addition”

...

Complete

...

Rahul has joined a collection “class-2-maths” which contains the “single-digit-addition” content and opens the content

...

Complete

Collection Carry Forward Mode

...

Pros

1. Simpler logic to implement within clients as they do know the context in which the user is opening the content

2. No extra actions or need for complex business rules either or client or server

Cons

1. There might be cases where a program is launched with a course which the user is already part of or completed. The user needs to redo the entire course again (consumption outside of Context will not be considered towards the specific Context)

2. User experience will be complex if the user sees two courses (one within a program and another outside of the program). The user might complete the course assuming it is part of program and would probably realise that he hasn’t taken it in the context of the program

3. There would be multiple steps required by user to reach a content within the context of a program. Will make the discovery and browsing experience complex. (may not always be possible to ensure that users ‘discover’ relevant content in the context they are supposed to)

No Context Mode

Any content or collection consumption is monitored at its level only and is carry forwarded into any context. This is the expected mode of SunbirdCB. In this mode there are two sub-modes as well:

Content Mode

In this mode we monitor everything at a content level only. The collectionid and contextid would be irrelevant.

...

API responses for various other actions done by the user:

...

Action

...

Progress

...

Rahul opens the “single-digit-addition” content after going into the course “class-1-maths” when batch-1 is active

...

Complete

...

Rahul opens the “single-digit-addition” content after searching for it

...

Not Started

...

batch-1 has expired and Rahul has rejoined batch-2 and opened the content “single-digit-addition”

...

Complete

...

Rahul has joined a collection “class-2-maths” which contains the “single-digit-addition” content and opens the content

...

Not Started

...

Rahul has joined a program “program-abc“ which contains “class-1-maths” as one of the collection. He opens the “single-digit-addition” content within this context

...

Complete

Copy Mode

In this mode, any content or collection that is tracked, is copied over into context based on the business rules configured.

Example Use-case

Rahul completes content “single-digit-addition” as part of batch “batch-1“ within the course “class-1-maths”other actions done by the user:

Pros

1. Simplicity. No extra actions or need for complex business rules either or client or server

2. Straight-forward simple user experience

Cons

1. Once a content has been complete it is complete irrespective of the time period or context

2. If a collection wants to force all users to retake the content, it would mean that one has to clone the content. This would duplicate the assets on the platform

3. If a content is modified it is still shown to user as complete. This may not have any meaning

4. If a user joins a collection, there might already be some progress indicated

Collection Mode

In this mode we monitor everything at a collection level only. The contextid would be irrelevant.

Entry in DB

API responses for various other actions done by the user:

Pros

1. Simplicity. No extra actions or need for complex business rules either or client or server

2. Straight-forward simple user experience

Cons

1. Once a collection (course) has been complete it is complete irrespective of the time period or context

2. If a programs wants to force all users to retake the collection, it would mean that one has to clone the collection. This would duplicate the assets on the platform

3. If a collection is modified (a new assessment added) it is still shown to user as complete. This may not have any meaning

4. If a user joins a program or creates a playlist, there might already be some progress indicated

Carry Forward Mode

While the instance can be configured in strict context mode, there will be multiple other modes that might be needed to overlay on top of the strict context mode to be able to solve business/program use-cases like:

Carry forward or account progress of a content or collection under certain conditions only. For ex:

1. Within a program carry forward the progress of a collection if the user has completed the collection in last 3 months

2. Within a program carry forward the progress of a collection if the user is undergoing the course within the program start and end date

3. Within a collection always account for the progress irrespective of time period

Carry forward can be done via two possible modes - Copy & Move.

Copy Mode

In this mode, any content or collection that is monitored, is copied over into context based on the business rules configured.

Let’s assume the following business rule:

An user content progress is copied into a course or a program if the user has completed the content in the last 3 months or within the program/course duration

Copy Content Mode

Entry in DB

API responses for various other actions done by the user:

Pros

1. Opens up many complex use-cases

2. UX can be made simpler. A content or collection can be shown only once

Cons

1. The complexity on the server side explodes exponentially. There needs to be multiple asynchronous jobs like:

   1. Copy the progress after evaluating the business rules on actions like “enroll”

   2. Copy the progress continuously whenever a content/collection is taken outside the context but within the acceptable business rules

2. Explainability of the data becomes very complex. Debugging any issue would be a nightmare. Support would be bombarded with explainability requests

3. UX would become complex.

   1. User would be confused why a progress is complete or not complete

   2. If a collection is part of multiple programs, usability will be very challenging if the collection doesn’t satisfy the business rules

4. UX should be designed specifically for it

5. There are other complicated behavior like if the collection is complete and added to learner passbook, would the program also add another entry or how would the learner passbook experience be?

Copy Collection Mode

Entry in DB

API responses for various other actions done by the user:

Pros

1. Opens up many complex use-cases

2. UX can be made simpler. A content or collection can be shown only once

Cons

1. The complexity on the server side explodes exponentially. There needs to be multiple asynchronous jobs like:

   1. Copy the progress after evaluating the business rules on actions like “enroll”

   2. Copy the progress continuously whenever a content/collection is taken outside the context but within the acceptable business rules

2. Explainability of the data becomes very complex. Debugging any issue would be a nightmare. Support would be bombarded with explainability requests

3. UX would become complex.

   1. User would be confused why a progress is complete or not complete

   2. If a collection is part of multiple programs, usability will be very challenging if the collection doesn’t satisfy the business rules

4. UX should be designed specifically for it

5. There are other complicated behavior like if the collection is complete and added to learner passbook, would the program also add another entry or how would the learner passbook experience be?

Move Mode

In this mode, any collection that is monitored, is moved over into context based on the business rules configured. Move mode ensures that there is only one copy for the active collection (open batch or active batch). It doesn’t support moving of content as content would have taken in multiple contexts and moving the content will lose relevance in respective contexts

Entry in DB

API responses for various other actions done by the user:

Move Mode

In this mode, any content or collection that is tracked, is moved over into context based on the business rules configured.

Example Use-case

Rahul completes content “single-digit-addition” as part of batch “batch-1“ within the course “class-1-maths”

Pros

1. Opens up complex use-cases reducing the cons of the copy mode

2. UX can be made simpler. A in-progress content or collection can be shown only once

Cons

1. There needs to be a post enrol action that evaluates the business rule and does the movement of progress

2. Explainability of the data is still complex. Support would be bombarded with explainability requests

3. UX would become complex.

   1. User would be confused why a progress is complete or not complete or where his earlier course is.

   2. If a collection is part of multiple programs, usability will be very challenging if the collection doesn’t satisfy the business rules

   3. Setting the right context is also complex. It is tough for UI to know if the course is part of the program, this would mean inverse lookup queries or APIs and providing the right choice to the user

4. UX should be designed specifically for it

5. There are other use-cases like if the collection is complete and added to learner passbook, would the program also add another entry or how would the learner passbook experience be?

Extended Enrolment Consumption:

• Every new instance adapting the sunbird platform will have to select one of the option from 3 context modes, this would allow the application to mange manage the user in avoiding the consumption of content more than once based of specific predefined rules

• Mode for any instance will be one to one mapping

• With the extended design, tracking progress and score monitoring of the user consumption can be done for any new context like program, event etc

...

{
    "userid": "<<userid>>"
    "collectionid" : "<<courseid>>",
    "contentid" :"<<contenid>>"
}

{
  "userIduserid": "<<userid>>",
  "collectionIdcollectionid": "<<courseid>>",
  "contentIdcontentid": "<<contentid>>"
}

{
    "userid": "<<userid>>",
    "collectionid": "<<courseid>>",
    "contextid": "<<programid>>",
    "contentid": "<<contentid>>"
}

{
  "userIduserid": "<<userid>>",
  "collectionIdcollectionid": "<<courseid>>",
  "contentIdcontentid": "<<contentid>>"
}

{
    "userid": "<<userid>>",
    "collectionid": "<<courseid>>",
    "contextid": "<<programid>>",
    "contentid": "<<contenid>>"
}

Content View Lifecycle:

When the user view the content in context of a collection and batch, for the first time its start, progress update and end triggers are processed. Revisit (2nd - nth view) of the content will be ignored to process and update the DB.

...

...

Handling collection and batch dependencies:

For view start, end and update, courseId collectionId and batchId contextId are non-mandatory. This would enable to track monitor the progress for any content which is not part of a coursecollection.
This is handled in two ways:

• If, collectionId and batchId contextId are part of the request, then, individual content progress and overall collection progress is captured and computed.

...

• In case of only userId and contentId, the progress is captured only for that content

...

Handling Collection Data types in DB:

• With normal collection types, the map values gets distributed to multiple sstables with append, which might lead to read latency issues

• To the handle the scenario, will consider the frozen collection types, which will helpful in avoiding tombstone and multiple sstable reads

...

Current vs New (Viewer-Service) APIs:

We need to continue supporting the current APIs before deprecate and delete. So, it requires to work with both the APIs with backward compatibility.

Enhance Current APIs to read summary from aggregate table.

Enhance the below APIs to read the progress and score metrics from user_activity_agg table.

1. Enrolment List API.

2. Content State Read API.

One time Data migration:

The content status and score metrics data should be updated to user_activity_agg table from user_enrolemnts and assessment_agg table for all the existing enrolment records.

API Spec

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "contextId": "{{batchId}}",
        "contentId": "{{contentId}}"
    }
}

{
    "id": "api.view.start",
    "ver": "v1",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "{{contentId}}": "Progress started"
    }
}

...

{
  "id": "api.summary.list",
  "ver": "v1",
  "ts": "2021-06-23 05:59:54:984+0000",
  "params": {
    "resmsgid": null,
    "msgid": "95e4942d-cbe8-477d-aebd-ad8e6de4bfc8",
    "err": null,
    "status": "success",
    "errmsg": null
  },
  "responseCode": "OK",
  "result": {
    "summary": [
      {
        "userId": "{{userId}}",
        "collectionId": "{{collectionId}}",
        "batchIdcontextId": "{{batchId}}",
        "enrolledDate": 1624275377301,
        "active": true,
        "contentStatus": {
          "{{contentId}}": {{status}}
        },
        "assessmentStatus": {
          "assessmentId": {
            "score": {{best_score}},
            "max_score": {{max_score}}
          }
        },
        "collection": {
          "identifier": "{{collectionId}}",
          "name": "{{collectionName}}",
          "logo": "{{logo Url}}",
          "leafNodesCount": {{leafNodeCount}},
          "description": "{{description}}"
        },
        "issuedCertificates": [{
          "name": "{{certName}}",
          "id": "certificateId",
          "token": "{{certToken}}",
          "lastIssuedOn": "{{lastIssuedOn}}"
        }],
        "completedOn": {{completion_date}},
        "progress": {{progress}},
        "status": {{status}}
      }
    ]
  }
}

...

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchIdcontextId": "{{batchId}}"
    }
}

Response: 
{
    "id": "api.summary.delete",
    "ver": "v1",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {}
}

...

{
    "id": "api.summary.download",
    "ver": "v1",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
      "url": "{{userId}}_viewer_summary.json"
    }
}

Conclusion:

<TODO>