[Design Brainstorming] - Viewing Service - Design

Introduction

This wiki explain the current design and implementation problems to handling the tracking and monitoring service data at scale and the proposed approach to handle it.

Background & Problem Statement

Design

Viewing Service:

Viewing Service collects the “content view updates” and generate events to process and provide summary to the users.

When a user starts viewing a content, a view entry created. There are three stages when a user view the content. They are start, progress and end. Considering these three stages we have 3 API endpoints to capture this information for each stage.

An event will be generated when a content view ends. The summary computation jobs will read these event to process and compute the overall summary of the collection.

The computed summary will be available from API interface to download and view.

Summary Computation Jobs - Flink:

The Flink jobs are used to read and compute the summary of a collection consumption progress when the user view ends. It also computes the score for the current view and best score using all the previous views.

The event is just a trigger to initiate the computation of the collection progress. The job fetches the raw data from DB to compute the overall progress.
When an assessment type content (Ex: QuestionSet) view ends, it expects the ASSESS events data to assessment submit API for score metrics computation.

Once the view ends, the progress and score will be updated asynchronously by the flink jobs.

Viewer-Service - Content Consumption Scenarios:

The user can consume a content by searching it in our platform (organically) or via a collection when the user enrolled to a course.

With Viewer-Service, we will support tracking individual content consumption also. Below details explain how the data will be stored for a content consumption in different scenarios.

User

Accessing Content

Table - user_content_consumption

A

class-1-maths (Collection)
- single-digit-addition (Content)

<By enrolling to Collection>

userId: A
collectionId: class-1-maths
batchId: class-1-maths
contentId: single-digit-addition

A

single-digit-addition (Content)

<By using platform search>

userId: A
collectionId: single-digit-addition
batchId: single-digit-addition
contentId: single-digit-addition

Content View Lifecycle:

When the user view the content in context of a collection and batch, for the first time its start, progress update and end triggers are processed. Revisit (2nd - nth view) of the content will be ignored to process and update the DB.

Shall we enable force ‘view end’ to handle the collection progress update sync issues?

View Start API should insert the row only if the row not exists.
View Update and End API should update the row only if the row exists.

Handling collection and batch dependencies

For view start, end and update, courseId and batchId are non-mandatory. This would enable to track the progress for any content which is not part of a course.
This is handled in two ways:

If, collectionId and batchId are part of the request, then, individual content progress and overall collection progress is captured and computed.

In case of only userId and contentId, the progress is captured only for that content

V1 vs V2 APIs:

We will create a version column for course_batch table and update the default value (1) for all the existing rows.

New batches creation will set the version as 2.
If the batch version is 2, it uses v2 API endpoints from viewing-service.
If the batch version not exists or 1, it uses the existing API endpoint.

API Spec

Content View Start

POST - /v2/view/start

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}",
        "contentId": "{{contentId}}"
    }
}

Response:

{
    "id": "api.view.start",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "{{contentId}}": "Progress started"
    }
}

Content View Update

POST - /v2/view/update

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}",
        "contentId": "{{contentId}}",
        "progress": 34
    }
}

Response:

200 OK:
{
    "id": "api.view.update",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "{{contentId}}": "SUCCESS"
    }
}

4XX or 5XX Error:
{
    "id": "api.view.update",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": ERR_Error_Code,
        "status": "failed",
        "errmsg": ERR_error_msg
    },
    "responseCode": "BAD_REQUEST"/"SERVER_ERROR",
    "result": {
    }
}

Content View Assess

POST - /v2/view/assess

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}",
        "contentId": "{{contentId}}",
        "assessments": [{
            {{assess_event}} //Mandatory for self-assess contents
        }]
    }
}

Response:

{
    "id": "api.view.assess",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "{{contentId}}": "SUCCESS"
    }
}

Content View End

POST - /v2/view/end

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}",
        "contentId": "{{contentId}}"
    }
}

Response:

{
    "id": "api.view.end",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "{{contentId}}": "Progress ended"
    }
}

Content View Read

POST - /v2/view/read

Request:

{
    "request": {
        "userId": "{{userId}}",
        "contentId": ["do_123", "do_1234"],
        "collectionId" : "{{collectionId}}", //optional
        "batchId": "{{batchId}}"   // optional
  
    }
}

Response:

{
    "id": "api.view.read",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
    	"userId": "{{userId}}",
    	"collectionId": "{{collectionId}}",
    	"batchId": "{{batchId}}",
        "contents": [{
          "identifier": "{contentId}",
          "progress": 45,
    	  "score": {{best_score}},
    	  "max_score": {{max_score}}
        }]
    }
}

Viewer Summary - All enrolments

GET - /v2/viewer/summary/list/:userId

Response:

{
  "id": "api.viewer.summary.list",
  "ver": "v2",
  "ts": "2021-06-23 05:59:54:984+0000",
  "params": {
    "resmsgid": null,
    "msgid": "95e4942d-cbe8-477d-aebd-ad8e6de4bfc8",
    "err": null,
    "status": "success",
    "errmsg": null
  },
  "responseCode": "OK",
  "result": {
    "summary": [
      {
        "userId": "{{userId}}",
        "collectionId": "{{collectionId}}",
        "batchId": "{{batchId}}",
        "enrolledDate": 1624275377301,
        "active": true,
        "contentStatus": {
          "{{contentId}}": {{status}}
        },
        "assessmentStatus": {
          "assessmentId": {
            "score": {{best_score}},
            "max_score": {{max_score}}
          }
        },
        "collection": {
          "identifier": "{{collectionId}}",
          "name": "{{collectionName}}",
          "logo": "{{logo Url}}",
          "leafNodesCount": {{leafNodeCount}},
          "description": "{{description}}"
        },
        "issuedCertificates": [{
          "name": "{{certName}}",
          "id": "certificateId",
          "token": "{{certToken}}",
          "lastIssuedOn": "{{lastIssuedOn}}"
        }],
        "completedOn": {{completion_date}},
        "progress": {{progress}},
        "status": {{status}}
      }
    ]
  }
}

Viewer Summary - Specific enrolment

POST - /v2/viewer/summary/read

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}"
    }
}

Response:

{
  "id": "api.viewer.summary.read",
  "ver": "v2",
  "ts": "2021-06-23 05:59:54:984+0000",
  "params": {
    "resmsgid": null,
    "msgid": "95e4942d-cbe8-477d-aebd-ad8e6de4bfc8",
    "err": null,
    "status": "success",
    "errmsg": null
  },
  "responseCode": "OK",
  "result": {
        "userId": "{{userId}}",
        "collectionId": "{{collectionId}}",
        "batchId": "{{batchId}}",
        "enrolledDate": 1624275377301,
        "active": true,
        "contentStatus": {
          "{{contentId}}": {{status}}
        },
        "assessmentStatus": {
          "assessmentId": {
            "score": {{best_score}},
            "max_score": {{max_score}}
          }
        },
        "collection": {
          "identifier": "{{collectionId}}",
          "name": "{{collectionName}}",
          "logo": "{{logo Url}}",
          "leafNodesCount": {{leafNodeCount}},
          "description": "{{description}}"
        },
        "issuedCertificates": [{
          "name": "{{certName}}",
          "id": "certificateId",
          "token": "{{certToken}}",
          "lastIssuedOn": "{{lastIssuedOn}}"
        }],
        "completedOn": {{completion_date}},
        "progress": {{progress}},
        "status": {{status}}
  }
}

Viewer Summary Delete

DELETE - /v2/viewer/summary/delete/:userId?all - To Delete all enrolments

Response:

Response: 
{
    "id": "api.viewer.summary.delete",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {}
}

DELETE - /v2/viewer/summary/delete/:userId - To Delete specific enrolments

Request:

{
    "request": {
        "userId": "{{userId}}",
        "collectionId" : "{{collectionId}}",
        "batchId": "{{batchId}}"
    }
}

Response:

Response: 
{
    "id": "api.viewer.summary.delete",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {}
}

Viewer Summary Download

GET - /v2/viewer/summary/download/:userId?format=csv

Response:

{
    "id": "api.viewer.summary.download",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
      "url": "{{userId}}_viewer_summary.csv"
    }
}

GET - /v2/viewer/summary/download/:userId

Response:

{
    "id": "api.viewer.summary.download",
    "ver": "v2",
    "ts": "2021-06-23 05:37:40:575+0000",
    "params": {
        "resmsgid": null,
        "msgid": "5e763bc2-b072-440d-916e-da787881b1b9",
        "err": null,
        "status": "success",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
      "url": "{{userId}}_viewer_summary.json"
    }
}

Conclusion:

<TODO>