Definition
For OGHR, Platform needs to enable user to undertake exams. Question set evaluation is currently on the client side and this implies that the correct answer is available to the client. We need to introduce the capability to process the user responses, evaluate and compute scores on the server side so that the correct answer does not have to be sent to the client side.
Background
In the present system, Question Set consist of following :
Assessment User : Active seeker of Assessment in the platform.
QuML Player : QuML Player has capability to play the questions in QuML format. It also has capability to evaluate and compute score. Response Validation, Score Computing is completely handled in player as of now. Once User Submits the overall response, client validated scores and response are sent to the backend as ASSESS events using Content State Update API.
Question Editor : Question Set Editor enables the sourcing of question and questionSet in the system.
Flink Jobs : Flink Jobs aggregates the content state using Collection Activity Aggregator, Collection Assessment Aggregator, Cert Pre-processor, Cert Generator jobs.
Question Types
Objective Types:
a) MCQ (Multiple Choice Questions)
b) MSQ (Multiple Select Questions)
c) MTF (Match The Following)
d) FTB (Fill in The Blanks)
Subjective Types
a) VSA - (Very Short Answer)
b) SA - (Short Answer)
c) LA - (Long Answer)
Building Blocks :
Building Blocks | API | Flink Jobs |
|---|---|---|
Sunbird InQuiry | Question Set Hierarchy API | |
Question List Read API | ||
QuestionSet Create API | ||
Question Create API | ||
Sunbird Lern | Content State Read | Collection Activity Aggregate |
Content State Update | Collection Assessment Aggregate | |
Enrollment List API | Cert Pre Processer | |
Cert Generator | ||
Sunbird RC | Cert Registry Download API |
Design Problems :
Allow the assessment evaluation to be done on server also for question sets that are marked accordingly.
Solution to the question needs to be excluded from the Question Read API/ Question List API for server side assessment evaluation.
Scalable server side response processing for Question Sets.
Content State Update needs to happen from the server side response processing API.
Response Processing can happen in two ways:
Entire Question Set Response Processing (Current solution scope)
Question by Question Response Processing
Current Workflow
Solution Proposed
Technical Design Details:
Creation side Enhancements:
Introduce “serverEvaluable“ attribute at question and questionSet Object
A new attribute in QuestionSet to be introduced to mark it for server side evaluation. The editor need to allow setting this flag on a questionSet/questions. The creation APIs would be updated to support this attribute. Any question that is marked as serverEvaluable:true can only be part of a questionSet that is also marked as serverEvaluable:true. Also a question set that is marked serverEvaluable:true should only contain questions with serverEvaluable:true.
(“serverEvaluable“).
//Question Set Object
"questionSet": {
"serverEvaluable": true //#true for Server Side Valuation Default:#false for client side validation
}
//Question Object
"question" : {
"serverEvaluable": true
}
Consumption side Enhancements
There are two modes of accessing questionSet post this proposed change:
a) client-evaluable mode : default (serverEvaluable attribute does not exist or is false)
In this mode, the player uses the questionSetHierarchy to fetch the hierarchy, question list API to fetch question body and the content state update API to submit the ASSESS events. There are no changes to this processing as part of this change.
b) server-evaluable mode : serverEvaluable:true
Content Compatibility needs to be set to higher value so that discovery on older clients dont happen for this questionSet
We are proposing that, in this mode, the player uses a new API for questionSetHierarchy, the existing question list API and a new submitAssessment API.
QuestionSet Hierarchy API (new POST API)
The current questionsetHierarchy API is a get call and does not take in arguments. Introduce a new POST method for QuestionSet Hierarchy API that can take in request body. This API will have payload as follows
"request": {
"contentID": "",
"collectionID": "",
"userID": "",
"attemptID": ""
}
This API would handle selection of a subset of questions and randomization (currently done by player) as indicated by the metadata in the questionSet. The API will also return a “QuestionSetToken“ which is a signed token which has user-id, content-id, collection-id,attempt-id+selected_questionid_list recieved as part of hierarchy payload. This token will further be used to validate the request during the submitAssessment API call on server.
"questionSet": {
"timeLimits": "{\"maxTime\":\"3600\"}",
"questionSetToken": "", //#Question Set token to be generated at hierarchy read API with combination of "Question Set ID + userID"
"serverEvaluable": true //#true for Server Side Valuation Default:#false for client side validation
}
QuestionSetToken : This key is almost equivalent of jwt token created as follows:
“questionSetToken“ = > {
“timestamp”: epoch
"contentID": "",
"collectionID": "",
"userID": "",
"attemptID": "",
“questionList“: [“<do-id-1>“,”<do-id-2>”,…..”<do-id-n>”]
}
2. Updates to QuestionList API
Question Read API : Any Question Associated with serverEvaluable behaviour to trim off response declaration from Question Set.
"serverEvaluable": true,
"responseDeclaration": {
"response1": {
"maxScore": 1,
"cardinality": "single",
"type": "integer",
-- To be Trimmed off ----
"correctResponse": {
"value": "0",
"outcomes": {
"SCORE": 1
}
},
-- To be Trimmed off --
}
},
There are multiple attributes which persists correct answer in QuML
a) responseDeclaration: (Shown above)
b) answer
c) editorState
3. SubmitAssessment API to evaluate user responses & score.
QuestionResponseValidateAPI (Sync API Behaviour):
QuestionSetToken generated in Hierarchy is sent as part of this request. This token will help validate that the responses are submitted for the questions that were given out to this user and also verify the time of submission.
API accepts the request payload similar to content state update API.
{
"request": {
"userId": "843a9940-720f-43ed-a415-26bbfd3da9ef",
"questionSetToken": "",
"assessments":[
{
"assessmentTs": 1681284869464,
"batchId": "0132677340746629120",
"collectionId": "do_213267731619962880127",
"userId": "843a9940-720f-43ed-a415-26bbfd3da9ef",
"attemptId": "5486724f41afb4997118e6d97695684f",
"contentId": "do_2129959063404544001107"
},
"events":[],
responses:[{
"identifier":"<question-id>",
"questionType": "",
"userResponse":[""]
}]
}],
"contents": [
{
"contentId": "do_2132671468826214401203",
"batchId": "0132677340746629120",
"status": 2,
"courseId": "do_213267731619962880127",
"lastAccessTime": "2023-04-12 12:56:45:687+0530"
},
]
}
}
The Above payload mimics ContentStateUpdate API to mimic responses as above.
Question Set Response Processing flow
QuestionList Validation using backend persistence (Future Reference)
In order to check for validity of Question inside a QuestionSet. Design proposes the value to be encrypted based on QuestionID List in hierarchy and persist this information against QuestionSetToken in stores like Redis and Cassandra. This is currently not considered.