...
Creator should be able to create an observation with rubrics against a framework which will have domain & criteria (criteria’s will be associated to domain).
Creator should be able to tag observation and its questions with Evidence Collection Method (ECM).
Creator should be able to decide on page numbers for questions under either ECM or Criteria
Creator should be able to define Scoring, Max and Min Scoring level (at each criteria level).
Gathering additional metadata at each framework category level should be driven be form configuration.
Consumer should be able to consume the Observation in two ways
by browsing taxonomy structure i.e: domain & criteria
by browsing list of Evidence Collection Method (ECM)
| Info |
|---|
Rubric is an algorithmic expression which can be used to decide the level of observation consumer. |
Approach 1: Creating hierarchy
...
with leafNodes directly
The taxonomy structure will be captured and stored at observation root node.
The hierarchy structure without ECM will be
frameworkTemplatemetadata will hold the taxonomy structure. sample value is as below:
| Code Block |
|---|
...
...
"frameworkTemplate": { " |
...
domain": |
...
[ |
...
|
...
|
...
{ |
...
|
...
" |
...
name": " |
...
<name_given_ |
...
by_ |
...
creator>", |
...
|
...
"domain": "domain1", |
...
" |
...
description": " |
...
" } |
...
|
...
|
...
|
...
|
...
|
...
], |
...
" |
...
criteria": |
...
[ { |
...
|
...
|
...
" |
...
name": |
...
"<name_given_by_creator>", |
...
"domain": "domain1", |
...
" |
...
description": " |
...
",
|
...
" |
...
criteria": "criteria1" |
...
|
...
|
...
|
...
|
...
}, |
...
{ " |
...
name": " |
...
<name_given_by_creator>", |
...
" |
...
domain": " |
...
domain1",
|
...
" |
...
description": |
...
"", " |
...
criteria": " |
...
criteria2" |
...
} ] |
...
|
...
} |
list of ECMs will be stored as
ecmmetadata at root levelall children will be stored at level 1
additional metadata
pageNumberneed to be stampedSample Hierarchy Structure is as below:
| Code Block |
|---|
{ "Observation": { " |
...
name": " |
...
observation", "description": "test observation", "primaryCtegory": "Observation", " |
...
identifier": " |
...
do_1_observation", |
...
"frameworkTemplate": |
...
[], "ecm": |
...
[ |
...
" |
...
ecm1" |
...
, |
...
"ecm2", "ecm3" ], |
...
"children": [ { "identifier": "do_1_question1", "name": "question1", |
...
" |
...
parent": " |
...
do_1_observation", |
...
"domain": "domain1", |
...
" |
...
criteria": " |
...
criteria1", "pageNumber": 1, |
...
" |
...
ecm": " |
...
ecm1" // ecm meta is optional }, { " |
...
identifier": " |
...
do_1_question2", "name": "question2", |
...
"parent": "do_1_observation", "domain": "domain1", |
...
"criteria": " |
...
criteria1", "pageNumber": 2, |
...
" |
...
ecm": " |
...
ecm2" |
...
// ecm meta is optional } ] |
...
}
} |
Pros:
This approach is simpler because it provides required views with minimum computation.
creation of different needs to perform two operation: filter by metadata and group by pageNumber
this approach requires minimal changes in the platform.
Cons:
player need to understand
pageNumbermetadata.player need to perform two operation: filter by metadata and group by pageNumber
storing the forms is still not solved with this approach
Approach 2: FrameworkTemplate and View
ObservationType - 'Self' (without ECM)
| Code Block | ||
|---|---|---|
| ||
{ "Observation": { "type": "self", "frameworkTemplate": { "domain": [], |
...
"criteria": [] }, "sections": [], |
...
"view": [], "children": [] } } |
| Code Block | ||
|---|---|---|
| ||
{ |
...
"Observation": { "name": "observation", |
...
"type": "self", "frameworkTemplate": { |
...
"domain": [ |
...
{ |
...
|
...
" |
...
label": |
...
"<name_given_by_creator>", |
...
|
...
" |
...
name": " |
...
domain1", " |
...
description": " |
...
" |
...
} |
...
], " |
...
criteria": [ { |
...
|
...
"name": "<name_given_by_creator>", |
...
|
...
Approach 2: FrameworkTemplate and View
Observation Type - 'Self' (without ECM)
| Code Block | ||
|---|---|---|
| ||
{ "Observation": {"domain": "domain1", "namedescription": "observation", "type": "self", "frameworkTemplatecriteria": {"criteria1" "domain": [ }, { "labelname": "<name_given_by_creator>", "namedomain": "domain1", "description": "", "criteria": "criteria2" } ], }, "criteriasections": [ { "nameid": "<namedo_given1_by_creator>section", "domainchildren": "domain1",[ "description": ""do_1_question", "criteria": "criteria1"do_2_question" ] }, }, { { "nameid": "<namedo_given2_by_creator>section", "domainchildren": "domain1",[ "descriptiondo_2_question": "", ] "criteria": "criteria2" } ], } "view": [ ] { }, "sectionstype": ["criteria", {"domain": "domain1", "idcriteria": "do_1_sectioncriteria1", "childrensection": [ "do_1_questionsection", "do_2_questionsection" ] }, ] { "id"children": "do_2_section", "children": [[ // existing structure with new question properties { "id": "do_21_question", ] "domain": "domain1", } "criteria": "criteria1" ], "view": [ }, { "typeid": "criteriado_2_question", "domain": "domain1", "criteria": "criteria1criteria2", } "section": [ ] } } |
ObservationType - 'External' (with ECM)
| Code Block |
|---|
{ "do_1_section", Observation": { "type": "external", "do_2_sectionecm": [], "frameworkTemplate": { ] } "domain": [], ] "childrencriteria": [] // existing structure with new}, question properties "sections": [], { "view": [], "idchildren": "do_1_question", [] } } |
| Code Block | ||
|---|---|---|
| ||
{ "Observation": { "domainname": "domain1observation", "identifier": "do_obs_1" "criteriatype": "criteria1external", "ecm": }["ecm1", "ecm2"], "frameworkTemplate": { "domain": [ "id { "name": "do<name_given_2by_questioncreator>", "domain": "domain1", "criteriadescription": "criteria2" } ], } } |
Observation Type - 'External' (with ECM)
| Code Block |
|---|
{ "Observationcriteria": [ { "name": "observation", { "typename": "external<name_given_by_creator>", "ecm "domain": ["ecm1domain1", "description"ecm2: ""], "frameworkTemplate "criteria": {"criteria1" "domain": [ }, { "name": "<name_given_by_creator>", "domain": "domain1", "description": "", } "criteria": "criteria2" ], } ] }, "criteriasections": [ { "nameid": "<namedo_given1_by_creator>section", "domainchildren": "domain1",[ "description": ""do_1_question", "criteriado_2_question": "criteria1" ] }, }, { { "nameid": "<namedo_given2_by_creator>section", "domainchildren": "domain1",[ "descriptiondo_2_question": "", "criteria": "criteria2"] } ] }, "sectionsview": [ { "idtype": "do_1_sectionecm", "childrenecm": ["ecm1", "section": [ "do_1_questionsection", "do_2_questionsection" ] } ], "children": [ { "id": "do_21_sectionquestion", "childrenparent": ["do_obs_1" "domain": "do_2_questiondomain1", ] "criteria": "criteria1", } ],"ecm": "ecm1" "view": [ }, { "ecmid": "ecm1do_2_question", "typedomain": "ecmdomain1", "sectioncriteria": [ "do_1_section"criteria2", "ecm": "do_2_section" ]ecm1" } ], "children": [ // existing structure with new question properties { "id": "do_1_question", "domain": "domain1", "criteria": "criteria1", "ecm": "ecm1" }, { "id": "do_2_question", "domain": "domain1", "criteria": "criteria2", "ecm": "ecm1" } ] } } |
Pros
Creating the views (ECM & non-ECM) for both editor and player becomes easier with the flattened structure.
It addresses all the possible scenarios of sequencing and presentation of question on the consumption side.
It allows to store the snapshot of the taxonomy with additional metadata.
This design allows to add any framework categories under
frameworkTemplatebased on the need and views based on any metadata (ex: ECM)
Cons
-
Open Questions:
QUESTION - The relation between the
criteriaandsections?
Behaviour:
...
Creator will see the framework hierarchical structure of domain & criteria for the first time, when creation of observation begins.
...
Once the observation is created, the taxonomy structure will be copied to the observation and same will be shown to the user from next loading of editor.
...
}
} |
| Info |
|---|
|
Pros
Creating the views (ECM & non-ECM) for both editor and player becomes easier with the flattened structure.
It addresses all the possible scenarios of sequence/order and presentation of question on the consumption side.
It allows to store the snapshot of the taxonomy with additional metadata.
This design allows to add any framework categories under
frameworkTemplatebased on the need and views based on any metadata (ex: ECM)
Cons
Every time the editor is loaded, mapping needs to be done between the taxonomy tree and questions in
childrenproperty.Proposed structure is different from the current structure supported by the platform.
If a question is deleted, it has to be removed from
sectionslist, but the question does not have any property to hold the section id to backtrack. Hence removing questions from particular `sections` becomes challenging.
Open Questions:
QUESTION - The relation between the
criteriaandsections?
Approach 3: Creating hierarchy folder structure as per the taxonomy structure
Domain, Criteria & Section will be created as nested folders under observation root node.
The hierarchy structure without ECM will be as below:
Code Block language json { "Observation": { "name": "observation", "description": "test observation", "primaryCtegory": "Observation" "identifier": "do_1_observation", "ecm":["ecm1","ecm2"] "children": [ { "identifier": "do_1_domain1", "name": "<domain name entered by user>", "domain": "domain1", "parent": "do_1_observation", "children": [ { "identifier": "do_1_criteria1", "name": "<criteria name entered by user>", "domain": "domain1", "criteria": "criteria1", "parent": "do_1_domain1", "children": [ { "identifier": "do_1_section1", "name": "section1", "domain": "domain1", "criteria": "criteria1", "parent": "do_1_criteria1", "children": [ { "identifier": "do_1_question1", "name": "question1", "parent": "do_1_section1", "domain": "domain1", "criteria": "criteria1" }, { "identifier": "do_1_question2", "name": "question2", "parent": "do_1_section1", "domain": "domain1", "criteria": "criteria1" } ] } ] } ] } ], "view": [ { "type": "criteria", "domain": "domain1", "criteria": "criteria1", "section": [ "do_1_section1" ] } ] } }
The hierarchy structure with ECM will be as below:
| Code Block |
|---|
{
"Observation": {
"name": "observation",
"description": "test observation",
"primaryCtegory": "Observation"
"identifier": "do_1_observation",
"ecm":["ecm1","ecm2","ecm3"]
"children": [
{
"identifier": "do_1_domain1",
"name": "<domain name entered by user>",
"domain": "domain1",
"parent": "do_1_observation",
"children": [
{
"identifier": "do_1_criteria1",
"name": "<criteria name entered by user>",
"domain": "domain1",
"criteria": "criteria1",
"parent": "do_1_domain1",
"children": [
{
"identifier": "do_1_section1",
"name": "section1",
"domain": "domain1",
"criteria": "criteria1",
"parent": "do_1_criteria1",
"children": [
{
"identifier": "do_1_question1",
"name": "question1",
"parent": "do_1_section1",
"domain": "domain1",
"criteria": "criteria1"
"ecm": "ecm1"
},
{
"identifier": "do_1_question2",
"name": "question2",
"parent": "do_1_section1",
"domain": "domain1",
"criteria": "criteria1",
"ecm": "ecm2"
}
]
}
]
}
]
}
],
"view": [
{
"type": "ecm",
"ecm": "ecm1"
"domain": "domain1",
"criteria": "criteria1",
"section": [
"do_1_section1"
]
},
{
"type": "ecm",
"ecm": "ecm2"
"domain": "domain1",
"criteria": "criteria1",
"section": [
"do_1_section1"
]
}
]
}
} |
Pros:
The hierarchy structure is as similar as currently supported by the platform.
storing forms for each folder is easy in the category definition
Cons:
loading the required presentation (criteria view, ecm view) at consumption requires more processing.
Rubrics at each criteria
| Code Block |
|---|
{
"Observation": {
"levels": {
"L1": {
"label": "Below Average"
},
"L2": {
"label": "Average"
},
"L3": {
"label": "Good"
}
}
}
} |
Behaviour:
Creator will see the framework hierarchical structure of domain & criteria for the first time, when creation of observation begins.
Once the observation is created, the taxonomy structure will be copied to the observation and same will be shown to the user from next loading of editor.
Label and some additional information (description, keywords, etc) can be updated by creator for the copied taxonomy structure.
Recommended Approach:
QuestionSet will have the hierarchy as usual. For Observation with rubrics, the editor will auto-create the question-set hierarchy using the framework structure.
The editor (or the question set category - Observation with Rubric) should use a config to specify the list of framework categories that are to be used to create the hierarchy.
Once the hierarchy is created, the editor shall not allow editing the structure of the hierarchy (i.e. the folders).
Regarding the pagination structure, lets add a new attribute “rendering_sequence” to the question_set object. This attribute has the following structure:
| Code Block | ||
|---|---|---|
| ||
{
"name": "name of the view, could be defaulted to the attribute used to group the questions, e.g.: ecm, criteria, score, etc",
"sequence": [
{ // one entry per section, one section may contain multiple pages
"value": "attribute value, e.g.: ecm-1, criteria-1, etc",
"name": "human readable name for the section, same as the attribute value by default",
"pages": [["list of question ids in page 1"], ["questions in page 2"]],
"index": 0
}
]
} |
No partial updates are allowed for rendering_sequence. The complete sequence will be overridden on update.
Platform will not validate the rendering_sequence. I.e, if the rendering_sequence has a question_id that doesn’t exist in the hierarchy, API will still update the value.
Old players which do not understand rendering_sequence can fall back to use the default hierarchy.
If the rendering_sequence has invalid data, player can choose to skip the question, fail the rendering of the question set or fall back to use the default hierarchy.
Rubrics should be defined on the sections of the original hierarchy using the outcome_declaration attribute (should have the same structure as response_declaration of questions).
Open Item: should Observations with rubric be a new primary category or a new object type? It depends on the PRD:
- if the editor workflow is generalised enough to embed in the question set editor or not
- if the behaviours like rendering_sequence and rubrics are applicable for any question set or not
Questions:
| Code Block |
|---|
Question Category definition
hierarchyCategories: [ domain, criteria ]
Framework1
{
domain
criteria
}
Framework2
{
topic
learningOutcome
} |
| Code Block |
|---|
Observation Type - 'External'
{
"name": "ecm",
"sequence": [
{
"value": "ECM1",
"name": "<Name_Of_The_ECM_1>",
"pages": [["do_Question_1", "do_Question_3"], ["do_Question_2"]],
"index": 0
},
{
"value": "ECM2",
"name": "<Name_Of_The_ECM_2>",
"pages": [["do_Question_4", "do_Question_5"], ["do_Question_6"]],
"index": 1
}
]
}
Observation Type - 'Self'
{
"name": "criteria",
"sequence": [
{
"value": "CRITERIA1",
"name": "<Name_Of_The_CRITERIA_1>",
"pages": [["do_Question_1", "do_Question_3"], ["do_Question_2"]],
"index": 0
},
{
"value": "CRITERIA2",
"name": "<Name_Of_The_CRITERIA_2>",
"pages": [["do_Question_4", "do_Question_5"], ["do_Question_6"]],
"index": 1
}
]
} |