Problem Statement
Currently, the resource creators have to create the entire observation, survey and observation with rubric form via excel sheet templates which is a hectic and tiresome process. Furthermore, the implementation/support team has to rework on this and upload it at the backend after checking. This creates a lot of friction in the creation of the form.
The creation of observation is not self-serviceable and requires a lot of hand holding at each step of the creation.
The creator doesn’t get an idea of the look and feel of the questions and question sets on the app while creating the form.
High Level Approach
A self-serviceable creation portal should be built for the resource creators to create observation or survey forms.
Personas / Narrative
Resource creators - To create the observation/survey via the question set editor
Reviewer - To review the observation/survey and give feedback
Sourcing Admin - To manage user role, create sourcing projects and access and track progress of sourcing activities.
Goals
The creator should be able to create questions and group them at the same time using the question set editor.
The creators should be able to create the observation form with ease in self service mode.
A review process for all the created forms is enabled on the portal.
The dependencies for creation should not come on the support or implementation team.
The creation of all the different resource types i.e. Observation and survey, should be at a single place.
Sourcing admin should be able to create a project to seek content for “Not for a target collection” and seek survey, observation and observation with rubric as content type in the sourcing project.
Sourcing Admin can manage user roles, access, and track progress of sourcing activities.
Solution Alignment
To enable the creation of the following two primary categories of content for the object type Question Set:
Observation - Observations are simple forms for collecting data for specific processes/functions.
The observations are recorded for/against an entity. An entity can be like a state, district, block, cluster or school. Data is collected for/against that selected entity (Institution/Individual). An observation form can be submitted multiple times by a user as per their need and requirement by generating multiple instances of the same form. For eg:Class observations for a class/teacher in a school by the CRPs for all schools in a cluster on a weekly basis.
An observation of the daily processes and functioning of the school by a BEO
Observation with rubric - For a comprehensive evaluation and/or accreditation purpose, such observations are conducted on the basis of a rubric driven quality framework. A framework consists of multiple domains and criteria. Levels with particular score ranges through the scoring for questions are also defined for the criteria. This provides a quantified measurement for both objective and subjective parameters to be assessed. Subjective parameters like effectiveness of teacher instructions or classroom quality and objective parameters like no. of toilets, student teacher ratio or sports equipment. For eg:
Comprehensive assessment of Schools by the HMs on the basis of a framework like Shala Siddhi or School Quality Assessment and Assurance, SQAA.
Survey - Surveys provide data as opinion / feedback of one who is filling the survey. These are not taken up or recorded against any specific entity. They can only be taken up once as the surveys are filled as feedback or information or ideas for an event/activity/process. Surveys can also be time bound and the creator can define the time period. After the defined time period, the survey expires and users will not be able to access the form. Though, data submitted by any user before expiring is always available.
The details and attributes for these three primary categories of the question set can be found in this document - https://docs.google.com/spreadsheets/d/1_6Dh0aHUGKcZIgW1-qFpSosD293lc5pUvgnBWIih8eY/edit?usp=sharing
Key Features
Plan of record
As a creator, I should be able to
Add Meta data for detail of Observation Creation (Like observation name, description, keywords, etc.)
The following meta data configuration at the observation level will affect the question behaviour also:
In observation without rubric - ‘Enable Scoring’ will add on the field of score in the questions which wouldn’t be present otherwise
In observation with rubric - 'Add evidence collection methods' will add on the field of ECM to tag to the question which wouldn’t be present otherwise
Create section and provide further details (mentioned in this story SB-22181)
Include various (text, date, MCQs both single and multi select and slider) question types
Configuring ability to add remarks or evidences for any question type.
Add Hints
Add sub questions
Allow multiple instances for questions in a section
Change question sequence
Save the draft
Submit for review
View comments, if any, by reviewer when changes are requested.
As a reviewer, I should be able to
Review and publish the submitted content
As an admin, I should be able to
Create a project to “seek content not for a target collection”
Seek survey, observation and observation with rubric as content type in the sourcing project.
Manage user roles, access, and track progress of sourcing activities.
Table comparing meta data for Survey, Observation, and Observation with Rubric
Field | Survey | Observation | Observation with Rubric | Tech design doc / notes | |
|---|---|---|---|---|---|
| 1 | Name | Yes | Yes | Yes | |
| 2 | Description | Yes | Yes | Yes | |
| 3 | Keywords | Yes | Yes | Yes | |
| 4 | Language | Yes | Yes | Yes | |
| 5 | Entity type | Not required | Select from School, Block, Cluster, District | Select from School, Block, Cluster, District | |
| 6 | Primary Audience (Suggested user type) | Where is the master list of audience stored? How do we control the list for different categories - Explanation, Observation, etc? | |||
| 7 | Scoring | Default No. Creator cannot change. | Default No. Creator can select Yes if required. | Default Yes. Creator cannot change. | |
| 8 | Observation Recorded by | Not required | Not required | Creator can choose to mark between Self and External. In External, the creator would be able to add ECMs for the observation with rubric. |
Future considerations
Creator will get all data of the observations/survey they create.
Copying and editing an existing observation.
Searching and using question from the global question bank
Copying and editing the questions from the question bank
Product Pipeline (Overall Summary)
The details of the stories and progress on those can be found in this document.
Sourcing project creation flow
(The sourcing project to seek content not for a target collection is a critical flow for observation and survey creation)
Detailed flow of creating an Sourcing project
Sourcing org admin logs into the system and starts creating a sourcing project
The system provides option to select the project scope
-Seek content for a certain frameworkAs the user selects the option “Seek content for a certain framework”, the following flow is triggered.
The use has following options to select as the project scope.
There will be a default framework selected as per system configuration.
Selecting framework type is mandatory. Only one framework type can be selected per project.
The metadata fields for the sourcing project will be captured based on the selected framework for the sourcing project.
For any framework, an observation, survey or observation with rubric can be created. However, a certain few things can be considered
- For students, observation might not be relevant as observations are done against entities and this entity structure is not applicable.
-There is no direct impact of the selected framework on observation without rubric and survey
-On the basis of the selected framework, the values for tree structure of domain and criteria would be auto-populated. As explained below, for observation with rubric the level concept category from framework will be taken in use for domain criteria structure.When a framework type is chosen, the framework available in the tenant of the selected framework type is taken as the framework to be used for the project. The assumption is that there is a maximum one framework of a given type that is configured in the tenant. In case there is no framework of the selected type configured at tenant level, the framework of that type configured at system level is taken. The assumption is that a maximum one framework of a given type is configured at the system level. In case there are no framework of the selected found either at tenant or system level, system shows an error message “Incorrect framework configuration. Please check with the system administrator”.
Users can select one or more values from the framework category. None of the attributes are mandatory. Users also have the option to select all.
Observations, observation with rubric and surveys will be the new primary categories in content type dropdown while creating the sourcing project.
Users have the option to select one or more content types - this should show all primary categories of all the non-collection object types (content, question set).
Apart from the project scope, the user enters other project details that are the same as the existing project creation flow, except selecting target collections.
Users can save the project as draft.
Users can reopen a saved project in draft state. The saved values are displayed. Users can modify any values and save.
Users can publish the project once all required details are provided.
The bulk upload option on the contribution page, would not be applicable for observation and survey since bulk upload is only for uploaded resources (like pdfs, MP4 etc.) and not for resources that are to be created on the portal.
Step-1
After logging in
Step-2
After clicking on ‘Create new project”
Step-3
Filling the project details
Step-4
Selecting framework attributes and contribution settings
Step 5
Publishing
Contribution flow for Survey and Observation
Key flows for Creation for Observation or Survey
Actions | Conditions | Expected Results |
Log in on the contribution portal | Received a link from sourcing admin for contribution | The creator will see the sourcing project’s detail page for which link was shared under “My projects” Tab. |
Creator is a contributor under the org | The creator will see the list of all open projects under “All projects” and assigned projects for contribution under “My Project” | |
Creator is contributing to open projects through nomination | The creator will see the list of open projects under “All projects” Tab. | |
Creators click on the “Open” button for a sourcing project. | Creator was not logging in through the link. | Should be able to see the details of the project and the option to create or upload content. |
Creator will click on “Create/Upload’ button to start contribution | The contribution date shouldn’t have passed, else this button will be disabled. | Creators will see a pop up with tile of different content types |
Creator selects the content type and clicks on continue. | One content type need to be selected. | Details page for that content type will open up |
Creator fills all the mandatory metadata fields. (If mandatory metadata is not filled the observation would not be sent for review.) | The fields on the details page will be different according to the selected content type. | Add section button will be enabled. |
Creator clicks on the ‘Add Section’ button. | A section is created and will get added in the section list on the left hand side | |
Creator edits the section name | The section name changes on the left side list of the sections too too | |
Creator selects if multiple instances of questions in section are required | If creator selects “Yes” | The creator needs to add the label, accordingly. |
After creating the section creator will click on the “Add question” | One of the question type needs to be selected | As per the selected question type the question details page would open up |
Creator will fill in all the details according to the question type. | The question will be created and added to the section. |
Jira stories
Please find here Link to stories on Jira for specific stories of the flow.
Please find the UI screens here
New Question categories that would be added
Slider - Slider easily captures range based response, it shows a range specified by creator, the user can drag the point to the option he wants to choose. (Please refer the ‘Slider’)
Text - These would be subjective response type questions for user to enter the responses in free text. Number input, which prompts only number response by user, will be a validation in text type question
Date - Date type questions for user to input date in set format. Creator can also set configuration for auto capturing the date of the day.
MCQ - The MCQs will be for providing multiple choice based question
Single select response questions (configured by default) let the users select just one option in an MCQ.
Multi-select response question (creator can set this configuration) lets user select multiple response in an MCQ.
The document Question category in observation also details the attributes for the different question types.
The document https://project-sunbird.atlassian.net/l/c/HixG4v2Z also details out the approach for the dependent questions' functionality.
Contribution flow for observation with rubric
Observation with rubric contribution flow
Framework and Domain criteria values for Observation with rubric
An observation with Rubric content would be created for a particular framework which would also be tagged in the ‘sourcing project not for a target collection’. The observation would have a Domain-criteria structure that would be defined through the values in these frameworks for seeking content in the sourcing project.
[Underlying assumption - The frameworks that would be used for the purpose of the Observation with rubric would have to have at least a tree structure. This is because the Domain - Criteria values can only be taken from the tree structure. The further explanation to the values being taken from the tree is as below.]
The observation with rubric will directly derive ‘the level concept’ values as the domain criteria values. The highest level concept will be taken as domain and the lowest level concept as criteria. For eg. If there are 3 level concept L1 concept, L2 concept and L3 concept, then, the L1 concept values will be taken as domain values in the Observation with rubric and the L3 concept values will be taken as criteria values.
The frameworks like CPD will have a particular flat list of concepts over different levels like L1 concepts, L2 concepts, etc. The values in these level concepts would have direct associations among each, like, multiple L2 concepts would be associated to a particular L1 concept, multiple L3 concept would be associated to a particular L2 concept.
Key flows for Creation
Actions | Conditions | Expected Results | Notes |
Log in on the contribution portal | Received a link from sourcing admin for contribution | The creator will see the sourcing project’s detail page for which link was shared under “My projects” Tab. | No changes required |
Creator is a contributor under the org | The creator will see the list of all open projects under “All projects” and assigned projects for contribution under “My Project” | ||
Creator is contributing to open projects through nomination | The creator will see the list of open projects under “All projects” Tab. | ||
Creators click on the “Open” button for a sourcing project. | Creator was not logging in through the link. | Should be able to see the details of the project and the option to create or upload content. | |
Creator will click on “Create/Upload’ button to start contribution | The contribution date shouldn’t have passed, else this button will be disabled. | Creators will see a pop up with tile of different content types | |
Creator selects the content type and clicks on continue. | One content type need to be selected | Details page for that content type will open up | Configure survey, observation, observation with rubric. |
Creator fills all the mandatory metadata fields. (Creator can choose to fill the details later but before publishing the form all mandatory fields need to be filled) | The fields on the details page will be different according to the selected content type. | The existing fields of name, keyword, description and language will not need any change. The new metadata fields added will be Observation Recorded by, entity type, suggested user type will be added for this primary category. | |
Select the entity type of the form. The values will be school, block, cluster, district and state and will be same in both observation with and without rubric | The selected entity will be defined as the entity type of the observation form and on consumption user would be able to add specific entity to the form for that type only. | The entity layers of school, cluster, block, district, state are as existing in the UDISE system which Diksha is following. Thus, across education system the list will always be same. | |
Select the ‘language’ of the form | Creator is able to select any language for their form from all available languages. | All language available in the Diksha system should be available here. If the language once selected is changed later by the creator, the change would not directly impact the creation of questions, however the language of the form would be indicated from this selection. Thus, if creator changes language later, they will get a pop up to alerting that the question have been created in different language. (*The language 2 for translation will only be taken as enhancement later and not in the first cut.) | |
Select in the ‘Observation Recorded by’ field, who would be taking up the observation an “external” team or if it will done by “self” | If External is chosen, option to ‘Add ECMs’ will come up. If Add ECMs is selected, creator must tag ECMs for all questions before publishing. Default selection will be ‘self’. | A creator can change from self to external and external to self later too. On changing from external to self, if there were ECMs, the ECM would be removed but the questions and other info would be kept. On changing from self to external, nothing would change until ECMs are also enabled. If ECMs are enabled all the questions added previously without ECM tagging would be grouped under ‘No ECM’. On the pagination screen the ‘No ECM’ label would also show up in the left side tree within which the questions without ECM tagging will come. If the creator goes back to question creation page of such a question, they would see the select ECM option to reselect from the available ECMs. | |
Select the list of users for whom this observation is recommended. This list will be pre-populated and will be same for both observation and survey. Contents of the list will be
| This is for recommendation and will not have any implications on the consumption side | As per the selection by the creator in this field of 'Suggested User Type', the user would see the relevant form in the library (whenever this would be enabled for observations) according to their profile. Eg, if creator selects teacher, only user declaring role as teacher on consumption would see this. | |
Creator should be able to choose ECM’s from the list. ECM - It is the method of data collection being used by the observer to collect data/responses for the form. Question in a single ECM can be from across different criteria and domain. | If they have chosen to add ECM and selected a few ECMs, the questions would also have a dropdown to select an ECM to tag to question. | The list of selected ECM should come on question creation page, so that questions can be grouped according to the ECM | ECMs would be existing in a configurable form. The form can be configured by the implementation team for the tenant. Any tenant can request to edit the list of ECMs. However, by default the following list would be configured at the Diksha system level:
If an ECM that is selected already is removed later on from the observation details page and there are questions already tagged to that ECM, then, on the pagination screen the ‘No ECM’ label would also show up in the left side tree within which the questions without ECM tagging will come. |
Select the domain from the tree structure | All the domains are auto-populated on the left pane where folder hierarchy is shown according to the framework selected. No domain from this list can be deleted by the creator. | Creator will be able to fill the domain name as per what they want the users to see. | |
Creator will see the list of criteria under that domain | All the criteria under that domain are auto-populated according to the framework selected. No criteria from this list can be deleted by the creator. | Creator will be able to fill all the details related to criteria. | |
Creator should be able to add questions | The added questions will be shown under the criteria. If ECMs were enabled for the external type observation with rubric then, creator must select ECM to tag to the question before saving it. | Question creation page will open up | If no questions are added to a criteria it will not be visible on consumption side. If there are no questions under any of the criteria of a domain, the domain will not be visible on the consumption side. |
Creator will fill in all the details according to the question type. | The question will be created and added to the section. | ||
Creator should be able to tag questions with the ECM | If the form was selected for ‘External” and the list of ECM’s was selected. The list will get auto populated | The questions will be grouped on the basis of ECM selected. | Note: The dependent question will always have the same ECM tagging as the parent question. The creator wouldn’t be able to tag a different ECM to the dependent question than the parent question. |
Creator should be able to provide score to each question. | Question type can only be MCQ (single and multi-select)to give scores. | Scores will be added to questions and the maximum, minimum score of criteria will get calculated. | |
Creator should be able to define score range for each level of the criteria, according to the min. and max. score (Of all the questions in that criteria) | Maximum and minimum possible scores of criteria would be indicated to the user. As the user adds the questions, edits the score in any question or delete a question, the maximum and minimum possible score too will change. | Level ranges are defined for that criteria. | Creator would only be able to define the score ranges for the criteria if at least one question with score has been added. Until such time the min. and max. score will be zero and score range fields will be disabled. If the creator, after defining the score range, adds more questions or delete any which impacts the min. and max possible score than parallelly the score range will be flagged and the creator would have to change the score range again to correct it. The score range then would also depend directly on the score range and would have to be validated to not be below the minimum possible score or above the max possible score. |
Creator should be able to define score range for each level of the domain, according to the min. and max. score (Of all the criteria in that domain) | Maximum and minimum possible scores of domain would be indicated to the user. As the user adds the questions, edits the score in any question or delete a question, the maximum and minimum possible score too will change. | Level ranges are defined for Domain | |
Creator should be able to give page numbers to all the questions. The questions will be grouped according to criteria or ECM. | Default page number will be '1’ for all the questions. The page no. would not continue across different criteria/ECM. For each criteria and ECM the creator would be able to mark the page nos. starting from 1. | Questions will be divided according to the page numbers on the consumption side. | The pagination structure would not allow any creator to keep questions across the different criteria and domain in the same page. The questions on a page would belong to only one ECM/criteria. |
Creator should be able to view the status of progress for each criteria. | The tree structure on left side will show just the domains. | Number of questions, minimum maximum score and level wise score range will be visible in one page. | Each of the values on the right side are just for view and would be shown from the activity done by user in the creation process. |
Jira stories
Here are Link to stories on Jira for specific stories of the flow and further details feature wise.
The Wireframe document provides a draft UI for observation with rubric.
The UI screens through Figma can be found here
Review Flow
Actions | Conditions | Expected results | Notes |
|---|---|---|---|
Reviewer opens the observation/survey/observation with rubric content to review | The creator sent such content for review. | The reviewer can only view the entire content with the same view as that of creator. | |
The reviewer can view any of the detail pages (Observation or survey, section, domain, criteria) to verify and review | The reviewer is able to check all metadata and created content for reviewing. | ||
The reviewer can add comments to any of the screen of the content. | The comment are stored for the particular screen of the content. | The reviewer will be adding comments at for the particular screen. This contextualizes the comments they are putting and also eases the process for reviewer. | |
After adding comments, the reviewer can send back the content for changes by creator | The reviewer select request change as option to notify and get the creator to rework and submit again the content. | ||
If the reviewer finds everything correct, he/she can Accept and publish the content | The reviewer when ‘Accept’, the content is published in the library. | ||
The reviewer can reject the content if it is to be completely discarded. | A rejected content would have the reject status and the content would not be editable anymore by the creator. | ||
The creator can view the comments of each screen for the entire content and make changes accordingly. | The reviewer has requested changes in the content and has put their comments. | The creator can make changes to the content according to comments when reviewer ‘Request changes’ and send it for review again. | A screen should only show the comments the are put up on that screen by the reviewer. This will help creator easily understand the comment and action required. |
Jira stories
Here are Link to stories on Jira for specific stories of the flow and further details feature wise.
The Wireframe document provides a draft UI for observation with rubric.
The UI screens through Figma can be found here
Add Comment