Page Comparison

Introduction

This wiki details the architecture of enabling reporting framework to operate at scale. It discusses the high level design problems to be solved and introduces the proposed architecture for the same.

Key Design Problems

TBA

...

Reporting Architecture

...

Druid Architecture

...

Druid Data Model

Raw Telemetry

15objectidobjectidContent Id16objecttypeobjecttypeContent Type17objectversionobjectverArray[String]edatatypeedatatypeEvent typeedatasubtypeedatasubtypeEvent subtypeedatamodeedatamodeSTART event Mode of startedatapageidedatapageidUnique pageidedatauriedatauriIMPRESSION event Relative URI of the contentedataidedataidEvent data IdedatadurationedatadurationDuration of the eventedataindexedataindexASSESS event Index of the question within a contentStringtopntopnSEARCH event topn resultsObjectvisitsvisitsIMPRESSION event object visit detailsObjectdevicelocstateStringdevicelocstatecodeCountry ISO code information for the devicedeviceloc_countryldatacountryCountry location information for the devicedeviceosdeviceosStringdevicemakedevicemakeStringdeviceiddeviceidStringdevicememdevicememIntdeviceidiskdeviceidiskIntdeviceediskdeviceediskIntdevicescrndevicescrnIntcameradevicecameraPrimary & secondary camera speccpudevicecpuProcessor nameuaspecMapfirstaccessfirstaccessFirst access of contentnameName of the content

	Dimension in Druid	Field in Telemetry	Description	Data Type
1	ets	ets	Event timestamp	Long
2	eid	eid	Event Id	String
3	syncts	syncts	Sync Timestamp	Long
4	@timstamp	@timstamp	Sync Timestamp in String	String
5	actor_id	actor.id	Actor Id of the event	String
56	actor_type	actor.type	Type of the actor	String
67	context_channel	context.channel	Channel Id	String
78	context_pdata_id	context.pdata.id	Producer Id	String
89	context_pdata_pid	context.pdata.pid	Producer Process Id	String
910	context_pdata_ver	context.pdata.ver	Producer version number	String
1011	context_env	context.env	Context Environment	String
1112	context_sid	context.sid	Session Id	String
1213	context_did	context.did	Device Id	String
1314	context_cdata_type	context.cdata.type	Correlation Data Type	Array[String]
1415	context_cdata_id	context.cdata.id	Correlation Data Id	Array[String]
16	context_rollup_	l1	context.rollup.	l1	Context level 1 rollup	String
17	context_rollup_	l2	context.rollup.	l2	Context level 2 rollup	String
18	context_rollup_	l3	context.rollup.	Content Version	String	18	tags	tags	Tags	l3	Context level 3 rollup	String
19	context_rollup_	l4	context.rollup.	l4	Context level 4 rollup	String
20	object_	id	object.	id	Content Id	String
21	object_	type	object.	type	Content Type	String
22	object_	version	object.	ver	Content Version	String
23	object_rollup_	l1	object.rollup.	l1	Content level 1 rollup	String
24	object_rollup_	l2	object.rollup.	l2	Content level 2 rollup	String
25	object_rollup_	l3	object.rollup.	l3	Content level 3 rollup	String
26	object_rollup_	l4	object.rollup.	l4	Content level 4 rollup	String
27	edata_pass	edata.pass	ASSESS event Field to identify pass or fail for assessments	tags	tags	Tags	Array[String]
28	edata_scoretype	edata.score	ASSESS event Assessment score	Doubletype	Event type	String
29	edata_resvaluessubtype	edata.resvalues	ASSESS event Assessment results	Array[Object]	subtype	Event subtype	String
30	edata_item_idmode	edata.item.idmodeASSESS event Assessment item id	START event Mode of start	String
31	edata_item_titlepageid	edata.item.titleASSESS event Assessment item titlepageid	Unique pageid	String
32	edata_item_maxscoreuri	edata.item.maxscoreuri	ASSESS event Assessment item max score	DoubleIMPRESSION event Relative URI of the content	String
33	edata_target_id	edata.target.idASSESS event Assessment item target id	Event data Id	String
34	edata_target_typeduration	edata.target.type	ASSESS event Assessment item target type	Stringduration	Duration of the event	Double
35	edata_ratingindex	edata.ratingindex	FEEDBACK event Ratings	StringASSESS event Index of the question within a content	Integer
36	edata_commentspass	edata.commentspass	FEEDBACK event CommentsASSESS event Field to identify pass or fail for assessments	String
37	edata_dirscore	edata.dirscore	SHARE ASSESS event directionAssessment score	StringDouble
38	edata_items_idresvalues	edata.items.idresvalues	SHARE event shared item ids	StringASSESS event Assessment results	Array[Object]
39	edata_itemsitem_typeid	edata.itemsitem.typeSHARE item typesid	ASSESS event Assessment item id	String
40	edata_itemsitem_origin_idtitle	edata.itemsitem.origin.idSHARE event source idtitle	ASSESS event Assessment item title	String
41	edata_itemsitem_origin_typemaxscore	edata.itemsitem.origin.type	SHARE event source type	Stringmaxscore	ASSESS event Assessment item max score	Double
42	edata_items_totarget_id	edata.itemstarget.to.id	SHARE event destination ASSESS event Assessment item target id	String
43	edata_items_totarget_type	edata.itemstarget.to.type	SHARE ASSESS event destination Assessment item target type	String
44	edata_staterating	edata.staterating	AUDIT FEEDBACK event current stateRatings	StringInteger
45	edata_prevstatecomments	edata.prevstateAUDIT event previous statecomments	FEEDBACK event Comments	String
46	edata_sizedir	edata.sizedir	SEARCH SHARE event result sizedirection	IntegerString
47	edata_filtersitems_dialcodesid	edata.filtersitems.dialcodesid	SEARCH event List of dialcodesSHARE event shared item ids	Array[String]
48	edata_	items_type	edata.	items.type	SHARE item types	Array[	String]
49	edata_	items_origin_id	edata.	items.origin.id	SHARE event source id	Array[	String]
50	edata_items_	origin_	ldata.state	State location information for the device	type	edata.items.origin.type	SHARE event source type	Array[String]
51	edata_	items_	to_	id	ldata.state_code	State ISO code information for the device	String
52	device_loc_city	ldata.city	City location information for the device	String
53	device_loc_country_code	ldata.country_code	edata.items.to.id	SHARE event destination id	Array[String]
52	edata_items_to_type	edata.items.to.type	SHARE event destination type	Array[String]
53	edata_state	edata.state	AUDIT event current state	String
54	edata_	prevstate	edata.	prevstate	AUDIT event previous state	String
55	edata_	size	edata.	Device OS name	size	SEARCH event result size	Integer
56	edata_filters_	dialcodes	edata.filters.	Device make and model	dialcodes	SEARCH event List of dialcodes	Array[String]
57	edata_topn_	identifier	edata.topn.	Physical device id if available from OS	identifier	SEARCH event topn results	Array[String]
58	edata_visits_	objid	edata.visits.	Total memory in mb	objid	IMPRESSION event unique id for object visited	Array[String]
59	edata_visits_	objtype	edata.visits.	Total interanl disk	objtype	IMPRESSION event type of object visited	Array[String]
60	edata_visits_	objver	edata.	Total external disk	visits.objver	IMPRESSION event version of object visited	Array[String]
61	edata_visits_	index	edata.visits.	Screen size in inches	index	IMPRESSION event index of object within list	Array[Integer]
62	device_loc_	state	devicedata.	state	State location information for the device	String
63	device_loc_	state_code	devicedata.	statecode	State ISO code information for the device	String
64	device_ sims	device.sims	Number of sim cards	Int loc_iso_state_code	devicedata.iso3166statecode	State ISO-3166 code information for the device	String
65	device_	loc_	ua_spec	User agent spec of the browser	city	devicedata.city	City location information for the device	String
66	device_	loc_	country_	code	devicedata.countrycode	Country ISO code information for the device	String
67	device_loc_	country	name	devicedata.country	Country location information for the device	String
68	contentdevice_object_type	objectType	Type of the contentos	devicedata.os	Device OS name	String
69	contentdevice_typemake	contentType	Type of the resourcedevicedata.make	Device make and model	String
70	contentdevice_media_type	mediaType	Type of the media of the resourceid	devicedata.id	Physical device id if available from OS	String
71	contentdevice_languagemem	language	List of languages in the content	Array[String]devicedata.mem	Total memory in mb	Integer
72	contentdevice_mediumidisk	meidum	Language medium of the board	Stringdevicedata.idisk	Total interanl disk	Integer
73	contentdevice_mimetypeedisk	mimeType	Mimetype of the resource in the content	Stringdevicedata.edisk	Total external disk	Integer
74	contentdevice_framework	framework	Stringscrn	devicedata.scrn	Screen size in inches	Integer
75	contentdevice_boardcamera	board	Board of affiliationdevicedata.camera	Primary & secondary camera spec	String
76	contentdevice_statuscpu	status	Status of the content - Draft, Published etc.devicedata.cpu	Processor name	String
77	contentdevice_mimetypes_count	mimeTypesCount	Count grouped by mimetypes	JSON Objectsims	devicedata.sims	Number of sim cards	Integer
78	contentdevice_typesuaspec_count	contentTypesCount	Counts grouped by each content type	JSON Object
79	content_version	versionKey	Version of the content	Double
80	content_last_submitted_on	lastSubmittedOn	Last submitted date of the contentagent	devicedata.uaspec.agent	User agent of the browser	String
79	device_uaspec_ver	devicedata.uaspec.ver	User agent version of the browser	String
80	device_uaspec_system	devicedata.uaspec.system	User agent system identification of the browser	String
81	contentdevice_lastuaspec_published_on	lastPublishedOn	Last submitted date of the contentplatform	devicedata.uaspec.platform	User agent client platform of the browser	String
82	userdevice_firstuaspec_name	firstname	First name of the userraw	devicedata.uaspec.raw	Raw user agent of the browser	String
83	userdevice_lastfirst_nameaccess	lastname	Lastname devicedata.firstaccess	First access of the userStringdevice	Long (Epoch)
84	usercontent_grade_list	grade_list	List of grades taught	Array[String]name	contentdata.name	Name of the content	String
85	usercontent_languageobject_list	language_list	List of languages known	Array[String]type	contentdata.objecttype	Type of the content	String
86	usercontent_subject_list	subject_list	List of subjects taught	Array[String]	87	user_name	username	Username of the usertype	contentdata.contenttype	Type of the resource	String
87	content_media_type	contentdata.mediatype	Type of the media of the resource	String
88	usercontent_typelanguage	userTypecontentdata.language	Type of user	StringList of languages in the content	Array[String]
89	usercontent_loc_state	user_state	State info of the user	Stringmedium	contentdata.medium	Language medium of the board	Array[String]
90	usercontent_loc_district	user_district	District info of the usermimetype	contentdata.mimetype	Mimetype of the resource in the content	String
91	user_loc_block	user_block	Block info of the user	String	content_framework	contentdata.framework		String
92	content_board	contentdata.board	Board of affiliation	String
93	content_status	contentdata.status	Status of the content - Draft, Published etc.	String
94	content_version	contentdata.pkgversion	Version of the content	Double
95	content_last_submitted_on	contentdata.lastsubmittedon	Last submitted date of the content	Long (Epoch)
96	content_last_published_on	contentdata.lastpublishedon	Last submitted date of the content	Long (Epoch)
97	content_last_updated_on	contentdata.lastupdatedon	Last updated date of the content	Long (Epoch)
98	user_grade_list	userdata.gradelist	List of grades taught	Array[String]
99	user_language_list	userdata.languagelist	List of languages known	Array[String]
100	user_subject_list	userdata.subjectlist	List of subjects taught	Array[String]
101	user_type	userdata.type	Type of user	String
102	user_loc_state	userdata.state	State info of the user	String
103	user_loc_district	userdata.district	District info of the user	String
104	user_loc_block	userdata.block	Block info of the user	String
105	dialcode_channel	dialcodedata.channel	Channel for which dialcode is generated	String
106	dialcode_batchcode	dialcodedata.batchcode	Batch for which dialcode belongs to	String
107	dialcode_publisher	dialcodedata.publisher	Publisher of the dialcode	String
108	dialcode_generated_on	dialcodedata.generatedon	Dialcode generated on	Long (Epoch)
109	dialcode_published_on	dialcodedata.publishedon	Dialcode published on	Long (Epoch)
110	dialcode_status	dialcodedata.status	Status of the dialcode	String
111	dialcode_object_type	dialcodedata.objecttype	Object type - DialCode as a value	String

Summary Events

20edata_page_summary_id35363738404144737578

	Dimension in Druid	Field in Summary event	Description	Data Type
1	ets	ets	Event timestamp	Long
2	eid	eid	Event Id	String
3	ver	ver	Version	String
4	syncts	syncts	Sync timestamp	Long
5	uid	uid	User Id	String
6	context_date_range_from	context.date_range.from	Start Date for the summary	StringLong (Epoch)
7	context_date_range_to	context.date_range.to	End Date for the summary	StringLong (Epoch)
8	context_rollup_l1	context.rollup.l1	Context level1 rollup	String
9	context_rollup_l2	context.rollup.l2	Context level2 rollup	String
10	context_rollup_l3	context.rollup.l3	Context level3 rollup	String
11	context_rollup_l4	context.rollup.l4	Context level4 rollup	String
12	dimension_channel	dimensions.channel	Channel Id as dimension from raw telemetry	String
13	dimensions_did	dimensions.did	Device Id as dimension from raw telemetry	String
14	dimensions_pdata_id	dimensions.pdata.id	Producer Id as dimension from raw telemetry	String
15	dimensions_pdata_pid	dimensions.pdata.pid	Producer Process Id as dimension from raw telemetry	String
16	dimensions_pdata_ver	dimensions.pdata.ver	Producer Process Ver as dimension from raw telemetry	String
17	dimensions_sid	dimensions.sid	Session Id as dimension	String
1718	dimensions_type	dimensions.type	Type of summary	String
1819	dimensions_mode	dimensions.mode	Mode of action in the session	String
1920	object_id	object.id	Content Id	String
object_type	object.type	Content Type	String	21	object_type	object.type	Content Type	String
22	object_version	object.ver	Content version	String
23	object_rollup_l1	object.rollup.l1	Object level1 rollup	String
24	object_rollup_l2	object.rollup.l2	Object level2 rollup	String
25	object_rollup_l3	object.rollup.l3	Object level3 rollup	String
26	object_rollup_l4	object.rollup.l4	Object level4 rollup	String
27	tags	tags	Tags attached to a summary event	Array[String]
28	edata_time_spent	edata.eks.time_spent	Time spent in the session excluding idle time	String
29	edata_time_difference	edata.eks.time_diff	Total time in a session including idle time	String
30	edata_interaction_count	edata.eks.interact_events_count	Total count of interact events in a session	Long
31	edata_summary_env	edata.eks.env_summary.env	High level env within the app (content, domain, resources, community)	String
32	edata_env_summary_count	edata.eks.env_summary.count	Count of times the environment has been visited	Integer
33	edata_env_summary_time_spent	edata.eks.env_summary.time_spent	Time spent per env	Double
34	Object level2 rollup	String
25	object_rollup_l3	object.rollup.l3	Object level3 rollup	String
26	object_rollup_l4	object.rollup.l4	Object level4 rollup	String
27	tags	tags	Tags attached to a summary event	Array[String]
28	edata_time_spent	edata.eks.time_spent	Time spent in the session excluding idle time	Double
29	edata_time_difference	edata.eks.time_diff	Total time in a session including idle time	Double
30	edata_interaction_count	edata.eks.interact_events_count	Total count of interact events in a session	Long
31	edata_env_summary_env	edata.eks.env_summary.env	High level env within the app (content, domain, resources, community)	Array[String]
32	edata_env_summary_count	edata.eks.env_summary.count	Count of times the environment has been visited	Array[Integer]
33	edata_env_summary_time_spent	edata.eks.env_summary.time_spent	Time spent per env	Array[Double]
34	edata_page_summary_id	edata.eks.page_summary.id	Page id	Array[String]
35	edata_page_summary_type	edata.eks.page_summary.type	Type of page e.g. view/edit	Array[String]
36	edata_page_summary_env	edata.eks.page_summary.env	Env of page	Array[String]
37	edata_page_summary_visit_count	edata.eks.page_summary.id	Page id	String	visit_count	Number of times each page was visited	Array[Integer]
38	edata_page_summary_time_typespent	edata.eks.page_summary.type	Type of page e.g. view/edit	String	time_spent	Time taken per page	Array[Double]
39	edata_pageitem_summaryresponses_visititem_countid	edata.eks.pageitem_summaryresponses.visit_count	Number of times each page was visited	String	itemId	Question Id passed in the ASSESS event	Array[String]
40	edata_pageitem_summaryresponses_time_spent	edata.eks.pageitem_summaryresponses.time_spenttimeSpent	Time taken per page	Double	spent in seconds from ASSESS event	Array[Double]
41	edata_item_responses_item_idpass	edata.eks.item_responses.itemIdQuestion Id passed in the pass	Pass response for a question from ASSESS event	Array[String]
3942	edata_item_responses_time_spentscore	edata.eks.item_responses.timeSpentTime spent in seconds score	Score from ASSESS event	String	Array[Integer]
43	edata_item_responses_max_passscore	edata.eks.item_responses.passPass response for a question maxScore	Max Score from ASSESS event	String	Array[Integer]
44	edata_item_responses_scoretimestamp	edata.eks.item_responses.scoreScore time_stamp	Timestamp for each response from ASSESS event	Array[Integer]
42	edata_item_responses_max_score	edata.eks.item_responses.maxScore	Max Score from ASSESS event	Array[Integer]
43	edata_item_responses_timestamp	edata.eks.item_responses.time_stamp	Timestamp for each response from ASSESS event	String
Long (Epoch)]
45	device_loc_state	devicedata.state	State location information for the device	String
46	device_loc_state_code	devicedata.statecode	State ISO code information for the device	String
47	device_loc_iso_state_code	devicedata.iso3166statecode	State ISO-3166 code information for the device	String
48	device_loc_statecity	ldatadevicedata.statecity	State City location information for the device	String
4549	device_loc_statecountry_codeldata	devicedata.state_codecountrycode	State Country ISO code information for the device	String
4650	device_loc_citycountry	ldatadevicedata.citycountry	City Country location information for the device	String
4751	device_loc_country_code	ldata.country_code	Country ISO code information for the device	String
48	device_loc_country	ldata.country	Country location information for the device	String
49	device_os	device.os	Device OS name	String
50	device_make	device.make	Device make and model	String
51	device_id	device.id	Physical device id if available from OS	String
52	device_mem	device.mem	Total memory in mb	Int
53	device_idisk	device.idisk	Total interanl disk	Int
54	device_edisk	device.edisk	Total external disk	Int
55	device_scrn	device.scrn	Screen size in inches	Int
56	device_camera	device.camera	Primary & secondary camera spec	String
57	device_cpu	device.cpu	Processor name	String
58	device_sims	device.sims	Number of sim cards	Int
59	device_ua_spec	ua_spec	User agent spec of the browser	Map
60	device_first_access	first_access	First access of the device	String
61	content_name	os	devicedata.os	Device OS name	String
52	device_make	devicedata.make	Device make and model	String
53	device_id	devicedata.id	Physical device id if available from OS	String
54	device_mem	devicedata.mem	Total memory in mb	Integer
55	device_idisk	devicedata.idisk	Total interanl disk	Integer
56	device_edisk	devicedata.edisk	Total external disk	Integer
57	device_scrn	devicedata.scrn	Screen size in inches	Integer
58	device_camera	devicedata.camera	Primary & secondary camera spec	String
59	device_cpu	devicedata.cpu	Processor name	String
60	device_sims	devicedata.sims	Number of sim cards	Integer
61	device_uaspec_agent	devicedata.uaspec.agent	User agent of the browser	String
62	device_uaspec_ver	devicedata.uaspec.ver	User agent version of the browser	String
63	device_uaspec_system	devicedata.uaspec.system	User agent system identification of the browser	String
64	device_uaspec_platform	devicedata.uaspec.platform	User agent client platform of the browser	String
65	device_uaspec_raw	devicedata.uaspec.raw	Raw user agent of the browser	String
66	device_first_access	devicedata.firstaccess	First access of the device	Long (Epoch)
67	content_name	contentdata.name	Name of the content	String
6268	content_object_type	objectTypecontentdata.objecttype	Type of the content	String
6369	content_type	contentTypecontentdata.contenttype	Type of the resource	String
6470	content_media_type	contentdata.mediatype	mediaType	Type of media of the content	String
6571	content_language	contentdata.language	List of languages	Array[String]
6672	content_medium	contentdata.medium	Language medium of the board	String
6773	content_mimetype	mimeTypecontentdata.mimetype	Mimetype of the content	String
6874	content_framework	contentdata.framework		String
6975	content_board	contentdata.board	Board of affiliation	String
7076	content_status	contentdata.status	Status of the content - Draft, Published etc.	String
71	content_mimetypes_count	mimeTypesCount	Count grouped by mimetypes	JSON Object
72	content_content_types_count	contentTypesCount	Counts grouped by each content type	JSON Object
the content - Draft, Published etc.	String
77	content_version	versionKeycontentdata.pkgversion	Version of the content	Double
7478	content_last_submitted_on	lastSubmittedOncontentdata.lastsubmittedon	Last submitted date of the content	String	Long (Epoch)
79	content_last_published_on	lastPublishedOncontentdata.lastpublishedon	Last submitted published date of the content	String
76	user_first_name	firstname	First name of the user	String
77	user_last_name	lastname	Last name of the user	String
Long (Epoch)
80	content_last_updated_on	contentdata.lastupdatedon	Last updated date of the content	Long (Epoch)
81	user_grade_listgrade_list	userdata.gradelist	List of grades taught	Array[String]
7982	user_language_listlanguage_list	userdata.languagelist	List of languages known	Array[String]
8083	user_subject_listsubject_list	userdata.subjectlist	List of subjects taught	Array[String]
8184	user_name	username	User name of the user	String	82	user_type	usertypetype	userdata.type	Type of the user	String
8385	user_loc_stateuser_	userdata.state	State info of the user	String
8486	user_loc_districtuser_	userdata.district	District info of the user	String
8587	user_loc_blockuser_	userdata.block	Block info of the user	String

Aggregates

Granularity → DAY

Druid field name	Druid source field	Aggregate Type
total_interactions	interaction_count	SUM
total_time_spent	time_spent	SUM
total_sessions	mid	COUNT

...

Report JSON Spec

JSON Schema

Code Block

language	js

[{
    id: String, // Required. Report ID.
    label: String, // Required. Report Label (will be shown up as menu)
    title: String, // Optional. Report title. Defaults to report label
    description: String, // Optional. Report description. HTML text can be included as description
    dataSource: String, // Required. Location of the data source to show the report. Can be an expression. For ex: /<report_id>/{{channel}}/report.json
    charts: [{ // Optional
 		datasets: [{
			data: Array[Number], // Required if `dataExpr` is not provided. Array of Number. Data points to show in the chart
			dataExpr: String, // Required if `data` is not provided. Expression pointing to the data in dataSource. For ex: {{data.noOfDownloads}}
			label: String // Required. Label to display on the chart
		}],
		labels: Array[String], // Required if `labelsExpr` is not provided. Labels to show on the x-axis
		labelsExpr: String, // Required if `labels` is not provided. Expression pointing to the data in dataSource. For ex: {{data.Date}}
		chartType: String, // Optional. Defaults to line. Available types - line, bar, radar, pie, polarArea & doughnut
		colors: [""], // Optional. Color to show for each dataset. Defaults to ["#024F9D"].
		options: { // Optional. options for display. Full set of options look at https://valor-software.com/ng2-charts/
			responsive: Boolean, // Defaults to true
			...
		}, 
		legend: Boolean // Optional. Whether to show the legend below/above the chart. Defaults to true and position to top.
    }],
    table: { // Optional
        "columns": Array[String], // Required if `columnsExpr` is not provided. Columns to show.
        "values": Array[Array[String]], // Required if `valuesExpr` is not provided. Column data.
        "columnsExpr": String, // Required if `columns` is not provided. Expression pointing to the data in dataSource. For ex: {{keys}}
        "valuesExpr": String // Required if `values` is not provided. Expression pointing to the data in dataSource. For ex: {{tableData}}
    },
    downloadUrl: String // Location to download the data as CSV
}]

...

Versions Compared

Old Version 19

New Version Current

Key