Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Introduction

The container for the OpenRAP 2.0 has been developed on top of ext-framework. The additions on top of ext-framework are:

...

Following is the high level component diagram of OpenRAP 2.0

Image Added


...

SDK

Following are SDK's built into the container.

...

Telemetry SDK

Code Block
languagejs
//* 
 * API to register a plugin and it's corresponding sync url to dispatch telemetry to.
 //* config = {batchSize: 200, syncInterval: 30}
register =* function(String@param pluginId, String
 * @param syncURL, Object config){};

// Get String - The server url to sync the telemetry SDK instanceto.
This can* be@param injectableconfig whenObject the- pluginTelemetry uses typescript
getInstancesync configuration
 */
register = function(String pluginId) : TelemetrySDK , String syncURL, Object config){};

//* 
API to* log/storeGet athe telemetry eventSDK sendinstance. =This function(String event) : Promise {};

// API to log/store an array of telemetry events
sendcan be injectable when the plugin uses typescript
 * @param pluginId String
 * @return Object TelemetrySDK
 */
getInstance = function(Array[String] eventspluginId) : PromiseTelemetrySDK {};

//* 
 * API to receivelog/store a telemetry event
batch of* telemetry@param eventsevent inString
zip stream*/
sendBatchsend = function(String batchEventevent) : Promise {};

//* 
 * API to receivelog/store aan batcharray of telemetry events in zip stream
sendBatch
 * @param events String[]
 */
send = function(byteString[] events) : Promise {};

//* 
list =* function()API :to TelemetryPacket[] {};

get = function(String packetId): TelemetryPacket {};

// API to force sync. This would create packets to be synced
sync = function(receive a batch of telemetry events in zip stream
 * @param batchEvent String
 */
sendBatch = function(String batchEvent) : Promise {};

//* 
 * API to getreceive thea statusbatch of telemetry events/batches in forzip astream
given plugin* id@param // Responseevents byte[] - {events: 3456, batches: 20}
getStatusZip stream of the events
 */
sendBatch = function(byte[] events) : Promise {};

Usage example

Code Block
languagejs
// Example to use telemetry SDK
telemetrySDK = TelemetrySDK.getInstance(this.manifest.id);

telemetrySDK.send({...});
telemetrySDK.send([{...}]);

telemetrySDK.sync();

File SDK

Code Block
languagejs
getInstance = function(String pluginId) : FileSDK {};

mkdir

/* 
 * API to list the telemetry packets
 * @param status String - Get the packets based on the status
 * @return TelemetryPacket Document[]
 */
list = function(String pathstatus?) : PromiseTelemetryPacket[] {};

copy/* =
function(String source,* StringAPI dest)to :get Promisethe {};telemetry data mvfor =a function(String from, String to) : Promise {};

rm = function(String file) : Promise {};

rmdirgiven packet.
 * @param packetId String
 * @return TelemetryPacket Document
 */
get = function(String pathpacketId) : PromiseTelemetryPacket {};
 zip
=/* function(String
path, String* fileName)API :to Promise {};

unzip = function(String fileName, String path) : Promise {};

getAbsPathforce sync. This would create packets to be synced
 */
sync = function(String path) : StringPromise {};

watch/* =
function(String[] paths, callback){};

Network SDK

Code Block
languagejs
isNetworkAvailable = function() : boolean {};

// Events
// Following are the events fired within the SDK
network:available
network:disconnected

Usage

Code Block
languagejs
EventManager.on('network:available', function() {
	// Do something.
	// May be sync telemetry.
	// Fetch content updates
	// Resume content downloads
	// etc
});

...

 * API to get the status of events/batches for a given plugin id
 * @return Object - The status of the telemetry for ex: {events: 3456, batches: 20}
 */
getStatus = function() : Promise {};

Usage example

Code Block
languagejs
getInstance = function(String pluginId) : HttpSDK {};

get// Example to use telemetry SDK
telemetrySDK = TelemetrySDK.getInstance(this.manifest.id);

telemetrySDK.send({...});
telemetrySDK.send([{...}]);

telemetrySDK.sync();

File SDK

Code Block
languagejs
getInstance = function(String url, Object optionspluginId) : FileSDK {};

mkdir = function(String path) : Promise {};

postcopy = function(String urlsource, Object data, Object optionsString dest) : Promise {};

patchmv = function(String urlfrom, Object data, Object optionsString to) : Promise {};

deleterm = function(String url, Object optionsfile) : Promise {};

headrmdir = function(String url, Object optionspath) : Promise {};

Download Manager

Following are the APIs for the download manager. The workflow of download manager is detailed out in the next few sections.

Code Block
languagejs
/* Method to get the instance of the download manager */
getInstance

zip = function(String path, String fileName) : Promise {};

unzip = function(String pluginIdfileName, String path) : DownloadManagerPromise {};

/*getAbsPath = function(String *path) Method: toString queue{};
the
downloadwatch of a file
 * @Param file - The file to download
 * @Param path - Path to download the file
 * @return downloadId - The download id reference
 */
download = function(String file, String path) : String {};

/* 
 * Method to queue the download of a file
 * @Param file - The file to download
 * @Param path - Path to download the file
 * @return downloadId - The download id reference
 */
download = function(String[] files, String folder) : String {};

/* 
 * Method to get the status of the download
 * @Param downloadId String
 * @return Download object 
 */
get= function(String[] paths, callback){};

Network SDK

Code Block
languagejs
isNetworkAvailable = function() : boolean {};

// EVENTS
// Following are the events fired within the SDK
network:available
network:disconnected

Usage

Code Block
languagejs
EventManager.on('network:available', function() {
	// Do something.
	// May be sync telemetry.
	// Fetch content updates
	// Resume content downloads
	// etc
});

Http SDK

Code Block
languagejs
getInstance = function(String pluginId) : HttpSDK {};

get = function(String url, Object options) : Promise {};

post = function(String url, Object data, Object options) : Promise {};

patch = function(String url, Object data, Object options) : Promise {};

delete = function(String url, Object options) : Promise {};

head = function(String url, Object options) : Promise {};

Download Manager

Following are the APIs for the download manager. The workflow of download manager is detailed out in the next few sections.

Code Block
languagejs
/* Method to get the instance of the download manager */
getInstance = function(String downloadIdpluginId) : DownloadObjectDownloadManager {};

/* 
 * Method to listqueue the download queueof baseda on thefile
status  * @Param@param status Stringfile - The statusfile of theto download
- Submitted,* Complete,@param InProgress,path Failed.- BlankPath optionto willdownload returnthe all statusfile
 * @return ArraydownloadId - ArrayThe ofdownload downloadid objectsreference
 */
listdownload = function(String status): DownloadObject[]file, String path) : String {};

DownloadObject/* =
{ 	id: String, // Download id
	status: String, // Submitted, InProgress, Complete, Failed.
	createdOn: Date,
	updatedOn: Date,
  	stats: {
  		totalFiles: Number, // Total files to download
		downloadedFiles: Number, // Total files downloaded so far
  		totalSize: Number, // Total number of bytes to download
  		downloadedSize: Number, // Total number of bytes downloaded so far
  	},
  	files: [{ // Status of each file within the given download
  		file: String, // File that is downloaded
  		source: String, // source from where it is downloaded
  		path: String, // Relative path where the file is downloaded to
  		size: Integer, // Total file size in bytes
  		downloaded: Integer // Downloaded until now
  	}]
}

// Events
// Following are the events fired within the SDK
"download:complete"
EventManager.dispatch("download:complete", {
	id: String, // Download Id
	files: [{
		file: String, // File that is downloaded
		source: String, // source from where it is downloaded
		path: String, // Relative path where the file is downloaded to
		size: Integer // file size in bytes
	}]
});

Settings SDK

Code Block
languagejs
/* Method to get the instance of the settings sdk */
getInstance = function(String pluginId) : SettingsSDK {};

put: function(String key, Object value) : Promise {};

get: function(String key) : Promise {};

Open Questions:

  • Does encryption needs to be part of container or plugin

Database Schema

Settings

Code Block
languagejs
{
	document: {
		"_id"* Method to queue the download of a file
 * @param file - The file to download
 * @param path - Path to download the file
 * @return downloadId - The download id reference
 */
download = function(String[] files, String folder) : String {};

/* 
 * Method to get the status of the download
 * @param downloadId String
 * @return Download object 
 */
get = function(String downloadId) : DownloadObject {};

/* 
 * Method to pause the download
 * @param downloadId String
 */
pause = function(String downloadId) : Promise {};

/* 
 * Method to cancel the download
 * @param downloadId String
 */
cancel = function(String downloadId) : Promise {};

/* 
 * Method to pause all the downloads for the given plugin
 * @param downloadId String
 */
pauseAll = function() : Promise {};

/* 
 * Method to cancel all the downloads for the given plugin
 * @param downloadId String
 */
cancelAll = function() : Promise {};

/* 
 * Method to list the download queue based on the status
 * @param status String - The status of the download - Submitted, Complete, InProgress, Failed. Blank option will return all status
 * @return Array - Array of download objects
 */
list = function(String status): DownloadObject[] {};

DownloadObject = {
	id: String, // SettingDownload id
		"value"status: ObjectString, // SettingSubmitted, valueInProgress, 	}Complete, Failed.
	indexcreatedOn: {}
}

Telemetry

A document database is created for each plugin telemetry storage - <plugin_id>_telemetry

Code Block
languagejs
{
	document: {
		"_id": StringDate,
	updatedOn: Date,
  	stats: {
  		totalFiles: Number, // midTotal offiles theto eventdownload
		"event"downloadedFiles: {}Number, // eventTotal 	},
	index: {}
}

TelemetryPackets

Code Block
languagejs
{
	document: {
		"_id": String, // packet id
		"pluginId": Stringfiles downloaded so far
  		totalSize: Number, // Total number of bytes to download
  		downloadedSize: Number, // PluginTotal idnumber the packet belongs to
		"status": String,of bytes downloaded so far
  	},
  	files: [{ // statusStatus of each file within the given packet. Synced/NotSynceddownload
  		"createdOn"file: DateString, // File that Dateis ofdownloaded
packet creation 		"updatedOn"source: DateString, // Datesource whenfrom thewhere documentit is updateddownloaded
  		"size"path: NumberString, // Relative Sizepath ofwhere the file is eventsdownloaded into
the packet 		"events"size: []Integer, // events Total file size in bytes
  	},
	indexdownloaded: [pluginId, status, updatedOn]
}

DownloadQueue

Code Block
languagejs
{
	document: {
		"_id": String,Integer // Downloaded until now
  	}]
}

// Download Id
		"pluginId": String, EVENTS
// PluginFollowing idare the downloadevents objectfired belongswithin to
		status: String, // Submitted, InProgress, Complete, Failed.
		createdOn: Date,
		updatedOn: Date,
  		stats: {
  			totalFiles: Numberthe SDK
"<plugin>:download:complete"
EventManager.dispatch("sunbirded:download:complete", {
	id: String, // Download Id
	files: [{
		file: String, // File that is downloaded
		source: String, // Totalsource filesfrom towhere download
			downloadedFiles: Number, // Total files downloaded so far
 it is downloaded
			totalSizepath: NumberString, // TotalRelative numberpath ofwhere bytesthe tofile downloadis downloaded to
			downloadedSizesize: Number,Integer // Totalfile numbersize ofin bytes
downloaded so far
  		},
  		files: [{ // Status of each file within the given download
  			file: String, // File that is downloaded
  			source: String, // source from where it is downloaded
  			path: String, // Relative path where the file is downloaded to
  			size: Integer, // Total file size in bytes
  			downloaded: Integer // Downloaded until now
  		}]
	},
	index: [pluginId, status, updatedOn]
}

Folder Structure

Following is the folder structure of the container when it is installed on a dekstop/raspberrypi device/server

Code Block
<app_base_dir>
	/logs
	|----/container.log
	|----/crash.log
	/<plugin_id>
	|----/files // Folder where the files are either download/imported
	|----/ecars // Folder where the ecar files are stored
	|----/logs // Plugin logs
	|--------/application.log
	|--------/crash.log
	|----/content
	|--------/<do_xxxxxx> // Path where the ecar files are extracted and served via http during play

Download Manager

Below explains the process of downloading content when a user request for it, the steps are 

  1. Download Manager is initialised with config and will be ready to take the request
  2. When a user request for the download Sunbird Ed Plugin forward request to download Queue
  3. If download manager is available it will download the content and once it is download
  4.  It will raise an event and plugin will listen to that event
  5. With event data it will extract the ECAR and index the content in content database

Image Removed

At low level the download manager works as

  1. When it starts downloading it creates <filename>.sud and <filename>.<n>.partial(s) files, <filename>.sud file contains the meta data about file.
  2. If internet is disconnected or system got shut down  then it will pause the download
  3. When the internet is connected again or system restarted it will read the file .sud  and continue the download from where it left
  4. When the file is download completed then it combines all the partials and creates final ECAR file.

For more details you can refer su-downloader3

Image Removed

Telemetry Sync

Here we have two approaches to sync telemetry efficiently below will explain the them with pros and cons and concludes on one.

Solution 1:

This solution is having two steps to sync the telemetry in first step there will be a process which will execute every <x> minutes(configurable) in this method below steps are performed

  1. Read the events form telemetry database
  2. Events will be divided into equal numbers as batches of size <x> events(configurable)
  3. The events will be compressed(.gzip) and stored in telemetry folder
  4. Same events will be deleted from Telemetry database.

Image Removed

Below diagram shows the second step process which is again a process which will execute in every <x> minutes(configurable)  below are the steps executed

  1. Check the internet connectivity if available read all the event(s) files from telemetry folder
  2. Sync the events to server and after each file is synced move them to telemetryArchive folder
  3. The files in telemetry archive folder will be deleted in <x> days(configurable)

Image Removed

Pros:

1. No dependency on database availability while syncing telemetry to server

2. There is clear separation in creation of compressed telemetry and syncing compressed telemetry to server

3. If the database is crashed or uninstalled by user still we  have telemetry in file system since we are reading and storing them in file system

Cons:

1. User can delete the telemetry files from file system

2. More number of I/O operation on file system

3. Possibility of telemetry sync failure is high

Solution 2:

This solution will have only one process which will be executed in every <x> minutes(configurable) . Below are the steps

  1. Check for network connection if available ready the events from telemetry database
  2. Events will be divided into equal numbers as batches of size <x> events(configurable)
  3. Sync the events to server and after each file is synced move them to telemetryArchive folder
  4. The files in telemetry archive folder will be deleted in <x> days(configurable)
  5. Same events will be deleted from Telemetry database
  6. Above steps from step3 to step5 will be executed in series for each batch of events

Image Removed

Pros:

1. Less chance of telemetry being lost due to user actions

2. Telemetry  sync failure is low

3. Less number of I/O operation which result in more performance

4. Easy to develop and test

5. There is single source of data which is not synced to server which will help in export process.

...

1. Dependency on database availability

2.  Possibility of duplicate telemetry sync if the process execution time is less

...

	}]
});

Settings SDK

Code Block
languagejs
/* 
 * Method to get the instance of the settings sdk
 */
getInstance = function(String pluginId) : SettingsSDK {};

/* 
 * Method to put the setting
 * @param key String - The key for the config/setting
 * @param value Object - The value of the setting
 */
put: function(String key, Object value) : Promise {};

/* 
 * Method to get the setting
 * @param key String - The key for the config/setting
 * @return value Object
 */
get: function(String key) : Promise {};

Open Questions:

  • Does encryption needs to be part of container or plugin


...

Database Schema

plugin_registry

Code Block
languagejs
{
	document: {
		"_id": String, // Plugin Id
		"config": {} // Plugin config
	},
	index: []
}

settings

Code Block
languagejs
{
	document: {
		"_id": String, // Setting id
		"value": Object // Setting value
	},
	index: []
}

telemetry

A document database is created for each plugin telemetry storage - <plugin_id>_telemetry

Code Block
languagejs
{
	document: {
		"_id": String, // mid of the event
		"event": {} // event
	},
	index: []
}

telemetry_packets

Code Block
languagejs
{
	document: {
		"_id": String, // packet id
		"pluginId": String, // Plugin id the packet belongs to
		"status": String, // status of the packet. Synced/NotSynced
		"statusMsg": String, // Associated message to the status
		"createdOn": Date, // Date of packet creation
		"updatedOn": Date, // Date when the document is updated
		"size": Number, // Size of the events in the packet
		"events": [] // events
	},
	index: [pluginId, status, updatedOn]
}

download_queue

Code Block
languagejs
{
	document: {
		"_id": String, // Download Id
		"pluginId": String, // Plugin id the download object belongs to
		status: String, // Submitted, InProgress, Complete, Failed.
		statusMsg: String, // Associated message to the status
		createdOn: Date,
		updatedOn: Date,
  		stats: {
  			totalFiles: Number, // Total files to download
			downloadedFiles: Number, // Total files downloaded so far
  			totalSize: Number, // Total number of bytes to download
  			downloadedSize: Number, // Total number of bytes downloaded so far
  		},
  		files: [{ // Status of each file within the given download
  			file: String, // File that is downloaded
  			source: String, // source from where it is downloaded
  			path: String, // Relative path where the file is downloaded to
  			size: Integer, // Total file size in bytes
  			downloaded: Integer // Downloaded until now
  		}]
	},
	index: [pluginId, status, updatedOn]
}

Folder Structure

Following is the folder structure of the container when it is installed on a dekstop/raspberrypi device/server

Code Block
<app_base_dir>
	/logs
	|----/container.log
	|----/crash.log
	/<plugin_id>/
	|----/files/ // Folder where the files are either download/imported
	|--------/file1.ar // Sample file
	|----/ecars/ // Folder where the ecar files are stored
	|--------/do_1234.ecar // Ecar file
	|----/logs/ // Plugin logs
	|--------/application.log
	|--------/crash.log
	|----/content/
	|--------/<do_xxxxxx>/ // Path where the ecar files are extracted and served via http during play
	|----/telemetry_archived/
	|--------/<packet_id>.json // Archived telemetry files that are either exported or synced.

Download Manager


Below explains the process of downloading content when a user request for it, the steps are 

  1. Download Manager is initialised with config and will be ready to take the request
  2. When a user request for the download Sunbird Ed Plugin forward request to download Queue
  3. If download manager is available it will download the content and once it is download
  4.  It will raise an event and plugin will listen to that event
  5. With event data it will extract the ECAR and index the content in content database


Image Added


At low level the download manager works as

  1. When it starts downloading it creates <filename>.sud and <filename>.<n>.partial(s) files, <filename>.sud file contains the meta data about file.
  2. If internet is disconnected or system got shut down  then it will pause the download
  3. When the internet is connected again or system restarted it will read the file .sud  and continue the download from where it left
  4. When the file is download completed then it combines all the partials and creates final ECAR file.

For more details you can refer su-downloader3

Image Added


Telemetry Sync

Image Added

Telemetry Events

...