Introduction
The container for the OpenRAP 2.0 has been developed on top of ext-framework. The additions on top of ext-framework are:
- SDK's like Global, Settings & Download Manager
- While the ext-framework has been designed to add plugins within a single application, the use case of OpenRAP 2.0 is that each application is a plugin within OpenRAP. Therefore OpenRAP container has been built on top of ext-framework to support this capability
Following is the high level component diagram of OpenRAP 2.0
SDK
Following are SDK's built into the container.
Global
/* * Plugin to register with the container on initialization. * pluginConfig = { * pluginVer: String, * apiToken: String, * apiBaseURL: String, * apiTokenRefreshFn: Function * } */ register = function(String pluginId, Object pluginConfig){}; /* * Get the plugin configuration. * @param pluginId String * @return pluginConfig */ get = function(String pluginId): pluginConfig {}
Telemetry SDK
// API to register a plugin and it's corresponding sync url to dispatch telemetry to. // config = {batchSize: 200, syncInterval: 30} register = function(String pluginId, String syncURL, Object config){}; // Get the telemetry SDK instance. This can be injectable when the plugin uses typescript getInstance = function(String pluginId) : TelemetrySDK {}; // API to log/store a telemetry event send = function(String event) : Promise {}; // API to log/store an array of telemetry events send = function(Array[String] events) : Promise {}; // API to receive a batch of telemetry events in zip stream sendBatch = function(String batchEvent) : Promise {}; // API to receive a batch of telemetry events in zip stream sendBatch = function(byte[] events) : Promise {}; // list = function() : TelemetryPacket[] {}; get = function(String packetId): TelemetryPacket {}; // API to force sync. This would create packets to be synced sync = function() : Promise {}; // API to get the status of events/batches for a given plugin id // Response - {events: 3456, batches: 20} getStatus = function() : Promise {};
Usage example
// Example to use telemetry SDK telemetrySDK = TelemetrySDK.getInstance(this.manifest.id); telemetrySDK.send({...}); telemetrySDK.send([{...}]); telemetrySDK.sync();
File SDK
getInstance = function(String pluginId) : FileSDK {}; mkdir = function(String path) : Promise {}; copy = function(String source, String dest) : Promise {}; mv = function(String from, String to) : Promise {}; rm = function(String file) : Promise {}; rmdir = function(String path) : Promise {}; zip = function(String path, String fileName) : Promise {}; unzip = function(String fileName, String path) : Promise {}; getAbsPath = function(String path) : String {}; watch = function(String[] paths, callback){};
Network SDK
isNetworkAvailable = function() : boolean {}; // Events // Following are the events fired within the SDK network:available network:disconnected
Usage
EventManager.on('network:available', function() { // Do something. // May be sync telemetry. // Fetch content updates // Resume content downloads // etc });
Http SDK
getInstance = function(String pluginId) : HttpSDK {}; get = function(String url, Object options) : Promise {}; post = function(String url, Object data, Object options) : Promise {}; patch = function(String url, Object data, Object options) : Promise {}; delete = function(String url, Object options) : Promise {}; head = function(String url, Object options) : Promise {};
Download Manager
Following are the APIs for the download manager. The workflow of download manager is detailed out in the next few sections.
/* Method to get the instance of the download manager */ getInstance = function(String pluginId) : DownloadManager {}; /* * Method to queue the download of a file * @Param file - The file to download * @Param path - Path to download the file * @return downloadId - The download id reference */ download = function(String file, String path) : String {}; /* * Method to queue the download of a file * @Param file - The file to download * @Param path - Path to download the file * @return downloadId - The download id reference */ download = function(String[] files, String folder) : String {}; /* * Method to get the status of the download * @Param downloadId String * @return Download object */ get = function(String downloadId) : DownloadObject {}; /* * Method to list the download queue based on the status * @Param status String - The status of the download - Submitted, Complete, InProgress, Failed. Blank option will return all status * @return Array - Array of download objects */ list = function(String status): DownloadObject[] {}; DownloadObject = { id: String, // Download id status: String, // Submitted, InProgress, Complete, Failed. createdOn: Date, updatedOn: Date, stats: { totalFiles: Number, // Total files to download downloadedFiles: Number, // Total files downloaded so far totalSize: Number, // Total number of bytes to download downloadedSize: Number, // Total number of bytes downloaded so far }, files: [{ // Status of each file within the given download file: String, // File that is downloaded source: String, // source from where it is downloaded path: String, // Relative path where the file is downloaded to size: Integer, // Total file size in bytes downloaded: Integer // Downloaded until now }] } // Events // Following are the events fired within the SDK "download:complete" EventManager.dispatch("download:complete", { id: String, // Download Id files: [{ file: String, // File that is downloaded source: String, // source from where it is downloaded path: String, // Relative path where the file is downloaded to size: Integer // file size in bytes }] });
Settings SDK
/* Method to get the instance of the settings sdk */ getInstance = function(String pluginId) : SettingsSDK {}; put: function(String key, Object value) : Promise {}; get: function(String key) : Promise {};
Open Questions:
- Does encryption needs to be part of container or plugin
Database Schema
Registry
{ document: { "_id": String, // Plugin Id "config": {} // Plugin config }, index: [] }
Settings
{ document: { "_id": String, // Setting id "value": Object // Setting value }, index: [] }
Telemetry
A document database is created for each plugin telemetry storage - <plugin_id>_telemetry
{ document: { "_id": String, // mid of the event "event": {} // event }, index: [] }
TelemetryPackets
{ document: { "_id": String, // packet id "pluginId": String, // Plugin id the packet belongs to "status": String, // status of the packet. Synced/NotSynced "createdOn": Date, // Date of packet creation "updatedOn": Date, // Date when the document is updated "size": Number, // Size of the events in the packet "events": [] // events }, index: [pluginId, status, updatedOn] }
DownloadQueue
{ document: { "_id": String, // Download Id "pluginId": String, // Plugin id the download object belongs to status: String, // Submitted, InProgress, Complete, Failed. createdOn: Date, updatedOn: Date, stats: { totalFiles: Number, // Total files to download downloadedFiles: Number, // Total files downloaded so far totalSize: Number, // Total number of bytes to download downloadedSize: Number, // Total number of bytes downloaded so far }, files: [{ // Status of each file within the given download file: String, // File that is downloaded source: String, // source from where it is downloaded path: String, // Relative path where the file is downloaded to size: Integer, // Total file size in bytes downloaded: Integer // Downloaded until now }] }, index: [pluginId, status, updatedOn] }
Folder Structure
Following is the folder structure of the container when it is installed on a dekstop/raspberrypi device/server
<app_base_dir> /logs |----/container.log |----/crash.log /<plugin_id> |----/files // Folder where the files are either download/imported |----/ecars // Folder where the ecar files are stored |----/logs // Plugin logs |--------/application.log |--------/crash.log |----/content |--------/<do_xxxxxx> // Path where the ecar files are extracted and served via http during play
Download Manager
Below explains the process of downloading content when a user request for it, the steps are
- Download Manager is initialised with config and will be ready to take the request
- When a user request for the download Sunbird Ed Plugin forward request to download Queue
- If download manager is available it will download the content and once it is download
- It will raise an event and plugin will listen to that event
- With event data it will extract the ECAR and index the content in content database
At low level the download manager works as
- When it starts downloading it creates <filename>.sud and <filename>.<n>.partial(s) files, <filename>.sud file contains the meta data about file.
- If internet is disconnected or system got shut down then it will pause the download
- When the internet is connected again or system restarted it will read the file .sud and continue the download from where it left
- When the file is download completed then it combines all the partials and creates final ECAR file.
For more details you can refer su-downloader3
Telemetry Sync
Here we have two approaches to sync telemetry efficiently below will explain the them with pros and cons and concludes on one.
Solution 1:
This solution is having two steps to sync the telemetry in first step there will be a process which will execute every <x> minutes(configurable) in this method below steps are performed
- Read the events form telemetry database
- Events will be divided into equal numbers as batches of size <x> events(configurable)
- The events will be compressed(.gzip) and stored in telemetry folder
- Same events will be deleted from Telemetry database.
Below diagram shows the second step process which is again a process which will execute in every <x> minutes(configurable) below are the steps executed
- Check the internet connectivity if available read all the event(s) files from telemetry folder
- Sync the events to server and after each file is synced move them to telemetryArchive folder
- The files in telemetry archive folder will be deleted in <x> days(configurable)
Pros:
1. No dependency on database availability while syncing telemetry to server
2. There is clear separation in creation of compressed telemetry and syncing compressed telemetry to server
3. If the database is crashed or uninstalled by user still we have telemetry in file system since we are reading and storing them in file system
Cons:
1. User can delete the telemetry files from file system
2. More number of I/O operation on file system
3. Possibility of telemetry sync failure is high
Solution 2:
This solution will have only one process which will be executed in every <x> minutes(configurable) . Below are the steps
- Check for network connection if available ready the events from telemetry database
- Events will be divided into equal numbers as batches of size <x> events(configurable)
- Sync the events to server and after each file is synced move them to telemetryArchive folder
- The files in telemetry archive folder will be deleted in <x> days(configurable)
- Same events will be deleted from Telemetry database
- Above steps from step3 to step5 will be executed in series for each batch of events
Pros:
1. Less chance of telemetry being lost due to user actions
2. Telemetry sync failure is low
3. Less number of I/O operation which result in more performance
4. Easy to develop and test
5. There is single source of data which is not synced to server which will help in export process.
Cons:
1. Dependency on database availability
2. Possibility of duplicate telemetry sync if the process execution time is less
Conclusion: We will go with the solution 2 since there is less chance of losing telemetry in the process of syncing it to server and possibility of the user being able to easily access the file system compare to database more.
Telemetry Events
<To be done>
Logging
<To be done>