...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Table of Contents
Background:
Currently, Deskstop App has a logging system in place. That is configurable and allows storing of backups on log rotation. Desktop App stores all type('all'|'trace'|'fatal'|'error'|'off'|'info'|'warn'|'debug') of logs in app.log file that has a rotation policy of 10 MB and stores compressed backup up to 3 files. Add has a dedicated error.log file that stores only error logs(10 MB | 3 backups). This is built on top of log4js. It is difficult to query based on date and there is no distinction between logs.
...
Solutions:
Logs Categorize:
Sunbird desktop app code can be categorized into different sections based on the task that it dose. Grouping logs based on categorizes help debug, sync and analyze the different aspects of the Desktop app. This task can have a different log folder and each file inside this can be stored based on the date they are created. This files can be in yyyy-MM-dd<logType>.log pattern.
...
| Log Category | Enabled | Syncs to platform | Log storage | Log format | |
|---|---|---|---|---|---|
| 1 | App Install / Update | Enabled by default. | No | Last 3 Logs will be stored | String |
| 2 | Application | Disabled, can be enabled when needed for a short amount of time. | No | Last 3 Logs will be stored | String |
| 3 | Crash | Enabled by default. | Yes | Last 10 Logs will be stored | String |
| 4 | Error | Enabled by default. | Yes | Last 500 Logs will be stored | String |
| 5 | Performance | Enabled by default. | Yes | Last 100 Logs will be stored | JSON |
App Install / Update:
These logs will be generated when the app is being installed on the system(Window or Ubuntu).
...
Additionally, there will be a few more types of log categories:
...
| language | actionscript3 |
|---|
...
These logs will be generated by hooking into Electron-builder installation hooks. We can keep 2-3 installation logs.
Application:
All logs generated by the app while its running can be collected into one file. By default, this log will be disabled. It can be enabled when it needs for a specified time then it can be disabled.
Crash:
These logs get generated when the app(Electron) crashes. Logs generated will be in minidump format and these logs need to be further processed to do any analysis. The app keeps the last 10 crash logs and sync to the platform for further processing using the network queue.
Syncing Crash logs to the platform:
Solution 1: Processing minidump in server
We require an API to sync and batch job process minidumps.
- API to store minidumps: This API will accept minidumps from all apps and stores this in the blob.
- Batch job: This job will retrieve minidumps from the blob and using electron symbols and minidump library will create meaning full data. This data can be stored in Druid or Elastic search for analysis.
Solution 2: Processing minidump in the desktop app
We require electron symbols and minidump parsing library to process crash logs. Once processed we can sync the processed data to error sync API.
Error:
All unhandled exceptions and unhandled rejections will be collected in these logs. The app keeps the last 500 crash logs and sync to the platform for further processing using the network queue.
Syncing Error logs to the platform:
Sunbird platform has error log aggregation API, same can be used here.
Performance:
This log gets generated for each task/API call etc, capturing time it took to complete, DID and other metrics(CPU, memory usage, etc). Perf logs will be generated for below tasks
- App startup
- Content Import
- Content Export
- Content Download
- Content Delete
- API calls
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
id: '<uuid>',
timeTaken: '<time taken to complete the task>',
createdOn: '<created date>',
type: '<network|system>',
subType: '<import,download,export etc>',
did:'<system device id>',
size: '<size of the object>',
extras: {<'task specific data'>}
} |
Logs generated will be in JSON format.
Syncing Perf log to the platform:
Solution 1: Sync to new API
We require new API sync this logs to the platform and to analyze/visualize.
Solution 2: Log Telemetry metrics event
We can convert perf log to telemetry metrics event. This will be synced along with all the telemetry events to platform.
Logs levels
The desktop app will have all traditional log levels and some custom levels.
| Level | Description | |||
|---|---|---|---|---|
| 1 | INFO | For logging information messages. | ||
| 2 | DEBUG | For logging messages for debugging purposepurposes. | ||
| 3 | ERROR | For logging errors. | ||
| 4 | FATAL | For logging errors that are fatal. | ||
| 5 | WARN | For logging warning messages. | ||
| 6 | TRACE | For logging messages to help trace errors. | ||
| 7 | PERF | For logging all performance information. Like how much time import took, time took for search content, etc. | ||
| METRICS | For logging system metrics in different stages. This can contain memory, storage, a resource used, etc. | |||
| 8 | DB_ERROR | For logging errors from Database. | ||
| 9 | NETWORK_ERROR | For logging network-related errors. | DIAG | For logging Diagnostic information like a number of tasks in queue, etc. |