Page Comparison

Overview

Learner Service provides APIs related to resources such as Organisations, Users (i.e. learner), Course Batches, Locations, Badges, Pages etc.

...

Transactional (generated due to DB update)
Instructional (e.g. manual sync, badge assertion, manual background job run)

Image RemovedImage Added

Audit Logging

For reliable audit logging, events (on DB updates) should be generated immediately, in order and avoid duplication especially in a multi-node DB installation. Following is a summary of the available alternatives explored.

Approach	Pros	Cons
Cassandra Trigger	Immediately triggered Maintains order (not tested in multi-node scenario)	Deprecated (refer Datastax documentation) Semi-private and subject to change Performance overhead
Use audit logging of Apache Cassandra 4.0	Out of the box product feature which is expected to meet performance and reliability design goals	Not yet released
Change Data Capture	No performance overhead during write	Data not immediately available (requires memtables flush or commit log limit) Every node has a separate commit log (binary format) Consumer needs to handle duplicate events in a multi-node setup
ecAudit (Ericsson Cassandra Audit)	Immediately logged (on the node where the query is executed) Read is straightforward as log is in text format	Correlation required in case of FAILED events Every node has a separate audit log Performance overhead

...

Code Block

USER EVENT

{
    "identifier": "3d45fbd8-b911-4cc5-b503-61215902d780",
    "ets": 1548833319693,
    "event": {
        "properties": {
            "id": {
                "nv": "3d45fbd8-b911-4cc5-b503-61215902d780"
            },
            "userId": {
                "nv": "3d45fbd8-b911-4cc5-b503-61215902d780"
            },
            "firstName": {
                "nv": "manzarul haque"
            },
            "email": {
                "nv": "manzarul07@gmail.com"
            },
            "emailVerified": {
                "nv": true
            },
            "phone": {
                "nv": "98988888889888888888"
            },
            "phoneVerified": {
                "nv": true
            },
            "channel": {
                "nv": "ap"
            },
            "rootOrgId": {
                "nv": "123456789"
            }
        }
    },
    "operationType": "UPSERT",
    "eventType": "transactional",
    "userId": "ANONYMOUS",
    "createdOn": "2019-01-30T12:58:39.693+0530",
    "objectType": "user"
}

ADDRESS EVENT

{
    "identifier": "12345",
    "ets": 1548833319693,
    "event": {
        "properties": {
            "id": {
                "nv": "12345"
            },
            "userIdtype": { // TODO
                "nv": "3d45fbd8-b911-4cc5-b503-61215902d780usr"
            },
            "cityuserId": {
                "nv": "Bangalore3d45fbd8-b911-4cc5-b503-61215902d780"
            },
            "zipCodecity": {
                "nv": "560000Bangalore"
            },
        }     },"zipCode": {
    "operationType": "UPSERT",     "eventType": "transactional"     "userIdnv": "ANONYMOUS560000",
            }
        }
    },
    "operationType": "UPSERT",
    "eventType": "transactional"
    "userId": "ANONYMOUS",
    "createdOn": "2019-01-30T12:58:39.693+0530",
    "objectType": "address"
}

EDUCATION EVENT

{
    "identifier": "3d45fbd8-b911-4cc5-b503-61215902d780",
    "ets": 1548833319693,
    "event": {
        "properties": {
            "id": {
                "nv": "6789"
            },
            "userId": {
                "nv": "3d45fbd8-b911-4cc5-b503-61215902d780"
            },
            "degree": {
                "nv": "B.SC"
            },
            "yearOfPassing": {
                "nv": "2010"
            },
            "addressId": {
                "nv": "1111"
            }
        }
    },
    "operationType": "UPSERT",
    "eventType": "transactional"
    "userId": "ANONYMOUS",
    "createdOn": "2019-01-30T12:58:39.693+0530",
    "objectType": "user_education"
}

ADDRESS EVENT

{
    "identifier": "1111",
    "ets": 1548833319693,
    "event": {
        "properties": {
            "id": {
                "nv": "1111"
            },
            "userIdtype": { // TODO
                "nv": "3d45fbd8-b911-"edu"
            },
            "userId": {
                "nv": "3d45fbd8-b911-4cc5-b503-61215902d780"
            },
            "city": {
                "nv": "Bangalore"
            },
            "zipCode": {
                "nv": "560000"
            }
        }
    },
    "operationType": "UPSERT",
    "eventType": "transactional"
    "userId": "ANONYMOUS",
    "createdOn": "2019-01-30T12:58:39.693+0530",
    "objectType": "address"
}

...

Indexer Mapping

Code Block

{
    "user": {
        "indexTypeindex": "user",
        "key": "id",
        "visibilityEnabled": true
    },
    "user_education": {
        "indexTypeindex": "user",
        "path": "education",
        "key": "userid",
        "visibilityEnabled": true
    },
    "user_job_profile": {
        "indexTypeindex": "user",
        "path": "jobProfile",
        "key": "userid",
        "visibilityEnabled": true
    },
    "user_org": {
        "indexTypeindex": "user",
        "path": "organisations",
        "key": "userid",
        "visibilityEnabled": true
    },
    "address": {
        "types": {
            "usr": {
                "indexTypeindex": "user",
                "path": "address",
                "key": "userid",
                "visibilityEnabled": true
            },
            "edu": {
                "indexTypeindex": "user",
                "path": "education.address",
                "key": "userid",
                "visibilityEnabled": true
            },
            "job": {
                "indexTypeindex": "user",
                "path": "jobProfile.address",
                "key": "userid",
                "visibilityEnabled": true
            },
            "org": {
                "indexTypeindex": "org",
                "path": "address",
                "key": "orgid" // TODO
            },
        }
    },
    "organisation": {
        "indexTypeindex": "org",
        "key": "id"

    },
    "course_batch": {
        "indexTypeindex": "cbatch",
        "key": "id"
    },
    "user_courses": {
        "indexTypeindex": "usercourses",
        "key": "id"
    },
    "location": {
        "indexTypeindex": "location",
        "key": "id"
    },
    "visibility": {
        "indexTypeindex": {
            "user": {
                "table": "user",
                "column": "profilevisibility",
                "indexType": "profilevisibility",
                "key": "id"
            }
        }
    }
}

Action Points

1) Test Cassandra Triggers and ecAudit with different data types (e.g. list, map, blob etc.)

Please refer Cassandra Audit Logging for Different Types

2) Can Cassandra Trigger be applied per column?

No (refer Apache Cassandra Trigger documentation)

3) Performance comparison of Cassandra Triggers and ecAudit

Please refer Performance Comparison of Cassandra Audit Log Options.

4) Drop data in user_education, user_job_profile and address tables based on discussion with Rahul and Rayulu

5) Rename indexType in mapping to ____ ?

index

Appendix

Learner Service Data Models Comparison

Versions Compared

Old Version 1

New Version Current

Key

Overview

Audit Logging

Action Points

Appendix