Overview

Jira Ticket: SB-10068 - Getting issue details... STATUS

As part of implementation of A/B framework, we have evaluated 4 frameworks' that are available for A/B testing. Planout (by Facebook), Wasabi (by Intuit), Alephbet & sixpack (by seatgeek)

We will list down the salient feature and comparison among this tools in following table

	Planout	Wasabi	Sixpack	Alephbet
Implemented By	Facebook (Python), Hubspot(Javascript), GlassDoor (Java)	Intuit	SeatGeek	GingerLime
Server Implementation	Multiple - Python, Java, PHP	Java	Python	None
Event logging mechanism	Add own logic for log-event - event object is received as an input	Full-fledged API - for assignment, impression, engagement	API for collection impression/engagement	Stores in javascript localstore, gives plug-in point
License information	BSD	Apache 2.0	BSD	MIT
Other dependency	None	Cassandra/MySQL	Redis	None
Other standout feature	Support's namespace - can be used to qualify experiment further	Has UI for configuring and looking at analytics data	Has UI for configuring and looking at analytics data	Has support for tracking adapters, as well as allows to implement custom tracking adapter.

It is desirable to use, just java-script implementation as minimal data storage or even just telemetry generation is enough for gathering data. So Planout/Alephbet are better fit for our purpose.

Sample Configuration for audience selection

Syntax

{
  "activeExperiments": [
    {
      "ExperimentName": "<Name of experiment>",
      "startDate": "2019-01-01",
      "endDate": "2019-01-31",
      "rendering": true/false,
      "deploymentVersion": 1.14"rules": {
        "stateCode": [
          "state1",
          "state2"..."stateN"
        ],
        "browser": [
          "browser1",
          "browser2"..."browserN"
        ],
        "userType": [
          "Student",
          "Teacher",
          ..."Other"
        ],
        "framework.id": [
          "NCF",
          ..."Framework2"
        ],
        "framework.board": [
          "CBSE",
          ..."ICSE"
        ],
        "framework.medium": [
          "English",
          ..."Telugu"
        ]
      }
    },
    {
      "ExperimentName": "<Name of experiment>""startDate": "2019-02-01",
      "endDate": "2019-02-28",
      "rendering": true/false"deploymentVersion": 1.14"rules": {
        "stateCode": [
          "state1",
          "state2"..."stateN"
        ],
        "browser": [
          "browser1",
          "browser2"..."browserN"
        ],
        "userType": [
          "Student",
          "Teacher",
          ..."Other"
        ],
        "framework.id": [
          "NCF",
          ..."Framework2"
        ],
        "framework.board": [
          "CBSE",
          ..."ICSE"
        ],
        "framework.medium": [
          "English",
          ..."Telugu"
        ]
      }
    }
  ]
}

Example configuration

{
  "activeExperiments": [
    {
      "ExperimentName": "TestLibPage",
      "startDate": "2019-01-01",
      "endDate": "2019-01-31",
      "rendering": true,
      "deploymentVersion": 1.14"rules": {
        'stateCode': [
          "AP",
          "KA"
        ],
        'browser': [
          "Chrome"
        ]
      }
    },
    {
      "ExperimentName": "TestLibPage",
      "startDate": "2019-02-01",
      "endDate": "2019-02-28",
      "rendering": true,
      "deploymentVersion": 1.15"rules": {
        "browser": [
          "Firefox"
        ]
      }
    }
  }
]

A/B testing available framework evaluation

Overview

Sample Configuration for audience selection