Versions Compared

Old Version 1

changes.mady.by.user AMOL N GOLVELKAR

Saved on

compared with

New Version Current

changes.mady.by.user AMOL N GOLVELKAR

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

Jira Ticket:

Jira Legacy
serverSystem JIRA
serverId2207a759-5bc8-39c5-9cd2-aa9ccc1f65dd
keySB-10068

As part of implementation of A/B framework, we have evaluated 4 frameworks' that are available for A/B testing. Planout (by Facebook), Wasabi (by Intuit), Alephbet & sixpack (by seatgeek)

We will list down the salient feature and comparison among this tools in following table 



PlanoutWasabiSixpackAlephbet
Implemented ByFacebook (Python), Hubspot(Javascript), GlassDoor (Java)IntuitSeatGeekGingerLime
Server ImplementationMultiple - Python, Java, PHPJavaPythonNone
Event logging mechanismAdd own logic for log-event - event object is received as an inputFull-fledged API - for assignment, impression, engagementAPI for collection impression/engagementStores in javascript localstore, gives plug-in point
License informationBSDApache 2.0BSDMIT

Other dependency

NoneCassandra/MySQLRedisNone
Other standout featureSupport's namespace  - can be used to qualify experiment furtherHas UI for configuring and looking at analytics dataHas UI for configuring and looking at analytics dataHas support for tracking adapters, as well as allows to implement custom tracking adapter.
Config Syntax
//Extend experiment class 
//and over-ride assign function to
//initiate variants
class MyExperiment extends 
PlanOut.Experiment {

assign(params, args) { 
params.set(‘foo’, new PlanOut
.Ops.Random
.UniformChoice({‘choices’: [‘a’, ‘b’], 
‘unit’: args.id})); 
}

}

//Following is way to get value out 
of experiment object
var exp =
 new MyExperiment({‘id’: user.id });
console.log(“User has foo 
param set to “ + exp.get(‘foo’));


They have back-end and UI for confguri
var session = new sixpack.Session();
session.participate('test-exp', ['alt-one'
, 'alt-two'], function (err, res) {
  if (err) throw err;
  alt = res.alternative.name
  if (alt == 'alt-one') {
    console.log('default: ' + alt);
  } else {
    console.log(alt);
  }
});
var button_color_experiment = new AlephBet.Experiment({
  name: 'button color', 
 // the name of this experiment; required.
  variants: {  
   // variants for this experiment; required.
    blue: {
      activate: function() { 
     // activate function to execute if variant is selected
        $('#my-btn').attr('style', 'color: blue;');
      }
    },
    red: {
      activate: function() {
        $('#my-btn').attr('style', 'color: red;');
      }
    }
  },
});


It is desirable to use, just java-script implementation as minimal data storage or even just telemetry generation is enough for gathering data. So Planout/Alephbet are better fit for our purpose.


Sample Configuration

...

for

...

audience selection


Code Block
languagejs
titleSyntax
{
  "activeExperiments": [
    {
      "ExperimentName": "<Name of experiment>", //Experiment name
      "startDate": "2019-01-01", //Start date from which experiment is applicable.
      "endDate": "2019-01-31", //End date when experiment expires
      "rendering": true/false, //toggle for enable/disable the experiement - regardless of toher paramters.
      "deploymentVersion": 1.14,
	  "rules": {
        "stateCode": [ //Value should be one of the configured
          "state1",
          "state2"..."stateN"
        ],
        "browser": [
          "browser1",
          "browser2"..."browserN"
        ],
        "userType": [
          "Student",
          "Teacher",
          ..."Other"
        ],
        "framework.id": [
          "NCF",
          ..."Framework2"
        ],
        "framework.board": [
          "CBSE",
          ..."ICSE"
        ],
        "framework.medium": [
          "English",
          ..."Telugu"
        ]
      }
    },
    {
      "ExperimentName": "<Name of experiment>""startDate": "2019-02-01",
      "endDate": "2019-02-28",
      "rendering": true/false"deploymentVersion": 1.14"rules": {
        "stateCode": [
          "state1",
          "state2"..."stateN"
        ],
        "browser": [
          "browser1",
          "browser2"..."browserN"
        ],
        "userType": [
          "Student",
          "Teacher",
          ..."Other"
        ],
        "framework.id": [
          "NCF",
          ..."Framework2"
        ],
        "framework.board": [
          "CBSE",
          ..."ICSE"
        ],
        "framework.medium": [
          "English",
          ..."Telugu"
        ]
      }
    }
  ]
}



Code Block
languagejs
titleJsonConfigurationExample configuration
{
  "activeExperiments": [
    {
      "ExperimentName": "TestLibPage",
      "startDate": "2019-01-01",
      "endDate": "2019-01-31",
      "rendering": true,
      "deploymentVersion": 1.14"rules": {
        'stateCode': 'AP',[
          "AP",
          "KA"
        ],
        'deviceTypebrowser': 'Android'
} [
          "Chrome"
        ]
      }
    },
    {
      "ExperimentName": "TestLibPage",
      "startDate": "2019-02-01",
      "endDate": "2019-02-28",
      "rendering": true,
      "deploymentVersion": 1.15"rules": {
        "browser": [
          "Firefox"
        ]
      }
    }
  }
]