rjjfox / ab-test-calculator

AB test calculator run through Streamlit

AB Test Calculator

An AB test significance calculator with options to use Bayesian or Frequentist statistics. Calculations come through basic SciPy.stats methods, the web app is built with Streamlit and hosted on Heroku. See it live here.

See also my AB test sample size calculator.

Table of Contents

• Getting Started
• Deployment
• Built With
• License

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Installing

Clone/fork the repo onto your local machine.

It is then recommended to use a virtual environment to install the dependencies using the requirements.txt file.

pip install -r requirements.txt

With these installed, you simply need to run

streamlit run app.py

Docker

Alternatively, with Docker, use the following command and then navigate to localhost.

docker run -p 80:8080 ryanfox212/ab-test-calculator

Deployment

I utilised Heroku's free tier to host the web app. To do the same, create a project on Heroku's website and then I recommend linking the project to a Github repository in the Deploy section.

Alternatively, you can use the Heroku CLI. Provided you have already created a project on the site, use

heroku git:remote -a [project-name]
git push heroku master

Use the same project name as the one you created. If successful, you can then go to https://[project-name].herokuapp.com/.

Built With

• Streamlit - The web application framework used
• SciPy - For the statistical methods
• Seaborn - For vizualisations
• Heroku - Cloud platform used for deployment

License

This project is licensed under the MIT License - see the LICENSE file for details.

ab-test-calculator's Issues

Add Bayes Factor to Bayesian analysis

Investigate and potentially add a Bayes Factor score to the Bayesian approach.

This will allow people to refer to a table that indicates how strong is the evidence towards A or B being the better experience.

Concerns are that this will become a kind of 'threshold' that people feel they need to meet. This leads to a more frequentist way of thinking.

Bayes Factor information

Add Docker installation capability to Readme

Containerize and upload to DockerHub. Add installation instructions to repo readme

Frequent 'Application error' on Heroku

H10 error without clear indication of what is causing the error.

The sample size app is working fine. Restarting the app brings it back online but only temporarily.

`2020-12-02T17:55:31.600161+00:00 heroku[router]: at=info method=GET path="/static/js/35.8b5aee2e.chunk.js" host=abtestcalculator.herokuapp.com request_id=6bed4e59-39e6-4eb4-91ba-054177676381 fwd="86.26.169.205" dyno=web.1 connect=7ms service=713ms status=200 bytes=924 protocol=https
2020-12-02T17:55:32.004219+00:00 heroku[router]: at=info method=GET path="/stream" host=abtestcalculator.herokuapp.com request_id=a07d1d46-eeb0-4f56-b346-372e71abaa39 fwd="86.26.169.205" dyno=web.1 connect=3ms service=7089ms status=101 bytes=3704 protocol=https
2020-12-02T18:31:56.360217+00:00 heroku[web.1]: Idling
2020-12-02T18:31:56.362124+00:00 heroku[web.1]: State changed from up to down
2020-12-02T18:31:57.289592+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2020-12-02T18:31:57.326310+00:00 app[web.1]: Stopping...
2020-12-02T18:31:57.909981+00:00 heroku[web.1]: Process exited with status 0
2020-12-04T23:48:36.484998+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/robots.txt" host=testcalculator.rfoxdata.co.uk request_id=e6332438-f459-4454-95a5-da0723859f0a fwd="66.249.64.131" dyno= connect= service= status=503 bytes= protocol=http
2020-12-07T18:09:57.005798+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=abtestcalculator.herokuapp.com request_id=aec55d2e-2e73-4727-8198-5c0ec3939a5c fwd="86.26.169.205" dyno= connect= service= status=503 bytes= protocol=https
2020-12-07T18:09:57.810181+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=abtestcalculator.herokuapp.com request_id=5a18804b-5ad4-47e1-8da8-a846fec70f17 fwd="86.26.169.205" dyno= connect= service= status=503 bytes= protocol=https
2020-12-07T18:11:04.649913+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=abtestcalculator.herokuapp.com request_id=a056875b-5aec-4a03-8c46-abf380e81cff fwd="86.26.169.205" dyno= connect= service= status=503 bytes= protocol=https
2020-12-07T18:11:05.248761+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=abtestcalculator.herokuapp.com request_id=95adaa28-9ccd-46c2-aa61-317457945f65 fwd="86.26.169.205" dyno= connect= service= status=503 bytes= protocol=https`