maccman / abba Goto Github PK
View Code? Open in Web Editor NEWA/B testing framework
License: MIT License
A/B testing framework
License: MIT License
Is there a reason why you didn't specify any versions for the gems in Gemfile? I'm asking, because it can lead to unexpected behaviour if you always retrieve the latest versions of the gems.
Is there a stable gemset with defined versions that you would suggest?
I'm getting the following error with HEAD (cc7c444) when loading /admin/experiments, only when running in production (thin start -e production
). Have I configured something incorrectly?
NoMethodError - undefined method `collection' for nil:NilClass:
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/mongo_mapper-0.12.0/lib/mongo_mapper/plugins/persistence.rb:49:in `collection'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/mongo_mapper-0.12.0/lib/mongo_mapper/plugins/querying.rb:61:in `query'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/mongo_mapper-0.12.0/lib/mongo_mapper/plugins/sci.rb:45:in `query'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/mongo_mapper-0.12.0/lib/mongo_mapper/plugins/identity_map.rb:90:in `query'
/home/username/abba/admin.rb:101:in `block in <top (required)>'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:1265:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:1265:in `block in compile!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:835:in `[]'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:835:in `block (3 levels) in route!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:851:in `route_eval'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:835:in `block (2 levels) in route!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:872:in `block in process_route'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:870:in `catch'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:870:in `process_route'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:834:in `block in route!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:833:in `each'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:833:in `route!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:936:in `dispatch!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:769:in `block in call!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:921:in `block in invoke'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:921:in `catch'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:921:in `invoke'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:769:in `call!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:755:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/path_traversal.rb:16:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/json_csrf.rb:17:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-1.4.3/lib/rack/logger.rb:15:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-1.4.3/lib/rack/commonlogger.rb:33:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:136:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:129:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-1.4.3/lib/rack/head.rb:9:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/rack-1.4.3/lib/rack/methodoverride.rb:21:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:99:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:1389:in `block in call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:1471:in `synchronize'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/sinatra-1.3.3/lib/sinatra/base.rb:1389:in `call'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/connection.rb:81:in `block in pre_process'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/connection.rb:79:in `catch'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/connection.rb:79:in `pre_process'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/connection.rb:54:in `process'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/connection.rb:39:in `receive_data'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/controllers/controller.rb:86:in `start'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/runner.rb:187:in `run_command'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/lib/thin/runner.rb:152:in `run!'
/home/username/.rvm/gems/ruby-1.9.3-p392/gems/thin-1.5.0/bin/thin:6:in `<top (required)>'
/home/username/.rvm/gems/ruby-1.9.3-p392/bin/thin:19:in `load'
/home/username/.rvm/gems/ruby-1.9.3-p392/bin/thin:19:in `<main>'
/home/username/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `eval'
/home/username/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `<main>'
Some version info:
$ ruby --version
ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]
$ bundle
Using i18n (0.6.1)
Using multi_json (1.5.0)
Using activesupport (3.2.11)
Using builder (3.0.4)
Using activemodel (3.2.11)
Using backports (2.6.5)
Using bson (1.8.0)
Using bson_ext (1.8.0)
Using coffee-script-source (1.4.0)
Using execjs (1.4.0)
Using coffee-script (2.2.0)
Using daemons (1.1.9)
Using eco-source (1.1.0.rc.1)
Using eco (1.0.0)
Using erubis (2.7.0)
Using eventmachine (1.0.0)
Using hike (1.2.1)
Using mongo (1.8.0)
Using plucky (0.5.2)
Using mongo_mapper (0.12.0)
Using rack (1.4.3)
Using rack-protection (1.3.2)
Using rack-test (0.6.2)
Using tilt (1.3.3)
Using sinatra (1.3.3)
Using sinatra-contrib (1.3.2)
Using sprockets (2.4.5)
Using sprockets-commonjs (0.0.6.pre)
Using stylus-source (0.31.0)
Using stylus (0.6.2)
Using thin (1.5.0)
Using uglifier (1.3.0)
Using useragent (0.4.16)
Using bundler (1.3.4)
config.yml:
ssl: false
username: username123
password: password123
Looks like you are using a 1-tailed test to calculate the probability. Is that correct? I see both ways used in different A/B testing tools (Optimzely use a 1-tailed test, I believe).
Curious as to the thinking behind the choice.
Odd that this happens only when running ABBA as a rack app under NGINX/Passenger, but when I do, I get this error message:
Error compiling CSS asset
Encoding::InvalidByteSequenceError: "\xE2" on US-ASCII
(in /var/code/abba/app/assets/stylesheets/experiment.css.styl)
(/var/code/abba/vendor/bundle/ruby/1.9.1/gems/execjs-1.4.0/lib/execjs/external_runtime.rb:173:in `read'
I understand that issue may be more appropriately answered in the ExecJS or the Stylus project, maybe even Passenger! I'm not sure exactly which of these projects this is best addressed in.
This doesn't happen when running under thin.
However, by adding this line at the top of the config.ru
file, the issue is solved for me given my preferred deployment setup:
Encoding.default_external = 'UTF-8'
Thought I'd open this issue in part to see if others have this problem.
In my head I expected this framework to report unique visitors, not total. So if the same user refreshes the page multiple times, I'm seeing a visit for each page load whereas I expect to see 1 visit.
I'm still pretty new to AB testing philosophy... am I "doing it wrong" to expect unique visits? If not, would you accept a patch for that?
Thanks!
--Bob
Is ABBA still working? I got this basic issue when I try to set up.
2020-07-21 08:01:58 - Mongo::OperationFailure - Database command 'update' failed: BSON field 'update.multi' is an unknown field.: /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/db.rb:610:in
rescue in command'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/db.rb:606:incommand' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:314:in
block in send_write_command'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:55:inblock in instrument' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:20:in
instrument'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:54:ininstrument' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:313:in
send_write_command'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/collection.rb:1104:insend_write' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo-1.12.5/lib/mongo/collection.rb:497:in
update'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo_mapper-0.13.1/lib/mongo_mapper/plugins/modifiers.rb:85:inmodifier_update' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo_mapper-0.13.1/lib/mongo_mapper/plugins/modifiers.rb:9:in
increment'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/mongo_mapper-0.13.1/lib/mongo_mapper/plugins/modifiers.rb:117:inincrement' /home/vagrant/abba/app/models/variant.rb:22:in
start!'
/home/vagrant/abba/app.rb:54:inblock in <top (required)>' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:inblock in compile!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
[]'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:inblock (3 levels) in route!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in
route_eval'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:inblock (2 levels) in route!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in
block in process_route'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:incatch' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in
process_route'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:inblock in route!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in
each'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:inroute!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in
block in dispatch!'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:inblock in invoke' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
catch'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:ininvoke' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in
dispatch!'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:inblock in call!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
block in invoke'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:incatch' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
invoke'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:incall!' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-1.6.4/lib/rack/logger.rb:15:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-1.6.4/lib/rack/head.rb:13:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:incall' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in
block in call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:insynchronize' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in
call'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/connection.rb:86:inblock in pre_process' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/connection.rb:84:in
catch'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/connection.rb:84:inpre_process' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/connection.rb:53:in
process'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/connection.rb:39:inreceive_data' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
run_machine'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:inrun' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/backends/base.rb:73:in
start'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/server.rb:162:instart' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/controllers/controller.rb:87:in
start'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/runner.rb:200:inrun_command' /home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/lib/thin/runner.rb:156:in
run!'
/home/vagrant/.rvm/gems/ruby-2.1.5/gems/thin-1.6.4/bin/thin:6:in<top (required)>' /home/vagrant/.rvm/gems/ruby-2.1.5/bin/thin:23:in
load'
/home/vagrant/.rvm/gems/ruby-2.1.5/bin/thin:23:in<main>' /home/vagrant/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in
eval'
/home/vagrant/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in<main>'
I'm trying to switch to ABBA from optimizely, cause i don't mind writing jquery test code on my own. Still, the main problem is that i'd like to define experiments on server side, not in js code snippet.
Current solution lacks flexibility, because i have to change the code, every time i want to change my test. It's an issue when you work with several clients, and don't controll website code.
I could manage this in Google Tag Manager, but A/B test scripts should load ASAP, and in GTM you cannot controll the order of firing scripts, and all A/B test vendors say, that GTM is not the right tool if you want to change page contents.
So the solution would be to add a parameter to javascript request, and concatenate test code, or request it from standard js.
You'd also have to add possibility to create experiments on the backend, but if now they're created on first request, it should be easy to simply make this request on the backend.
do you think it's possible?
I was getting an error while deploying to Heroku
ruby: symbol lookup error: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.0/lib/rubyeventmachine.so: undefined symbol: rb_enable_interrupt
Given the fact that Ruby is not explicitly specified, it was using 2.0.0.
Got it to work by specifying ruby '1.9.3' in Gemfile
Please update to Ruby 2.0.0
Hi Alex,
I really interested on your awesome abba app but unfortunately I dont know RoR or sinatra but I really wanna know how it work. May I know how to run the /test/start.html and /test/complete.html? I've setup everything and the admin is works, but when I access these url in browser
http://localhost:3000/start and http://localhost:3000/complete
I got 406, and broken image
when I remove get '/start', _:provides => 'image/gif'_
I got _experiment required_
Any idea how to test your app in my local?
Thanks
Hi,
I'm trying to extend your chart by a line showing the visitors/converstions over time in order to have a better feeling for the shown values.
The data is already selected but I have very serious problems to get the data into the chart.
I'm new to d3 and maybe I oversee the point where you put the elements into the chart.
Could you please elaborate a little bit how the chart works?
Thanks!!
Full disclosure: I authored the original ABBA library.
This looks like a great library for people who want to run A/B tests themselves, I haven't seen any other package that takes care of the whole stack and makes it this easy.
However, I'm wary of the statistics here, for a few reasons:
The package appears to give one-tailed p-values. This is debatable, but I believe that doesn't line up with how most people interpret their A/B tests and thus leads to an excess of false positives.
The package supports multiple comparisons but includes no correction or warning for it. Granted, very few A/B testing packages address this, but it can also lead to a (potentially serious) excess of false positives.
The package computes a table of four normal tail probabilities itself using a fairly crude method:
a = 50.0
norm_dist = []
(0.0..3.1).step(0.01) { |x| norm_dist << [x, a += 1 / Math.sqrt(2 * Math::PI) * Math::E ** (-x ** 2 / 2)] }
Z_TO_PROBABILITY = [90, 95, 99, 99.9].map { |pct| [norm_dist.find { |x,a| a >= pct }.first, pct] }.reverse
resulting in
z-value computed tail prob actual tail prob
2.75 .001 .0030
2.26 .01 .0119
1.63 .05 .0516
1.27 .1 .1020
For the largest z-value, the computed tail probability is low by a factor of 3, which can be substantial for people trying to run high-confidence tests -- again, it's overestimating confidence which can again lead to excess false positives. It'd be simpler and more accurate to hardcode a lookup table -- tables of these values are readily available. (One could also make use of a statistical library or directly code a numerical approximation to the standard normal CDF.)
The package uses the non-pooled Z-test for two proportions in case where I believe the pooled test is more appropriate (because the null hypothesis is that the control and the variation have the same conversion rate). This tends to overestimate Z-values and thus, again, confidences.
The Z-test can be inaccurate for small samples, though this doesn't actually contribute much error since the package requires that the total number of conversions is at least 25 which keeps us in pretty safe territory. With highly uneven sampling weights we could get into small-sample trouble, but it's definitely less of an issue.
As an example of the potential for numerical accuracy issues, consider an experiment with 6/500 conversions in baseline and 20/500 conversions in one variation. The package reports 99.9% confidence, or a one-tailed p-value <= 0.001. Fisher's Exact Test gives a one-tailed p-value of 0.0042. The original ABBA gives a two-tailed p-value of 0.0063, corresponding to a one-tailed p-value of roughly 0.0033. So we're underestimating the one-tailed p-value by a factor of 3-4 and the two-tailed p-value (which is probably more appropriate) by a factor of 6-8. This is pretty substantial -- our long-run false-positive rate will be 6-8x higher than we expect (ignoring multiple testing issues).
I think this would be a really awesome contribution to the world of A/B testing with some more robust statistics. I'd suggest using the (original) ABBA JS library (or perhaps a port of the Python version to Ruby), which also gets you some nice confidence intervals on proportions and improvements. Together the two would make a pretty sweet solution to do-it-yourself A/B testing.
As I followed the documentation and the examples in the /public/tests/start.html
and /public/tests/complete.html
I could not find anything to set the conversion for a variant to indicate that a user (which faced a variant) has done targetted action or not.
any help would be appreciated.
In app/assets/stylesheets/experiments.css.styl there is some "\xE2"
and it makes me this error:
InvalidByteSequenceError: "\xE2" on US-ASCII
/var/lib/gems/1.9.1/gems/execjs-1.4.0/lib/execjs/external_runtime.rb:173:in `read'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.