[QUESTION] MAB with Thompson Sampling not converging about ax

[seabull]

Here is an example output when I ran the above code using _SEM_DEFAULT=0.3 and _NUMBER_OF_TRIALS=5. I wonder why it doesn't converge to arm 0_4? I tried to increase the number of trial and it doesn't converge.

_data=[array([0.25345768, 0.06306603, 0.12302662, 0.49030093, 0.99286429]), array([0.25071666, 0.06221013, 0.12561016, 0.50398922, 1.01736567]), array([0.24777093, 0.06145106, 0.1239555 , 0.49263727, 1.0022855 ]), array([0.24869022, 0.06223818, 0.12602859, 0.49545971, 1.00161501]), array([0.2458025 , 0.06365979, 0.12473368, 0.49971953, 0.99040744]), array([0.24614296, 0.06176861, 0.12681383, 0.50680577, 0.99104223]), array([0.25083241, 0.06126039, 0.12636907, 0.48952094, 0.99879117])]
Running iteration 1...
Trial status = TrialStatus.CANDIDATE
Trial metadata: {'name': '0', 'weights': OrderedDict([(Arm(name='0_0', parameters={'arm_name': 'treatment_id0'}), 0.2), (Arm(name='0_1', parameters={'arm_name': 'treatment_id1'}), 0.25), (Arm(name='0_2', parameters={'arm_name': 'treatment_id2'}), 0.15), (Arm(name='0_3', parameters={'arm_name': 'treatment_id3'}), 0.05), (Arm(name='0_4', parameters={'arm_name': 'treatment_id4'}), 0.35)]), 'arms_by_name': ['0_0', '0_1', '0_2', '0_3', '0_4'], 'deploy_name': None, 'abandoned_arms': []}
***** arm_name='0_0' i_arm=0 data[i_arm]=0.25083240649895866
***** arm_name='0_1' i_arm=1 data[i_arm]=0.06126038517367513
***** arm_name='0_2' i_arm=2 data[i_arm]=0.12636907064177577
***** arm_name='0_3' i_arm=3 data[i_arm]=0.48952094033977517
***** arm_name='0_4' i_arm=4 data[i_arm]=0.998791170255513
Running iteration 2...
Trial status = TrialStatus.CANDIDATE
Trial metadata: {'name': '1', 'weights': OrderedDict([(Arm(name='0_4', parameters={'arm_name': 'treatment_id4'}), 0.8477000000000001), (Arm(name='0_3', parameters={'arm_name': 'treatment_id3'}), 0.1081), (Arm(name='0_0', parameters={'arm_name': 'treatment_id0'}), 0.027000000000000003), (Arm(name='0_2', parameters={'arm_name': 'treatment_id2'}), 0.0106), (Arm(name='0_1', parameters={'arm_name': 'treatment_id1'}), 0.006600000000000001)]), 'arms_by_name': ['0_0', '0_1', '0_2', '0_3', '0_4'], 'deploy_name': None, 'abandoned_arms': []}
***** arm_name='0_0' i_arm=0 data[i_arm]=0.24614295933957336
***** arm_name='0_1' i_arm=1 data[i_arm]=0.06176860919832933
***** arm_name='0_2' i_arm=2 data[i_arm]=0.1268138286664056
***** arm_name='0_3' i_arm=3 data[i_arm]=0.5068057719258409
***** arm_name='0_4' i_arm=4 data[i_arm]=0.9910422269798573
Running iteration 3...
Trial status = TrialStatus.CANDIDATE
Trial metadata: {'name': '2', 'weights': OrderedDict([(Arm(name='0_4', parameters={'arm_name': 'treatment_id4'}), 0.8433), (Arm(name='0_3', parameters={'arm_name': 'treatment_id3'}), 0.11170000000000001), (Arm(name='0_0', parameters={'arm_name': 'treatment_id0'}), 0.025800000000000003), (Arm(name='0_2', parameters={'arm_name': 'treatment_id2'}), 0.0109), (Arm(name='0_1', parameters={'arm_name': 'treatment_id1'}), 0.008300000000000002)]), 'arms_by_name': ['0_0', '0_1', '0_2', '0_3', '0_4'], 'deploy_name': None, 'abandoned_arms': []}
***** arm_name='0_0' i_arm=0 data[i_arm]=0.24580249651917732
***** arm_name='0_1' i_arm=1 data[i_arm]=0.06365978691900227
***** arm_name='0_2' i_arm=2 data[i_arm]=0.12473368410256476
***** arm_name='0_3' i_arm=3 data[i_arm]=0.4997195250309848
***** arm_name='0_4' i_arm=4 data[i_arm]=0.9904074372295137
Running iteration 4...
Trial status = TrialStatus.CANDIDATE
Trial metadata: {'name': '3', 'weights': OrderedDict([(Arm(name='0_4', parameters={'arm_name': 'treatment_id4'}), 0.8441999999999998), (Arm(name='0_3', parameters={'arm_name': 'treatment_id3'}), 0.11109999999999998), (Arm(name='0_0', parameters={'arm_name': 'treatment_id0'}), 0.025899999999999996), (Arm(name='0_2', parameters={'arm_name': 'treatment_id2'}), 0.010599999999999997), (Arm(name='0_1', parameters={'arm_name': 'treatment_id1'}), 0.008199999999999999)]), 'arms_by_name': ['0_0', '0_1', '0_2', '0_3', '0_4'], 'deploy_name': None, 'abandoned_arms': []}
***** arm_name='0_0' i_arm=0 data[i_arm]=0.24869022287212142
***** arm_name='0_1' i_arm=1 data[i_arm]=0.062238180699438886
***** arm_name='0_2' i_arm=2 data[i_arm]=0.12602858804337366
***** arm_name='0_3' i_arm=3 data[i_arm]=0.4954597070422151
***** arm_name='0_4' i_arm=4 data[i_arm]=1.0016150080832735
Running iteration 5...
Trial status = TrialStatus.CANDIDATE
Trial metadata: {'name': '4', 'weights': OrderedDict([(Arm(name='0_4', parameters={'arm_name': 'treatment_id4'}), 0.8509000000000001), (Arm(name='0_3', parameters={'arm_name': 'treatment_id3'}), 0.1027), (Arm(name='0_0', parameters={'arm_name': 'treatment_id0'}), 0.024900000000000002), (Arm(name='0_2', parameters={'arm_name': 'treatment_id2'}), 0.0136), (Arm(name='0_1', parameters={'arm_name': 'treatment_id1'}), 0.007900000000000003)]), 'arms_by_name': ['0_0', '0_1', '0_2', '0_3', '0_4'], 'deploy_name': None, 'abandoned_arms': []}
***** arm_name='0_0' i_arm=0 data[i_arm]=0.24777092672773257
***** arm_name='0_1' i_arm=1 data[i_arm]=0.06145105606923228
***** arm_name='0_2' i_arm=2 data[i_arm]=0.12395549743144098
***** arm_name='0_3' i_arm=3 data[i_arm]=0.49263726642852995
***** arm_name='0_4' i_arm=4 data[i_arm]=1.0022854962492587

[esantorella]

Hi, sorry for the slow follow-up. It's possible that the model is still very uncertain about which arm is the best. It would help to produce the first and last plots in this tutorial to get a sense of uncertainty levels.

Do you know the standard error of the data in this example? If so, you can pass that to the model as well; that should lead to better model fits, although it wouldn't necessarily lead to faster convergence.