Poor results of Learning to Rank model about reconfigsrc

[Gu-Youngfeng]

According to the follow-up experimental results, I found that the method classification performed well among comparative methods. Here is a results on 18 datasets using 4 comparative methods.

• M1: ReConfig
• M2: Rank-based
• M3: Classification
• M4: Random Deletion

TABLE I. RDTie on 18 datasets using four comparative methos

Datasets	M1 M2 M3 M4	M1 M2 M3 M4	M1 M2 M3 M4	M1 M2 M3 M4
	Top-1	Top-3	Top-5	Top-10
Apache	8.0 8.9 8.5 9.6	4.0 3.8 3.3 4.2	2.7 2.1 1.6 2.3	1.5 0.8 0.6 1.2
BDBJ	11.3 10.9 10.7 10.5	6.3 5.8 5.8 5.6	4.1 3.6 3.8 3.5	2.1 1.4 1.7 1.5
clasp	27.7 31.1 20.9 29.5	10.9 12.5 9.1 10.2	7.7 7.8 5.1 7.1	6.4 4.3 3.9 5.6
lrzip	24.1 24.4 18.6 25.0	10.0 10.4 7.8 9.7	6.1 6.0 4.7 6.7	4.1 2.9 2.8 5.0
noc-obj1	11.4 12.2 9.9 14.2	5.2 4.5 3.5 4.9	3.5 2.4 1.9 2.9	2.8 1.0 1.2 1.9
noc-obj2	10.6 10.3 6.7 9.3	4.2 4.5 3.0 4.4	3.2 2.8 2.2 3.4	2.3 1.4 1.3 2.8
snw-obj1	7.4 7.5 6.6 7.3	3.3 3.2 2.8 3.2	2.5 2.3 2.0 2.3	1.2 0.9 0.8 0.9
snw-obj2	6.9 6.8 6.9 6.6	2.2 2.2 2.2 2.1	1.0 1.0 1.0 1.0	0.4 0.5 0.3 0.4
sqlite	183.8 202.5 217.7 196.3	95.4 100.4 97.8 103.1	67.3 66.8 65.2 71.1	40.3 36.3 42.1 41.6
wc+rs-3d-c4-obj1	16.7 16.7 15.7 16.1	6.3 6.4 5.8 5.8	4.4 4.3 4.0 4.0	2.2 1.8 1.7 1.8
wc+rs-3d-c4-obj2	4.7 5.0 4.4 4.7	2.0 2.2 1.9 2.1	1.4 1.4 1.3 1.5	0.7 0.5 0.6 0.8
wc+sol-3d-c4-obj1	18.8 18.6 18.3 19.0	7.4 7.9 7.3 7.9	5.2 5.3 5.1 5.4	2.9 2.6 2.7 3.2
wc+sol-3d-c4-obj2	4.7 4.7 4.7 4.6	1.9 2.0 1.8 1.9	1.2 1.2 1.0 1.2	0.7 0.5 0.5 0.6
wc+wc-3d-c4-obj1	14.8 15.5 13.7 15.3	3.7 4.2 4.5 4.7	2.0 2.1 2.7 3.0	1.1 0.7 1.5 1.6
wc+wc-3d-c4-obj2	6.0 6.1 5.8 6.4	2.6 2.5 2.5 2.8	1.7 1.5 1.7 1.9	1.1 0.5 0.9 1.3
wc-3d-c4-obj1	37.3 38.8 28.0 37.0	11.7 15.4 9.5 13.3	5.7 9.4 4.9 7.8	3.2 4.8 2.9 4.8
wc-3d-c4-obj2	23.3 24.4 17.1 25.8	10.2 12.4 6.9 12.5	7.2 8.6 4.1 8.7	5.1 5.4 2.1 5.7
WGet	27.1 25.9 27.0 25.1	11.9 11.2 11.9 11.8	7.1 6.7 7.1 7.3	3.6 3.5 3.3 3.5

Throught the Table 1, we can find that M3 can achieve the lowest measurement (i.e., RDTie) in the most of datasets under the condition of Top-1, Top-3, Top-5, Top-10. Can we draw a conclusion that Classification is the best choose to solve the tie issue problem in configurable systems?