flashrepo / flash-singleconfig Goto Github PK

In our previous paper, we used a rank based progressive sampling technique to optimize for software systems. However progressive sampling uses a validation set to perform iterative sampling and that adds additional cost to the sampling process. Caution: none of the previous papers including our paper does not talk about the validation set.

This new technique is does not use any validation technique and hence requires far fewer evaluations hence lowering the cost of optimization by a factor of [2.28-46.45].

This technique is inspired by both active learning and rank based sampling. The process starts by

1. evaluating an initial set of samples (30 in our experiments).
2. A model (CART) is build using these configurations. This model is used to predict performance scores for the rest of the samples (|dataset|-30).
3. The best configuration (based on predicted performance score) is chosen, evaluated and added to the training set. Now, the training set has 31 samples where has the testing set has (|dataset|-31) samples.
4. Continue till the stopping criterion is reached. In our case, the stopping criterion is: if the newly added point is not better (lower) than the best score of the training set then lose life. The number of lives used in the experiment is 10.

• When would this method not work? Would adding random points improve the performance?
• Keep the Rank Difference - Comparable?
• Show that the rank difference is actually affected by the size of validation set
• Stat test

How to read

y-axis: Rank Difference
x-axis: Software System (randomly assigned)

Lower the better

How to read

y-axis: Number of Evaluations (log scale)
x-axis: Software System (randomly assigned)
Lower the better