[Feature Request]: Visualization of optimization progress and function evaluation values about ax

[lena-kashtelyan]

Hello again, @quanvuong!

1. You can use render(ax_client.get_optimization_trace()) to see the objective values so far (API reference). We will add parameter values on hover to this plot, to make it easier to understand what the most promising parameters so far are. The resulting plot currently will look something like this:
   .

2. Length scale / sensitivity analysis –– in the works, thank you for bringing it up!

3. Extracting visualization plot –– we will look into a more easily serializable way of providing the plots, thanks for the feedback!

4. Will add parameter values on hover to plot shown in 1), would that, in combination with the contour plot, be sufficient for the purpose of detecting whether the search space needs adjustment?

[lena-kashtelyan]

Hello, @quanvuong! Just for context, which one of our APIs are you using and would want to see this functionality implemented in?

[quanvuong]

Hi!

I'm using the Service API. The Service API allows me to manually logs the hyper-parameter values and their corresponding function evaluation values. But it would be nice if this feature was already implemented.

[lena-kashtelyan]

@quanvuong, just so that I understand, does the following code not achieve this purpose? What should we add?

from ax.service.ax_client import AxClient
from ax.utils.measurement.synthetic_functions import branin
from ax.utils.notebook.plotting import render

ax = AxClient()

ax.create_experiment(
    parameters=[
        {"name": "x", "type": "range", "bounds": [-5.0, 10.0]},
        {"name": "y", "type": "range", "bounds": [0.0, 15.0]}
    ]
)

for _ in range(6):
    params, idx = ax.get_next_trial()
    ax.complete_trial(idx, branin(*(params.values())))

ax.get_best_parameters()

render(ax.get_contour_plot())

Just for ease of reference, this is what it looks like executed in a notebook:

[quanvuong]

Thanks!

For example:

1. The contour plot does not show the function evaluation values, the corresponding hyper-parameter in a ranked manner. This would be useful during optimization to get a sense of what is the most promising hyper-parameter values so far.

2. Length Scale or some sort of sensitivity analysis. Before running BayesOpt, I might not know what the important hyper-parameters are and what hyper-parameters the performance is relatively insensitive too. It would be super useful to show how sensitive the performance is with respect to each hyper-parameter. This would inform subsequent experiments and understanding.

3. I might run BayesOpt inside a docker image or on a server. So it's not easy to extract out visualization plot. In this case, textual logs would be useful as well.

[quanvuong]

Another feature that would be helpful is:

4. Viewing the hyper-parameter values of the best function evaluations so far. This helps to see if the optimized hyper-parameter values hit their respective bounds, meaning that the bounds for some of the hyper-parameters are too tight. This usually means that better hyper-parameter values can be found by loosening the bounds the hyper-parameter values are searched over.

[quanvuong]

yes, that would be great. Thanks!

[lena-kashtelyan]

Parameter values on-hover are now part of the optimization trace plot for the Service API and more textual logs are now enabled in the latest stable version, 0.1.6. Examples in the tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html.

Sensitivity analysis plot and extraction of plots in serializable way are still in the works.

[2timesjay]

We have a pending change to the service API:

render(ax_client.get_feature_importances())

Will show a ranked bar chart of feature importances for different metrics.

[2timesjay]

Also, @quanvuong, while we consider a slightly more elegant option you can serialize and visualize plots as follows:

[lena-kashtelyan]

I believe this should address all the issues brought up in this task: textual logs and parameter values on hover (included in the latest stable release), sensitivity analysis plots (included on master), and a workaround for serializing the plots. Closing the issue, but feel free to follow up, @quanvuong!