Effect of `evaluate!(mach, ...)` is unpredictable. Retrain on all data and re-evaluate? about mlj.jl

[CameronBieganek]

If we set aside learning networks, it seems like we don't really need evaluate!. The non-mutating evaluate is sufficient and more intuitive.

Is there a way to write a non-mutating evaluate that works on learning networks? Or is there already a way to use evaluate on learning networks that is not immediately obvious to me? I'm rusty on MLJ, since I unfortunately have to use Python at work. :(

[ablaom]

@CameronBieganek @tylerjthomas9

[tylerjthomas9]

I think that this is a fantastic change. As you said, the current mach returned by evaluate! is not useful. It may add some complications that evaluate and evaluate have different default values for retrain, but I think that it makes logical sense to not retrain by default with evaluate(model, data...).

[OkonSamuel]

If we set aside learning networks, it seems like we don't really need evaluate!. The non-mutating evaluate is sufficient and more intuitive.

Is there a way to write a non-mutating evaluate that works on learning networks? Or is there already a way to use evaluate on learning networks that is not immediately obvious to me? I'm rusty on MLJ, since I unfortunately have to use Python at work. :(

I agree with @CameronBieganek that it would have been best if evaluate just returned a performance evaluation object and does not mutate the machine.
We could still have the evaluate! method as an internal method for convenience.
For learning networks now that the data isn't attached to the network anymore, we can make it such that calling evaluate on a machine passes a deepcopy of the machine to the internal evaluate! method.
I don't know how this will affect models that wraps external c libraries.

[ablaom]

This is great feedback, thanks.

I'm torn between dumping mutation altogether (advantage: simplicity) and proceeding as I propose above (convenience). Some context: In a summer project we are working on auto logging of workflows using MLFlow, and in this context it seemed natural to log a training score, and a serialised set of learned parameters, each time a model is "evaluated". This saves having to specify metrics a second time if wanting the training score.

Is there a way to write a non-mutating evaluate that works on learning networks? Or is there already a way to use evaluate on learning networks that is not immediately obvious to me?

@CameronBieganek Not sure what you mean here. Perhaps you mean calling evaluate! on "learning network machines" (a special kind of machine), now deprecated? Generally the idea with learning networks is that they should be "exported" as stand-alone composite models in serious use. There was a simplification to this exporting process you may have missed. Learning network machines, once an intermediate step in the export process, have been deprecated as seen as unnecessary complication.

[ablaom]

Okay, I now remember the reason for the existing behaviour. The use case is evaluating models that support "warm restart". If using Holdout() (or other single test/train pair of row indices) and you re-evaluate the machine after changing a model hyperparameter then, for some hyperparameters like n_iterations, the re-evaluation will not require retraining the machine from scratch.

On another matter, perhaps a better way to get training scores, is to add a resampling strategy InSample() which has the single train/test pair of row indicestrain=rows and test=rows.