The ml-19-20 from andreacini

ml-19-20's People

Contributors

Stargazers

Watchers

ml-19-20's Issues

lab 4 model performance: select best model

@andreacini I found a possible issue in lab 4 in the last snippets (18)

In the grid search based on the hold out validation, the accuracies are saved in array acc_list.

To get the index of the best accuracy you have on line 16:
imax = np.argmax(acc)
should be:
imax = np.argmax(acc_list)
Otherwise, imax will always return either 0 or 1.

The accuracy should also be saved as a single element:
acc = model.evaluate(X_val, y_val)
should be:
_, acc = model.evaluate(X_val, y_val)
Method evaluate returns a tuple and the method argmax from numpy checks only the first elements of the tuple, the loss and not the accuracy.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=True)

X_atr, X_val, y_atr, y_val = train_test_split(X_train, y_train, test_size=0.3, shuffle=True)

# Model selection
acc_list = []
for (neurons, activation) in model_parameters:
    print("Training NN with {} neurons and {} epochs".format(neurons, epochs))

    model = create_ffnn(neurons=neurons, activation=activation)
    model.fit(X_atr, y_atr, epochs=epochs, verbose=0)

    # acc = model.evaluate(X_val, y_val) # split in -, acc
    _, acc = model.evaluate(X_val, y_val)
    acc_list.append(acc)

#imax = np.argmax(acc) # returns always 0 or 1
imax = np.argmax(acc_list)  
print("Best model parameters:", model_parameters[imax])

# Performance of best model
(neurons, activation) = model_parameters[imax]
best_model = create_ffnn(neurons=neurons, activation=activation)
best_model.fit(X_train, y_train, epochs=epochs, verbose=0)
_, final_acc = best_model.evaluate(X_test, y_test)
print("Best model accuracy:", final_acc)

# Final trained model
final_model = create_ffnn(neurons=neurons, activation=activation)
final_model.fit(X, y, epochs=epochs, verbose=0)

Recommend Projects