Neural Network giving NaN values about onedal

[Vika-F]

Thank you for your interest to our library.
Please give us several days to run the analysis on this example.
It would help us a lot if you share the exact modifications that you do in NeuralNetConfiguratorDistr.java that lead to NaN values in the results.

[mayank-kgp]

@Vika-F I am trying neural network on MNIST data. MNIST data contains 10 labels. So I am changing number of neurons in last layer to 10.
Link for exact line number -
https://github.com/01org/daal/blob/daal_2018_beta_update1/examples/java/com/intel/daal/examples/neural_networks/NeuralNetConfiguratorDistr.java#L60

[Vika-F]

Thank you for the clarification.
We will analyze the issue and come back to you with the results.

[Vika-F]

Per our analysis, there is a bug in the distributed training of a neural network in the present version of Intel DAAL: weights and biases are not properly initialized in the beginning of the computations.
We plan to fix it in the future versions of the library.

The workaround for this bug is below: add the piece of the code

if (i == 0) {
    /* Retrieve training model of the neural network on master node */
    TrainingModel trainingModelOnMaster = net.getResult().get(TrainingResultId.model);
    /* Retrieve training model of the neural network on local node */
    TrainingModel trainingModelOnLocal  = netLocal[0].input.get(DistributedStep1LocalInputId.inputModel);

    /* Set weights and biases on master node using the weights and biases from local node */
    trainingModelOnMaster.setWeightsAndBiases(trainingModelOnLocal.getWeightsAndBiases());

    /* Set initialization flag parameter as true in all forward layers of the training model on master node */
    ForwardLayers forwardLayers = trainingModelOnMaster.getForwardLayers();
    for (int j = 0; j < forwardLayers.size(); j++) {
        forwardLayers.get(j).getLayerParameter().setWeightsAndBiasesInitializationFlag(true);
    }
}

into the example NeuralNetDenseDistr.java#L156

[mayank-kgp]

Thanks @Vika-F .
It is working now.