scisharp / sianet

An easy to use C# deep learning library with CUDA/OpenCL support

Trello is used to track SiaNet devlopment activities. You are welcome to watch any task and track progress. Suggestion will be put on the wishlist and then will be planned out for development

https://trello.com/b/bLbgQLgy/sianet-development

A C# deep learning library

Developing a C# wrapper to help developer easily create and train deep neural network models.

• Easy to use library, just focus on research
• Multiple backend - CNTK, TensorFlow, MxNet, PyTorch, ArrayFire
• CUDA/ OpenCL support for some of the backends
• Light weight libray, built with .NET standard 2.0
• Code well structured, easy to extend if you would like to extend with new layer, loss, metrics, optimizers, constraints, regularizer

A Basic example

The below is a classification example with Titanic dataset. Able to reach 75% accuracy within 10 epoch.

//Setup Engine. If using TensorSharp then pass SiaNet.Backend.TensorSharp.SiaNetBackend.Instance. 
//Once other backend is ready you will be able to use CNTK, TensorFlow and MxNet as well.
Global.UseEngine(SiaNet.Backend.ArrayFire.SiaNetBackend.Instance, DeviceType.CPU);

var dataset = LoadTrain(); //Load train data
var test = LoadTest(); //Load test data

var (train, val) = dataset.Split(0.25);

//Build model
var model = new Sequential();
model.EpochEnd += Model_EpochEnd;
model.Add(new Dense(128, ActivationType.ReLU));
model.Add(new Dense(64, ActivationType.ReLU));
model.Add(new Dense(1, ActivationType.Sigmoid));

//Compile with Optimizer, Loss and Metric
model.Compile(OptimizerType.Adam, LossType.BinaryCrossEntropy, MetricType.BinaryAccurary);

// Train for 100 epoch with batch size of 32
model.Train(train, 100, 32, val);

var predictions = model.Predict(test);
predictions.Print();

Training Result

Complete Code: https://github.com/SciSharp/SiaNet/blob/master/Examples/BasicClassificationWithTitanicDataset/Program.cs

More examples: https://github.com/SciSharp/SiaNet/blob/master/Examples

API Docs

https://scisharp.github.io/SiaNet/

Contribution

Any help is welcome!!!

sianet's Issues

Model Load not working

Model.load is not loading layers etc

Saving seems OK and this works :

Dim modelcntk As CNTK.Function = CNTK.Function.Load(modelfile, CNTK.DeviceDescriptor.GPUDevice(0))

But cant load in Sequential Model as it has to be a Sequential and above code loads a function

Training Stop criteria as the sequential model is stopping only after finishing the given epochs.

Ability to stop the training using certain conditions is met. For eg. when the accuracy reached 95% stop the training and save the model.

Model.evaluate error on softmax

Model.evaluate does not return full softmax array to get the index of the predicted class. It only returns value.
eg. in a multiclass prediction model.evaluate, expected is : (0.10, 0,05, 0,85) so the winner is 3d value but it returns only the single value of 0.85

Is this project rewritten with cuda instead of cntk? Is there a demo now?

Is this project rewritten with cuda instead of cntk? Is there a demo now?

CrossEntropy issue on LSTMs

I think LSTM cannot work on Classification problems on current version but only on Regression problems. Tried to convert sample code to do Multi Classification but got rising error "Operation 'TransposeTimes" at

private static Function CrossEntropy(Variable labels, Variable predictions)
{
return CNTKLib.CrossEntropyWithSoftmax(predictions, labels);
}

Examples for creating datasets?

Hello, awesome project, I was checking the examples, I don't know how to construct datasets, could make a feed forward neural network to solve the XOR problem and showing how to construct dataset? That'd would be very helpful though. Thanks.

Implement Evaluate method

Would you like to implement the evaluate method?

is it possible to make CRNN using SiaNet ?

Hi,
SiaNet is great ! thanks for your work.
is it possible to make CRNN currently ?
thanks!

GPU Access?

What's the key to enabling GPU access? I've cloned your repository and run the examples and CPU works, GPU does not. I have a NVidia GPU with Cuda 8 drivers and other toolkits like ConvNetSharp use it.

Any tips?

Thanks,
Darren

Missing predict function

Hi,

Unless I am mistaking, I don't see any predict function, or any other way to actually use our trained models.

We can only train and evaluate. Am I missing something ?

Add recurrent_dropout and recurrent_regularizer to LSTM Layer

Hello,

Would be great to consider adding ecurrent_dropout and recurrent_regularizer parameters to LSTM Layer
See: https://keras.io/layers/recurrent/#lstm

Multithread predictions

Hello i am having memory access violation when i attempt to multi thread image predictions. this is happening at line

// Start evaluation on the device
modelFunc.Evaluate(inputDataMap, outputDataMap, GlobalParameters.Device);

within ImageNet.cs is there a way to multithread the functionality ?

Thanks

min max must be reversed in Rmsprop Optimizer

Shape is missing in example

Throws exception that Dense is missing Shape in your example :)

Can you add a DEMO of LSTM for time series ,Please

SiaNet is great ! Thanks for your work.
Can you add a DEMO of LSTM for time series .
thanks!

SiaNet.Examples.CPUOnly Error in new version

Add TensorFlow.NET as a backend?

TensorFlow.NET is a full binding for Google's TensorFlow.
It can build graph, train in GPU and do inference.

GPU memory leak on big validation data-set

Training with about 1million records (each having 30 values) memory usage on GPU is quite low with my model.
However, as soon as I add 5,000 validation record to the mix, after first epoch, it skyrockets to 3.6GB. Any higher and I can't run it on my GPU.

I double checked the code and it seems that the library indeed does run the validation on batches with the same size as the training batches (in my case 64). However, the memory leak is still there. Can somebody check the reason behind it?

Model is pretty simple:

            _model = new Sequential();
            _model.Add(new Reshape(Shape.Create(1, _train.XFrame.Shape[1]), Shape.Create(_train.XFrame.Shape[1])));
            _model.Add(new Dropout(0.5));
            _model.Add(new LSTM(150, Shape.Create(1, _train.XFrame.Shape[1])));
            _model.Add(new Dense(_train.YFrame.Shape[1]));

Where's the code??

I'm deeply confused as to what's the extent of this repository.
I have been trying to test the library but something is wrong: The examples projects are left unfinished and nowhere close to what the wiki advertises.
Trying to actually get a prediction from the code, the Xor example does not seem to work (predicts NAN on the samples).
Then trying to test Housing Prediction, where are SplitXY and SplitTrainTest advertised in the wiki example?
Then trying to implement something similar, Dataframe.Reshape and CSVFrame[params string[] columns] don't seem to do anything interesting.
Where does the rabbit hole stop?
Is there anything functional at all?

Example of image classification on custom image dataset

Xframe.Shuffle accidentally erases data

Example not reducing loss

Thanks a lot for making a start on this great library. It's much appreciated!
I have high hopes for CNTK on .NET so great work on pioneering a Keras style library!!

I've just tried to run the example. It runs fine, but it's not actually learning. (Based on last commit Wednesday, 8 November 2017 8:45:12 AM):

Epoch: 1, Loss: 2.30605627276727, Accuracy: 0.890608324439701
Epoch: 2, Loss: 2.30215639003322, Accuracy: 0.892077323717949
Epoch: 3, Loss: 2.30211564978006, Accuracy: 0.890541622198506
Epoch: 4, Loss: 2.30210599838159, Accuracy: 0.889840411324786
Epoch: 5, Loss: 2.30213043173197, Accuracy: 0.890474919957311
Epoch: 6, Loss: 2.30217392449705, Accuracy: 0.890992254273504
Epoch: 7, Loss: 2.30216704488691, Accuracy: 0.890641675560299
Epoch: 8, Loss: 2.30213429097436, Accuracy: 0.889823717948718
Epoch: 9, Loss: 2.30213692781129, Accuracy: 0.889891275346852
Epoch: 10, Loss: 2.30221930833963, Accuracy: 0.890591613247863

Am I missing something?

Feature Request: Add Mish activation

Mish is a novel activation function proposed in this paper.
It has shown promising results so far and has been adopted in several packages including:

TensorFlow-Addons	SpaCy (Tok2Vec Layer)	Thinc - SpaCy's official NLP based ML library
Eclipse's deeplearning4j	Hasktorch	Echo AI
CNTKX - Extension of Microsoft's CNTK	FastAI-Dev	Darknet
Yolov3	BeeDNN - Library in C++	Gen-EfficientNet-PyTorch
dnet	ruby-dnn	blackcat-tensors
DL4S	HuggingFace Transformers	PAGI
OpenCV	Odin-AI	Mini DNN
Efficient Segmentation Networks	TF Semantic Segmentation	Dynastes
DLib	Copernicus	AllenNLP
PyWick

All benchmarks, analysis and links to official package implementations can be found in this repository

Mish also was recently used for a submission on the Stanford DAWN Cifar-10 Training Time Benchmark where it obtained 94% accuracy in just 10.7 seconds which is the current best score on 4 GPU and second fastest overall. Additionally, Mish has shown to improve convergence rate by requiring less epochs. Reference -

Mish also has shown consistent improved ImageNet scores and is more robust. Reference -

Additional ImageNet benchmarks along with Network architectures and weights are avilable on my repository.

Summary of Vision related results:

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10:

How can i insert an array List as input of my neural network?

I have multiples inputs , but most of them are List array and i dont know how can i configure my neural network to accept list array as an input.
Here i have a example XOR(but the inputs are just floats and i cant change to List array):

`
using SiaNet;
using SiaNet.Data;
using SiaNet.Engine;
using SiaNet.Events;
using SiaNet.Layers;
using System;

namespace Sia_XOR
{
class Program
{
static void Main(string[] args)
{
//1º- Setup Engine
Global.UseEngine(SiaNet.Backend.ArrayFire.SiaNetBackend.Instance, DeviceType.CPU);

        //2º -Prep Data
        var (x, y) = PrepDataset();
        DataFrameIter trainSet = new DataFrameIter(x, y);

        //Build model with simple fully connected layers
        var model = new Sequential();
        model.EpochEnd += Model_EpochEnd;
        model.Add(new Dense(64, ActType.ReLU));
        model.Add(new Dense(1, ActType.Sigmoid));

        //Compile with Optimizer, Loss and Metric
        model.Compile(OptimizerType.SGD, LossType.MeanSquaredError, MetricType.BinaryAccurary);
        // Train for 1000 epoch with batch size of 2
        model.Train(trainSet, epochs: 300, batchSize: 2);

        //Create prediction data to evaluate
        DataFrame2D predX = new DataFrame2D(2);
        predX.Load(0, 0, 0, 1); //Result should be 0 and 1
        var rawPred = model.Predict(predX);
        Global.CurrentBackend.Round(rawPred).Print();
        Console.ReadLine();
    }

    #region DataSet

    private static (DataFrame2D, DataFrame2D) PrepDataset()
    {
        // We will prepare XOR gate dataset which will be treated as classification problem.
        // More about this: https://medium.com/@jayeshbahire/the-xor-problem-in-neural-networks-50006411840b
        DataFrame2D x = new DataFrame2D(2);
        x.Load(new float[] { 0, 0, 0, 1, 1, 0, 1, 1 });

        DataFrame2D y = new DataFrame2D(1);
        y.Load(new float[] { 0, 1, 1, 0 });
        return (x, y);
    }

    #endregion

    private static void Model_EpochEnd(object sender, EpochEndEventArgs e)
    {
        Console.WriteLine("Epoch: {0}, Loss: {1}, Metric: {2}", e.Epoch, e.Loss, e.Metric);
    }
}

}`

Shuffle addition on training

It would be nice to have a shuffle option on each epoch while Model.Train

add SELU Activation function

Error building clean pull (XYFrame)

Hey! Trying to build from code. Is this not the intended way of getting dev-version? Solution is missing the XYFrame class, which is defined in SiaNet/Models (https://github.com/deepakkumar1984/SiaNet/blob/master/SiaNet/Model/TrainTestFrame.cs), but the file is not included in the project. Same goes for DataFrame class. After including the signatures of those classes don't seem to match the usage (wrong constructor, cast errors etc.)
Am I missing something out here, or is the current state just not compiling?

Error using Accuracy metric on the timeseries example

I've tried using the Accuracy metric on the TimeSeriesPrediction example and I get an error caused by the fact that Predictions variable has 1 dynamic axis while the Labels variable has 2 dynamic axes.

System.ApplicationException: Operation 'TransposeTimes': Operand 'Output('Hardmax571_Output_0', [1], [#])' has dynamic axes, that do not match the dynamic axes '[*, #]' of the other operands.

Is this a bug or is this intended behavior?

ReShape Error

System.ArgumentOutOfRangeException:“Values for 1 required arguments 'Input('Input79', [100 x 66], [, #])', that the requested output(s) 'Input('Input79', [100 x 66], [, #])' depend on, have not been provided.

The azure blob storage is down

The azure blob storage is down so the downloader doesn't work.

Seed Value Addition

It would be nice to add the Seed value to any of Initialization that use them so as to be able to reproduce the training results

Nice work 👍

Server for downloading example datasets could not be resolve.

Hi,

When SiaNet downloads the example datasets, it always show error message "System.Net.WebException: The remote name could not be resolve: 'sianet.blob.core.windows.net'" (Line 182 in Downloader.cs file).

Please let me know where I can find all example datasets.

Thank you.

InvalidCastException

1; class Sequential : Evaluate(Value data) ,
2 class DataFrameTrainPredict: Evaluate(object testData)
Sequential .Evaluate invokes DataFrameTrainPredict.Evaluate,and in DataFrameTrainPredict.Evaluate there are "DataFrame data = (DataFrame)testData;" , so it is a InvalidCastException when Value is Cast into DataFrame.

Regulizer - add gradientClippingThresholdPerSample and gradientClippingWithTruncation

It would be nice to add gradientClippingThresholdPerSample and gradientClippingWithTruncation at your class Regulizers. You do have the L1, L2 but you could add those because on a lot of examples in CNTK they are using them

OneHotEncode Example

I found OneHotEncode method, can you give an example of using OneHotEncode?

Issues training with large batch -

From @Sam7:
I was using just started moving an existing implementation that was using AleaTK to use your SiaNet.
It seems to be running but the prediction result is always around 0.
I used to have batch size of 10000 for the AleaTK implementation which worked well, since the model is very small and I have stacks of training data.
But with SiaNet it's always 0 except when I go to a VERY small batch size of 32.
Any idea why that could be? (I'm using the XYFrame method not the Image one)

Image Classification using CNN

Hi,
Thank you for the great work you did. I have the following simple CNN for image classification:
`
featureShape = new Shape(new int[] { imageWidth, imageHeight, 3 });
var inputShape = new Shape(new int[] { imageWidth, imageHeight, 3 });
frameX = new DataFrame(featureShape);

            labelShape = new Shape(new int[] { 3 });
            frameY = new DataFrame(labelShape);

            model = new Sequential(inputShape);
            model.Add(new Conv2D(16, Tuple.Create(3, 3), Tuple.Create(2, 2), activation: new ReLU(), weightInitializer: new SiaNet.Model.Initializers.Xavier(), useBias: true, biasInitializer: new Model.Initializers.Ones()));

            model.Add(new MaxPool2D(Tuple.Create(3, 3)));
            model.Add(new Conv2D(32, Tuple.Create(3, 3), Tuple.Create(2, 2), activation: new ReLU(), weightInitializer: new SiaNet.Model.Initializers.Xavier(), useBias: true, biasInitializer: new Model.Initializers.Ones()));


            model.Add(new MaxPool2D(Tuple.Create(3, 3)));


            model.Add(new Flatten());
            model.Add(new Dense(200, activation: new ReLU()));
           // model.Add(new Dropout(0.5));
            model.Add(new Dense(3, activation: new Softmax()));

var compiledModel = model.Compile();
compiledModel.EpochEnd += CompiledModel_EpochEnd;
compiledModel.BatchStart += CompiledModel_BatchStart;
compiledModel.BatchEnd += CompiledModel_BatchEnd;
compiledModel.TrainingStart += CompiledModel_TrainingStart1;
compiledModel.TrainingEnd += CompiledModel_TrainingEnd1;

            // prepare dataset
            trainingData = new DataFrameList(featureShape, labelShape);
            PrepareDataSet2(@"C:\Authorized Model\Keras Dataset\Fire\", new float[] { 1,0,0 }, true);
            PrepareDataSet2(@"C:\Authorized Model\Keras Dataset\Person\", new float[] { 0, 1, 0 }, true);
            PrepareDataSet2(@"C:\Authorized Model\Keras Dataset\Police\", new float[] { 0,0,1 }, true);

            compiledModel.Fit(trainingData, 40, 150, new Adam(), new CrossEntropy());

and the PrepareDataSet2 method:

    static void PrepareDataSet2(string folderName, float[] classLabel, bool train = true)
    {
        int width = int.Parse(ConfigurationManager.AppSettings["ImageWidth"]);
        int height = int.Parse(ConfigurationManager.AppSettings["ImageHeight"]);
        Console.WriteLine("Loading dataset for class: " + classLabel);
        var files = Directory.EnumerateFiles(folderName, "*.*", SearchOption.AllDirectories)
                    .Where(s => s.ToLower().EndsWith(".jpeg") || s.ToLower().EndsWith(".jpg") || s.ToLower().EndsWith(".bmp") || s.ToLower().EndsWith(".png") || s.ToLower().EndsWith(".gif"));

        foreach (var file in files)
        {
            using (Bitmap full = (Bitmap)Bitmap.FromFile(file))
            {
                using (Bitmap bt = full.Resize(width, height, true))
                {
                    float[] imgMatrix = bt.ParallelExtractCHW().ToArray();
                    //float[] imgMatrix = bt.ToByteArray().ToArray().Select(d => (float)d).ToArray();
                    if (imgMatrix.All(d => d == 0))
                    {
                        throw new Exception("All values in the image re 0!!");
                    }
                    //frameX.Add(imgMatrix);

                   
                    if (train)
                    {
                        trainingData.AddFrame(imgMatrix, classLabel);
                    }
                    else
                    {
                        frameX.Add(imgMatrix);
                    }
                }
            }
        }
        Console.WriteLine("Finished loading dataset for class: " + classLabel);
    }

`
The issue is: i do training for this CNN and i use Predict method to do classification for new images but i always get the same result for prediction (0,0,1).
Appreciate your support?

Embedding layer implementation and Reader for Text dataset for training

Got a request via email to implement embedding layer required for text classification.