prcastro / nervoso.jl

Simple Neural Networks library written in Julia

Hi there 👋

My name is Paulo and I'm a Machine Learning Engineer working at Nubank!

• 🔭 I’m currently building tools to make LLMs more accessible and easy to use.
• 🌱 I’m currently learning:
  • 🤖 Creating AI agents using LangChain
  • 🎨 Low-level graphics programming with C
  • 🕹️ Game development with Unity and Godot
• 👯 I’m looking to collaborate on Open Source projects ranging from MLOps to Graphics programming
• 😱 I'm fascinated by retro games programing, and I love building demo projects aimed at old consoles and computers
• 😄 Pronouns: he/him

nervoso.jl's Issues

Code Coverage

We must get both Coveralls and Codecov working. By working, I mean a badge on README with the correct % of code coverage.

Momentum gradient descent

I should implement this and see if it help finding the optimum weights.

Worked examples

We should add worked examples on the documentation

• XOR
• Digits

Batch Gradient Descent

Implement batch GD

Usage with MLBase

While I don't want to make MLBase a dependency, I belive we should document how to use this library alongside MLBase and also delegate performance evalutation functionality to MLBase.

This would not only make this lib more useful, but also reduce the amount of code to maintain.

Tests

Write a suit of tests, probably using FactCheck

• Write tests
• Coveralls

Reference documentation

We need to write a reference, documenting each function/type/etc exported by the library. However, it should be automated, extracting the metadata from each function and putting into a markdown file. This way, we don't need to update the documentation by hand each time we write a new function.

Change package name

Change the package name to something nicer

Not converging

Having problems with bth xor and digits datasets

Documentation

Not only insert a nice/helpful docstring on each function/type, but also document the interface expected by the package (how to create new error/activation functions, etc) and conceptually how it works (like a PDF deriving the algorithm).

• Docstrings
• README
• Interface documentation
• Conceptual PDF

Regularization

We should make regularization available.

Would it be feasible to make it extendable? The user could define a function of the weights to be minimized alongside with the error, and we use this function (its derivative) to update the weights? Or should we hardcode some of them (L1, L2 and Dropout, for instance) into Nervoso?

Useful link:
http://neuralnetworksanddeeplearning.com/chap3.html#regularization

Automatic differentiation

This is what Theano do when used in Deep Learning with Python. We can do it by using packages from JuliaDiff to compute the derivatives instead of using a derivative dictionary. While this could cause some slowdown on differentiation (and at import, because we would depend on a package), this wouldn't be much, and would make much easier to extend the library with new cost/activation functions.

AdaGrad

Implement AdaGrad

Move docs from README to ReadTheDocs

Precompilation

Should we add the line

VERSION >= v"0.4.0-dev+6521" && __precompile__(true)

to the beginning of the module file (NeuralNetsLite.jl file) so that the Module is precompiled?
What can we benefit on doing this?

Change errorprime API

An idea is to change someerrorprime to be the derivative of the error with respect to the output, and handle the chain rule outside (like in here https://github.com/prcastro/NeuralNetsLite.jl/blob/master/src/feedforward/network.jl#L103)