abap34 / deepshiba.jl

DeepShiba

deepshiba.jl's Issues

Don't work badge

The status of the badge is unkown..... despite passing the test.

There is currently not much need to provide this.
So I propose to remove it.

Adding type annotations

It is in the final stages to design the core of DeepShiba
For this reason, I think it is useful to add type annotations for the structures and functions.
This will improve performance and readability of the source code.

Support for higher order differentiation

Currently, DeepShiba builds a computational graph when operations are performed on ShibaObject.Variable.

Also, grads are stored by the usual Julia language types; Int and Float, for example.

Therefore, the graph is not built for back propagation.

That means that "back propagation for back propagation", i.e. higher order differentiation, is not possible.

No type annotation on new functions

The new functions in functions.jl are not type-annotated.

[discussion] run faster

This issue is for discussing how to make DeepShiba run faster.

Bugs in back propagation

There is a bug in the back propagation.

using DeepShiba

function goldstain(x, y)
    z = (1 + (x + y + 1)^2 * (19 - 14x + 3x^2 - 14y + 6x * y + 3y^2)) * (30 + (2x - 3y)^2 * (18 - 32x + 12x^2 + 48y - 36x * y + 27y^2))
    return z
end

x = variable(1.0, "x")
y = variable(1.0, "y")
z = goldstain(x, y)

backward!(z)
print(z)
print(x)
print(y)

Expected output

(
data:1876.0
grad:1
creator:Mul
)
x(
data:1.0
grad:-5376.0
)
y(
data:1.0
grad:8064.0
)

Actual output

(
data:1876.0
grad:1
creator:Mul
)
(
data:1.0
grad:-84799.0
)
(
data:1.0
grad:6772.0
)

Improve type annotation

Today, DeepShiba is type-annotated for many functions and structures.

The purpose of this is to prevent errors by deliberately narrowing the scope of operation.

However, in many situations, abstract types are used, which raises concerns from a speed and memory efficiency perspective.

Experimentation and improvement are needed.

Improve the output

Deepshiba needs to make the output in REPL and print() more visible.

The current output is not user-friendly.

Dealing with branching graphs

Currently, DeepShiba does not support branching graphs that use functions with multiple arguments, such as addition.
Also, their reverse propagation will not work properly.
These need to be implemented.

Documenting a function

DeepShiba's functions are currently undocumented.
It would be better if these had a docstring.

A better test

DeepShiba underwent a major and destructive change.

As a result, a lot of test code stopped working and was removed. (#29)

As a result, the current DeepShiba does not have adequate testing, which leads to bugs.

DeepShiba needs a better testing framework and useful functions.

overloading an operator

The current DeepShiba requires the use of redundant expressions. For example, when adding, you need to use Add(), which is not user friendly.
We should be able to use + in a more intuitive way.
Users can more easily take advantage of DeepShiba by overloading existing Julia operators to support DeepShiba objects.

Implementing a function to visualize a graph

A Function, visualize graphs is very useful for debugging.

Add plot features

The current plot function is a single theme and the output value is a single value.

It would be useful to add themes and output values, e.g. grads, to this.

Display a plot in window or Jupyter

The current DeepShiba plot only supports exporting to files.

This is unfriendly for users who want to improve their models interactively.

More Functions

Some mathematical functions are missing from the current DeepShiba. For example, sin and cos, they need to be added.

Support for transformation function

Problems

Currently, DeepShiba assumes that only mathematical functions... , can be back-propagated regardless of the values upstream of the graph.

However, this needs to be modified.
Because DeepShiba needs to add functions that require values to propagate from upstream of the graph - for example, the reshape function.

How to solve

This requires the design of a more flexible function structure.
In the current version of DeepShiba, each function's type is a ShibaObject.Func.

DeepShiba will be change the system of it,
first, functions will be a new type which supertype is ShibaObject.Func.
And that's behaivior controlled by multiple dispach.

This can lead to many improvements.
We need to do some validation, but for now, here's an example

1.Memory Efficiency

So far, each function on the graph has its own forward and backward.
However, they are now separated from the structure by multiple dispatch. This reduces the size of the structure to a much smaller size.

2. Ease of debugging

The current Func has a field -- "name".
This is for debugging purposes, and if it is not set, the output may not be as clear as it should be.
This is especially true for plots.
However, if you define a type for each function, they can easily be done with typeof()!

Support for GPUs

The current DeepShiba does not support GPUs.

With GPU support, DeepShiba will run faster.

Doesn't pass the test.

I feel it's a very funny mistake.
https://travis-ci.org/github/abap34/DeepShiba.jl/builds/703524347?utm_source=github_status&utm_medium=notification

Circular Reference

Currently, DeepShiba uses a circular reference internally.

function (f::Func)(vars::Variable...)
    f.inputs = vars
    xs = [x.data for x in vars]
    ys = f.forward(xs...)
    ys = as_tuple(ys)
    outputs = [Variable(y, f, nothing, f.generation + 1, nothing) for y in ys]
    f.generation = maximum([x.generation for x in f.inputs])
    f.outputs = outputs
    length(outputs)  == 1 ? outputs[1] : outputs
end


function backward!(var::Variable)
    (isnothing(var.grad)) && (var.grad = ones_like(var.data))
    funcs = PriorityQueue{Func, Int}()
    seen_set = Set{Func}()
    funcs[var.creator] = 1
    while !(isempty(funcs))
        f = dequeue!(funcs)
        gys = [output.grad for output in f.outputs]
        inputs = get_data.(f.inputs)
        gxs =  f.backward.(inputs) .* gys
        gxs = tuple(gxs...)
        for (x, gx) in zip(f.inputs, gxs)
            if isnothing(x.grad) 
                x.grad = gx
            else
                x.grad = x.grad + gx
            end
            if (!(isnothing(x.creator))) 
                if !(x.creator in seen_set)
                    funcs[x.creator] = x.generation
                end
            end
        end
    end
end

Variable refers to Func and Func refers to Varible.
This is not good for memory efficiency.
Improvement is needed.

create numerical differentiation

Functions that perform numerical differentiation can be used for testing and other purposes.

remove example

The current DeepShiba has grown through a series of disruptive changes.

Therefore, the example needs to be modified repeatedly, which is inefficient.

Speeding up the plot

The current DeepShiba also uses priority queue when doing plots.
But when plotting, just need to scan through all the elements, and the order of the elements does not matter.
Therefore, DeepShiba will use a simpler structure with no order.

This will greatly speed up the plotting process

create simple core file.

Need to implement a core file for hypothetical use.

clean up file structures

DeepShiba has added a lot of features, and the number of functions in a single file is enormous.
This should be subdivided and organized by directory.

Support for Broadcast

The current DeepShiba does not support broadcast.

However, it is a very powerful feature and is needed when defining complicated functions.

It should be added as soon as possible.

Improve text output

The current Variable in DeepShiba always returns the same string when requested for text output.

However, when an array of Variables is defined, for example, the current polite output is inconvenient.

Plotting from script does not work properly.

If you try to plot directly from script, the window disappears instantly.

This is an example of code that does not work properly.

using DeepShiba

x = variable(2.0)
y = variable(3.0)
f(x, y) = 2x^2 + 12y
z = f(x, y)
plot(z)

Implement back propagation

The most core feature, back propagation, needs to be implemented.

Separation of core files

DeepShiba is becoming more complex.
Long source code has a negative impact on readability and it's management.

Change the implementation of back propagation from a recursive structure to a loop-based one

The current back propagation is implemented using a recursive structure.
However, this is a concern in terms of scalability and speed of operation, and needs to be changed to a loop-based implementation.