Comments (17)
This is an unusually annoying depwarn, because it seems to be triggered upon every invocation of a Flux call. :)
from flux.jl.
Yup. I've just been using --depwarn=no
(as per usual) but should probably sort this out ASAP.
from flux.jl.
You should just stop overloading .+
etcetera in 0.6.
from flux.jl.
That's not a reasonable requirement in any case where we can't broadcast arbitrary Julia functions. As it turns out there are quite a lot of cases like that, including TensorFlow, MXNet, and many of the GPU libraries. This is a real use case and it's unfortunate that the discussions in Base didn't take it into account whatsoever.
from flux.jl.
Why can't you broadcast arbitrary Julia functions?
from flux.jl.
Many libraries that provide an array abstraction do so in a numpy-like fashion – you get a set of "vectorised" operations like +
, *
etc. but anything that accesses individual elements is either breaking the abstraction barrier or unusably slow.
from flux.jl.
If you only support a small set of operators on your data, there are plenty of binary operators to choose from that you can define. You don't have to use .+
. Dot operators now carry with them an expectation of fusion and support for in-place operations like x .+= foo.(x.^2) .- 3
without temporary arrays.
In the longer term, the whole Matlab/numpy-like style, where only certain vectorized operations are fast (at the cost of lots of temporary arrays), kind of defeats the point of Julia.
from flux.jl.
I also don't see how that applies to Flux and DataFlow, which are pure-Julia packages as far as I can tell.
from flux.jl.
In the long term, yes, I'd love to have this stuff all implemented in Julia and compile GPU code on the fly etc. But that isn't going to happen immediately, so interop with existing libraries is the only reasonable option right now.
Can you elaborate on how, say, broadcasting +
, sin
etc should be written over GPU arrays, if not with .+
and sin.
? (Bearing in mind we want to be generic over array types.)
I expect it would be possible to implement broadcasting syntax in a trait-like way in which the container can choose whether to fuse, which would solve the problem for us.
from flux.jl.
I also don't see how that applies to Flux and DataFlow, which are pure-Julia packages as far as I can tell.
Flux has a lazy dependency on TensorFlow, and/or MXNet
from flux.jl.
TensorFlow allows you to define efficient custom operations in C++, and it's also possible in MXNet; why couldn't you do that from Julia?
Anyway, basically .+
means fusing broadcast
now in Julia, so if you want something that is not a broadcast
call you should use a different symbol (e.g. ⊕
or ⨦
) or function name. (I'd like to update Julia that you can use e.g. +̃
or +′
as operators.)
from flux.jl.
It's technically possible, it's just a big project, given that we need robust GPU compilation among other things. The right solution to this is not "wait until 2025".
The thing is, I do want broadcast. The semantics are all the same, and changing the user API (especially for something so common) for an implementation detail is not reasonable.
Not being able to write generic code that works over a range of implementation strategies kind of defeats the point of Julia.
from flux.jl.
Not having fusion for user-defined container types and operations in Julia would be a much bigger sacrifice than saying that you need to rename if you explicitly want non-fusing operations.
from flux.jl.
I'm not arguing we should trade one for the other, but I'm repeating myself now.
I expect it would be possible to implement broadcasting syntax in a trait-like way in which the container can choose whether to fuse, which would solve the problem for us.
from flux.jl.
I expect it would be possible to implement broadcasting syntax in a trait-like way in which the container can choose whether to fuse, which would solve the problem for us.
Nope, because fusion happens at a syntactic level (at lowering time), before types are known.
Changing fusion to a compile-time optimization that depends on inference is a complete redesign (and would also result in semantics that depend on inference). It's something that's been tried many times in many languages and has always failed to achieve genericity for user-defined types and functions. That is a "wait until 2050" solution.
from flux.jl.
if should_fuse(x, y)
broadcast((x, y) -> x + y, x, y)
else
broadcast(+, x, y)
end
This is still a syntactical transformation that doesn't depend on inference. should_fuse
can default to true
and be compiled away in the base case (just like promotion rules), leaving you identical code to the current output. But overriding it for GPUArray
etc would solve our problem.
from flux.jl.
I see, yes, that would be possible.
from flux.jl.
Related Issues (20)
- Significant time spent moving medium-size arrays to GPU, type instability HOT 10
- ConvTranspose errors with symmetric non-constant pad
- SamePad() for even sized filters.
- Dense layers with shared parameters HOT 5
- Implementation of `AdamW` differs from PyTorch HOT 10
- `gpu` should warn if cuDNN is not installed HOT 2
- Cannot take `gradient` of L2 regularization loss HOT 1
- Create a flag to use Enzyme as the AD in training/etc. HOT 14
- test Enzyme gradient for loss functions
- test Enzyme gpu support
- Enzyme fails with MultiHeadAttention layer HOT 13
- Enable github Discussions
- Stacked RNN in Flux.jl?
- Add option to throw error on passing wrong precision floats to layers HOT 3
- Potential bug of RNN training flow
- why is my `withgradient` type unstable ? HOT 1
- is `Flux.huber_loss` type-unstable ?
- Can't load a Fluxml trained & saved model. Getting ERROR: CUDA error: invalid device context (code 201, ERROR_INVALID_CONTEXT) HOT 1
- ConvTranspose with padding on cpu throws exception HOT 1
- DifferentiationInterface testing HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux.jl.