Comments (11)
@shoyer Interesting, can you write up a small example that I can hack on?
from opt_einsum.
I don't know how interesting an example it is, but here's the one that turned up in our project:
sizes = {'a': 29, 'b': 29, 'c': 4, 'd': 4, 'f': 3, 'i': 3, 'j': 10, 'o': 3, 's': 50}
einsum_string = 'sabj,sfab,ifo,cdj,siac->sobd'
The first three terms are fixed, the last two are vary in each iteration.
from opt_einsum.
Sorry it took me a bit to get back to this. If you look at the full contraction we see:
Complete contraction: sabj,sfab,ifo,cdj,siac->sobd
Naive scaling: 9
Optimized scaling: 6
Naive FLOP count: 1.453e+09
Optimized FLOP count: 1.482e+07
Theoretical speedup: 98.028
Largest intermediate: 5.046e+05 elements
--------------------------------------------------------------------------------
scaling BLAS current remaining
--------------------------------------------------------------------------------
6 TDOT siac,ifo->focas sabj,sfab,cdj,focas->sobd
6 False focas,sfab->ocabs sabj,cdj,ocabs->sobd
6 False ocabs,sabj->ocjbs cdj,ocjbs->sobd
6 TDOT ocjbs,cdj->sobd sobd->sobd
The possible intermediates to form are:
sabj,sfab,ifo->objia
(N^7 scaling, 13x cost)sabj,sfab->abjf
(N^7 scaling, 12x cost)sfab,ifo->sabio
(N^6 scaling, 0.9x cost)sabj,ifo->sabjifo
(N^9 scaling, ...)
So it doesn't look quite worth it for this particular example, I think I know how to build something that automatically detects this however.
from opt_einsum.
So it doesn't look quite worth it for this particular example
Sorry, I should have clarified -- we already did this analysis by hand and came to the same conclusion :). But yes, in general this would be nice to have. I suspect the pattern of updating only some tensors inside an optimization and/or simulation loop is pretty common.
from opt_einsum.
Well at least we are coming up with the same thing! Any thoughts on the API if we have a function that looks like the following:
constant_contract(expression, constant_tensors, tensors)
where the order in the expression would be constant_tensors + tensors
. This could return:
- a packet:
(new_expression, temporaries)
so you could callcontract(new_expression, *(temporaries + args))
- a function that takes
func(tensors)
- Something else?
from opt_einsum.
I think this is also possible now, to an extent, using dask
as the backend with #17 since it will reuse intermediaries if the arrays are kept the same.
from opt_einsum.
Good point, I wonder if it would be worth it to revisit this in a more explicit manner however. If I think about the ML libraries it would be much better if we could reuse parts that would stay on a GPU.
from opt_einsum.
Indeed. I have a bit of my library that can view any tensor network as a scipy.LinearOperator
, allowing for various nice iterative decompositions just involving matvec
. Obviously between matvec
calls the tensors are all constant apart from the new vector. I think the contractions generally start with the vector and contract everything into it (and so there are no reusables), but it would be nice to automatically detect when this is not the case.
My suggestion for an api would be:
>>> # assume we have four shapes and matching arrays with 2 & 4 constant
>>> expr = contract_expression("abc,bcd,cde,ef", shape1, array2, shape3, array4)
>>> out = expr(array1, array3)
I.e. if you supply arrays to a contract_expression
it internalizes them, and then to the expression itself you only supply arrays matching the remaining shapes.
EDIT: I think this api would be pretty easy to implement, but short of just using dask internally, working out the reusables would be a bit more involved.
from opt_einsum.
What if someone does contract_expression("i,i->", (1, 2, 3), np.array([1, 2, 3]))
expecting to take an inner product? I think I was stumped for a clean API last time this came up.
from opt_einsum.
Ah yes. I think you'd have to explicitly enforce a 'if it doesn't have a shape, it's not an array' policy.
Producing helpful errors to that effect would be possible since shapes are only 1D and wouldn't match the expression if viewed as arrays but I see your point.
Another option would just be:
contract_expression("abc,bcd,cde,ef", shape0, array1, shape2, array3, constant=[1, 3])
from opt_einsum.
I could be more comfortable with that. shape0
/shape2
could also be arrays with shape attributes as well.
from opt_einsum.
Related Issues (20)
- allow `backend=None` as shorthand for 'auto' HOT 1
- Could `opt_einsum` understand repeated inputs?
- Zero dimension tensors (as zero entries) lead opt_einsum slower than einsum HOT 1
- Fink optimal path for caching constant intermediates HOT 3
- When I use opt_einsum optimizes torch.einum, the running time after optimization increases. HOT 5
- Thoughts on making the numpy dependency optional? HOT 5
- Way to keep tensor Ids from being rearranged to cretae custom paths HOT 2
- 'out' option in 'contract' makes a copy HOT 6
- Contigugous loss after opt-einsum that may affect the efficiency of subsequent operations HOT 8
- Return type of opt_einsum.contract -- linter errors HOT 2
- Add Aesara as a backend HOT 4
- Multiple GPUs HOT 2
- Suggestion: Multiple einsum steps HOT 2
- Setting memory_limit in contract_path causes "ValueError: invalid subscript 'Ų' in einstein sum subscripts string, subscripts must be letters" HOT 7
- contract_expression inconsistent with contract
- contract_path with `optimize=bool` raises HOT 3
- Sub-optimal contraction path when using broadcasting
- New release with Python 3.12 support? HOT 1
- Setup.py says install requires Numpy >= 1.7 HOT 1
- `contract` failed to find required methods from `jax.numpy` backend HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opt_einsum.