juliaarrays / arraysofarrays.jl Goto Github PK

Add conversion to/from Slices (JuliaLang/julia#32310).

It is sometimes that case I'm constructing a VectorOfArrays and I know what size the backing array will be as well as the number of arrays they will constitute. Would it be useful to add a:

function Base.sizehint!(A::VectorOfArrays{T,N}, n, s::Int) where {T,N}

Overload for this situation? Happy to put together a PR if there's agreement this is helpful.

As stated in the doc and the header of the readme.md, you claimed that this is a package " for efficient storage and handling of nested arrays.". However, you have not provided any performance evaluations! Can you please think about it and provide proof for what you've stated? How much is this efficient? Compared to what is this package efficient?

First off, awesome idea/package! It's been quite useful in my own work.

I was really excited to use your package in conjunction with StructArrays but ran into an issue (that I attempted to fix with a PR to StructArrays) as explained here. The owner of that package pointed out, however, that the real issue is the following:

julia> x=nestedview(rand(2,3,10), 2);
julia> x[1] isa eltype(x)
false

What are your thoughts on modifying eltype to be consistent with the true return type of getindex?

-Colin

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

ROOT uses Int32 for all (jagg) offsets internally, everywhere.

sizehint! method uses mul which I guess should be prod ?

Hello,

Is there a way to easily reconsitute a VectorOfVectors once it has been flattened?

using ArraysOfArrays

# original VectorOfVectors 
VV = VectorOfVectors{Float64}()
push!(VV, rand(3))
push!(VV, rand2())

# need to flatten for AD 
VV_flat = flatview(VV)

# wish I could do this
VV_nested = nestedview(VV_flat)
VV_nested == VV # please be true

In my use case, it is convenient to use a VectorOfVectors since it simplifies indexing groups of parameters that relate to different aspects of a model. Sadly, ForwardDiff doesn't play nice with this structure as it wants a single vector, hence the initial flattening. I would think this is not so problematic, except that reconsituting the nested view within the function to be optimized seems complicated once dual numbers are involved.

Thanks

(copied from Slack)

Here's one example:

Statistics.std(X::AbstractVectorOfSimilarArrays{T,M}; corrected::Bool = true) where {T,M} =
    std(flatview(X); corrected = corrected)
which results in the following behavior:
julia> A=rand(2,3,4);

julia> B=nestedview(A, Val(2));

julia> C=[Array(el) for el in B];

julia> std(B)
0.32135578668613923

julia> std(C)
2×3 Array{Float64,2}:
 0.163392  0.393621  0.343109
 0.290079  0.365516  0.135774

I think the fix for this one is easy: just remove all the overloads for now. If we continue with the plan to merge SpecialArrays/JuliennedArrays here then how dims is handled changes anyways since they both support slicing along arbitrary dimensions and not just the first M.

Add array type ContiguousUnitRanges or so that wraps an elem_ptr vector and presents it as a vector of ranges. Allow VectorOfArrays to use arbitrary vectors of unit-ranges as partition vector. Using ContiguousUnitRanges will then result in a memory- and performance-optimized instance with immutable elements lengths.

MWE:

julia> A = VectorOfVectors([rand(2), rand(5), rand(3), rand(7)])
julia> flatview(view(A, 2:3)) == A.data