Apparently property arrays are limited to 32767 elements about cellblender

[GoogleCodeExporter]

Original comment by t.m.bartol on 9 Apr 2013 at 6:25

• Changed title: *Apparently property arrays are limited to 32767 elements *

[GoogleCodeExporter]

Apparently property arrays are limited to 32767 elements.

This mean that you can't have a region with more than this many
faces.  One possible fix is to store large regions as multiple arrays.
I had thought this limitation was removed in a recent 2.6x version of Blender 
but
apparently it has now returned.  Yikes!

Original comment by t.m.bartol on 9 Apr 2013 at 6:28

[GoogleCodeExporter]

It seems that for regions with a large number of faces storing them by index is 
rather inefficient. Maybe there's a way to go to a range based approach, i.e. 
instead of [1,2,3, ...., 1e6] we'd do range(1:1e6). Clearly, this only works 
well for 
(a number of) large contiguous sets of indices but maybe there is some logic we 
can 
add that would figure that out and either store it as a list of indices or a 
list of ranges whatever seems most efficient. This assumes that we can store 
ranges in property arrays.

Original comment by [email protected] on 11 Apr 2013 at 6:59

[GoogleCodeExporter]

Hi Markus,

  What a great idea!  I think I know just what to do:  I'll sort the list
of indices and then run-length encode it.  The
format for a run-length encoded list is simply a linear array of pairs of
numbers (or a two dimensional array) where the
first number in a pair is the starting index for a contiguous run and the
second number is the length of the run.  To make
things even more compact  I could use positive numbers to encode the
starting index and a negative number to encode
the length of the run.  This way if a run is length 2 or less I'd just list
the indices directly as positive numbers and not list
the length of the run.  But is the run is length 3 or longer then I'd list
it's starting index followed by a negative number for
its run-length.  I've done things like this before and you usually get a
factor of 2 or more compression.  In this case I'd expect even more
compression because we're not compressing random data but numbers that
sample (without replacement) a contiguous list.  We expect to need fewer
runs and for runs to get longer and longer as regions get larger and larger
(i.e. when regions sample a large fraction of the whole object). For
example, in the case of a region that encompasses a whole object of N faces
the encoding would simply be [0, -N]  instead of the whole array  [0, 1, 2,
3, 4, ... N].

Original comment by t.m.bartol on 12 Apr 2013 at 3:39

[GoogleCodeExporter]

Hi Tom,

Great idea - run-length encoding should work really well in this case and 
result 
in significant space saving. Also, unless somebody has >15k distinct regions 
there 
should be no issues fitting everything into a single property array ;)

Original comment by [email protected] on 12 Apr 2013 at 1:27

[GoogleCodeExporter]

RightOh!  Still, in the interest of semi-infinite scalability I suppose I
could write the code to allow 32767 banks of region arrays (with 32767
regions each, so 2^30 regions total), each one allowing 32767 banks of
run-length encoded face lists (with 32767 rle codes, no sure how many total
faces this means in the worst case).  Not too much extra effort really and
we'd (probably) never need to upgrade this. The reason I worry is that
adding banks later will break backward compatibility so there is an
advantage to do it now and be done with it.   Or is this just silly?  If
someday we can do a whole pyramidal neuron we'd need regions for 1e4
spines, 1e4 spine heads, 1e4 spine necks, etc... so we'd probably exceed
this limit (frightening to think about).  Would we really be able to handle
an object that big in Blender?   Anyhow,  I'll do some quick python testing
of rle (just pure python tests not using Blender) to see what the practical
limits are with no support for region banks and report back what I find...

Original comment by t.m.bartol on 12 Apr 2013 at 4:17

[GoogleCodeExporter]

This was fixed on 05/13/2013 in commit 1bf71d5f6d7fc716db7810f8a1e0cc1a827c4614.

Original comment by [email protected] on 4 Mar 2014 at 4:21

• Changed state: Fixed