ServiceUnavailable handling about cloud-volume

[macrintr]

When I followed up, investigating if the CloudVolume object had been corrupted, I got another exception:

julia> size(vol)
Exception AttributeError: "'Storage' object has no attribute '_threads'" in <bound method Storage.__del__ of <cloudvolume.storage.Storage object at 0x7fb237c6ee50>> ignored
ERROR: MethodError: no method matching size(::CloudVolume.CloudVolumeWrapper)
Closest candidates are:
  size{N}(::Any, ::Integer, ::Integer, ::Integer...) at abstractarray.jl:48
  size(::BitArray{1}) at bitarray.jl:39
  size(::BitArray{1}, ::Any) at bitarray.jl:43
  ...

The error is expected, but the exception wasn't.

[macrintr]

Happened again. Just writing to track it. The first instance happened after about 20 hours (sometime around Sunday early morning). This second instance happened after about 32 hours (just now, Monday afternoon).

[macrintr]

Just watching my upload with @wongwill86, and noticed another 503 exception:

Exception in thread Thread-614553:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cloudvolume/storage.py", line 61, in _consume_queue
    super(Storage, self)._consume_queue(terminate_evt)
  File "/usr/local/lib/python2.7/dist-packages/cloudvolume/threaded_queue.py", line 133, in _consume_queue
    interface = self._initialize_interface()
  File "/usr/local/lib/python2.7/dist-packages/cloudvolume/storage.py", line 55, in _initialize_interface
    return self._interface_cls(self._path)
  File "/usr/local/lib/python2.7/dist-packages/cloudvolume/storage.py", line 335, in __init__
    self._bucket = self._client.get_bucket(self._path.bucket_name)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/client.py", line 173, in get_bucket
    bucket.reload(client=self)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/_helpers.py", line 99, in reload
    _target_object=self)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
ServiceUnavailable: 503 GET https://www.googleapis.com/storage/v1/b/neuroglancer?projection=noAcl: unknown error

This didn't cause an error in my script, though, so the upload is still chugging. There's no missing data for the chunks around the exception, either.

[william-silversmith]

Hmm do you know what block you were processing when the error occurred? Is the error legit? If you can reproduce it I can help figure this out.

[william-silversmith]

Also, why was it expected that _threads was not available?

[macrintr]

The 503 exception has happened 5 times now. When I've restarted the ingest, all the chunks in the immediate vicinity are present. In that instance yesterday, where the exception didn't cause my ingest to stop, all the chunks were present. To recreate what I'm doing, just try a long ingest job on a single thread. Here's my Julia ingest script:

using H5SectionsArrays

module CloudVolume

using PyCall
@pyimport cloudvolume as cv
const pyslice=pybuiltin(:slice)

function cached(f)
    cache=Dict()
    function my_f(args...)
        if !haskey(cache, args)
            cache[args] = f(args...)
        else
            println("restoring from cache")
        end
        return cache[args]
    end
end

CachedVolume = cached(cv.CloudVolume)

immutable CloudVolumeWrapper
    val
    function CloudVolumeWrapper(storage_string, scale_idx=0)
        return new(CachedVolume(storage_string, scale_idx))
    end
end

function Base.getindex(x::CloudVolumeWrapper, slicex::UnitRange, 
                                        slicey::UnitRange, slicez::UnitRange)
    return squeeze(get(x.val, 
            (pyslice(slicex.start,slicex.stop+1),
            pyslice(slicey.start,slicey.stop+1),
            pyslice(slicez.start,slicez.stop+1))),4)
end

function Base.setindex!(x::CloudVolumeWrapper, img::Array, slicex::UnitRange, 
                                        slicey::UnitRange, slicez::UnitRange)
    x.val[:__setitem__]((pyslice(slicex.start,slicex.stop),
                            pyslice(slicey.start,slicey.stop),
                            pyslice(slicez.start,slicez.stop)), 
                            img)
end

end

vol = CloudVolume.CloudVolumeWrapper("gs://neuroglancer/pinky40_alignment/prealigned")
ba = H5SectionsArray("data/registry_prealigned_with_dropped_slices.txt");

ng_to_z = readdlm("data/ng_to_alembic_z.csv", Int64)
ng_to_alembic = Dict(ng_to_z[i,1] => ng_to_z[i,2] for i in 1:size(ng_to_z,1))

z_range = 504:8:1004
x_range = 11264:1024:67000
y_range = 8192:1024:43000

x_restart_range = 51200:1024:67000

for (zstart, zstop) in zip(z_range[1:end-1], z_range[2:end])
    for z in zstart:zstop
        alembic_z = ng_to_alembic[z]
        dir = "gs://seunglab_alembic/datasets/pinky_40percent/3_prealigned"
        src_fn = "1,$(alembic_z)_prealigned.h5"
        dst_fn = "/home/thomas_macrina/data/$(src_fn)"
        if !isfile(dst_fn)
            cmd = `gsutil -m cp $(dir)/$(src_fn) /home/thomas_macrina/data/`
            println(cmd)
            Base.run(cmd)
        end
    end

    if zstart == 504
        for (xstart, xstop) in zip(x_restart_range[1:end-1], x_restart_range[2:end])
            for (ystart, ystop) in zip(y_range[1:end-1], y_range[2:end])
                h5_slice = (xstart:xstop-1, ystart:ystop-1, zstart+1:zstop)
                ng_slice = (xstart:xstop, ystart:ystop, zstart:zstop) # lineup with final z indices
                # println((h5_slice, ng_slice))
                vol[ng_slice...] = ba[h5_slice...]
            end
        end
    else
        for (xstart, xstop) in zip(x_range[1:end-1], x_range[2:end])
        	for (ystart, ystop) in zip(y_range[1:end-1], y_range[2:end])
        		h5_slice = (xstart:xstop-1, ystart:ystop-1, zstart+1:zstop)
        		ng_slice = (xstart:xstop, ystart:ystop, zstart:zstop) # lineup with final z indices
        		# println((h5_slice, ng_slice))
                vol[ng_slice...] = ba[h5_slice...]
        	end
        end
    end

    for z in zstart:zstop-1
        alembic_z = ng_to_alembic[z]
        src_fn = "1,$(alembic_z)_prealigned.h5"
        cmd = `rm /home/thomas_macrina/data/$(src_fn)`
        println(cmd)
        Base.run(cmd)
    end
end

[macrintr]

As for the _thread exception -- that wasn't expected. The MethodError in that output that I copied above was expected, though -- that's what size(vol) should throw.

[william-silversmith]

Hmm, ok. For some reason I was thinking 403 instead of 503. I understand what you're asking now. For the moment, I would just catch it on the client side and retry the chunk that failed. CloudVolume internally already retries 7 times before it gives up and reraises 503. It's probably inevitable in a big job that you encounter some of these.

I think the right way to handle a 503 that survives those retries would probably be to back off completely on all threads for a moment before resuming. Maybe 503 shouldn't even be a fatal error, it should retry indefinitely (or at least much more aggressively, maybe 100 times).

Thoughts?

[william-silversmith]

This seems to have stopped being a problem as we got better at understanding the limitations of the cloud API. Closing for now.