balena / python-pqueue Goto Github PK

My queue is initialized with a path like /home/user/store; pqueue's _saveinfo method creates a temporary file and then moves it to the directory in question.

It works fine if both, the source and the destination are on the same partition, but it fails if they're on different partitions. Here is a stack trace:

> pqueue.py(116)_saveinfo()
-> tmpfd, tmpfn = tempfile.mkstemp()
(Pdb) n
> pqueue.py(117)_saveinfo()
-> os.write(tmpfd, pickle.dumps(self.info))
(Pdb) n
> pqueue.py(118)_saveinfo()
-> os.close(tmpfd)
(Pdb) n
> pqueue.py(120)_saveinfo()
-> os.rename(tmpfn, self._infopath())
(Pdb) n
OSError: (18, 'Invalid cross-device link')
> /pqueue.py(120)_saveinfo()
-> os.rename(tmpfn, self._infopath())
(Pdb) pp tmpfn
'/tmp/tmpgGWKxT'
(Pdb) pp self._infopath()
u'/home/user/test/info'

Here's the output of df -h

user@machine ~/test $ df -h
Filesystem           Size  Used Avail Use% Mounted on
udev                 3.8G   12K  3.8G   1% /dev
tmpfs                779M  1.5M  777M   1% /run
/dev/sda1             30G   12G   17G  41% /
none                 4.0K     0  4.0K   0% /sys/fs/cgroup
none                 5.0M     0  5.0M   0% /run/lock
none                 3.8G  362M  3.5G  10% /run/shm
none                 100M   40K  100M   1% /run/user
/dev/sda3            881G  791G   45G  95% /home
/home/user/.Private  881G  791G   45G  95% /home/user

The classification tags on pypi page of this library indicate that it's Operating System :: OS Independent, but it does not work on Windows actually. I tested on Windows 10, either when you create a queue and start putting things into it, A WindowsError 183 raises, or when you load a queue from existing file, the Unix-only function os.ftruncate being used will cause an AttributeError because, well, by definition there is no os.ftruncate on Windows.

Hello,

When I try to create a pqueue in a directory that already has queue files I get an error saying that one of the queue files can not be found. This is quite a strange error since I have multiple different pqueue directories and the others seem to be fine.

I can fix this error by manually creating the missing file, however I was wondering if there was an automatic way to do this? Something, say for example like a try catch where the file is created if not found, or the next file is searched for (in my case 309 didn't exist but 310 did).

With the following example, if I run it and then comment out the put loop and then run it again, I get 99 printed.

def worker():
    while True:
        try:
            item = q.get()
            print item
        except:
            pass
        finally:
            q.task_done()

for i in range(3):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in range(100):
    q.put(item)

Dear

First thank you for this module. I am using it but when I try the pop, this happens:

from pqueue import Queue
q = Queue("try")
q.put(1)
q.put('A')
q.pop()
Traceback (most recent call last):
File "", line 1, in
AttributeError: Queue instance has no attribute 'pop'
q.get()
1
q.get()
'A'

Anyway, is there a way to delete "get" data that I don't want to store anymore?

Hello,

I can't find exactly what is causing it or when it happens, but about once a week the cache file gets corrupted and I have to manually delete it. Any idea what could be happening ?

Traceback (most recent call last):
   File "telemetry_server.py", line 289, in upload_cached_records
     cache_object = self.cache_queue.get(block=False)
   File "/usr/lib/python3.8/queue.py", line 180, in get
     item = self._get()
   File "/usr/local/lib/python3.8/dist-packages/pqueue/pqueue.py", line 87, in _get
     data = pickle.load(self.tailf)
 _pickle.UnpicklingError: invalid load key, '\x00'.

Just a quick thank you for the lib - played with a few over this past week - happy I found this one :)