waveform-computing / compoundpi Goto Github PK

Hello, Dave Compoundpi seems perfect for my application based on reading the documentation, however im having trouble setting up my network settings, and am very weak in that area of computer science. I'm a university student, and would definetely be will to issue citations if you have any relevant published work on compoundpi, should my project get off the ground. My email is [email protected] if you would prefer to contact me that way.

I am currently running around 60 Rasberry Pi 3's on a 100mb wired ethernet configuration. Capturing an image works great, even at the highest resolution using the V2.1 cameras. However, when running the download command, I run into this error message which causes CPI to be unresponsive:

Warning: time delta of 192.168.1.148 is >0.25s
cpi> download
Downloaded 20161008-144351313362-192.168.1.191.jpg
[########## ] 43%Traceback (most recent call last):
File "/usr/bin/cpi", line 9, in
load_entry_point('compoundpi==0.4', 'console_scripts', 'cpi')()
File "/usr/lib/python2.7/dist-packages/compoundpi/terminal.py", line 138, in
call
return self.main(args) or 0
File "/usr/lib/python2.7/dist-packages/compoundpi/cli.py", line 227, in main
proc.cmdloop()
File "/usr/lib/python2.7/cmd.py", line 142, in cmdloop
stop = self.onecmd(line)
File "/usr/lib/python2.7/dist-packages/compoundpi/cli.py", line 301, in
onecmd
return Cmd.onecmd(self, line)
File "/usr/lib/python2.7/dist-packages/compoundpi/cmdline.py", line 239, in
onecmd
return cmd.Cmd.onecmd(self, line)
File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
return func(arg)
File "/usr/lib/python2.7/dist-packages/compoundpi/cli.py", line 1602, in
do_download
self.client.download(address, f.index, output)
File "/usr/lib/python2.7/dist-packages/compoundpi/client.py", line 1456, in
download
self._protocol.do_send(index, self.bind[1]), [address])
File "/usr/lib/python2.7/dist-packages/compoundpi/client.py", line 777, in
transact
raise CompoundPiTransactionFailed(errors)
CompoundPiTransactionFailed: 1 errors encountered while executing
192.168.1.248: no response

I have been able to download one picture at a time using download and sometimes using a range of no more than 10 IP addresses, but have never been able to just run download and gather everything.

Please let me know if there is anything I can do to help.

There's too many options in config which should be specified (or potentially specified) directly in commands (capture count, delay, recording format, etc). Additionally, the tab-completion leaves something to be desired, and the grammar is weird in places ("set" for configuration options on the client, but a whole bunch of stuff other than "set" for configuration options on the server). Ideally we ought to switch to a simple parser (pyparsing? hand-crank a rdp?) and use its output to improve completion.

To prevent "address in use" errors on startup of (esp.) cpi and cpigui shortly after shutting one of them down.

PING is the one command in the protocol which (at the moment) can't be repeated because it resets the sequence number on the receiving servers (it has to do this to permit the client to be restarted). If we added a randomly seeded "magic cookie" argument to the command, we could detect (and ignore) repeated PING commands on the servers which in turn would allow all client commands to have a retry mechanism (more useful for unicast commands, but still useful).

As mentioned in #5 we need tests for the UI components (command line and GUI); this is less important than ensuring the server and client logic remains fully tested but it should still be done before we can do a full 1.0 release.

Qt-based presumably

In the case of partial/failed captures there needs to be a way to clear down the memory buffer on the cpid servers (might be handy to include free memory in the status output as well... although that'll require a change in the protocol ...)

Any chance to get RAW support?

Thanks for a great project!

@waveform80 I am facing this error while downloading images from the server using a python script. Can you help me here? I have followed the documentation strictly.
IOError: [Errno 2] No such file or directory: '192.168.43.171-0.jpg'

Instead of specifying a recording length in advance we really need a recording start and recording stop message. Might be worth thinking about SD card storage for this stuff, but that brings in the question of exactly where the daemon has privs to write to (given it drops root privs after init)

Given the client implementation is split into its own class, it's reasonably simple to build one's own client (for batch processing). Add a chapter with an example of doing this.

The status command should warn about excessive differences in the reported timestamp, and different framerates & resolutions.

Enjoying Compound Pi, thanks for providing it to the community!

I appreciate that the genesis of Compound Pi (it seems) is simultaneous image capture and presentation, for which downloading to the Client makes total sense.

My application is video-based, with files likely larger than the available ram on each Server, so I'd like to save the files on each Server's local SD card, instead of having to download it to the Client.

Can this be done? Is it possible to create an option for this in each Server's configuration?

Thanks very much,
Doug

Broadcast isn't supported under IPv6, and multicast is considerably easier for users to setup (no need to know the correct netmask of the network - just use "find" and you're practically done)

Hi there,

After a few tests is seems clear that the 8Mp PiCam v2 is not fully supported(?) or am I overlooking something? Thus far I am unable to pull an actual 8Mp image from a server; RPI-3 with a v2 cam. All servers are fully updated/upgraded to the latest version of Raspbian & firmware.

Any future plans for support? Might there be an interim workaround?

kind regards,
Mike @ STEAMfruit

Add protocol support for start/stop recording. No idea how to integrate this into the GUI yet so this may be a batch-only feature for 0.4.

Add ability to specify quality when capturing images/recording videos

Identify fails and crashes cpi when an address is explicitly specified

The current video support produces the wrong filename extension when downloading MJPEG recordings (because the server only knows that the file is a video file, not what sort of video file). Need to extend the protocol to support something like MIME types for the file-type field in LIST, and extend the client accordingly.

Both CLI and GUI clients should have some command/option for duplicating settings from one Pi to all others

Hi @waveform80. I was wondering if this project can be run on android somehow by treating the phones as servers? Cheers 😄

Managed to crash one of the Pi servers in the test with Kathrin. Still not sure how - try and reproduce to investigate.

It'd be nice to have something better than "Communicating..." in the status bar...

The network protocol should include a VERSION command for the client to determine what version the various servers are running. Perhaps this should be the output of the PING command?

The protocol could be expanded to include specific sub-lists of servers targetted by a broadcast which would improve synchronization when a subset of servers are specified. The major problem becomes message size when dealing with a sufficiently large network of servers. Investigate the practicality of this sometime before 1.0

The string protocol is nice and easy to debug, but there's enough variability in parsing time to make it a problem, at least when dealing with an array of Pi 1's (Pi 2's parsing time is almost constantly 0.2ms, Pi 1's varies between 0.4ms and 1.2ms which is enough to warrant refactoring the protocol).

The concept of the protocol can probably remain the same (a separated protocol module with class decorators used to handle parsing and construction of messages), but messages will be strictly binary. This will probably restrict data-types for parameters to something simple to handle (32-bit integer and 64-bit float only ... perhaps a one-byte for a bool). This might be a good opportunity to integrate #33 as well (make part of the header a variable length list of units?).

Allow user to specify user and group to run as. This will almost always be root (due to the requirement for /dev/mem access for GPIO), but can conceivably be something else so we should permit the user to select otherwise.

When I tried

cpi> iso 800

I got

Traceback (most recent call last):
  File "/usr/bin/cpi", line 9, in <module>
    load_entry_point('compoundpi==0.3', 'console_scripts', 'cpi')()
  File "/usr/lib/python2.7/dist-packages/compoundpi/terminal.py", line 138, in
__call__
    return self.main(args) or 0
  File "/usr/lib/python2.7/dist-packages/compoundpi/cli.py", line 172, in main
    proc.cmdloop()
  File "/usr/lib/python2.7/cmd.py", line 142, in cmdloop
    stop = self.onecmd(line)
  File "/usr/lib/python2.7/dist-packages/compoundpi/cmdline.py", line 236, in
onecmd
    return cmd.Cmd.onecmd(self, line)
  File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
    return func(arg)
  File "/usr/lib/python2.7/dist-packages/compoundpi/cli.py", line 954, in
do_iso
    value, self.parse_arg(arg[1] if len(arg) > 1 else None))
NameError: global name 'value' is not defined

There's an example for downloading with the line

with io.open('/home/pi/skygrow/test_images/%s.jpg' % addr,'w') as output:

this actually breaks a pipe in our setup (5 servers, raspi2 client)
This however works:

with io.open('/home/pi/skygrow/test_images/%s.jpg' % addr,'wb') as output:

GUI client uses netifaces to discover available interfaces but this isn't included as a dep in either the PyPI or deb packages

Now that python-daemon is finally py3 compatible should we move stuff over? It'd certainly clean up the command line client a whole lot (Cmd is an old-style class in py2 which makes sub-classing it a pain, and then there's the whole argparse thing). Not a definite thing, but something to consider sometime prior to 1.0.

The GUI currently ignores top level exceptions - need to fix this properly

Probably not an issue, wasn't sure where to post... Apologies if this is in the way.

I'm attempting to setup an array of pi cameras, and have one of the camera/pi units act as a control master, while still acting as a camera to synchronously take photos with the other units.
Is it possible to get client like functionality on one of the pis?

The unicast responses from multiple Pis interfere with one another, particularly on low bandwidth networks like Wifi. Change the return channel to use TCP instead (which will involve changing the find command to open the TCP channel in response to PING).

The GUI needs documenting in a similar manner to the command line docs (section per function, screenshots, blah blah blah)

Need to add controls for ensuring cameras take equivalent exposures, with equivalent AWB settings, etc. etc. This can wait until 0.3 though...

Should probably also add controls for manual white balance (see waveform80/picamera#74)

When I run cpid on the lxterminal I get the next log:
I'm running the compoundpi server on a raspberry pi zero with raspbian
Compundpi --version 0.4
Status Active: running
What can I do i cant find the servers on the client running on raspbian jessie on a pi 3
Traceback (most recent call last):
File "/usr/sbin/cpid", line 9, in
load_entry_point('compoundpi==0.4', 'console_scripts', 'cpid')()
File "/usr/lib/python2.7/dist-packages/compoundpi/terminal.py", line 138, in call
return self.main(args) or 0
File "/usr/lib/python2.7/dist-packages/compoundpi/server.py", line 213, in main
signal.SIGINT: self.interrupt,
File "/usr/lib/pymodules/python2.7/daemon/daemon.py", line 352, in enter
self.open()
File "/usr/lib/pymodules/python2.7/daemon/daemon.py", line 344, in open
self.pidfile.enter()
File "/usr/lib/pymodules/python2.7/lockfile.py", line 223, in enter
self.acquire()
File "/usr/lib/pymodules/python2.7/daemon/pidlockfile.py", line 109, in acquire
super(TimeoutPIDLockFile, self).acquire(timeout, *args, **kwargs)
File "/usr/lib/pymodules/python2.7/daemon/pidlockfile.py", line 59, in acquire
super(PIDLockFile, self).acquire(*args, **kwargs)
File "/usr/lib/pymodules/python2.7/lockfile.py", line 239, in acquire
raise LockFailed("failed to create %%s" %% self.unique_name)
LockFailed: failed to create /var/run/cam1.MainThread-1015

It's become plain from various experiments that a method of avoiding (or at least significantly reducing the chance of) single frame skip in synchronized captures, is to initialize the camera components simultaneously. Need to experiment to see whether the Python based startup is sufficiently consistent to make it worth implementing this feature, but if it is additional protocol commands should be introduced to permit shutting down and starting up the camera synchronously.

Come to think of it this may be worthwhile anyway as a debugging tool (e.g. the ability to have cpid relinquish control of the camera temporarily in order to perform some debugging then have it grab it again).

The GUI (and possibly the CLI) should provide some means of calling an external tool with a selection of the captured images

Default timeout of 5 seconds turns out to be insufficient on wireless networks. Might even go for higher than 10, depending on experience

Uses an integer conversion, should be float

It would be useful (especially in the GUI, but also in the protocol generally) to permit a live feed from an individual camera for calibration and configuration. This'll require adding commands to the protocol to start and stop the preview, and should probably work something along the lines of the SEND command (in terms of port selection and subsequent connections, but obviously without an automatic cut-off).

Not sure what format the preview should be (H264? MJPEG stream? JPEG frames separated out from such a stream?) - this'll probably depend on what we can display easily in Qt. It'd be especially useful if the preview could appear as an integral part of the configuration dialog but we'll have to see about that nearer the time.

Change client library to use a list instead of a set for server storage, and provide some mechanism in the CLI and GUI clients to permit ordering of servers.

Add ability to enable/disable software denoise

This'll be a pain in the ass given the dreadful state of Python distros on Windows but it's probably worth doing. Need to add PySide compat. into the GUI portion and figure out how on earth to build netifaces for win64

Need a test suite - particularly for the server

This probably isn't an issue but wasn't sure where else to post.

I'm trying to develop a web based GUI for pi camera using the Flask framework, except I'm running into port address issues and I don't know enough about socket.io or networking to know how to try and solve it.

If you create an instance of CompoundPiClient in or above the Flask instance in the same script file you get.

  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use

Which doesn't happen if you run a Flask instance and cpi separately, all seem fine and dandy, but not if they are in the same script.

Any ideas?

Docs should at least include an example of constructing a video from multiple JPEGs with ffmpeg.

cpid daemon should reload config on SIGHUP. Remember to update Raspbian initscript as part of this ticket.