zeromq / filemq

FileMQ is a publish-subscribe file service based on 0MQ

FileMQ implementation in C

Why FileMQ?

"Request-reply is just a vulgar subclass of publish-subscribe" -- me, in "Software Architecture over 0MQ".

It's the API my 5-year old son can use. It runs on every machine there is. It carries binary objects of any size and format. It plugs into almost every application in existence. Yes, it's your file system.

So this is FileMQ. It's a "chunked, flow-controlled, restartable, cancellable, async, multicast file transfer ØMQ protocol", server and client. The protocol spec is at http://rfc.zeromq.org/spec:19. It's a rough cut, good enough to prove or disprove the concept.

What problems does FileMQ solve? Well, two main things. First, it creates a stupidly simple pub-sub messaging system that literally anyone, and any application can use. You know the 0MQ pub-sub model, where a publisher distributes files to a set of subscribers. The radio broadcast model, where you join at any time, get stuff, then leave when you're bored. That's FileMQ. Second, it wraps this up in a proper server and client that runs in the background. You may get something similar to the DropBox experience but there is no attempt, yet, at full synchronization, and certainly not in both directions.

How to build and use

This code needs the freshest possible libsodium, libzmq, and CZMQ. To build:

git clone git://github.com/jedisct1/libsodium.git
cd libsodium
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

git clone git://github.com/zeromq/libzmq.git
cd libzmq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

git clone git://github.com/zeromq/czmq.git
cd czmq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

git clone git://github.com/hintjens/filemq.git
cd filemq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

Contribution process:

http://rfc.zeromq.org/spec:22

Internals

I originally designed FileMQ as a training tool for Chapter 6 of the 0MQ Guide (soon in dead tree format from O'Reilly!) and sort of a reusable stack for your own experiments in protocol development. Many concepts in FileMQ are borrowed from other protocols, such as the original AMQP (which, you may not know, had a file transfer class before that was taken out to a dark alley and beaten to death by two red-hatted goons from a firm who shall remain nameless). Of course AMQP borrowed much from other protocols, like its SASL authentication dialog, which came from BEEP, and which I happily used in FILEMQ as well.

SASL is the simple authentication and security layer, and a neat way to make secure protocols over 0MQ. It works for the specific case where peer A talks to peer B over a bidirectional connection. That is any classic client-server model, which is what FILEMQ (the protocol, in upper case), gives us.

Incidentally, it's client-to-server, not peer-to-peer. Fully-symmetric protocols are nasty. Far easier to define client and server as "roles" and allow any node to be either, or both. Which is what the filemq main application is.

If you look at that program (src/filemq.c) you'll see it creates a server object, a client object, runs them until someone interrupts the process, and then quits. Nice and simple.

The whole thing is written in C, but using my special sauce of class-orientation and generic containers that makes it almost as nice as Python to write, and to read. I'm sure Erlang still outclasses this, but for sheer coverage, it's hard to beat perfect C.

So there's a client class, and a server class, and then a bunch of utility classes, some of which are more or less reusable for other applications. There's a neat configuration file parser and the SASL class (which is still empty but will hopefully grow). The directory and file management classes may be a little specific to FileMQ's use case.

The real magic sauce in FileMQ is in the model directory, which you can safely ignore until you feel confident enough to go take a look. As a first start, check out the zeromq/zproto project, and fmq_msg.xml. You will need to grab gsl from https://github.com/zeromq/gsl if you want to generate the code. Check the generate script to see how we run gsl.

So that's code generation. You can, if you absolutely hate the gsl tool (and people do), you can get the same results with 10x more work using XSTL. There are other ways; in some languages you can parse the XML at runtime. That is, BTW, a rather stupid idea since it means anyone running the code needs the protocol spec at hand.

Now, take a look at fmq_client.xml and fmq_server.xml, and their matching compilers in client_c.gsl and server_c.gsl. These churn out quite ordinary multi-threaded clients and servers for any given state machine. It should take you a week or so to understand what the heck is actually going on. I'm not going to explain too much, it'll spoil the fun.

Let me know how you get on. I'll expand this README as you ask questions.

filemq's Issues

File name incorrect in fmq_client.c

If you start up a server process serving files from the physical directory '/tmp' to the virtual directory '/xyz/foo', then set up a client subscribing to the virtual directory '/xyz/foo' into the physical directory '/home', the files are written to the directory '/home/xyz/foo'.

This seems counter-intuitive. I would expect that, in the above scenario, if you create a file '/tmp/file' it should be served to '/home/file', since I'm subscribing to the same virtual directory. Perhaps I misunderstand, but it would seem to me that what's wanted is a clone of the contents of '/tmp' from the server into '/home' of the client.

FileMQ does not build under OS X

I tried to build filemq and go this error:

  CC       src/src_libfilemq_la-fmq_msg.lo
  CC       src/src_libfilemq_la-fmq_server.lo
In file included from src/fmq_server.c:23:
In file included from ./src/filemq_classes.h:25:
In file included from ./src/../include/filemq.h:18:
./include/filemq_library.h:53:9: error: 'WITH_DRAFTS' macro redefined [-Werror]
#define WITH_DRAFTS
        ^
./src/platform.h:185:9: note: previous definition is here
#define WITH_DRAFTS 1
        ^
1 error generated.
make[1]: *** [src/src_libfilemq_la-fmq_server.lo] Error 1
make: *** [check-recursive] Error 1

I don't have C skills and haven't the slightest clue what this means but I'd be happy to provide more info if required.

Can't compile with MSVC

MSVC has the following issues:

• Requires explicit casting
• Doesn't support strcasecmp
• Name conflict with member variable "virtual" (used in fmq_patch)
• Name conflict with local varible "new" (used in fmq_dir)

I propose the following changes:

• Cast all necessary variables
• Define strcasecmp as _stricmp for Win32
• Rename "virtual" to "virtualf" (for virtual file name)
• Rename "new" to "_new" to avoid these conflicts

I've got these changes ready in my fork, can we discuss?

GSL files?

Hello. I cannot find the .gsl files anywhere. I see that they were originally committed by @hintjens under /model, but that no longer exists.

Did the code generation behavior change at some point?

make check fail

CC src/src_libfilemq_la-fmq_msg.lo
CC src/src_libfilemq_la-fmq_server.lo
In file included from src/fmq_server.c:71:0:
src/fmq_server_engine.inc: In function ‘s_client_free’:
src/fmq_server_engine.inc:274:23: error: ‘__builtin___snprintf_chk’ output truncated before the last format character [-Werror=format-truncation=]
"%6d:%-33s", self->unique_id, string);
^
In file included from /usr/include/stdio.h:862:0,
from /usr/local/include/zmq.h:59,
from /usr/local/include/czmq_prelude.h:217,
from /usr/local/include/czmq_library.h:23,
from /usr/local/include/czmq.h:37,
from src/../include/filemq_library.h:25,
from src/../include/filemq.h:18,
from src/filemq_classes.h:25,
from src/fmq_server.c:23:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64:10: note: ‘__builtin___snprintf_chk’ output between 41 and 46 bytes into a destination of size 40
return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
__bos (__s), __fmt, __va_arg_pack ());
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/fmq_server.c:71:0:
src/fmq_server_engine.inc: In function ‘s_server_handle_protocol’:
src/fmq_server_engine.inc:274:23: error: ‘__builtin___snprintf_chk’ output truncated before the last format character [-Werror=format-truncation=]
"%6d:%-33s", self->unique_id, string);
^

Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

fmq_hash.c: The buffer variable is not alligned on 64-bit boundary in fmq_hash_test function

When I ran the make check command, this message was emitted

make check-TESTS
make[2]: Entering directory `/home/rafael/0MQ/filemq/src'
Running self tests...

• fmq_patch: OK
• fmq_chunk: OK
• fmq_file: OK
• fmq_dir: OK
• fmq_msg: OK
• fmq_sasl: OK
  lt-fmq_selftest: fmq_hash.c:79: fmq_hash_update: Assertion `((long) buffer & 7L) == 0' failed.

• fmq_hash: /bin/bash: line 5: 14443 Aborted (core dumped) ${dir}$tst
  FAIL: fmq_selftest

I reviewed the source code (fmq_hash.c) and I saw that the buffer was declared as a local variable. In this way the buffer is not alligned on 64-bit boundary.

In order to fix this I declared the buffer as global variable. Ugly but works for me.

// --------------------------------------------------------------------------
// Self test of this class

byte buffer [1024]; // 2013.03.18 rgp delared as global

int
fmq_hash_test (bool verbose)
{
printf (" * fmq_hash: ");
memset (buffer, 0xAA, 1024);

fmq_hash_t *hash = fmq_hash_new ();
fmq_hash_update (hash, buffer, 1024);
byte *data = fmq_hash_data (hash);
size_t size = fmq_hash_size (hash);
assert (data [0] == 0xDE);
assert (data [1] == 0xB2);
assert (data [2] == 0x38);
assert (data [3] == 0x07);
fmq_hash_destroy (&hash);

printf ("OK\n");
return 0;

}

Kind regard from Mexico City
Rafael

Client does not detect dead server

Start a server, start a client, then kill the server. The client does not detect the dead server (with or without heartbeat=1). Re-starting the server still leaves the original client running, but unconnected.

It appears as if zmq_poll does not properly detect broken connections (but I'm not absolutely sure of this).

Latest commit doesn't build with msvc

Let me know how to fix this so I can help and also create a release version - last one was 4 years ago

zfile_rmdir() undefined

When I try to build filemq, I hit the following error:

fmq_dir.c: In function 'fmq_dir_remove':
fmq_dir.c:433:9: error: implicit declaration of function 'zfile_rmdir' [-Werror=implicit-function-declaration]
cc1: all warnings being treated as errors
make[2]: *** [fmq_dir.lo] Error 1

I built and installed libzmq and czmq and zfile_rmdir() isn't defined anywhere. I pulled everything from the git repositories, so I have the latest source. Is there perhaps an older revision of something I should be using?

Matt

unknown type name ‘Bool’

Problem: filemq does not compile on Ubuntu out of the box.

fmq_selftest.c:30:5: error: unknown type name ‘Bool’
fmq_selftest.c:33:19: error: ‘TRUE’ undeclared (first use in this function)

Solution: use bool type from C99

i am getting errors in make check step will you please help me out

Making check in src
make[1]: Entering directory /home/neotrade/Downloads/filemq/src' /bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c -o fmq_client.lo fmq_client.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -fPIC -DPIC -o .libs/fmq_client.o libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -o fmq_client.o >/dev/null 2>&1 mv -f .deps/fmq_client.Tpo .deps/fmq_client.Plo /bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c -o fmq_msg.lo fmq_msg.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c fmq_msg.c -fPIC -DPIC -o .libs/fmq_msg.o libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c fmq_msg.c -o fmq_msg.o >/dev/null 2>&1 mv -f .deps/fmq_msg.Tpo .deps/fmq_msg.Plo /bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_server.lo -MD -MP -MF .deps/fmq_server.Tpo -c -o fmq_server.lo fmq_server.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_server.lo -MD -MP -MF .deps/fmq_server.Tpo -c fmq_server.c -fPIC -DPIC -o .libs/fmq_server.o fmq_server.c: In function 'sub_patch_add': fmq_server.c:332:13: error: passing argument 3 of 'zhash_insert' discards 'const' qualifier from pointer target type [-Werror] zdir_patch_digest (patch), zdir_patch_vpath (patch)); ^ In file included from /usr/local/include/czmq_library.h:168:0, from /usr/local/include/czmq.h:30, from fmq_server.c:33: /usr/local/include/zhash.h:46:5: note: expected 'void *' but argument is of type 'const char *' zhash_insert (zhash_t *self,const char *key,void *item); ^ cc1: all warnings being treated as errors make[1]: *** [fmq_server.lo] Error 1 make[1]: Leaving directory/home/neotrade/Downloads/filemq/src'
make: *** [check-recursive] Error 1

Both filemq and track crash when delete file from original-directory

Hi,
got a core dump when deleting one file from the original-directory when run "filemq" or "track":
HUGZ:
lt-filemq: zhash.c:219: zhash_insert: Assertion `key' failed.
Aborted (core dumped)

gdb shows:
#4 0x00007ffff7bad4f9 in zhash_insert (self=0x7ffff0002650, key=0x0, value=0x7ffff00046c0) at zhash.c:219
#5 0x00007ffff7deb0a7 in sub_patch_add (self=0x7ffff0001a80, patch=0x7ffff0178cd0) at fmq_server.c:329

0x00007ffff7deca4b in mount_refresh (args=, ctx=, pipe=) at fmq_server.c:409

in #5, the patch->op = patch_delete; patch->digest = 0;
looks like the sub_patch_add() doesn't handle the patch_delete case correctly?

Thanks,
-Steve

Assertion failure during server processing of icanhaz message from client

I have the latest versions of libzmq, czmq, and filemq all building on Windows 7, and am running filemq_selftest using two separate processes, one as the server via the -s option, and the other as the client.

When the server starts, it creates .\fmqroot\send. I drop a file in that directory.
When I start the client, they start chattering to sync up.

On receipt of the icanhaz message from the client, the server gets to this point in the call stack:

store_client_subscription()
    mount_sub_store()
        sub_patch_add()

...in the call to sub_patch_add(), the server first calls zdir_patch_digest_set(), which apparently intends to generate a SHA1 hash for the file I added. However, to generate the hash, it seems to want to use openssl to do this, which is not currently available on my system, and wasn't listed as a build dependency for czmq, where zdir_patch_digest_set() is implemented.

The result is that, since there is no implementation available to calculate the SHA1 hash, zdir_patch_digest_set() is a no-op, so the digest field of __zdir_patch_t is left with a value of 0.

Since this file is a new entry in the server cache, it calls zhash_insert(), using the result of zdir_patch_digest() as the key value, which has a value of 0. zhash_insert then fails on an assertion requiring a non-zero value for the key.

Seems like the dependency on OpenSSL for filemq to operate correctly should be explicitly mentioned so that the czmq libraries can be built properly so this error does not occur.

Or, is there some other alternative I can use as a workaround in the meantime?

Also, it seems like the selftest for czmq and/or filemq should include test functions that would catch this problem.

Failed to build FileMQ, "cannot link with -lczmq", czmq is installed already.

Hi I getting this error: "configure: error: cannot link with -lczmq, install libczmq.", on ./configure && make check. I have installed libzmq, czmq and libsodium right before compilation, so it's the latest from repo. How can I fix it?

compilation issue

Hello,

This might a trivial issue ... but i am not able to compile filemq.

Got the following errors:
Making all in src
make[1]: Entering directory /root/git/filemq/src' make all-am make[2]: Entering directory/root/git/filemq/src'
/bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c -o fmq_client.lo fmq_client.c
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -fPIC -DPIC -o .libs/fmq_client.o
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -o fmq_client.o >/dev/null 2>&1
mv -f .deps/fmq_client.Tpo .deps/fmq_client.Plo
/bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c -o fmq_msg.lo fmq_msg.c
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_GNU_SOURCE -DLINUX -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c fmq_msg.c -fPIC -DPIC -o .libs/fmq_msg.o
In file included from fmq_msg.c:43:0:
../include/fmq_msg.h:115:21: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:119:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:141:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:146:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:151:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:154:9: error: unknown type name 'zhash_t'
../include/fmq_msg.h:155:9: error: unknown type name 'zhash_t'
../include/fmq_msg.h:158:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:163:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:165:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:166:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:169:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:171:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:172:9: error: unknown type name 'byte'
../include/fmq_msg.h:174:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:175:9: error: unknown type name 'byte'
../include/fmq_msg.h:176:9: error: unknown type name 'zhash_t'
../include/fmq_msg.h:177:9: error: unknown type name 'zchunk_t'
../include/fmq_msg.h:180:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:185:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:190:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:195:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:200:1: error: unknown type name 'zmsg_t'
../include/fmq_msg.h:220:9: error: unknown type name 'zhash_t'
../include/fmq_msg.h:221:9: error: unknown type name 'zhash_t'
../include/fmq_msg.h:232:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:233:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:239:9: error: unknown type name 'uint64_t'
../include/fmq_msg.h:240:9: error: unknown type name 'byte'
../include/fmq_msg.h:242:9: error: unknown type name 'uint64_t'

[...]

I have tried with :
git clone git://github.com/hintjens/filemq.git
cd filemq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig

as specified, after installing libsodium, libzmq and czmq in /usr/local . (from latest git)
I also tried with stable version of libzmq (4.0.4)
and czmq tag 2.2.0

What is my mistake ? Sorry about my poor C skills ... and thanks for your help.

"make check" error

Hello,
when I run "make check", I got the following error. Would you please help me with that?
Thanks
<xmp>
Making check in src
/bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_DARWIN_C_SOURCE -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c -o fmq_client.lo fmq_client.c
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_DARWIN_C_SOURCE -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -fno-common -DPIC -o .libs/fmq_client.o
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_DARWIN_C_SOURCE -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_client.lo -MD -MP -MF .deps/fmq_client.Tpo -c fmq_client.c -o fmq_client.o >/dev/null 2>&1
mv -f .deps/fmq_client.Tpo .deps/fmq_client.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_DARWIN_C_SOURCE -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c -o fmq_msg.lo fmq_msg.c
libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -pedantic -Werror -Wall -Wno-unused -D_DARWIN_C_SOURCE -D_REENTRANT -D_THREAD_SAFE -g -O2 -MT fmq_msg.lo -MD -MP -MF .deps/fmq_msg.Tpo -c fmq_msg.c -fno-common -DPIC -o .libs/fmq_msg.o
fmq_msg.c: In function 'fmq_msg_decode':
fmq_msg.c:388:9: error: implicit declaration of function 'zsys_error' [-Werror=implicit-function-declaration]
fmq_msg.c: In function 'fmq_msg_encode':
fmq_msg.c:432:17: error: implicit declaration of function 'zhash_first' [-Werror=implicit-function-declaration]
fmq_msg.c:432:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:434:21: error: implicit declaration of function 'zhash_cursor' [-Werror=implicit-function-declaration]
fmq_msg.c:434:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char ' but argument is of type 'int'
fmq_msg.c:436:21: error: implicit declaration of function 'zhash_next' [-Werror=implicit-function-declaration]
fmq_msg.c:436:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:445:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:447:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char *' but argument is of type 'int'
fmq_msg.c:449:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:483:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:485:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char *' but argument is of type 'int'
fmq_msg.c:487:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:548:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:550:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char *' but argument is of type 'int'
fmq_msg.c:550:21: error: passing argument 2 of '__builtin___memcpy_chk' makes pointer from integer without a cast [-Werror]
fmq_msg.c:550:21: note: expected 'const void *' but argument is of type 'int'
fmq_msg.c:552:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:559:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:561:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char *' but argument is of type 'int'
fmq_msg.c:561:21: error: passing argument 2 of '__builtin___memcpy_chk' makes pointer from integer without a cast [-Werror]
fmq_msg.c:561:21: note: expected 'const void *' but argument is of type 'int'
fmq_msg.c:563:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:590:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:592:21: error: passing argument 1 of 'strlen' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq_prelude.h:209:0,
from /opt/local/include/czmq.h:19,
from fmq_msg.c:43:
/usr/include/string.h:82:9: note: expected 'const char *' but argument is of type 'int'
fmq_msg.c:592:21: error: passing argument 2 of '__builtin___memcpy_chk' makes pointer from integer without a cast [-Werror]
fmq_msg.c:592:21: note: expected 'const void *' but argument is of type 'int'
fmq_msg.c:594:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c: In function 'fmq_msg_recv':
fmq_msg.c:659:5: error: implicit declaration of function 'zsock_resolve' [-Werror=implicit-function-declaration]
fmq_msg.c:659:5: error: passing argument 1 of 'zsocket_type' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq.h:52:0,
from fmq_msg.c:43:
/opt/local/include/zsockopt.h:126:17: note: expected 'void *' but argument is of type 'int'
fmq_msg.c:666:5: error: passing argument 1 of 'zsocket_type' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq.h:52:0,
from fmq_msg.c:43:
/opt/local/include/zsockopt.h:126:17: note: expected 'void *' but argument is of type 'int'
fmq_msg.c: In function 'fmq_msg_recv_nowait':
fmq_msg.c:684:5: error: passing argument 1 of 'zsocket_type' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq.h:52:0,
from fmq_msg.c:43:
/opt/local/include/zsockopt.h:126:17: note: expected 'void *' but argument is of type 'int'
fmq_msg.c:691:5: error: passing argument 1 of 'zsocket_type' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq.h:52:0,
from fmq_msg.c:43:
/opt/local/include/zsockopt.h:126:17: note: expected 'void *' but argument is of type 'int'
fmq_msg.c: In function 'fmq_msg_send':
fmq_msg.c:718:5: error: passing argument 1 of 'zsocket_type' makes pointer from integer without a cast [-Werror]
In file included from /opt/local/include/czmq.h:52:0,
from fmq_msg.c:43:
/opt/local/include/zsockopt.h:126:17: note: expected 'void *' but argument is of type 'int'
fmq_msg.c: In function 'fmq_msg_print':
fmq_msg.c:1144:13: error: implicit declaration of function 'zsys_debug' [-Werror=implicit-function-declaration]
fmq_msg.c:1161:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:1164:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:1171:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:1174:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:1203:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c:1206:28: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
fmq_msg.c: In function 'fmq_msg_test':
fmq_msg.c:1798:5: error: unknown type name 'zsock_t'
fmq_msg.c:1798:5: error: implicit declaration of function 'zsock_new' [-Werror=implicit-function-declaration]
fmq_msg.c:1798:22: error: initialization makes pointer from integer without a cast [-Werror]
fmq_msg.c:1800:5: error: implicit declaration of function 'zsock_connect' [-Werror=implicit-function-declaration]
fmq_msg.c:1802:5: error: unknown type name 'zsock_t'
fmq_msg.c:1802:23: error: initialization makes pointer from integer without a cast [-Werror]
fmq_msg.c:1804:5: error: implicit declaration of function 'zsock_bind' [-Werror=implicit-function-declaration]
fmq_msg.c:2046:5: error: implicit declaration of function 'zsock_destroy' [-Werror=implicit-function-declaration]
cc1: all warnings being treated as errors
make[1]: ** [fmq_msg.lo] Error 1
make: *** [check-recursive] Error 1
</xmp>

SIGSEGV during running test building filemq jar file (using libzmq in 4.1.x zeromq branch)

hi , some env parameters:

[amilkowski@localhost java]$ java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
[amilkowski@localhost java]$

[amilkowski@localhost java]$ uname -a
Linux localhost.localdomain 3.18.17-13.el7.x86_64 #1 SMP Wed Jul 22 14:20:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

using zeromq (lib) from https://github.com/zeromq/zeromq4-1
and using filemq from git master

T E S T S

Running org.filemq.TestFmqHash

• fmq_hash: OK
  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec
  Running org.filemq.TestFmqFile
• fmq_file: OK
  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.008 sec
  Running org.filemq.TestFmqMsg
• fmq_msg: #

  A fatal error has been detected by the Java Runtime Environment:

  SIGSEGV (0xb) at pc=0x00007f81b279da10, pid=25052, tid=140195789207296

  JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)

  Java VM: Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed mode linux-amd64 compressed oops)

  Problematic frame:

  C [libzmq.so.5+0x36a10] non-virtual thunk to zmq::socket_base_t::hiccuped(zmq::pipe_t*)+0x10

  Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

  An error report file with more information is saved as:

  /opt/local/src/git/zeromq/filemq/java/hs_err_pid25052.log

  If you would like to submit a bug report, please visit:

  http://bugreport.sun.com/bugreport/crash.jsp

  The crash happened outside the Java Virtual Machine in native code.

  See problematic frame for where to report the bug.

  /bin/sh: line 1: 25052 Aborted (core dumped) /usr/java/jdk1.7.0_75/jre/bin/java -Djava.library.path=/opt/local/src/git/zeromq/jzmq/dist/lib -jar /opt/local/src/git/zeromq/filemq/java/target/surefire/surefirebooter2020896169079705910.jar /opt/local/src/git/zeromq/filemq/java/target/surefire/surefire6545309803011920537tmp /opt/local/src/git/zeromq/filemq/java/target/surefire/surefire_01891127949912442224tmp

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.522 s
[INFO] Finished at: 2015-08-10T16:21:19-04:00
[INFO] Final Memory: 39M/3926M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project filemq: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

czmq path issue

When I build filemq, the file libfmq.la contains an incorrect path to czmq.

I run configure like so:

./configure --with-libzmq=/usr/local/libzmq3 --with-libczmq=/usr/local/czmq --prefix=/usr/local/filemq

Then I run make check install

Inside /usr/local/filemq/lib/libfmq.la I see this line:

dependency_libs=' -L/usr/local/czmq/lib -L/usr/local/libzmq3/lib -lm -lcrypto /usr/local/lib/libczmq.la /usr/local/libzmq3/lib/libzmq.la -lrt -lpthread'

/usr/local/lib/libczmq.la doesn't exist, and should be instead /usr/local/czmq/lib/libczmq.la

This was causing the Zyre build to fail. Correcting the path above fixed my issue.

My environment is Fedora 18 Linux, x86_64

osx 10.7 sha1_init/update/final deprecated

Building fmq on 10.7/clang, i get few deprecated function in use. This is specific to osx and i don't know why those openssl functions are deprecated on 10.7.

fmq_hash.c:47:5: error: 'SHA1_Init' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Init (&self->context);
^
fmq_hash.c:74:5: error: 'SHA1_Update' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Update (&self->context, buffer, length);
^
fmq_hash.c:85:9: error: 'SHA1_Final' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Final (self->hash, &self->context);

patch: https://gist.github.com/4466305

Compilation errors in fmq_client.c and fmq_server.c

I followed the directions for downloading and installing:

git clone git://github.com/hintjens/filemq.git
cd filemq
./autogen.sh
./configure && make check

I got the following errors:

fmq_client.c: In function 'fmq_client_subscribe':
fmq_client.c:149:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_client.c: In function 'fmq_client_set_inbox':
fmq_client.c:161:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_client.c: In function 'fmq_client_set_resync':
fmq_client.c:172:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_client.c: In function 'process_the_patch':
fmq_client.c:599:13: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
make[2]: *** [fmq_client.lo] Error 1

fmq_server.c: In function 'fmq_server_bind':
fmq_server.c:116:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_server.c: In function 'fmq_server_publish':
fmq_server.c:133:5: error: too many arguments to function 'zstr_sendm'
/usr/local/include/zstr.h:56:5: note: declared here
fmq_server.c:134:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_server.c: In function 'fmq_server_set_anonymous':
fmq_server.c:145:5: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
fmq_server.c: In function 'server_control_message':
fmq_server.c:666:9: error: too many arguments to function 'zstr_send'
/usr/local/include/zstr.h:51:5: note: declared here
make[2]: *** [fmq_server.lo] Error 1

Clarify License

Hello,

All source files, except src/filemq_client.c and src/filemq_server.c, has MPL Version 2.0 license header. LICENSE file has MPL version 2.0 text. That is OK.

But on github webpage top right corner GPLv3 license is mentioned. And there are two license files COPYING and COPYING.LESSER which says GPLv3 and LGPLv3. This seems to be inconsistent.

It seems that filemq has MPL version 2.0 license because almost all sources has MPL Version 2.0 license header. But could you please clarify license removing inconsistency ?

Thanks,
Rahul

FileMQ uses deprecated CZMQ API's

There are a lot of references to deprecated czmq API's in FileMQ. Not sure if it will be acceptable, but I am working on fixing this for the c code. I do not have any expertise for Java.

CPU Usage went up to 99%

generating msg, client and server files breaks filemq

Problem: If I generate the protocol files by calling ./generate in filemq/model the compilation of the library breaks. Error message:

fmq_msg.c:772:1: error: conflicting types for 'fmq_msg_send_ohai'
In file included from fmq_msg.c:27:0:
../include/fmq_msg.h:108:5: note: previous declaration of 'fmq_msg_send_ohai' was here

The generated files differ considerably to the ones in master.

Any ideas?

filemq doesn't build with gcc 7.1 due to Wformat-truncation

Hi,

From: src/fmq_server_engine.inc:
//  Set log file prefix; this string will be added to log data, to make
//  log data more searchable. The string is truncated to ~20 chars.

The code intend to truncate the string to ~20 chars but this behaviour
throw an error with new compiler (gcc >= 7.1).

See:
http://autobuild.buildroot.net/results/434/43452430edf44cbcef6ea3a5ab80ac0ff406208f

This can be avoided by disabling this warning with -Wno-format-truncation.

Best regards,
Romain

a

Server sends full contents, ignoring cache

When a new connection is made with RESYNC=1, the server ignores the cache and sends all files. It should only send the files not already present on the client.

The check should (probably) be done in mount_sub_store (fmq_server.c:464-473).