sleepybishop / nanorq

fast compact raptorq (rfc6330) codec in c

🌕 📦📦📦.📦 🛰️ 📦📦 📡⛵

nanorq's Issues

Time testing

Hi, I would like to ask if the precomputation in encoding refers to the pre-encoding in Raptor codes? If so, should it be included in the timing test function

Make: missing gf2.h

Hi,

I am trying to compile the code but got this error when running make:
$ make
cc -O3 -g -std=c99 -Wall -I. -Iinclude -Ideps/oblas -march=native -funroll-loops -ftree-vectorize -fno-inline -fstack-protector-all -D_DEFAULT_SOURCE -D_FILE_OFFSET_BITS=64 -c -o encode.o encode.c
cc -O3 -g -std=c99 -Wall -I. -Iinclude -Ideps/oblas -march=native -funroll-loops -ftree-vectorize -fno-inline -fstack-protector-all -D_DEFAULT_SOURCE -D_FILE_OFFSET_BITS=64 -c -o lib/bitmask.o lib/bitmask.c
cc -O3 -g -std=c99 -Wall -I. -Iinclude -Ideps/oblas -march=native -funroll-loops -ftree-vectorize -fno-inline -fstack-protector-all -D_DEFAULT_SOURCE -D_FILE_OFFSET_BITS=64 -c -o lib/io.o lib/io.c
lib/io.c: In function ‘ioctx_mmap_file’:
lib/io.c:365:5: warning: ignoring return value of ‘ftruncate’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
365 | ftruncate(fd, mapsize);
| ^~~~~~~~~~~~~~~~~~~~~~
lib/io.c: In function ‘mmapio_destroy’:
lib/io.c:321:5: warning: ignoring return value of ‘ftruncate’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
321 | ftruncate(_io->fd, _io->filesize);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/io.c: In function ‘mmapio_seek’:
lib/io.c:211:7: warning: ignoring return value of ‘ftruncate’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
211 | ftruncate(_io->fd, _io->filesize + _io->filesize);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc -O3 -g -std=c99 -Wall -I. -Iinclude -Ideps/oblas -march=native -funroll-loops -ftree-vectorize -fno-inline -fstack-protector-all -D_DEFAULT_SOURCE -D_FILE_OFFSET_BITS=64 -c -o lib/params.o lib/params.c
cc -O3 -g -std=c99 -Wall -I. -Iinclude -Ideps/oblas -march=native -funroll-loops -ftree-vectorize -fno-inline -fstack-protector-all -D_DEFAULT_SOURCE -D_FILE_OFFSET_BITS=64 -c -o lib/precode.o lib/precode.c
In file included from include/precode.h:8,
from lib/precode.c:1:
include/wrkmat.h:7:10: fatal error: gf2.h: No such file or directory
7 | #include "gf2.h"
| ^~~~~~~
compilation terminated.
make: *** [: lib/precode.o] Error 1

Would you please advise what I can do to resolve it?

Thanks a lot!

Question

How to know the total number of repair symbols, when the encoder is initialized?

Decode support

Hi,
I noticed that the decode example only exists in the stable branch, not in the dev branch.
Does that mean decode is not yet available after the refactoring or only that there is no example yet?

lib/precode.c:121: precode_matrix_choose: Assertion `0' failed.

First, thank you for your work on the library.

I was trying to write a test case that covers recovery failure, and I received the assertion error in the title.

In summary, using exactly 10 source symbols of 1448 bytes and dropping ESI 4, 5, 6, 10, and 11.

Per a suggestion in another issue, I was able to modify benchmark.c to reproduce this:
benchmark.zip

./benchmark 1448 10 50
adding symbol 0 to sbn 0.
adding symbol 1 to sbn 0.
adding symbol 2 to sbn 0.
adding symbol 3 to sbn 0.
adding symbol 7 to sbn 0.
adding symbol 8 to sbn 0.
adding symbol 9 to sbn 0.
adding symbol 12 to sbn 0.
adding symbol 13 to sbn 0.
adding symbol 14 to sbn 0.
benchmark: lib/precode.c:121: precode_matrix_choose: Assertion `0' failed.
Aborted (core dumped)

FWIW, the Rust RaptorQ implementation seems to be able to perform a successful recovery with this data set.

Make 'stable' branch as default

Using ioctx_from_mem to create a non file based transfer FEC

Hi, Im trying to create redundancy for a video streaming device over GPRS LTE so that at low bandwidth or low signal areas RaptorQ can persist connections and act adaptively to spikes of low or high performance, but when i am trying to implement this library with ioctx_from_mem i have come across several issues that have made it hard for me to implement this library in my project. Is there any way you can help me out with from_mem implementation?

My guidelines are simple, i will be using a udp connection to transfer the data blocks that nanorq generates and repack them at the transmission point. For this i have created a linked list data structure that consists of Node>Blocks>Packets. My problem starts with encoding and decoding processes. When i encode with from_mem using an array of randomly generated uint8_t values i seem to have alot of zeroes for a size of uint8_t[10000] and the generated data size goes up nearly 3x! From my research RaptorQ library exchanges cpu usage to data transmitted which makes me question whether i was successful in my implementation of the encoding.

Assuming that i was successful in the encoding process when i try to decode the first packet gets decoded with no problem, but the rest fails. I am guessing this is on my part as well since i am a novice to C. Also i failed to understand where the output for this data is so that it can be used, i have guessed that the decoder_add_symbol restores if any symbols are missing and outputs the fixed packet to the packet data but when i inspected the code it seems that the io gets manipulated and not the packet.

If you can create a guideline for me to follow that would be amazing. I can also share the current version of my code if you want to inspect it

Thank you for your time.

Add Licence Info

Hi,

Could you please add some information about which licence you are using for this project? I may use it in a project and want to make sure it's all in line!

Many thanks.

error

when i testing, happen error:adding symbol 16777217 failed.

Rename sched.h/c

There is already file named sched.h on the system. Including nanorq sources in some projects leading to broken build. Renaming file to other name, like schedx.h fixes the problem.

P.S. Ubuntu 22.04, almost everything is by default

'octmul.h' file not found

Can't compile for target ARM since octmul.h referenced from oblas_neon.c is missing.

Build Error - Intrinsic _mm256_loadu2_m128i Not Recognized

I am unable to build this project. Tried it on several Linux machine and the build errors are the same. It seems it does not recognize the intrinsic _mm256_loadu2_m128i.

I am using gcc 9.4.0:
$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Please advise how I can build this.

$ make
cc -O3 -g -std=c11 -Wall -Iinclude -Ideps/ -fPIC -march=native -funroll-loops -ftree-vectorize -Wno-unused -Wno-sequence-point -fstack-protector-all -DOBLAS_AVX2 -c -o deps/obl/oblas_lite.o deps/obl/oblas_lite.c
deps/obl/oblas_lite.c: In function ‘obl_axpy’:
deps/obl/oblas_lite.c:122:33: warning: implicit declaration of function ‘_mm256_loadu2_m128i’; did you mean ‘_mm256_loadu_si256’? [-Wimplicit-function-declaration]
122 | const __m256i urow_lo = _mm256_loadu2_m128i((__m128i *)u_lo, (__m128i *)u_lo);
| ^~~~~~~~~~~~~~~~~~~
deps/obl/oblas_lite.c:234:9: note: in expansion of macro ‘OBL_SHUF’
234 | OBL_SHUF(obl_axpy, a, b, OBL_SHUF_XOR);
| ^~~~~~~~
deps/obl/oblas_lite.c:122:33: error: incompatible types when initializing type ‘__m256i’ {aka ‘const __vector(4) long long int’} using type ‘int’
122 | const __m256i urow_lo = _mm256_loadu2_m128i((__m128i *)u_lo, (__m128i *)u_lo);
| ^~~~~~~~~~~~~~~~~~~
deps/obl/oblas_lite.c:234:9: note: in expansion of macro ‘OBL_SHUF’
234 | OBL_SHUF(obl_axpy, a, b, OBL_SHUF_XOR);
| ^~~~~~~~
deps/obl/oblas_lite.c:123:33: error: incompatible types when initializing type ‘__m256i’ {aka ‘const __vector(4) long long int’} using type ‘int’
123 | const __m256i urow_hi = _mm256_loadu2_m128i((__m128i *)u_hi, (__m128i *)u_hi);
| ^~~~~~~~~~~~~~~~~~~
deps/obl/oblas_lite.c:234:9: note: in expansion of macro ‘OBL_SHUF’
234 | OBL_SHUF(obl_axpy, a, b, OBL_SHUF_XOR);
| ^~~~~~~~
deps/obl/oblas_lite.c: In function ‘obl_scal’:
deps/obl/oblas_lite.c:122:33: error: incompatible types when initializing type ‘__m256i’ {aka ‘const __vector(4) long long int’} using type ‘int’
122 | const __m256i urow_lo = _mm256_loadu2_m128i((__m128i *)u_lo, (__m128i *)u_lo);
| ^~~~~~~~~~~~~~~~~~~
deps/obl/oblas_lite.c:240:5: note: in expansion of macro ‘OBL_SHUF’
240 | OBL_SHUF(obl_scal, a, a, OBL_NOOP);
| ^~~~~~~~
deps/obl/oblas_lite.c:123:33: error: incompatible types when initializing type ‘__m256i’ {aka ‘const __vector(4) long long int’} using type ‘int’
123 | const __m256i urow_hi = _mm256_loadu2_m128i((__m128i *)u_hi, (__m128i *)u_hi);
| ^~~~~~~~~~~~~~~~~~~
deps/obl/oblas_lite.c:240:5: note: in expansion of macro ‘OBL_SHUF’
240 | OBL_SHUF(obl_scal, a, a, OBL_NOOP);
| ^~~~~~~~
deps/obl/oblas_lite.c: In function ‘obl_axpy’:
deps/obl/oblas_lite.c:121:23: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
121 | const __m256i mask = _mm256_set1_epi8(0x0f);
| ^~~~
deps/obl/oblas_lite.c:234:9: note: in expansion of macro ‘OBL_SHUF’
234 | OBL_SHUF(obl_axpy, a, b, OBL_SHUF_XOR);
| ^~~~~~~~
make: *** [: deps/obl/oblas_lite.o] Error 1

Is is possible to upgrade lib oblas to the latest reversion for nanorq ?

Hello,

Thanks for your good job on this projects, I notice that lib oblas has lots of updates, Is it possible to use the lastest reversion for nanorq ?

Thanks.

gf2.h not found

Hi,
I've added your oblas libraries under deps, but it doesn't contain the gf2.h file. When I try to make nanorq I get the following error:
In file included from include/precode.h:8:0,
from lib/precode.c:1:
include/wrkmat.h:7:10: fatal error: gf2.h: No such file or directory
#include "gf2.h"
^~~~~~~

Where can I get gf2.h. please?
Bob.

include/util.h:8:10: fatal error: octmat.h:

I have this error. Any help? Thanks

include/util.h:8:10: fatal error: octmat.h:

can not interop with libraptorq

I test the lib and found it is really faster than libraptorq, but unfortunately, the data encoded by libraptorq can not decoded by this lib and vice versa. Is it really comply with the RFC?

nanorq_decode_block() sometimes return error data

I try the lib and found that nanorq_decode_block() sometimes return error data. when overhead = 0, the decoder should have a probability of 1/256 that the repair symbol is not enough and should return 0 for couldn't decode. but this lib always return data, and the data is not correct.

question

How should I use it