cyroxin / elembuf

An efficient and simple to use buffer/array for data manipulation.

Elembuf

An efficient and simple to use buffer/array for data manipulation.

Description

A circular buffer can be thought of as an array that can eliminate program side copying, making data reads from sockets or other IO more efficient. The simple idea is that if elements/array items are popped/removed from the front, the free slots can be used to fill the back of the array without moving the existing data. A simple way to understand the concept is that a circular buffer is a faster concatenator (concat/cat/~) as all free data reserves are more easily taken advantage of, however, circular buffers are limited to a maximum size.

Elembuf is an implementation of a circular buffer. It is however different from a regular circular buffer in that it is as compact as an array, infact it is indistinquishable from a regular dynamic array and it can even be directly cast to it. Allocated memory of an Elembuf is internally memory mapped to work like a mirror of itself and it achieves higher speeds by being allocated to a custom memory position. The mirroring properties of the implementation give additional benefits to encryption and compression algorithms. You may however use Elembuf for any purpose which a regular array is used for, such as when parsing data, reading files or assembling data. Elembuf has a page sized or larger maximum array byte length and works directly with the OS, thus it does not use the garbage collector.

Because construction speed is slightly slower than with a regular array, reusage is key to the function of an Elembuf. You should construct the buffer at the start of the program, or at server boot, and change the source where data is received instead of deconstructing the array/buffer. This way you can take advantage of the filling speed.

Elembuf currently works for Windows, Linux, Mac and other Posix compatible systems.

Getting Started

Windows/Mac

You can download the repo and open the solution file in the repository called "Elembuf.sln" (VS17 & VS19) with visuald in visual studio. The source files should automatically link and you may easily edit the source code and add your own code into the solution.

Linux

Linux users may use vscode! Although not as good of an experience as visual studio, it does come with most functionality out of the box. It is as simple as opening the project folder in vscode and installing two extensions (code-d & C/C++ for Visual Studio Code).

Note that you are not restricted to the use of a specific IDE or editor. The source files can be used as is, regardless of what environment is used, as long as they are linked properly.

Dub

You can use elembuf as a project dependency by creating a file called app.d with the following contents and running it using dub:

#!/usr/bin/env dub
/+ dub.sdl:
name "app"
dependency "elembuf" version="~>1.2.3"
+/

module app;

import elembuf;
import std.stdio;

void main() {
    auto bufchar = buffer(""); // char[]
    assert(bufchar == "");
  
    bufchar ~= "Hello world!"; // Does not use the GC
    bufchar.writeln;
}

$ dub app.d

Hello world!

Benchmarks

Comparing an optimized array against elembuf in concating data.

Windows 10 - AMD A8-6410 x64 - 4GB memory - LDC release, 100k runs.
Bench [buffer construction + destr]:75 ╬╝s and 3 hnsecs
Bench [buffer runtime]:167 ╬╝s and 7 hnsecs
Bench [array construction + destr]:15 ╬╝s and 7 hnsecs
Bench [array runtime]:185 ╬╝s and 3 hnsecs
Reuses needed: 3

Linux MX-18.3 (Linux) - AMD A8-6410 x64- 4GB memory - DMD release -nobounds, 100k runs.
Bench [buffer construction + destr]:24 μs and 4 hnsecs
Bench [buffer runtime]:18 μs and 9 hnsecs
Bench [array construction + destr]:2 μs and 9 hnsecs
Bench [array runtime]:19 μs and 3 hnsecs
Reuses needed: 53

Linux MX-18.3 (Posix) - AMD A8-6410 x64 - 4GB memory - DMD release -nobounds, 100k runs.
Bench [buffer construction + destr]:36 μs and 3 hnsecs
Bench [buffer runtime]:19 μs
Bench [array construction + destr]:2 μs and 9 hnsecs
Bench [array runtime]:19 μs and 4 hnsecs
Reuses needed: 83

Contributions

Feel free to contribute by sending a pull request or sending in bug reports through issues. If you are unsure on what to help with, you can gather ideas from projects. Remember to include tests in your pull requests!

Maintainers

• Cyroxin - Cyroxin

elembuf's Issues

Program exited with code -11

I still having trouble using this lib, now I'm doing:

    cast(BufferT)elements << [value];

https://github.com/mingwugmail/liblfdsd/blob/master/comparison/elembuf_sync_queue.d#L30

$ make elembuf_sync_q
DFLAGS="-O4 --release --boundscheck=off" /project/ldc2/bin/dub elembuf_sync_queue.d

Program exited with code -11

Can you check what's going wrong, or am I using the lib correctly.

add align(8) to mail

https://github.com/Cyroxin/Elembuf/blob/master/source/buffer.d#L292

		__gshared ptrdiff_t mail = 0; // Thread sync variable, must fit a pointer.

Since this var is atomicLoad & atomicStore, better make it aligned to word size, i.e.

		  align(8) __gshared ptrdiff_t mail = 0; // Thread sync variable, must fit a pointer.

can not decl shared Buffer!() bufchar ?

$ cat cb.d

#!/usr/bin/env dub
/+ dub.sdl:
name "app"
dependency "elembuf" version="~>1.1.1"
+/

import buffer;

void main() {
        shared Buffer!() bufchar = Buffer!()();
        assert(bufchar == "");
}

compile error:

$ dub.exe cb.d
cb.d(10,19): Error: cannot implicitly convert expression opCall(((Buffer!(char, false) __tmpfordtor7 = opCall();) , __tmpfordtor7).buf) of type Buffer!(char, fal
se) to shared(Buffer!(char, false))
cb.d(10,19):        bufchar = opCall(((Buffer!(char, false) __tmpfordtor7 = opCall();) , __tmpfordtor7).buf) is the first assignment of bufchar therefore it repr
esents its initialization
cb.d(10,19):        opAssign methods are not used for initialization, but for subsequent assignments
cb.d(11,9): Error: array equality comparison type mismatch, shared(char[]) vs string
C:\project\dmd2\windows\bin64\dmd.exe failed with exit code 1.

ldc2: simple Program exited with code -11

$ cat cb.d

#!/usr/bin/env dub
/+ dub.sdl:
name "app"
dependency "elembuf" version="~>1.1.1"
+/

import buffer;

void main() {
        Buffer!() bufchar = Buffer!()();
        assert(bufchar == "");
}

with ldc2/bin/dub

$ dub cb.d
Program exited with code -11