curent / dime Goto Github PK

I think it's safe to say this program will only be run on trusted networks, but better safe than sorry, right?

After adding the Python interfaces, it might be helpful to create a docker image that runs the server.

e.g.

d.sendlocal('matlab', 'pi', 3.14);
d.synclocal()

% struct with fields:
% 
%   pi: 3.1400

Make it so, instead of a Matlab client joining with a unique name, it joins a named group. Messages sent to a group are sent to all members of said group. Clients should be able to join more than one group. Optionally, sub-groups (groups of groups) may be an option, but this might be too much to maintain.

Hi Good Morning,

Can anyone please help me in running Makefile. I'm facing following issue.

As mentioned in the comment, I uncommented the following lines:

I'm using mingw compiler on my system.

Your help will be appreciable.

Thanks.

e.g. the server could accept connections from both a TCP socket and a Unix domain socket, and these connections would both be communicating on the same server.

Waits for a group to be populated on the server-side

Something like:

import dime
import numpy as np

d = dime.DimeClient("python", "ipc", "/tmp/dime.sock")

d["a"] = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
d.send_var("python", "a")
del d["a"]
d.sync()

print(d["a"])
# [[1 2 3]
#  [4 5 6]
#  [7 8 9]]

Note that #3 needs to be completed as a dependency. Also, the server should be aware of its clients' platforms, and instruct them to change their serialization method depending on what other types of clients are connected.

See #11 for more info

Right now, dime_socket_recvpartial handles both the recv() syscall and interpreting the data from it. However, this could theoretically end up with us getting two messages in one recv() call, and the second message will never be read. These two functionalities need to be split to ensure this doesn't happen

From README.md:

That having been said, multi-threading may not be avoided in the future. A technique used by several HTTP servers to tackle the C10k problem (e.g. Nginx, lighttpd) is to have a fixed-size pool of worker threads that each poll a set of client connections dispatched to them by the main thread. (An excellent article on the I/O performance of threads vs poll/select can be found here.) This approach may be used by this code in the future, if such scalability is desired.

Note that multi-threading would greatly increase the complexity of the server, and might actually make it slower due to locks on tables starving certain threads. Still, for maximum performance and scalability it may be worth considering in the future.

The run-time flag -v should make the server print additional info about who connected, what messages are sent/received, etc.

Command that tells the clients to block until a variable is queued. When one is queued, send the client a message to unblock.

Includes adding Doxygen annotations to source code, and updating README.md with relevant info on compilation/running the server.

Both ANDES and AGVis have messages when they begin to run. Dime has no such message so it may be confusing for new users to know it has been started.

For example, AGVis has a message appear on run like this:

    _   _____   ___     | Version 3.1.3.post40+g51174f2
   /_\ / __\ \ / (_)___ | Python 3.8.17 on Linux, 07/31/2023 03:57:50 PM
  / _ \ (_ |\ V /| (_-< | 
 /_/ \_\___| \_/ |_/__/ | Web-based geographical visualizer.

Working directory: "/home/zmalk/work/agvis"
AGVis will serve static files from directory "/home/zmalk/work/agvis/agvis/static"
at the URL http://127.0.0.1:8810. Open your web browser and navigate to the URL to access the application.

Starting AGVis... Press Ctrl+C to stop.

A few things:

• Buffers for send/recv are allocated dynamically. A static byte array might make more sense if we can ensure a single-threaded, non-reentrant server
• We can skip a potentially expensive memcpy and read directly from the buffer if the contents therein are contiguous
• The ring buffers can be shrunk when they are empty

Instead of using a queue proper, we can just use a dynamic array and clear it every time we read from it (since all inputs are consumed at once)

See also: whether or not a Python client has the same issue

Largely the same motivations and issues as #6

Transfer the documentation to PyData Sphinx Theme, see CURENT/agvis#22

e.g. running

d = dime('matlab', 'ipc', '/tmp/dime.sock');
d.send_var('matlab', 'a');
d.send_var('matlab', 'b');
 ...
d.send_var('matlab', 'z');
d.sync();

works fine, but

d = dime('matlab', 'ipc', '/tmp/dime.sock');
d.send_var('matlab', 'a', 'b', ..., 'z');
d.sync();

Blocks forever on the last line. The server is receiving extra 'send' commands after 'sync' is called, so I suspect this is a client-side issue.

If the serialization changes during a send_var call, then the clients should resend their data with the new method. Otherwise, clients on the other end may not be able to decode the variables.

Hi,

I'm using DimeClient with python.
What kind of bindata is required for the sync_r method to receive from the server?

Please help!

Without packing DiME as a package, it is challenging to install necessary DiME dependencies for users.

DiME has the following dependencies build-essential, autotools-dev, autoconf, and libev-dev, to support the installation of OpenSSL, zlib, and Jansson.

To follow the installation style of ANDES and AMS, it can be better to pack a C project with the necessary dependencies and publish it on Conda and PyPI.

The installation of DiME is expected to be simple, like conda install dime or pip install dime.

Then starting a DiME server in the command line can be like this:

dime --path /tmp/dime --port 8818

Similarly, using DiME in Python can be like this:

import dime

# NOTE: start a DiME server
server = dime.server(path='/tmp/dime', port=8818)

# NOTE: start a DiME client and join a server
client = dime.client(protocol, varargin)
client.join(server)
...

With this feature, we can improve the usability of DiME and integrate AGVis as a built-in visualizer for ANDES and AMS.

Here are some possible references for packing DiME:
Conda documentation on creating and publishing packages
PyPI documentation on creating and publishing packages
Blog post on publishing C libraries on Conda and PyPI
Github repository with example setup.py files for packaging C projects
Conda-forge documentation on building and uploading packages
PyPI documentation on using twine to upload packages

See Interactive use through Python code for more discussion.

This will likely save memory and definitely save some CPU time.

Hi,

I have posted a query on ANDES community. Below is the link for reference.

https://github.com/cuihantao/andes/discussions/303

Your help will be really appreciable.

It appears sometimes dime likes to run forever when initialized. It would be good to add a timeout of some arbitrary length of time to avoid multiple hidden dime servers running in the background for no reason.

This is a problem because the Unix socket persists after the program is closed unless it is unlinked explicitly

Hi,

While trying to connect using tcp protocol, I'm facing below issue.

Any help will be appreciable.

Thanks,
Ammy

Also, use a dual-stack IPv4/IPv6 socket when supported by the host OS.

Depends on #34

Make the hash table implementation preserve the order in which elements are inserted, and iterate through them via that order, similar to Python's collections.OrderedDict. Currently two possible ways of doing this:

• Create an implicit doubly-linked list within the hash table. Has the issue of maintaining said list when elements are relocated
• Do what Python does, and split the table into two arrays (one for key-value pairs, and one for lookup based on the hash value). Has issues with locality for pure lookup purposes

Right now, all the tests are managed by a bunch of shell scripts, and that's becoming a little cumbersome

Note that this will in all likelihood cut our throughput in half, but we're already some 40x over our goal with that so it's probably fine

Changing the workspace from 'base' to 'caller' breaks the current Matlab tests.

(Pretty sure Matlab doesn't even support this protocol)

Ties in with #9 to some extent - a thread-local string unifying errors from Jansson and the C standard library

This was the original plan for #35, but since ltbvis assumes a Chrome browser for proper functionality, and Chrome has non-standard raw TCP socket support, that will be implemented first.

Currently, data is marshalled via Matlab's undocumented getByteStreamFromArray and getArrayFromByteStream. Although this is the fastest method for pure Matlab clients, other types of clients (i.e. a Python binding) may exist in the future, (and relying on undocumented functions is as stable and future-proof as a box of moldy dynamite) so an open standard would be preferable. Ideally, the server would send a signal to continue using getByteStreamFromArray/getArrayFromByteStream until a non-Matlab client connects, at which point the alternate marshaller is used.

It seems that Nich West has developed tests for DiME in the folder test.

Setting these tests automatically running for pull requests will be helpful for maintenance.

Reference:
Continuous integration
Required status check