sociomantic-tsunami / dlsproto Goto Github PK
View Code? Open in Web Editor NEWDistributed Log Store protocol definition, client, fake node, and tests
License: Boost Software License 1.0
Distributed Log Store protocol definition, client, fake node, and tests
License: Boost Software License 1.0
Having two nodes, A
and B
, disconnecting a single node during running GetRange
request, makes GetRange
request never end. Same happens if the request uses a single node and it disconnects - the request never ends.
Since the DLS node never resumes the request, disconnecting the client should simply count towards finished nodes. In general, I think we should take this in the careful consideration when implementing the Continue
after disconnect - this should not be allowed if the client requested stop
.
Remove trusty support from all projects. Trusty is already more than 3 years old and the latest LTS (Xenial) has already been out there for more than a year, so it should be enough to maintain the latest LTS only now.
If the GetRange is waiting on the Stopped message from the node and it never arrives, and yet node stays connected, the entire request will hang and it will not be able to proceed. Simple registering a timer to resume a receive fiber with the timeout message should be enough. Client should then continue with the stopping the request (perhaps it should also raise node_error
).
sociomantic-tsunami/dmqproto#3 discusses a (hopefully) much simpler and more reliable way of implementing a suspendable request. If the reworking of Consume is successful, GetAll should be reworked in the same way.
Client should allow inspecting the number of bytes/records transferred for each connection.
e.g. https://github.com/sociomantic-tsunami/dlsproto/blob/neo/src/dlsproto/client/UsageExamples.d#L323-L330 is logging started
, finished
, and stopped
as errors. This is harmless enough, but makes for a somewhat confusing usage example.
The templates Const, Immut and Inout were used for transitioning from D1 to D2. Now that swarm has been converted, they should be replaced by plain D2 keywords.
Node should provide client with the summary of the iteration, with the information such as total number of records in the range and the number of records in the range matching the given filters.
Similarly to what we already have with Ocean, we should add all other open source projects to dlang Jenkins to make sure new DMDs don't break them.
Sometimes when implementing features of a DLS reader we want to know exactly how much data we're dealing with for a certain range. It guides us in how we end up implementing our algorithms, whether we can keep some X range of data in memory, etc.
The current workaround is that we do these tests manually with getRange. But that of course causes a lot of network traffic for data which is ultimately just discarded.
Ideally it would be supported for for getRange and its filter/regex equivalents.
Like https://github.com/sociomantic-tsunami/dhtproto/blob/neo/src/dhtproto/client/UsageExamples.d.
This makes the examples simpler for end-users to follow.
https://github.com/sociomantic-tsunami/swarm/releases/tag/v4.4.0
The legacy clients have implemented the algorithms for making sure that all nodes are alive and sending traffic. It is implemented via means of monitoring the events when the individual nodes finish running the request and observing the time for the remaining nodes to finish. In the case when client sees that the individual node needs a long time to complete the request, it would "timeout" and it would stop the request.
Ideally, the client should have a mean of aborting the request on a connection that seems staled, without client needing to track the individual nodes behaviours.
To move forward with D2 we need to provide automatically converted libraries for projects that are built only for D2.
Every time a new tag is pushed, we have to convert it to D2, tag the converted version with a +d2
build metadata information appended (so tag v2.3.4
--- conversion D2 ---> v2.3.4+d2
) and push it back (making sure the new tag is not converted again!).
It's often the case that clients can decide if they want to inspect old data, given the sufficient amount of the fresh data. If we allow client to iterate from the end to the start of the time range (in chunks), client can stop iteration if they decide they can ignore the older data.
These requests are already unsupported by the protocol, but they can't be deprecated since
we can't get rid of the deprecations in D1 due to the call chain analysis bug where the deprecated
struct is triggering the deprecation itself. Since they are doing nothing since two last major, we can just
remove them.
This issue is automatically created to act as a card for
Sociomantic D2 migration project board.
All other issues and pull request in this repository that are related to D2
migration should be referenced from this issue. It is highly appreciated
to provide short status reports on any breakthroughs in form of issue comments.
Please adjust this issue column in https://github.com/orgs/sociomantic/projects/4
board to reflect current stage.
If this is not a D project (GitHub language detection is imperfect), please
both close the issue and remove matching card from the linked project.
In case we have several tests running in parallel, each with the separate node, we need to make
client's connect to the different ports. This is possible by overriding DlsTestCase.prepare
method, but it should be easier to configure this (probably via static
variable, configurable from TestRunner
s).
A fairly common usage is for a client to request several chunks of data in different ranges / channels. How exactly they do this varies by application. Some might simply do sequential GetRanges, others might start a certain number of GetRanges in parallel, etc. Existing client code in fact second guesses the DLS, attempting to optimise the throughput by a certain GetRange assignment strategy.
It occurs to me that this is not ideal. The DLS nodes ought to be the ones deciding the most efficient way to serve requests, not the clients. The node implementation can change over time, meaning that clients' attempts at gaming the system might become outmoded or even horribly inefficient.
We should think about adding a request to get data from multiple ranges/channels, allowing the node to decide the best way to serve the data.
The node could inform the client about the current iteration progress. The progress doesn't need to be fine grained, reporting progress in block should be sufficient.
Generally, trying it again will have the same result.
If there's a bug in the node's implementation, or the routing error in the network, the node might not
be able to proceed with sending the data to the client. Perhaps setting the inactivity timeout could abort the iteration on the given node, without user having to setup/maintain inactivity timer in their code (which is different than total-time timeout, as the timer needs to be reset each time we get data from the node).
Swarm implements record batching utility inside swarm.neo.util.Batch
and we should adapt dlsproto to use it. Also, it could be that the usual batch size (of 64k) is very small for the amounts of data such as DLS - the amount of batches received is very high, leading to lots of bandwidth used just for signalling.
14:59:59 ./submodules/dlsproto/src/dlsproto/client/internal/SuspendableRequest.d(474): Deprecation: function swarm.neo.connection.RequestOnConnBase.RequestOnConnBase.EventDispatcher.receiveAndHandleEvents is deprecated - Use nextEvent instead
14:59:59 ./submodules/dlsproto/src/dlsproto/client/internal/SuspendableRequest.d(639): Deprecation: function swarm.neo.connection.RequestOnConnBase.RequestOnConnBase.EventDispatcher.sendReceive is deprecated - Use nextEvent instead
14:59:59 ./submodules/dlsproto/src/dlsproto/client/internal/SuspendableRequest.d(699): Deprecation: function swarm.neo.connection.RequestOnConnBase.RequestOnConnBase.EventDispatcher.receiveAndHandleEvents is deprecated - Use nextEvent instead
The issue can be reproduced running a GetRange Request with the data below
The scenario is as follows:
dlsproto.client.request.internal.GetRange.GetRangeHandler.forceStopRequest()
It looks like that #60 introduces a bug where on rare occasions not all connections are able to finish the GetRange request. This doesn't seem trivially reproducible, but it requires a lot of data.
The documentation in the examples/tests for the method connectionNotifier()
in module dlstest.DlsClient
is incorrect for the tag neo-alpha-1
/***********************************************************************
Connection notifier used by the client (see the outer class' ctor).
Params:
node_address = address/port of node which notification refers to
e = exception instance indicating an error (null indicates
connection success)
***********************************************************************/
private void connectionNotifier ( RawClient.Neo.ConnNotification info)
{
with (info.Active) switch (info.active)
{
case connected:
log.trace("Neo connection established (on {}:{})",
info.connected.node_addr.address_bytes,
info.connected.node_addr.port);
if (this.connect_task && this.connect_task.suspended())
this.connect_task.resume();
break;
case error_while_connecting:
with (info.error_while_connecting)
{
this.connection_error = true;
log.error("Neo connection error: {} (on {}:{})",
getMsg(e),
node_addr.address_bytes, node_addr.port);
}
break;
default:
assert(false);
}
}
node_disconnected
notification of GetRange does not mention that the request will be restarted on that node when the connection is back up.The neo client already has a built-in task-blocking API, so could be used directly in test cases. (This is what dhtproto does, for example.)
With each record batch, the node sends the index in the bucket file (either as a file offset or a count index) of the last record included. If an error happens, after re-connection the client can tell the node the last index so it can carry on from that point.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.