Comments (13)
I was expecting something like this:
// Just allocate a buffer flyweight object without having it point to real memory.
private static final UnsafeBuffer BUFFER = new UnsafeBuffer(0);
// This tries to reserve lengthOfMyData bytes on the underlying logbuffer.
// If we managed to reserve the appropriate number of bytes, then change BUFFER's internals to the right pointer + length.
final boolean result = publication.offer(BUFFER, lengthOfMyData);
if (result) {
// Do optional bounds checking to make sure we don't go past lengthOfMyData.
BUFFER.putByte(byte1);
BUFFER.putByte(byte2);
.....
}
from aeron.
The statement is that the data is copy-free in the processing the log buffers. This is very unusual in a message transport.
However it is funny that you mention because this week I'm working on an additional API that will allow the claiming of a range in the log buffer so it can be written directly into. This will not really matter when going off box but should make a difference in the IPC case on box. The API will look something like:
final LogClaim logClaim = new LogClaim(); // Can be stored for future use
if (publication.claim(messageLength, logClaim))
{
final MutableDirectBuffer buffer = logClaim.buffer();
final int offset = logClaim.offset();
// insert data directly into the buffer or wrap with a flyweight
logClaim.publish();
}
from aeron.
Awesome!
from aeron.
I've pushed a commit that offers this functionality. 8ed70c4
Works like the following:
final BufferClaim bufferClaim = new BufferClaim(); // Can be stored for future use
if (publication.tryClaim(messageLength, bufferClaim))
{
final MutableDirectBuffer buffer = bufferClaim .buffer();
final int offset = bufferClaim .offset();
// insert data directly into the buffer or wrap with a flyweight
bufferClaim .commit();
}
from aeron.
What will happen to the state of the LogBuffer
if client code calls tryClaim()
, but never calls BufferClaim.commit()
. Would it be useful to support passing in a lambda and ensure that the commit is always called?
public boolean tryClaim(final int length, final BufferClaim bufferClaim, Consumer<BufferClaim> callback)
{
boolean claimed = tryClaim(length, bufferClaim);
if (claimed)
{
try
{
callback.accept(bufferClaim);
}
finally
{
bufferClaim.commit();
}
}
return claimed;
}
from aeron.
It is a good point and something people have to be carefully about. The issue with such a lambda is that is it likely to be capturing and thus will allocate and have an impact on latency.
from aeron.
You can work around the capturing, by allowing arguments to be floated through. The Disruptor's EventSink interface does this. See the description about using Java 8, it has a couple of notes around capturing Lambdas.
from aeron.
In your example bb
will be captured and thus allocation will take place.
from aeron.
Not in the second example, where a method reference is used.
from aeron.
OK I see what you are doing. This can work for a single argument which might be sufficient for many cases. Could be an option consider.
from aeron.
I supported up to three and varargs (although varargs will potentially allocate). In the multi-producer case, failing to call commit/publish will leave the Disruptor (and similarly the LogBuffer - I think) in an invalid and unrepairable state. The most common case of this occurring is when the caller fails to do their exception handling correctly (commit must be in a finally block). With the Disruptor I recommend the publishEvent call as the default approach only only use next()/publish() if some advanced use case is required. I would suggest something similar for Aeron.
from aeron.
Varargs definitely allocates. Its also probably worse than a capturing lambda as well, because it needs to initialise the array of arguments.
I think really the only people who should be using this API are advanced users of the IPC case who are being bottlenecked by the additional copying cost of the current API. Do we want to caveat that in the javadoc for this class?
from aeron.
This is even more advanced that the Disruptor as you get direct access to the underlying buffer. Definitely an advanced feature only.
from aeron.
Related Issues (20)
- aeronmd.c Closing multi-publisher IPC publication HOT 2
- Cannot set thread affinity in shared or sharednetwork modes for c media driver HOT 3
- list-members(ClusterTool) command does not show isLeader accurately
- big latency while transmit small packets cross different AWS zone over Aeron comparing with raw UDP HOT 2
- AeronCluster.AsyncConnect can forget to close subscription HOT 1
- [C++] `ReplayMerge` with multicast live destination doesn't merge. HOT 10
- AeronCluster.java decoding order issue HOT 3
- OpenTelemetry Integration
- ArchiveException: ERROR - response for correlationId=15, error: 59232 position not aligned to a data header HOT 8
- Invoke fileChannel's force method before close HOT 2
- Heartbeats being sent, despite no publishing. HOT 5
- `ReplayMerge::doWork` throws exceptions without descriptions.
- ReplayMerge join position is greater than the replay position HOT 4
- AeronCluster client (gateway) - SIGSEGV HOT 1
- Set thread name to "client-conductor" fails.
- aeron ping-pong example build should detect sendmmsg
- code examples for C or C++ HOT 2
- Archive ConductorServiceTimeoutException when using `useConductorAgentInvoker`
- [C Media Driver]: Custom poller and receiver functions HOT 2
- Entire cluster of 3 members getting stuck if one of the followers gets stuck HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aeron.