Comments (7)
If it's purely a writer API, that would be more doable indeed.
For a writer API, you'd need to:
- fetch the config of the cluster and parse the list of nodes that have sequencing enabled
- figure out which node holds the sequencer for a log (this is done through consistent-hashing the log_id against the list of nodes in the config that are enabled for sequencing, see HashBasedSequencerLocator). If the node turns out to be down, you'll have to remove it from the list and try to rehash (you can later improve on this by fetching a list of live nodes from a random node in the cluster).
- establish a connection to that node, negotiate a protocol version and send an APPEND message and interpret the APPENDED message that you'd get in response (it may contain redirects and such).
All the message definitions (and serializers/deserializers for the protocol) are here. You can start with the protocol header, then see HELLO and ACK messages that are used to establish a connection. The messages used for communicating writes from clients to servers are APPEND, the response will come as an APPENDED message.
Have fun! :D
from logdevice.
You would have to read way less C++ if you went for the bindings vs. trying to figure out the existing logic and protocol :D
Sorry, I won't have time to help you find all the definitions in the code. Try getting an IDE that would do that for you (e.g. vim with ctags). Writing documentation on the protocol also won't be at the top of our priority list any time soon. It may be a good idea to generally describe how the reader/writer work though.
The only currently supported platform is x86-64. In theory any little-endian 64-bit platform should work (?), but we haven't tried other platforms, so in practice there may be some breakage.
from logdevice.
LogDevice has a very heavy client, especially on the read path (e.g. just ClientReadStream-related code is 10K lines). Developing an alternative client and maintaining it would be a very complex project.
Which language are you interested in developing a client library for? Could it bind to the C++ client instead?
from logdevice.
Yes, this refreshing design would make such development quite interesting. Personally I'd like to give the writer a try with Go. The langue has quite a few useful features (like channels) to support the heavy lifting. A plain binding would probably work and of course there are downsides to that such as slow compile times and limitations on the threading model.
from logdevice.
Just to be clear: this can be a fun thing to play with, but I'd heavily discourage you or anyone else create/use a client that's not just bindings to the C++ client unless it provides very clear benefits. Duplicating complex logic in different languages and then maintaining it isn't something we should be doing unless there's a very clear win.
from logdevice.
Clear. For me the clear win is no C++ which is almost impossible to read. ;-) Where is request_id_t defined or Status? For some reason each message has its own serdes code and it looks like the process is dumping raw structs and primitives here and there which makes it AMD64 only? In other words, people like me need a lot of help or some documentation. 😕
from logdevice.
Thanks for the feedback anyway. :-) Maybe as a finishing touch document the AMD64-only requirement somewhere?
from logdevice.
Related Issues (20)
- Inconsistent column type for node_id in Sequencer and Nodes table HOT 3
- Settings abort-on-failed-check, abort-on-failed-catch and external-loglevel don't correctly subscribe to settings update HOT 1
- Tests failed in master due to some folly singletons being used without calling folly init
- Make DSCP reflection configurable in server settings HOT 1
- ModuleNotFoundError: No Module named 'libfb' HOT 1
- LogDevice build failing on Ubuntu bionic HOT 4
- logdevice/build_tools/fedora.deps file doesn't exist HOT 2
- With latest code, linking integration_test fail with SIGSEGV
- The log range specified must be positive 62-bit with start smaller or equal than end of range, supplied range is 0..200000 HOT 3
- Can not build on Ubuntu 19.10 HOT 3
- Will sequencer become a bottleneck for large size record with high throughput HOT 4
- Replication policy cross multiple cities with one special deferred? HOT 4
- Can not build on Ubuntu 18.04.4 HOT 2
- Conditional add question HOT 2
- Run quickstart through docker image failed HOT 1
- I want to integrate Kubernetes with VirtualLog. HOT 1
- Docker build fbthrift compilation thrift/lib/py3/test/gen-py3/testing/clients.cpp failed
- Is this project no longer maintained? HOT 12
- [email protected]
- LogDevice cluster start error --- NOSEQUENCER and sequencer:[invalid NodeID]
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logdevice.