delhivery / moirai Goto Github PK
View Code? Open in Web Editor NEWPath prediction for Delhivery Transportation Network
Path prediction for Delhivery Transportation Network
Is your feature request related to a problem? Please describe.
Some routes in the system are generated on a one time basis. Additionally, only objects at this route's source are expected to be routed via this ad-hoc route.
Atropos should add support so that it can consume one time routes and generate new routing solutions for applicable objects.
Describe the solution you'd like
Expose a separate endpoint to consume ad-hoc routes. For each ad-hoc route, identify list of objects available at route source.
Copy the current graph and add the ad-hoc route to the copy. For each identified object, request routing against the forked graph. Store this data against its own namespace so as not to muddle standardized routing.
Describe the bug
While atropos listens for changes to route/node configurations, these changes aren't propagated to the underlying graph thus leading to stale/incorrect outcomes.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The generated route should use the new route in its solution.
Is your feature request related to a problem? Please describe.
Currently, processing time against nodes is stored against a flat file. Use third party API to build interfaces to configure processing time at nodes so that the node stream can directly provide this information and there is a manageable and single source of truth for all nodes.
Describe the solution you'd like
Use the third party API to store inbound/outbound and cutoff times for each node. Build embeddable interfaces against these so that the node service can directly overlay them in its own UI.
Is your feature request related to a problem? Please describe.
Currently, data is pushed to elasticsearch in a singular index. This leads to constantly increasing data size on the cluster. A date based rolling index would allow for a timely deletion strategy of old dataset.
Describe the solution you'd like
Moirai should postfix requested index with creation date thus creating rolling indices based on creation date. Additionally, it should also implement a index lifecycle policy if not existant to drop indices older than current_day - 2.
Describe alternatives you've considered
An alternate would be to delete records from elasticsearch as part of clotho as per lifecycle of bags/shipments. However, this approach is ineherently more expensive on the ES cluster.
Is your feature request related to a problem? Please describe.
Currently, the solver works concurrently for the payload in the queue. This leads to added costs in terms of thread initialization and deinitialization, and increases linearly with number of tasks/threads to spawn.
We should instead move to a solution where number of threads is tied to hardware concurrency and each thread solves for a batch of requests sequentially.
Describe the solution you'd like
Is your feature request related to a problem? Please describe.
This is a meta issue to track progress of parallelism in moirai. The current approach does not scale vertically or horizontally.
Describe the solution you'd like
Is your feature request related to a problem? Please describe.
Currently, the kafka listener listens across partitions for loads. We should add the ability to spawn multiple processes each managing their own offsets and syncrhonization mechanism so that they do not overlap with each other. Additionally, each consumer should attempt to pick a separate partition first before competing with another solver process to read messages on the same partition.
Describe the solution you'd like
Describe the bug
Currently, routing requests are only consumed for objects that generate an explicit scan. For objects that were recommended on a route but weren't routed as per the system, the proposed route becomes stale since no scan is generated on idle items.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
When the proposed route is departed without the corresponding object associated with it, a automatic request for a new route should be generated either by matching against the departed route information or using a scheduler that re-generates routes automatically at the time of proposed departure.
Describe the bug
Some routes in the system are detected as invalid (non-existant source/target), even though the sources/targets are valid and active in the system.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Is your feature request related to a problem? Please describe.
Currently, Atropos directly pushes its output against Elasticsearch cluster. While this is good enough to start consuming data, it leads to overwriting of older paths with newer ones for the same object on future routing requests against it.
Support for multiple-sinks would allow us to route this to Kafka where it can be consumed by Data Warehouse or other services that directly want to create their own local replica.
Describe the solution you'd like
Writer should support factory pattern to write to as many sinks as the user requests. This would allow us to reuse the same writer to write to Elasticsearch/Kafka and any additional sinks if required (such as HTTP/Redis PUBSUB etc)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.