onyx-platform / onyx Goto Github PK

Submitting a catalog that doesn't conform to the specification of the informational model throws an unhelpful assertion inside the Coordinator. The catalog should be validated using something a library like Schema to obtain helpful error messages.

This is a particularly rough edge case. If a virtual peer receives a grouping task, it's capable of being starved from receiving the sentinel segment off the queue due to the way that HornetQ pins messages as it groups. If a consumer closes out, it might not necessarily requeue the sentinel in a server node where other consumers can reach it. Hence, the other virtual peers may deadlock and wait forever. This only affects batch mode - streaming mode is fine.

If the Coordinator and Peer are running on the same machine, they'll log to the same file. This can be a little bit of a pain during development. Each should log to its own file.

There's been some confusion around what the difference between the "event map", "lifecycle event map", and "context map" are. They are all the same thing. This should be fixed in the docs. I think I'd like to choose "lifecycle event" as the canonical term.

As of 0.3.0, the only strategy for balancing peers across jobs and tasks is round robin/breadth-first, respectively. This ticket should break out the algorithms used for planning and coordination into functions behind multimethods, and allow for a greedy strategy. A greedy strategy will try to complete an entire job before moving on to the next.

Coverage Protection is described here: https://github.com/MichaelDrogalis/onyx/blob/12c72be61d056446b1b7fe0a54a33782bbedc03b/doc/design/masterless.md#partial-coverage-protection

This was removed by the underlying mechanism changed for 0.5.0. Reimplement this API function.

With any task proceeded by a sequential task, the following exception will be thrown:

org.hornetq.api.core.HornetQInternalErrorException: HQ119000: ClientSession closed while creating session
    type: #<INTERNAL_ERROR>
org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal      ClientSessionFactoryImpl.java:  782
        org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession      ClientSessionFactoryImpl.java:  366
                               sun.reflect.NativeMethodAccessorImpl.invoke0      NativeMethodAccessorImpl.java      
                                sun.reflect.NativeMethodAccessorImpl.invoke      NativeMethodAccessorImpl.java:   57
                            sun.reflect.DelegatingMethodAccessorImpl.invoke  DelegatingMethodAccessorImpl.java:   43
                                            java.lang.reflect.Method.invoke                        Method.java:  606
                                clojure.lang.Reflector.invokeMatchingMethod                     Reflector.java:   93
                           clojure.lang.Reflector.invokeNoArgInstanceMember                     Reflector.java:  313
                                            onyx.queue.hornetq/eval20966/fn                        hornetq.clj:  201
                                                clojure.lang.MultiFn.invoke                       MultiFn.java:  231
                                       onyx.peer.operation/start-lifecycle?                      operation.clj:   55
                                           onyx.peer.transform/eval21156/fn                      transform.clj:   95
                                                clojure.lang.MultiFn.invoke                       MultiFn.java:  231
                    onyx.peer.task-lifecycle-extensions/merge-api-levels/fn      task_lifecycle_extensions.clj:   19
                                             clojure.lang.ArrayChunk.reduce                    ArrayChunk.java:   63
                                                  clojure.core.protocols/fn                      protocols.clj:   98
                                                clojure.core.protocols/fn/G                      protocols.clj:   19

The virtual peer will shutdown and instantly reboot, continuing as normal. This bug is mostly harmless. It is causes by the concurrent optimizations set for the HornetQ configuration. The Session Factory is swapped out in favor of a different factory, but the new factory doesn't "stick" for new tasks. Virtual peers reuse the old Session Factory that has been closed. The exception is thrown. After reboot, a fresh Session Factory is used.

Harmless, but annoying to see in the logs.

I never got around to adding a test for this because I implemented JGroups before the test suite used embedded HornetQ. This is pretty straightforward to test with an embedded cluster.

Considering a feature that will let a task complete when only one (not all) of its upstream inputs have pushed the sentinel onto the input stream. This would aid use cases where a privileged kill stream is utilized.

Just a placeholder, needs more thought.

Logs gets pretty huge after a short time. Add Rotor to Timbre to fix that.

Reproduced with the grouping test in Onyx core by turning up the number of virtual peers. Observed that two peers can continually take the sentinel segment off the queue and re-enqueue it infinitely, neither of them able to complete the task.

This API will compare it to a UUID and fail. Do the cast inside the function so users don't need to worry about it.

See here https://github.com/MichaelDrogalis/onyx/blob/0.4.x/doc/user-guide/coord-peer-config.md#jgroups-configuration.

I think :hornetq.udp/refresh-timeout should be :hornetq.jgroups/refresh-timeout (along with the others)

The catalog should off an optional onyx/max-peers parameter that takes an integer value representing the maximum number of peers that may be executing an instance of that task at any single point in time.

This plugin should be capable of reading off the Hadoop file system and writing segments back to it. The point of input and partitioning should be a single file, and the partitioning will happen over the byte sequence representing the file distributed over blocks in the cluster.

Key it under onyx.core/job-id.

Aggregators are at a disadvantage, performance-wise, to transformers and groupers. Aggregators can only hold a single session open which needs to be reused across pipeline iterations in the peer. The reason for this is that if multiple sessions were used, all sessions needs to be read from at the same time since groupers will pin particular message ids to consumers. These sessions shouldn't be closed, otherwise the messages will be repinned. Further, once the sentinel is read, all other sessions will block indefinitely.

The goal of this issue is to speed up this aggregators using an alternate design approach.

When we're running locally, it might be better for performance to use LMAX Disruptor instead of HornetQ for messaging. Requires benchmarking.

Hi, I wonder why virtual peer use 15 threads to process data? Are there other considerations?
(-> (fn1 event) fn2 fn3), isn't this simpler?

If a task throws an exception, the Peer crashes and isn't able to service additional work.

Reproduced in core.async plugin tests with a high number of virtual peers. Sometimes, closing a peer will block as it tries to flush its pipeline. The pipeline will block on reading from an ingress queue. This queue should always provide the sentinel value. Something is hanging on to the sentinel as a consumer and never committing it back to the queue, hence the hang.

The exception right now is confusing. start-lifecycle?, inject-lifecycle-resources, inject-temporal-resources,close-temporal-resources, andclose-lifecycle-resources` all need a type check on their return values.

If both of these are the same, the same multimethod for lifecycle resources gets dispatched to. This is really confusing. See #36

When a peer dies, it should attempt to recover by rebooting itself. See core.async test for failure.

This issue serves as a placeholder for the creation of another repository - onyx-dashboard. This dashboard will serve as a point of monitoring the status of what's happening inside Onyx by querying ZooKeeper. The data in ZooKeeper is immutable, and compressed with Fressian.

There needs to be an API function that takes a job ID and halts any peer execution of that job's tasks. The job's tasks will no longer be eligible for execution.

Allow value-level parameterization through the catalog.

[{...
  :my/param 42
  :my/other-param 44
  :onyx/params [:my/param :my/other-param]}]

It would be pretty great to Jepsen both HornetQ and Onyx itself. Partitioning virtual peers and coordinators fit the bill nicely.

Messed this one up during the redesign. Fix this before shipping 0.5.0.

If you supply a core async channel (or other non-serializable object) in a task map within a catalog, and submit-job, it will hang silently.

Ideally there should be some kind of validation of the catalog for serializabilty.

This silently fails right now, the exception gets swallowed up and everything halts.

From the mailing list:

"Also, it'd be really nice to specify a group-by operation on the input queue of each function, because then you could make things like wordcount really easy--you'd be able to say "send all instances of the same word to the the same downstream task", which would enable workflows that require an implicit sort/shuffle step."

https://groups.google.com/forum/#!topic/onyx-user/xniQcgCPEn8

Seems like this function should block, but actually returns a future.

In a very early version of Onyx, if the batch size of messages didn't accrue within a certain period of time, Onyx wouldn't attempt to keep reading and time out. This was removed due to a bug in HornetQ that didn't preserve sequential ordering. This is useful for sparse message streams, so I have found a workaround to add this back in.

Submitting a workflow that doesn't conform to the specification of the informational model throws an unhelpful assertion inside the Coordinator. The workflow should be validated using something a library like Schema to obtain helpful error messages.

Unsure of how I want it to look, but make it less verbose.

Should enable better searchability of the docs.

Observing high processor load after starting and stopping Onyx many times in the same repl session. Reproducing what @prasincs saw a few weeks ago.

Command section is mostly correct, but needs a final pass to ensure that it's update to date.

Reads block forever in aggregation readers. Fixed in 0.4.0-SNAPSHOT.

As of release 0.3.0, Clojure is the only supported language for Onyx. Java users can use the APIs that Clojure offers to tap some of the Onyx functionality, but this becomes problematic for areas such as lifecycle extensions that rely on implementations of multimethods.

Furthermore, EDN isn't the friendliest cross-language data format to send catalogs and workflows through. Part of this issue should explore options that Java users have on this front.

The Coordinator logs very infrequently as of 0.3.0. Logging using Dire should be implemented on events like job submission, task completion, peer birth/death, etc.

Some of the examples are incorrect due to design changes, or require pictures for better explanation.

A Kafka plugin should be created that offers both input and output functionality. Additionally, it should be capable of working with Kafka partitions.

Seeing this happen exactly twice, every single time. Never noticed until now.

In 0.4.0, we're going to move away from the tree/map based workflow to a vector-of-vectors. This will properly support multi input streams to any task, and continue to support multiple output streams. It will look like this:

[[:in-1 :inc]
 [:in-2 :inc]
 [:in-3 :inc]
 [:inc :out]]

Tasks:

• Function to convert map to DAG to preserve backward-compatibility
• Read from HornetQ queue's using Go blocks
• Reserve space in peer local atom to cache sentinel value's that have been seen
• Expand ZooKeeper sentinel reduction process to work across multiple input queues
• Alter planning to construct queue chains based off DAGs
• Intercept map at job submission time and conver to DAG
• Ignore input channels that are known to be exhausted
• Alternate input priority with round robin
• Change aggregate to read like transform does
• Alter validation to allow for DAG workflow
• Update documentation about map and vector workflows
• Update documentation to remove ack thread
• Update documentation to specify that read-batch must return a map
• Remove status check from pipeline

Hard coded for 250ms.

As mentioned in #2, workflow should be validated such that only input tasks are missing incoming edges, only output tasks should be missing output edges, and DAG should not have any cycles (dependency will throw an exception for you when creating the graph).

Removed while construction took place on 0.5.0.