clojusc / mesomatic

A simple and idiomatic Clojure facade around the Mesos JAVA API

mesomatic

A simple and idiomatic Clojure facade around the Mesos JAVA API

Contents

• About
• Resources
• Usage
• Examples
• Namespaces
• Type Conversions
• Release Notes
• Contributor Resources
• Donating

About

Mesomatic provides facilities to interact with Apache Mesos from clojure. It provides a simple and idiomatic facade around the Mesos JAVA API and facilities to help when writing mesos frameworks.

Mesomatic versions match the API version they target, a trailing minor indicates the patch release number, for instance version 1.0.1-r0 will target mesos 1.0.1.

Note that the clojusc Github org has volunteered to maintain the library originally created by pyr at pyr/mesomatic. The new location, clojusc/mesomatic, is now the offical home for the library.

Resources

• An excellent Mesos presentation from ApacheCon 2014
• pyr's Euroclojure 2015 talk on Mesomatic

Usage

Add this to your leiningen profile:

:dependencies [[clojusc/mesomatic "1.0.1-r1"]]

If you want to use the core.async facade, you will need to pull it in as well:

:dependencies [[clojusc/mesomatic "1.0.1-r1"]
               [clojusc/mesomatic-async "1.0.1-r1"]]

Examples

Be sure to examine the example frameworks built with mesomatic.

Namespaces

• mesomatic.types: contains a facade to and from all protobuf types.
• mesomatic.scheduler: facades for schedulers and scheduler-drivers
• mesomatic.executor: facades for executors and executor-drivers
• mesomatic.async.executor: produce executor callbacks on a channel
• mesomatic.async.scheduler: produce scheduler callbacks on a channel
• mesomatic.helpers: utility helpers for cluster decisions

Type Conversions

To go to and from protobuf types, mesomatic uses two simple functions:

• pb->data: yields a data structure from a mesos type, usually in the form of a record.
• data->pb: converts a data structure to a mesos type.
• ->pb: convert a plain map to a mesos type hinted at by a keyword

By yielding records, mesomatic provides elements which are homomorphic to maps and can easily be converted back to protobuf.

Special Cases

A few cases do not yield records:

• Scalar values (Protos.Value.Scalar) yield doubles.
• All enums yield keywords.
• Set values (Protos.Value.Set) yield sets.
• Some types containing a single repeated field are unrolled as a seq of their content, such as Protos.Value.Ranges.

Release Notes

1.0.1

• Target mesos 1.0.1
• Support for GPU resources
• Updates for API changes in Java bindings

Contributor Resources

• http://mesos.apache.org/documentation/latest/upgrades/
• https://github.com/ContainerSolutions/minimesos
• https://github.com/ContainerSolutions/minimesos-docker
• https://github.com/katacoda/minimesos-examples
• https://github.com/mesos/elasticsearch/tree/master/system-test/src/systemTest/java/org/apache/mesos/elasticsearch/systemtest

Donating

A donation account for supporting development on this project has been set up on Liberapay here:

• https://liberapay.com/clojusc-mesomatic/donate

You can learn more about Liberapay on its Wikipedia entry or on the service's "About" page.

https://liberapay.com/clojusc-mesomatic/donate

mesomatic's Issues

Enhancement: Allow scheduler drivers to be created with implicit acknowledgements

This was done by adding another arity to the scheduler-driver. Branch is here:

• https://github.com/clojusc/mesomatic/commits/add-implicit-acknowledgements

Needs to be rebased after other branches are merged.

Enhancement: Add missing task status reason constants

Just ran into a bug while trying to create a container-based executor where a task status error reason wasn't defined. I looked to see if there were any others missing, and added those too.

Will open a PR shortly ...

Decide upon a system / integration test strategy that runs on Travis CI

Blocks #43

Bug: Executor method doesn't need data conversion

send-status-update! doesn't need to call data->pb.

Also, some of the other executor methods call data->pb when they should instead call pb->data.

Finally, there is a miss-named Java method that needs to be fixed.

This work was done in the following branch:

• https://github.com/clojusc/mesomatic/tree/fix-executor-registered

Possible executor issue

I haven't had enough time to really dig into this, but I thought I'd open up a ticket in the event that there is a known issue or I can be pointed in the right direction (for how to really do this) more quickly :-)

In mesomatic.executor a MesosExecutorDriver is created:

• https://github.com/pyr/mesomatic/blob/6c01b6214a9d4a6f7c6d56fbd6d40bad02da5f6a/src/mesomatic/executor.clj#L57

I had naively attempted to pass an async executor here, saw an error message, and instead passed a map that can be converted to ExecutorInfo, e.g.:

(require '[mesomatic.types :as types])
(def executor-info {:name "'Hello, World!' Executor"})
(types/->pb :ExecutorInfo executor-info)

So that's the background; the error comes in when using the info data structure for creating a driver:

(require '[mesomatic.executor :as executor])
(executor/executor-driver executor-info)

ClassCastException org.apache.mesos.Protos$ExecutorInfo 
  cannot be cast to org.apache.mesos.Executor  
  mesomatic.executor/executor-driver (executor.clj:57)

Line 57 is this:

(let [d (MesosExecutorDriver. (->pb :ExecutorInfo executor))]

When running this in the REPL, the same thing occurs:

(import org.apache.mesos.MesosExecutorDriver)
(org.apache.mesos.MesosExecutorDriver. (types/->pb :ExecutorInfo executor-info))

ClassCastException org.apache.mesos.Protos$ExecutorInfo 
  cannot be cast to org.apache.mesos.Executor  
  clojusc.mesomatic.hello.dev/eval11689 (form-init4159658069332216610.clj:1)

Feature: Add new "accept-offers" method

The following "offers" method (using the new "operations" type) is missing from the sync and async Mesomatic schedulers:

• accept-offers

See: https://mesos.apache.org/api/latest/java/org/apache/mesos/MesosSchedulerDriver.html

Specific tasks:

• Add the above method to the mesomatic.scheduler/Scheduler protocol
• Update mesomatic.scheduler/wrap-driver to include the missing offer API method

This work will need to take advantage of the new Operation type being defined in #11 -- as such, the branch for this feature will be dependent upon the work done in the type branch https://github.com/clojusc/mesomatic/tree/add-opperation-type.

Task: Port Mesos Java Example to Mesomatic

In creating a minimal "Hello World" Mesomatic example, I've found a couple of different bugs in Mesomatic ... but more than that, my own lack of confidence around the workings of Mesos and Mesomatic were exposed. So! Instead of continuing down that path, I'd like to port a know, working example to Mesosmatic:

• https://github.com/apache/mesos/blob/master/src/examples/java/TestFramework.java
• https://github.com/apache/mesos/blob/master/src/examples/java/TestExecutor.java

I'll be putting this code here:

• https://github.com/clojusc/mesomatic-examples

If, once created, it is something that would be beneficial to include in a mesomatic/examples dir, I'm all for that as well.

Once this is working (and any missing features are added to Mesomatic to support this), I will return to the super-minimal "Hello World" example and get that finished up.

Usage:

• Since lein is a near-ubiquitous tool in the Clojure community, it will be used to run the framework

Dependencies:

• This example will use a bare minimum of libraries in order to clearly illustrate a core usage pattern (mesomatic and mesos; maybe cli for parsing args passed to main)

Dev plan:

• Create test-framework.clj as a port of TestFramework.java
• Create test-executor.clj as a port of TestExecutor.java
• Create core.clj with main function
• Add lein alias(es) in project.clj for running framework and/or executor
• Create tickets for all bug fixes/feature additions
• Submit PRs for completed work on bug fix/new feature branches:
  • DONE - https://github.com/clojusc/mesomatic/tree/add-opperation-type
  • DONE - https://github.com/clojusc/mesomatic/tree/add-accept-offers - see issue #12; this has an open PR for it here
  • DONE - https://github.com/clojusc/mesomatic/tree/add-missing-methods - see issue #13; will rebase after PR #16 is merged
  • DONE - https://github.com/clojusc/mesomatic/tree/add-missing-source - see issue #17; will rebase after PR #16 is merged
  • DONE - https://github.com/clojusc/mesomatic/tree/cond-to-case (aesthetic only) - see issue #18; will rebase after PR #16 is merged
  • DONE - https://github.com/clojusc/mesomatic/commits/fix-executor-registered - see issue #19; will rebase after previous three are merged
  • DONE - https://github.com/clojusc/mesomatic/tree/add-implicit-acknowledgements - see issue #20; will rebase after previous branches are merged
• Test all Mesomatic bug fixes/feature additions in a separate (cumulative) branch
• Demonstrate the ability to send messages from the executor back to the scheduler
• Ensure proper shutdown of framework upon successful completion of tasks
• Perform a final compare & contrast between Java and Clojure versions to ensure implementation parity

Plan to switch to HTTP api?

I was wondering if there was any plan to switch mesomatic to use the new HTTP api since the java lib is being deprecated.

Write clojure.test compatible layer for sys/int testing

Depends on #44
Blocks #43

Enhancement: Add missing scheduler driver methods

Once the work in issue #12 is completed, the following scheduler driver methods will be the only ones not yet implemented by Mesomatic:

• acknowledge-status-update
• suppress-offers

Add mapping for REASON_TASK_KILLED_DURING_LAUNCH

What?

Add a mapping for REASON_TASK_KILLED_DURING_LAUNCH:

http://mesos.apache.org/documentation/latest/task-state-reasons/#for-state-task_killed

Similar to past issue #28.

Why?

This reason comes through quite frequently in the wild, so it would be great to have first-class support for it. As it stands right now, we're having to "roll our own" to handle this reason cleanly.

Feature: Add new Operation type

There is a new type in Mesos called "operation" which is used by the new "offers" functions in the driver API. We need to add this to Mesomatic.

In particular, Mesomatic will need to add a set of functions and a protocol for org.apache.mesos.Protos$Offer$Operation in types.clj.

Note that the javadocs generated by Mesos only have placeholders for the new PB additions and don't have the actual docs. However, one can view the source here:

• src/java/generated/org/apache/mesos/Protos.java

In particular, this may be helpful:

$ grep Offer.Operation src/java/generated/org/apache/mesos/Protos.java |grep public

Specific tasks:

• Update import section in types.clj
• Create pb->data for operation types (Protos$Offer$Operation$Type/LAUNCH, RESERVE, UNRESERVE, CREATE, DESTROY)
• Create record for Operation
• Create method for pb->data Protos$Offer$Operation
• Update Operation record and its data->pb method to support LAUNCH type
• Update Operation record and its data->pb method to support RESERVE type
• Update Operation record and its data->pb method to support UNRESERVE type
• Update Operation record and its data->pb method to support CREATE type
• Update Operation record and its data->pb method to support DESTROY type
• Update ->pb with map conversions

Missing types in FrameworkInfo$Capability

We currently define:

• TASK_KILLING_STATE
• GPU_RESOURCES
• REVOCABLE_RESOURCES

Mesos 1.0.1 defines:

• UNKNOWN
• REVOCABLE_RESOURCES
• TASK_KILLING_STATE
• GPU_RESOURCES
• SHARED_RESOURCES
• PARTITION_AWARE

Add integration tests for version 1.3.1 to 1.4.1

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '1.3.1|1.4.1'|grep -v rc

1beaede refs/tags/1.3.1
60e6544 refs/tags/1.4.1

git log-short 1beaede..60e6544 src/java

4af1688 2017-08-30 Added several logs to the C++ part of the v1-v0 adapter. <Alexander Rukletsov>
bd9138c 2017-08-10 Extracted JNI code into a protected function for clarity. <Alexander Rukletsov>
f26fb42 2017-08-10 Ensured JAVA HTTP adapter propagates a subscription error. <Alexander Rukletsov>
3ee2986 2017-06-28 Fixed parameters in JNI call for disconnected event. <Alexander Rukletsov>
3451933 2017-05-26 Updated Mesos build library to use protobuf 3.3.0. <Zhitao Li>

git diff 1beaede..60e6544 src/java

Enhancement: convert cond to case in types.clj

There's one cond left in mesomatic.types that is able to be converted to a case.

This makes the code more readable, and brings the data->pb function in line with the other functions in the namespace that are amenable to using case (the ones that aren't checking against Java fields).

The work for this was done in this branch:

• https://github.com/clojusc/mesomatic/commits/cond-to-case

Add integration tests for version 1.1.3 to 1.2.2

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '1.1.3|1.2.2'|grep -v rc

ce77d91 refs/tags/1.1.3
f954bb0 refs/tags/1.2.2

git log-short ce77d91..f954bb0 src/java

7e5c2b2 2017-06-28 Fixed parameters in JNI call for disconnected event. <Alexander Rukletsov>
cbbc5a4 2016-10-28 Fixed MesosNativeLibrary to use '_NUM' MESOS_VERSION macros. <Joris Van Remoortere>
afdd707 2016-10-27 Populated `MasterInfo` in the v0 Java adapter. <Anand Mazumdar>

git diff ce77d91..f954bb0 src/java

Add integration tests for version 1.0.4 to 1.1.3

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '1.0.4|1.1.3'|grep -v rc

4154f66 refs/tags/1.0.4
ce77d91 refs/tags/1.1.3

git log-short 4154f66..ce77d91 src/java

a72d20e 2017-06-28 Fixed parameters in JNI call for disconnected event. <Alexander Rukletsov>
e105363 2016-10-28 Fixed MesosNativeLibrary to use '_NUM' MESOS_VERSION macros. <Joris Van Remoortere>
d9236ab 2016-10-27 Populated `MasterInfo` in the v0 Java adapter. <Anand Mazumdar>
22d3f56 2016-10-13 Corrected usage of "it's" in Mesos. <Neil Conway>
6c69a3c 2016-10-07 Fixed typo in comment. <Neil Conway>
d4492f3 2016-10-02 Fixed usage of "can not" vs. "cannot". <Neil Conway>
04b9498 2016-08-22 Renamed `JNIMesos` to `V1Mesos` for scheduler shim. <Anand Mazumdar>
bea8c3c 2016-08-10 Trimmed unneeded extra space between right angle brackets. <Gaojin CAO>
68c27d1 2016-07-19 Added native implementation for the V0 Mesos Adapter. <Anand Mazumdar>
bca68f6 2016-07-19 Added native implementation for v1 Mesos interface. <Anand Mazumdar>
5855532 2016-07-19 Added java implementations for the V0/V1 implementation for Mesos. <Anand Mazumdar>
c1ba4a5 2016-07-19 Added v1 Scheduler/Mesos interface in Java. <Anand Mazumdar>
07a1802 2016-07-19 Added helper functions for v1 JNI `construct()`/`convert()`. <Anand Mazumdar>
3e115ac 2016-07-25 Clean up indentation in Java scheduler driver. <Aleksandar Prokopec>

git diff 4154f66..ce77d91 src/java

"Asked to abort the driver"

I'm in a bit of a pickle and am hoping you can help. Periodically our scheduler disconnects from mesos and from the mesos logs on the scheduler side, I see:

I20161007 22:18:10.539175 28448 authenticatee.cpp:336] Authentication success
I20161007 22:18:10.539458 28455 sched.cpp:472] Successfully authenticated with master [email protected]:5050
I20161007 22:18:10.540596 28445 sched.cpp:708] Framework registered with 20150506-201345-1963923116-5050-3887-0002
I20161007 22:28:13.041359 28452 sched.cpp:1942] Asked to abort the driver
I20161007 22:28:13.041808 28452 sched.cpp:1178] Aborting framework '20150506-201345-1963923116-5050-3887-0002'

But I can't find any corresponding error that would imply why the driver is aborting. Have any of you seen something like this before? What steps can you recommend to debug? I've poured over both the scheduler and mesos master logs around the time to see if there was anything that jumped out but unfortunately not.

Support for Mesos >= 1.3.0?

Mesomatic depends on Mesos 1.0.1. Are there any plans to update it to support more recent versions of Mesos?

Cannot create executor-driver with expected initialization

When attempting to create a mesomatic executor driver like so:

(require [clojure.core.async :as a]
         [mesomatic.async.executor :as async-executor]
         [mesomatic.executor :as executor])
(def ch (a/chan))
(def exctr (async-executor/executor ch))
(executor/executor-driver exctr)

the following error occurs:

IllegalArgumentException Don't know how to create ISeq from: 
  mesomatic.executor$wrap_executor$reify__9882  
  clojure.lang.RT.seqFrom (RT.java:542)

Double-checking this, I found that:

• org.apache.mesos.MesosExecutorDriver.java expects an instance of org.apache.mesos.Executor during init
• mesomatic.executor/executor-driver seems to expect a map that
  • is then converted to protobuff data
  • and passed to MesosExectorDriver's init

Add support for principal secret

When defining a principal, mesomatic should also support the principal's secret. This is needed if authentication is enabled on the mesos masters.

I believe this is achieved by using the Credentials.Builder:

public static final class Protos.Credentials.Builder
extends <any>
implements Protos.CredentialsOrBuilder

 Credentials used for framework authentication, HTTP authentication
 (where the common 'username' and 'password' are captured as
 'principal' and 'secret' respectively), etc.

Support system / integration tests

• Decide upon a system / integration test strategy that runs on Travis CI - see #44
• Create the necessary Clojure glue for the selected approach (clojure.test-compatible) - see #45
• Create some basic integration tests, targeted initially at just the changes between 0.28.2 and 1.0.1 - see #46

Args for 5-arity MesomaticSchedulerDriver init in wrong order

Java code:

public MesosSchedulerDriver(Scheduler scheduler,
                            FrameworkInfo framework,
                            String master,
                            boolean implicitAcknowledgements,
                            Credential credential) { ... }

Mesomatic code:

(MesosSchedulerDriver. scheduler
  (->pb :FrameworkInfo framework)
  master
  (->pb :Credential credential)
  implicit-acknowledgements?))))

Totally my fault -- I'll submit a fix and PR...

Auto-teardown for exceptions not working in async driver

The official Mesos examples come with an "exception" framework that simply shows how a framework behaves when a framework encounters an exception.

When running analogous code in Mesomatic using the standard (synchronous) driver, the same behaviour is seen. However, when using the async driver, the framework hangs.

Add integration tests for version 1.0.1 to 1.0.4

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '1.0.1|1.0.4'|grep -v rc

3611eb0 refs/tags/1.0.1
4154f66 refs/tags/1.0.4

git diff 3611eb0..4154f66 src/java

Add integration tests for version 1.2.2 to 1.3.1

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '1.2.2|1.3.1'|grep -v rc

f954bb0 refs/tags/1.2.2
1beaede refs/tags/1.3.1

git log-short f954bb0..1beaede src/java

d888361 2017-06-28 Fixed parameters in JNI call for disconnected event. <Alexander Rukletsov>

git diff f954bb0..1beaede src/java

Enhancement: Add missing task status constants

The task source constants are missing in Mesomatic.

Work for this has been done in the following branch:

• https://github.com/clojusc/mesomatic/tree/add-missing-source

Once PR #16 is merged, I'll rebase and submit a PR for this one.

Create integration tests for just the changes between 0.28.0 and 1.0.1

Depends on #45
Blocks #43

cd mesos/apache-repo/
$ git show-ref --abbrev=7 --tags|egrep '0.28.0|1.0.1'|grep -v rc

961edbd refs/tags/0.28.0
3611eb0 refs/tags/1.0.1

git diff 961edbd..3611eb0 src/java

Consider using a Clojure protobuf library

A not insignificant portion of mesomatic is concerned with converting Clojure to and from protobuf data.

Questions to be answered:

• How significantly smaller would the mesomatic codebase be if we used a library like protobuf?
• Would it be possible to preserve 100% of the current mesomatic API and usage?
• What differences are there between mesomatic serialization and how the protobuf project does it?
• Same question for deserialization ...
• Would a move to a protobuf library be facilitated by anyt refactoring tasks in mesomatic?

NoSuchMethodError for org.apache.mesos.Protos$Value$Set.getItemList

This is a new one for me ... I'm not sure what has changed, but I wasn't getting this error last week :-/

(I might have still had an older jar file in my path, which would explain the sudden occurrence.)

When I kick off a new framework and the first resource offer is being processed, I get this error:

Exception in thread "Thread-7" java.lang.NoSuchMethodError: org.apache.mesos.Protos$Value$Set.getItemList()Lcom/google/protobuf/ProtocolStringList;
    at mesomatic.types$fn__819.invokeStatic(types.clj:596)
    at mesomatic.types$fn__819.invoke(types.clj:594)
    at clojure.lang.MultiFn.invoke(MultiFn.java:229)
    at mesomatic.types$fn__929.invokeStatic(types.clj:675)
    at mesomatic.types$fn__929.invoke(types.clj:667)
    at clojure.lang.MultiFn.invoke(MultiFn.java:229)
    at clojure.core$mapv$fn__6953.invoke(core.clj:6627)
    at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
    at clojure.core.protocols$fn__6742.invokeStatic(protocols.clj:75)
    at clojure.core.protocols$fn__6742.invoke(protocols.clj:75)
    at clojure.core.protocols$fn__6684$G__6679__6697.invoke(protocols.clj:13)
    at clojure.core$reduce.invokeStatic(core.clj:6545)
    at clojure.core$mapv.invokeStatic(core.clj:6618)
    at clojure.core$mapv.invoke(core.clj:6618)
    at mesomatic.types$fn__1298.invokeStatic(types.clj:1031)
    at mesomatic.types$fn__1298.invoke(types.clj:1024)
    at clojure.lang.MultiFn.invoke(MultiFn.java:229)
    at clojure.core$mapv$fn__6953.invoke(core.clj:6627)
    at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
    at clojure.core.protocols$fn__6742.invokeStatic(protocols.clj:75)
    at clojure.core.protocols$fn__6742.invoke(protocols.clj:75)
    at clojure.core.protocols$fn__6684$G__6679__6697.invoke(protocols.clj:13)
    at clojure.core$reduce.invokeStatic(core.clj:6545)
    at clojure.core$mapv.invokeStatic(core.clj:6618)
    at clojure.core$mapv.invoke(core.clj:6618)
    at mesomatic.scheduler$wrap_scheduler$reify__2859.resourceOffers(scheduler.clj:113)

Looking at mesomatic.types, this is the code that seems to be bombing out:

(defmethod pb->data Protos$Value$Set
  [^Protos$Value$Set value-set]
 (set (.getItemList value-set)))

And, indeed, when I look at the Protobuf docs, I don't see this method defined for ProtocolStringList:

• https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/ProtocolStringList

I'm not sure if Protos$Value$Set is supposed to be a ProtocolStringList or not ...