yahoo / storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.

Storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN. It is still a work in progress.

Contributors

• Andy Feng (@anfeng)
• Robert Evans (@revans2)
• Derek Dagit (@d2r)
• Nathan Roberts (@ynroberts)
• Xin Wang (@vesense)

Mailing list

Feel free to ask questions on storm-yarn's mailing list: http://groups.google.com/group/storm-yarn

New features:

Based on the project developed by yahoo, we have added following new features.

1. We have updated the version of Apache Storm from 0.9.0 to 1.0.1.

2. We have added StormClusterChecker class, in order to monitor the storm cluster. It can adjust the number of supervisors based on the usage of system resources.

3. We have added the function, namely removeSupervisors() in order to monitor resources. Its functionality is just opposite to that of addSupervisors().

4. We have updated the logging framework from logback to log4j2.

How to install and use

Prerequisite

• Install Java 8 and Maven 3 first. These two software are necessary to compiling and packaging the source code of storm-on-yarn.

• Make sure Hadoop YARN have been properly launched.

• The storm-on-yarn implementation does not include running Zookeeper on YARN. Make sure the Zookeeper service is independently launched beforehands.

Download and build

1. Download the source code of storm-on-yarn, e.g., execute the command git clone <link> to get the source code.

2. Edit pom.xml in storm-on-yarn root directory to set the Hadoop version.

3. To package items, please execute the following command under storm-on-yarn root directory.

    mvn package
   

You will see the following execution messages.

17:57:27.810 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/storm.zip --appname storm-on-yarn-test --output target/appId.txt
  17:57:59.681 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn getStormConfig ./conf/storm.yaml --appId application_1372121842369_0001 --output ./lib/storm/storm.yaml
  17:58:04.382 [main] INFO  com.yahoo.storm.yarn.TestIntegration - ./lib/storm/bin/storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology exclamation-topology
  17:58:04.382 [main] INFO  com.yahoo.storm.yarn.TestIntegration - ./lib/storm/bin/storm kill exclamation-topology
  17:58:07.798 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn stopNimbus ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:10.131 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn startNimbus ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:12.460 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn stopUI ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:15.045 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn startUI ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:17.390 [main] INFO  com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn shutdown ./conf/storm.yaml --appId application_1372121842369_0001
  

If you want to skip the tests, please add -DskipTests .

   mvn package -DskipTests

Deploy:

After compiling and building the whole project of storm-on-yarn, next you need to install storm-on-yarn and Storm on the Storm Client machine, which is used for submitting the YARN applications to YARN ResourceManager (RM) later.

Please refer to the following  guide, step by step to deploy on the Storm Client machine.

1. Copy the packaged storm-on-yarn project to Storm Client machine, downloading the project of storm-1.0.1. and put the decompressed project of storm-1.0.1 into same directory as the storm-on-yarn project. As shown below,

So far, you have put storm-on-yarn and storm in the right place on Storm Client machine. You do not need to start running the Storm cluster, as this will be done by running storm-on-yarn later on.

2. When executing storm-on-yarn commands, commands like "storm-yarn", "storm" and etc., will be frequently called. Therefore, all paths to the bin files containing these executable commands must be included to the PATH environment variable.

Hence you are suggested to add storm-1.0.1/bin and $(storm-on-yarn root directory)/bin to your PATH environment variable, like this:

3. Storm-on-yarn will replicate a copy of Storm code throughout all the nodes of the YARN cluster using HDFS. However, the location of where to fetch such copy is hard-coded into the Storm-on-YARN client. Therefore, you will have to manually prepare the copy inside HDFS.
    The storm.zip file (the copy of Storm code) can be stored in HDFS under path "/lib/storm/[storm version]/storm.zip".

Following commands illustrate how to upload the storm.zip from the local directory to "/lib/storm/1.0.1" in HDFS.

  hadoop fs -mkdir /lib 
  hadoop fs -mkdir /lib/storm
  hadoop fs -mkdir /lib/storm/1.0.1
  zip -r storm.zip storm-1.0.1
  hadoop fs -put storm.zip /lib/storm/1.0.1/

At this point, the deployment step is complete.

Run:

Everything should be ready. Now you can start your storm-on-yarn project.

The storm-on-yarn project have a set of specify commands and it use storm-yarn [command] -[arg] xxx as the comman format.

To launch the cluster you can run

storm-yarn launch [storm-yarn-configuration]

[storm-yarn-configuration], which is usually a .yaml file, including all the required configurations during the launch of the Storm cluster.

In this project, we provide two quick ways to create the storm-yarn-configuration file:

a) Edit the storm.yaml file under storm-1.0.1/conf folder

b) Copy the $(storm-on-yarn root directory)/src/main/master_defaults.yaml to storm-1.0.1/conf and rename it to master.yaml, and then edit it where necessary.

In the simplest case, the only configuration you need to add is the Zookeeper cluster information and set the port of supervisor:

When you write a right configure file and execute the command above, the Hadoop YARN will return a application ID, if you deploy completely right. like this:

And, you can see the graphic below on YARN UI and Storm UI respectively.

So far, the storm-on-yarn has run and you can submit topology.

You can run the following command to submit a topology.

  storm jar ***.jar topologyName <arg> -c nimbus=

For example :

  storm jar ~/WordCount2/testWord-1.0-SNAPSHOT.jar storm.topology.WordCountTopology wordcount -c nimbus=192.168.1.25

Other details:

1. storm-yarn has a number of new options for configuring the storm ApplicationManager (AM), e.g.,

   • master.initial-num-supervisors, which stands for the initial number of supervisors to launch with storm.
   • master.container.size-mb, which stands for the size of the containers to request (from the YARN RM).

2. The procedure of "storm-yarn launch" returns an Application ID, which uniquely identifies the newly launched Storm master. This Application ID will be used for accessing the Storm master.

To obtain a storm.yaml from the newly launch Storm master, you can run

  storm-yarn getStormConfig <storm-yarn-config> --appId <Application-ID> --output <storm.yaml>

storm.yaml will be retrieved from Storm master.

3. For a full list of storm-yarn commands and options you can run

    storm-yarn help
   

4. Storm-on-yarn is now configured to use Netty for communication between spouts and bolts.

It's pure JVM based, and thus OS independent.

If you are running storm using zeromq (instead of Netty), you need to augment the standard storm.zip file the needed .so files. This can be done with the not ideally named create-tarball.sh script

  create-tarball.sh storm.zip

5. Ideally the storm.zip file is a world readable file installed by ops so there is only one copy in the distributed cache ever.

Commands:

Command	Usage
storm-yarn launch	launch storm on yarn
storm-yarn help	get help
storm-yarn version	view storm version
storm-yarn addSupervisors/removeSupervisor	add/remove supervisor
storm-yarn startNimbus/stopNimbus	start/stop Nimbus
storm-yarn startUI/stopUI	start/stop Web UI
storm-yarn startSupervisors/stopSupervisor	start or stop all supervisors
storm-yarn shutdown	shutdown storm cluster

Arguments：

Argument	Usage
-appname	(Only for storm-yarn launch) Application Name. Default value – "Storm-on-Yarn"
-appId	(Required) The storm clusters app ID
-output	Output file
-container	(Required for removeSupervisors) the supervisor to be removed
-supervisors	(Required for addSupervisors) The # of supervisors to be added

Known Issues:

There is no failover when nimbus goes down. Still working on it.

There is no simple way to get to the logs for the different processes.

License

The use and distribution terms for this software are covered by the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.html).

Acknowledgement

This work was partially supported by SingAREN/AWS Cloud Credit for Research Program 2016.

storm-yarn's Issues

Allow storm-yarn AppMaster to choose dynamic port to avoid port conflict.

master.thrift.port should be dynamic determined.

Use master_defaults.yaml to specify default values container memory size

master_defaults should add default values for:

• master.timeout.secs
• master.container.size-mb

Currently, the default value of master.container.size-mb is hard coded in LaunchCommand.java.

the dafault value of master.timeout.secs is hard coded in StormOnYarn.java.

Not running nimbus in YARN

It seems like there may be a case to be made for not running nimbus in yarn.

Nimbus is a relatively lightweight process, and the cluster only needs one of them running at any given time. Running nimbus in YARN actually reduces the availability of SOY which is not really the Right Thing(TM).

It substantially complicates cluster operations in terms of job submission to rely on the nimbus container never failing. It also makes job submission difficult.

Running nimbus separately is not unlike running an independent map/reduce history server. So it makes a lot of sense in my opinion and there are apparent design patterns in other yarn based frameworks.

At least until the time when SOY is able to properly handle topology submission along with starting the framework, this seems like a good modification to make to simplify running SOY.

I would propose adding an external nimbus configuration option in order to prevent SOY from starting one, if a user would prefer to operate that way. Thoughts?

Resources allocation including CPU usage

In some use cases, the workers are CPU intension. Currently, only memory is specified for container allocation. CPU usage is needed to be considered.

cannot launch application.

I am having trouble building and running storm-yarn.

mvn clean package -DskipTests builds a jar storm-yarn-1.0-SNAPSHOT.jar

when I execute storm-yarn launch master_default.yaml, I get a Runtime exception, caused by a missing snakeyaml dependency

Exception in thread "main" java.lang.NoClassDefFoundError: org/yaml/snakeyaml/Yaml
        at com.yahoo.storm.yarn.Util.createConfigurationFileInFs(Util.java:133)
        at com.yahoo.storm.yarn.StormOnYarn.launchApp(StormOnYarn.java:178)
        at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:359)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:67)
        at com.yahoo.storm.yarn.Client.execute(Client.java:121)
        at com.yahoo.storm.yarn.Client.main(Client.java:127)
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.Yaml
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        ... 6 more

Adding the maven assembly plugin, builds a jar including all dependency jars:

pom.xml
<plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <executions>
                    <execution>
                            <phase>package</phase>
                            <goals>
                                    <goal>single</goal>
                            </goals>
                    </execution>
            </executions>
            <configuration>
                    <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
            </configuration>
</plugin>

however, the yarn client implementation complains about double definition of master_defaults.yaml or defaults.yaml

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Found multiple master_defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/opt/storm/stormOnYARN/storm-yarn-1.0-SNAPSHOT.jar!/master_defaults.yaml, file:/opt/storm/master_defaults.yaml]
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:384)
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:424)
        at com.yahoo.storm.yarn.Config.readStormConfig(Config.java:52)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:57)
        at com.yahoo.storm.yarn.Client.execute(Client.java:121)
        at com.yahoo.storm.yarn.Client.main(Client.java:127)
Caused by: java.io.IOException: Found multiple master_defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/opt/storm/stormOnYARN/storm-yarn-1.0-SNAPSHOT.jar!/master_defaults.yaml, file:/opt/storm/master_defaults.yaml]
        at org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:410)
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:363)
        ... 5 more

I have placed my storm zip in hdfs:/lib/storm

$ hdfs dfs -ls /lib/storm/1.2.2/
-rw-r--r--   3 root supergroup  168876919 2020-03-31 18:44 /lib/storm/1.2.2/storm.zip

I have also placed storm on my client machine

$ ls -l /opt/storm
total 165264
drwxr-xr-x 12 root root      4096 Mar 31 18:20  storm-1.2.2
drwxr-xr-x  8 root root      4096 Mar 31 19:59  stormOnYARN

master_defaults.yaml/master.yaml/storm.yaml

master.host: "localhost"
master.thrift.port: 55058

master.initial-num-supervisors: 1
master.container.priority: 0
master.container.size-mb: 4096
master.heartbeat.interval.millis: 1000
master.timeout.secs: 1000
yarn.report.wait.millis: 10000
nimbusui.startup.ms: 10000

storm.thrift.transport: "org.apache.storm.security.auth.SimpleTransportPlugin"
nimbus.thrift.port: 55057
nimbus.queue.size: 100000
nimbus.thrift.threads: 64
nimbus.thrift.max_buffer_size: 1048576

drpc.thrift.transport: "org.apache.storm.security.auth.SimpleTransportPlugin"
drpc.port: 55056
drpc.queue.size: 128
drpc.worker.threads: 64
drpc.max_buffer_size: 1048576

ui.port: 55055

storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 1048576
storm.messaging.netty.max_retries: 100
storm.messaging.netty.min_wait_ms: 1000
storm.messaging.netty.max_wait_ms: 5000

# Configuration parameter that allows the launching machine to specify the JAVA_JOME
# used when the application is executed on the YARN cluster.
#
#storm.yarn.java_home: "/usr/java/default"
storm.yarn.java_home: "/usr/lib/jvm/java-8-openjdk-amd64"

# Configuration parameter that allows the launching machine to specify the yarn classpath
# used when the application is executed on the YARN cluster.  To find this value, run
# "yarn classpath" on the target machines.
#
#storm.yarn.yarn_classpath: ""
storm.yarn.yarn_classpath: "/opt/hadoop/etc/hadoop:/opt/hadoop/etc/hadoop:/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/yarn/lib/*"

TLDR

I cant seem to get the correct configuration or build.

• Bundling dependencies seems to include master_default.yaml from storm jar, and my own master_default.yaml. This leads to config conflicts exception.
• When building without the dependencies I get runtime exception due to missing dependencies, specifically snakeyaml.

storm on yarn is not support Fair scheduler？

I tested the storm-on-yarn for capacity scheduler,and it‘s ok.But,when I switch the yarn scheduler to fair scheduler ,I finded that the supervisor doesn't running. There is only nimbus and core process. Why it can't start supervisor ,can you tell me? my email :[email protected]. Thanks very much.

storm ui port 8080 has conflict with Hadoop 2.0.0-cdh4.3.0 node manager

A workaround for now is is to add the followings in /etc/hadoop/conf.pseudo/mapred-site.xm and restart nodemanager.

mapreduce.shuffle.port 8081 Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers.

Don't overwrite original storm.yaml in HDFS. (find a new way to pass in the nimbus host)

Currently when launching a supervisor the AM will overwrite the original storm.yaml file so that nimbus.host can be output. This allows the supervisors to find nimbus to download jars and config files from nimbus. We need to stop overwriting this file because it causes the distributed cache to think the file has changed and fail to launch the supervisor. We should write it out once for the supervisors once nimbus comes up. After that we should not touch it again.

Ideally long term we should have the nodes discover nimbus through zk, so that if there is a failover to a new nimbus server the supervisors continue to function properly. We may also want to replace downloading the files from nimbus, and replace it with downloading the files from HDFS. This would make it so the supervisors do not need the address of nimbus at all. But both of these are longer term and require changes in storm itself.

Delegation Token Refreshes

According to https://issues.apache.org/jira/browse/YARN-941 - delegation tokens may be refreshed by simply restarting the application master. This makes intuitive sense.

I believe storm stores topology submission information in Zookeeper, so this could actually work quite well for storm on yarn in that the AM could inspect its delegation tokens and simply exit with a non-final error code every time the token expires. The topologies, etc would then be re-launched with the same processing and durability guarantees as normal storm (although, perhaps the restarts would be more noticeable).

storm-yarn based on storm 1.1.x when will be released

CDH 4.4.0 is not supported

In hadoop-0.23 branch, if we change pom file to use CDH 4.4.0, mvn test will generate the following errors. CDH 4.4.0 has some API change from CDH 4.3.0.

[INFO] Compiling 14 source files to /Users/afeng/dev/storm/storm-yarn/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/client/AMRMClientImpl.java:[29,41] error: cannot find symbol
[ERROR] symbol: class AMResponse
location: package org.apache.hadoop.yarn.api.records
/Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/client/AMRMClientImpl.java:[179,6] error: cannot find symbol
[ERROR] symbol: class AMResponse
location: class AMRMClientImpl
/Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/client/AMRMClientImpl.java:[179,44] error: cannot find symbol
[ERROR] symbol: method getAMResponse()
location: variable allocateResponse of type AllocateResponse
/Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/MasterServer.java:[72,31] error: cannot find symbol
[ERROR] symbol: method getAMResponse()
location: variable allocResponse of type AllocateResponse
/Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/MasterServer.java:[78,65] error: cannot find symbol
[ERROR] symbol: method getAMResponse()
location: variable allocResponse of type AllocateResponse
/Users/afeng/dev/storm/storm-yarn/src/main/java/com/yahoo/storm/yarn/MasterServer.java:[93,31] error: cannot find symbol

Support Hadoop 0.23 in addition to 2.x

We should enable storm-yarn to be used in Hadoop 0.23 YARN clsuter.

what different between storm and storm-yarn?

i use CDH4 yarn framework, i want build storm cluster on yarn framework,is storm can?
or is storm-yarn can? or both can?

There is a problem with the pom.xml file

Elder martial brother Feng and I found the following error when running MVN package on 2021.07.06-22:22: --- [error] failed to execute goal on project storm yard: could not resolve dependencies for project com.yahoo.storm.yard: Storm yard: jar: 1.0-snapshot: failed to collect dependencies at org.apache.storm: Storm core: jar: 1.2.2: failed to read artifact descriptor for org.apache.com storm:storm-core:jar:1.2.2: Could not transfer artifact org.apache.storm:storm-core:pom:1.2.2 from/to central ( http://repo1.maven.org/maven2/ ): Failed to transfer file: http://repo1.maven.org/maven2/org/apache/storm/storm-core/1.2.2/storm-core-1.2.2.pom. Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1]-------。 The problem was found in http-https.

launch Zookeeper as a part of storm-yarn?

Currently, we assume that Zookeeper has been deployed before launching storm cluster. It might be better if we could launch Zookeeper with storm-yarn as well.

Any thoughts?

i'm running zhe mvn package command,but not success.

hy@hy:~/work/storm-yarn$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building storm-yarn 1.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ storm-yarn ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default) @ storm-yarn ---
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ storm-yarn ---
[INFO] Surefire report directory: /home/hy/work/storm-yarn/target/surefire-reports

---

T E S T S

Running com.yahoo.storm.yarn.TestStormMaster
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hy/.m2/repository/ch/qos/logback/logback-classic/1.0.6/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hy/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23:21:00,294 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
23:21:00,295 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
23:21:00,295 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/home/hy/work/storm-yarn/target/test-classes/logback.xml]
23:21:00,419 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
23:21:00,427 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
23:21:00,473 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
23:21:00,537 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN
23:21:00,537 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
23:21:00,538 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
23:21:00,540 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@1d9c7028 - Registering current configuration as safe fallback point
SLF4J: Actual binding is of type [ch.qos.logback.classic.selector.DefaultContextSelector]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.607 sec
Running com.yahoo.storm.yarn.TestStormCluster
23:21:03.221 [main] WARN c.yahoo.storm.yarn.TestStormCluster - Still cannot connect to nimbus server.
23:21:03.220 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:04.227 [main] WARN c.yahoo.storm.yarn.TestStormCluster - Still cannot connect to nimbus server.
23:21:04.227 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:05.228 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:05.229 [main] WARN c.yahoo.storm.yarn.TestStormCluster - Still cannot connect to nimbus server.
23:21:06.231 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:06.232 [main] WARN c.yahoo.storm.yarn.TestStormCluster - Still cannot connect to nimbus server.
23:21:07.233 [main] WARN c.yahoo.storm.yarn.TestStormCluster - Still cannot connect to nimbus server.
23:21:07.233 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:08.234 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:60000hread] org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)evel org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:634)ogger{36} - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socketsg
23:21:08.692 [SyncThread:0hread] org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:445)evel org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:445)ogger{36} - Unexpected Exception: sg
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:161)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98)
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.08 sec
Running com.yahoo.storm.yarn.TestIntegration
23:21:09.002 [main] WARN o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23:21:10.457 [Thread-13] WARN o.a.h.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-resourcemanager.properties,hadoop-metrics2.properties
23:21:11.388 [main] WARN o.a.h.y.s.n.c.m.ContainersMonitorImpl - NodeManager configured with 4 G physical memory allocated to containers, which is more than 80% of the total physical memory available (3.7 G). Thrashing might happen.
Aug 02, 2013 11:21:11 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver as a provider class
Aug 02, 2013 11:21:11 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices as a root resource class
Aug 02, 2013 11:21:11 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Aug 02, 2013 11:21:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'
Aug 02, 2013 11:21:11 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:12 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
23:21:12.859 [main] WARN o.a.h.y.s.n.c.m.ContainersMonitorImpl - NodeManager configured with 4 G physical memory allocated to containers, which is more than 80% of the total physical memory available (3.7 G). Thrashing might happen.
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices as a root resource class
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver as a provider class
Aug 02, 2013 11:21:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'
Aug 02, 2013 11:21:12 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:13 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:13 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Aug 02, 2013 11:21:13 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
bin/storm-yarn: line 19: yarn: command not found
23:21:16,487 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
23:21:16,488 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
23:21:16,488 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/home/hy/work/storm-yarn/target/conf/logback.xml]
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/home/hy/work/storm-yarn/target/conf/logback.xml]
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/home/hy/work/storm-yarn/lib/storm-0.9.0-wip19/storm-console-logging-0.9.0-wip19.jar!/logback.xml]
23:21:16,637 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
23:21:16,643 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
23:21:16,684 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
23:21:16,810 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN
23:21:16,811 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
23:21:16,813 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
23:21:16,817 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@14b7e998 - Registering current configuration as safe fallback point
23:21:16,487 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
23:21:16,488 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
23:21:16,488 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/home/hy/work/storm-yarn/target/conf/logback.xml]
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/home/hy/work/storm-yarn/target/conf/logback.xml]
23:21:16,490 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/home/hy/work/storm-yarn/lib/storm-0.9.0-wip19/storm-console-logging-0.9.0-wip19.jar!/logback.xml]
23:21:16,637 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
23:21:16,643 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
23:21:16,684 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
23:21:16,810 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN
23:21:16,811 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
23:21:16,813 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
23:21:16,817 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@14b7e998 - Registering current configuration as safe fallback point
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/Options
at com.yahoo.storm.yarn.LaunchCommand.getOpts(LaunchCommand.java:34)
at com.yahoo.storm.yarn.Client.execute(Client.java:124)
at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.Options
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 3 more
23:21:31.872 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure
java.io.FileNotFoundException: target/appId.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_02]
at java.io.FileInputStream.(FileInputStream.java:138) ~[na:1.7.0_02]
at java.io.FileInputStream.(FileInputStream.java:97) ~[na:1.7.0_02]
at java.io.FileReader.(FileReader.java:58) ~[na:1.7.0_02]
at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:102) ~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_02]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_02]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_02]
at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_02]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) [junit-4.10.jar:na]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit-4.10.jar:na]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) [junit-4.10.jar:na]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) [junit-4.10.jar:na]
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) [junit-4.10.jar:na]
at org.junit.runners.ParentRunner.run(ParentRunner.java:300) [junit-4.10.jar:na]
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) [surefire-junit4-2.10.jar:2.10]
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) [surefire-junit4-2.10.jar:2.10]
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) [surefire-junit4-2.10.jar:2.10]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_02]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_02]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_02]
at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_02]
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) [surefire-api-2.10.jar:2.10]
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) [surefire-booter-2.10.jar:2.10]
23:21:31.894 [Container Monitor] WARN o.a.h.y.s.n.c.m.ContainersMonitorImpl - org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.
23:21:32.021 [Container Monitor] WARN o.a.h.y.s.n.c.m.ContainersMonitorImpl - org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.
23:21:32.131 [Thread[Thread-19,5,main]] ERROR o.a.h.s.t.d.AbstractDelegationTokenSecretManager - InterruptedExcpetion recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep interrupted
23:21:32.142 [ApplicationMaster Launcher] WARN o.a.h.y.s.r.a.ApplicationMasterLauncher - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
23:21:32.152 [ResourceManager Event Processor] ERROR o.a.h.y.s.r.ResourceManager - Returning, interrupted : java.lang.InterruptedException
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.472 sec <<< FAILURE!

Results :

Failed tests: com.yahoo.storm.yarn.TestIntegration: expected: but was:<java.io.FileNotFoundException: target/appId.txt (No such file or directory)>

Tests run: 3, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36.650s
[INFO] Finished at: Fri Aug 02 23:21:33 EDT 2013
[INFO] Final Memory: 14M/167M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10:test (default-test) on project storm-yarn: There are test failures.
[ERROR]
[ERROR] Please refer to /home/hy/work/storm-yarn/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

When I exucute storm-yarn with storm 1.2.2, It`s throws problem

-bash-4.1$ storm-yarn getStormConfig -appId application_1565674728089_0006 -output /home/hadoop/.storm/storm.yaml
2019-08-13 15:11:32,454 INFO client.RMProxy: Connecting to ResourceManager at binghe100/192.168.175.100:8032
2019-08-13 15:11:32,789 INFO yarn.StormOnYarn: application report for application_1565674728089_0006 :N/A:-1
2019-08-13 15:11:32,789 INFO yarn.StormOnYarn: Attaching to N/A:-1 to talk to app master application_1565674728089_0006
Exception in thread "main" java.lang.IllegalArgumentException: invalid port: -1
at org.apache.storm.security.auth.ThriftClient.(ThriftClient.java:64)
at org.apache.storm.security.auth.ThriftClient.(ThriftClient.java:46)
at com.yahoo.storm.yarn.MasterClient.(MasterClient.java:48)
at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:38)
at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:117)
at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:90)
at com.yahoo.storm.yarn.Client.execute(Client.java:121)
at com.yahoo.storm.yarn.Client.main(Client.java:127)
-bash-4.1$ yarn application -kill application_1565674728089_0006

storm-yarn shutdown

We should have a command to remove Storm cluster, and release all associated YARN resources.

storm-yarn shutdown --appId <appID>

support Apache Hadoop 0.23.x (and CDH 4.x)

Our current implementation assumes that users are using Apache Hadoop 2.x, which has slight different API than Apache Hadoop 0.23.x and CDH 2.0.

DRPC servers on YARN

Currently, Storm-YARN launch Nimbus, UI and Supervisor servers. We should enable DRPC servers to be launched if requested by community.

Yarn monitoring principle？

What is the yarn monitoring principle? Can other components be monitored? For example, kafka, mongo, HA works by restoring RM data or even component HA？

How to generate StormMaster.java

hello,
I'm studying storm-yarn's source code and try to move JStorm (java version of storm) on yarn, I see there is one generated source StormMaster.java by thrift, but I'm new to thrift, could someone give me some tip of generating this file?
Thanks!

Damarcus Jones Ai Chips

https://github.com/yarnpkg/yarn/blob/master/LICENSE

question : compile storm_master.thrift

i have changed storm_master.thrift file , when i compiled the file by thrift 0.9.2 , that thrift7 doesn't exist anywhere , how can I compile the file.

Dynamic ports for Nimbus and UI

Currently, Nimbus and UI are binding to a port # given by storm configuration. On a host assigned by YARN, it's potentially the assigned port was already in use.

We should enhance storm core to allow port to be dynamically assigned for Nimbus, UI and etc.

when i execute zhe storm-yarn getStormConfig command,it's happen exception.

[bigdata@tbe192168147047 storm-yarn]$ bin/storm-yarn getStormConfig storm.yaml --appId application_1375431632296_0002
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/bigdata/apps/hadoop-2.0.4/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/bigdata/apps/storm-yarn/lib/storm-0.9.0-wip19/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
13/08/02 17:00:00 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/08/02 17:00:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/08/02 17:00:01 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/08/02 17:00:01 INFO yarn.StormOnYarn: application report for application_1375431632296_0002 :N/A:0
13/08/02 17:00:01 INFO yarn.StormOnYarn: Attaching to N/A:0 to talk to app master application_1375431632296_0002
13/08/02 17:00:01 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
Exception in thread "main" java.lang.IllegalArgumentException: invalid port: 0
at backtype.storm.security.auth.ThriftClient.(ThriftClient.java:37)
at com.yahoo.storm.yarn.MasterClient.(MasterClient.java:48)
at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:38)
at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:113)
at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:75)
at com.yahoo.storm.yarn.Client.execute(Client.java:139)
at com.yahoo.storm.yarn.Client.main(Client.java:145)

New apache-storm-0.9.1-incubating.zip throws "Can not find storm home entry"

Use of the new apache-storm-0.9.1-incubating.zip file throws the following error:

Exception in thread "main" java.lang.RuntimeException: Can not find storm home entry in storm zip file.
at com.yahoo.storm.yarn.Util.getStormHomeInZip(Util.java:115)
at com.yahoo.storm.yarn.StormOnYarn.launchApp(StormOnYarn.java:242)
at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:372)
at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:69)
at com.yahoo.storm.yarn.Client.execute(Client.java:142)
at com.yahoo.storm.yarn.Client.main(Client.java:148)

Looking at the Util.getStormHomeInZip suggests the cause is due to the new naming convention "apache-storm-0.9.1" vs the previous format of "storm-0.9.1". getStormHomeInZip is looking for the home directory to begin with storm-.

I got around this by downloading the apache-storm-0.9.1-incubating.zip, extracting to a directory on my laptop, then renaming the root directory to storm-0.9.1-incubating before rezipping back up.

Real fix is to update getStormHomeInZip to search for a directory which contains storm- and not begin storm-version.

Storm supervisor configs overwrite with each other

Hi,

When starting supervisor container, it will try to write config under /user/xx/.storm/appattempt_xx/conf. When starting multiple supervisors, they will share the same path. Then one supervisor may overwrite the setting of another supervisor, which will results in following errors.

2014-01-02 14:37:31,183 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 39 cluster_timestamp: 1388020955932 } attemptId: 1 } id: 3 } state: C_COMPLETE diagnostics: "Resource hdfs://IDHV22-01:8020/user/yarn/.storm/appattempt_1388020955932_0039_000001/conf changed on src filesystem (expected 1388644474340, was 1388644474889\n" exit_status: -1000

storm.zip visibility should be detected automatically

It would be nice if we auto-detected the visibility based on the file system permissions.

If you look in ClientDistributedCacheManager.isPublic in the mapreduce project for Hadoop. There is some code I wrote there a while ago that will do this same thing. You should be able to copy it.

http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java?revision=1431166&view=markup

line 235

Proposal: Merge storm-yarn into storm source tree

I think it is a good idea to merge storm-yarn into storm source tree. This approach have been adopted by spark. spark-yarn is a internal part of spark.

Advantages:

1. More easy to manage the dependance.
2. We can finally throw away the zmq code and dependance for build and test.
3. In storm-yarn, we don't need to install both, don't need to manage STORM_HOME in storm-yarn anymore. There will be single installation, and this can be build into single rpm package. Very convenient for management.
4. No longer need local storm.zip package. We can set the jars(storm.jar storm-yarn.jar) and libraries as local resources, and use this to start container on another machine.

Steps:

1. First need to change the groupId to "storm"
2. Better we need to use pom to manage storm, so that we can more easily aggregate storm-yarn as a module.
3. Add interface in storm-yarn scripts, so that we can submit storm job with storm-yarn script file directly (no need to getStormConfig)
4. We should not try to copy storm.zip into HDFS and use that zip to deploy. Instead, we can just add lib and jars into local resource. Otherwise, there are two copy of storm. One is in STORM_HOME, the other is in storm.zip.

I successfully created a storm cluster,but don't find supervisor process

yarn log
13/09/23 16:28:58 INFO yarn.MasterServer: Starting the AM!!!!
13/09/23 16:28:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is inited.
13/09/23 16:28:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/23 16:29:00 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is started.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Storm master host:127.0.1.1
13/09/23 16:29:00 INFO yarn.MasterServer: launch nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting nimbus...
13/09/23 16:29:00 INFO yarn.MasterServer: launch ui
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting UI...
13/09/23 16:29:00 INFO yarn.MasterServer: launch 1 supervisors
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: adding 1 supervisors...
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Added 1 supervisors, but not requesting containers now.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server -Dstorm.home=./storm/storm-0.9.0-wip19/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/lib/ -Xmx1024m -Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml -Dlogfile.name=nimbus.log backtype.storm.daemon.nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server -Dstorm.home=./storm/storm-0.9.0-wip19/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/lib/ -Xmx768m -Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml -Dlogfile.name=ui.log backtype.storm.ui.core
13/09/23 16:29:00 INFO yarn.MasterServer: Got a registration response minimumCapability {, memory: 1024, }, maximumCapability {, memory: 8192, },
13/09/23 16:29:00 INFO yarn.MasterServer: Max Capability memory: 8192
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Max Capability is now memory: 8192
13/09/23 16:29:00 INFO yarn.MasterServer: Starting HB thread
13/09/23 16:29:00 INFO yarn.MasterServer: Starting launcher
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Creating new ContainerRequest with memory: 8192 and 1
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Requested container ask: Capability[memory: 8192]Priority[0]ContainerCount[1]
13/09/23 16:29:00 INFO yarn.MasterServer: Starting Master Thrift Server

storm log
2013-09-23 16:30:12 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:60000
2013-09-23 16:30:12 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) ~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
2013-09-23 16:30:14 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:60000
2013-09-23 16:30:14 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) ~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

zookeeper log

2013-09-23 16:19:56,405 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58588
2013-09-23 16:19:56,406 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58588
2013-09-23 16:19:56,420 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560001 with negotiated timeout 20000 for client /127.0.0.1:58588
2013-09-23 16:20:49,758 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58590
2013-09-23 16:20:49,761 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58590
2013-09-23 16:20:49,784 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560002 with negotiated timeout 20000 for client /127.0.0.1:58590
2013-09-23 16:20:49,811 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560002
2013-09-23 16:20:49,817 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58590 which had sessionid 0x14149e0d3560002
2013-09-23 16:20:49,825 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58591
2013-09-23 16:20:49,829 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58591
2013-09-23 16:20:49,896 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560003 with negotiated timeout 20000 for client /127.0.0.1:58591
2013-09-23 16:28:36,046 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x14149e0d3560003, likely client has closed socket
2013-09-23 16:28:36,048 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58591 which had sessionid 0x14149e0d3560003
2013-09-23 16:28:39,379 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560001
2013-09-23 16:28:39,399 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58588 which had sessionid 0x14149e0d3560001
2013-09-23 16:28:56,000 - INFO [SessionTracker:ZooKeeperServer@314] - Expiring session 0x14149e0d3560003, timeout of 20000ms exceeded
2013-09-23 16:28:56,001 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560003

JAVA_HOME is picked up from the client/deploying machine

I'm deploying from Mac OS X onto a YARN cluster running on RHEL.
I've tested the YARN cluster with Pig, and all appears to be well.

Storm-Yarn works like a champ, but the logs from the Resource Manager show:
"/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home//bin/java: No such file or directory"

I haven't looked deep enough yet, but it appears to be picking up my JAVA_HOME from the client machine and using that when it exec's on the YARN cluster.
I believe I can hack around it by deploying from a Linux machine, but wanted to make you guys aware.

This is the actual exception shown in the ResourceManager.

Application application_1380078062907_0003 failed 2 times due to AM Container for appattempt_1380078062907_0003_000002 exited with exitCode: 127 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)

Improvement of sleep in loop

Hi, I found an interesting case in this project.
It has “Thread.sleep()” in a “while(true)” loop. I found some discussion in Stackoverflow and other website. It can be refactored by using a executor service.
https://ejrh.wordpress.com/2012/07/13/sleeping-in-loops-considered-harmful/
https://stackoverflow.com/questions/3535754/netbeans-java-new-hint-thread-sleep-called-in-loop
or
Maybe use CountDownLatch.await to automatically awake and shutdown.

Detailed websites and lines:
286 | https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
160 | https://github.com/yacy/yacy_grid_mcp/blob/development/src/main/java/net/yacy/grid/mcp/Service.java
37 | https://github.com/yacy/yacy_grid_mcp/blob/development/src/main/java/net/yacy/grid/mcp/Caretaker.java

Best regards

worker's log not in the yarn container

worker's log not in the yarn container (together with supervisor's log).

storm-yarn Version update is slow

Storm-yarn Version update is slow

Hadoop 2.2.0 support needed

I am using Apache Hadoop-2.2.0 with storm-yarn from wget https://github.com/yahoo/storm-yarn/archive/master.zip.

When running the storm-yarn using mvn package the tests are failing with below error:

00:34:28.901 [main] INFO com.yahoo.storm.yarn.TestConfig - storm_home: lib/storm-0.9.0-wip21
00:34:28.903 [main] INFO com.yahoo.storm.yarn.TestIntegration - Will be using storm found on PATH at lib/storm-0.9.0-wip21
00:34:29.083 [main] INFO com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/storm.zip --appname storm-on-yarn-test --output target/appId.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/cloud/Manisha/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/cloud/nirmal/storm-yarn-master/lib/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
00:34:31.206 [mainhread] org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)evel org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)ogger{36} - Unable to load native-hadoop library for your platform... using builtin-java classes where applicablesg
00:34:32.465 [Socket Reader #1 for port 37121] INFO org.apache.hadoop.ipc.Server - IPC Server listener on 37121: readAndProcess from client 192.168.41.106 threw exception [org.apache.hadoop.ipc.RpcServerException: Connection context not established]
Exception in thread "main" org.apache.hadoop.ipc.RpcServerException: Connection context not established
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy8.getNewApplication(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:127)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:135)
at com.yahoo.storm.yarn.StormOnYarn.launchApp(StormOnYarn.java:140)
at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:372)
at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:55)
at com.yahoo.storm.yarn.Client.execute(Client.java:139)
at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException): Connection context not established
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy7.getNewApplication(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:167)
... 14 more
00:34:47.523 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure
java.io.FileNotFoundException: target/appId.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_45]
at java.io.FileInputStream.(FileInputStream.java:146) ~[na:1.7.0_45]
at java.io.FileInputStream.(FileInputStream.java:101) ~[na:1.7.0_45]
at java.io.FileReader.(FileReader.java:58) ~[na:1.7.0_45]
at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:101) ~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) [junit-4.10.jar:na]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit-4.10.jar:na]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) [junit-4.10.jar:na]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) [junit-4.10.jar:na]
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) [junit-4.10.jar:na]
at org.junit.runners.ParentRunner.run(ParentRunner.java:300) [junit-4.10.jar:na]
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) [surefire-junit4-2.10.jar:2.10]
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) [surefire-junit4-2.10.jar:2.10]
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) [surefire-junit4-2.10.jar:2.10]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) [surefire-api-2.10.jar:2.10]
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) [surefire-booter-2.10.jar:2.10]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) [surefire-booter-2.10.jar:2.10]
00:34:47.524 [main] INFO com.yahoo.storm.yarn.TestIntegration - java.lang.NullPointerException
00:34:47.525 [main] INFO c.yahoo.storm.yarn.EmbeddedZKServer - shutdown embedded zookeeper server with port 60000
00:34:47.527 [main] INFO com.yahoo.storm.yarn.TestIntegration - shutdown MiniYarn cluster
00:34:47.546 [main] INFO org.mortbay.log - Stopped [email protected]:0
00:34:47.648 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 35805
00:34:47.652 [IPC Server listener on 35805] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 35805
00:34:47.655 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.657 [main] INFO o.a.h.y.s.n.c.m.ContainersMonitorImpl - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread
00:34:47.658 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 60910
00:34:47.661 [IPC Server listener on 60910] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 60910
00:34:47.662 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.664 [Public Localizer] INFO o.a.h.y.s.n.c.l.ResourceLocalizationService - Public cache exiting
00:34:47.678 [main] INFO org.mortbay.log - Stopped [email protected]:0
00:34:47.780 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 43639
00:34:47.782 [IPC Server listener on 43639] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 43639
00:34:47.782 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.784 [main] INFO o.a.h.y.s.n.c.m.ContainersMonitorImpl - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread
00:34:47.785 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 37231
00:34:47.786 [IPC Server listener on 37231] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 37231
00:34:47.787 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.788 [Public Localizer] INFO o.a.h.y.s.n.c.l.ResourceLocalizationService - Public cache exiting
00:34:47.804 [main] INFO org.mortbay.log - Stopped [email protected]:0
00:34:47.807 [Thread[Thread-13,5,main]] ERROR o.a.h.s.t.d.AbstractDelegationTokenSecretManager - InterruptedExcpetion recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep interrupted
00:34:47.809 [main] INFO o.a.h.m.impl.MetricsSystemImpl - Stopping NodeManager metrics system...
00:34:47.814 [main] INFO o.a.h.m.impl.MetricsSystemImpl - NodeManager metrics system stopped.
00:34:47.815 [main] INFO o.a.h.m.impl.MetricsSystemImpl - NodeManager metrics system shutdown complete.
00:34:47.816 [ApplicationMaster Launcher] WARN o.a.h.y.s.r.a.ApplicationMasterLauncher - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
00:34:47.816 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 33651
00:34:47.819 [IPC Server listener on 33651] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 33651
00:34:47.819 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.819 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 37121
00:34:47.825 [IPC Server listener on 37121] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 37121
00:34:47.825 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.825 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 33039
00:34:47.832 [IPC Server listener on 33039] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 33039
00:34:47.834 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 54908
00:34:47.834 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.849 [IPC Server listener on 54908] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 54908
00:34:47.849 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder
00:34:47.849 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - NMLivelinessMonitor thread interrupted
00:34:47.850 [ResourceManager Event Processor] ERROR o.a.h.y.s.r.ResourceManager - Returning, interrupted : java.lang.InterruptedException
00:34:47.853 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - AMLivelinessMonitor thread interrupted
00:34:47.854 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - AMLivelinessMonitor thread interrupted
00:34:47.854 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.228 sec <<< FAILURE!

sleep() used in junit tests should be avoided

Currently, we have various sleep() calls in junit test. We should come up a mechanism to avoid such sleeps.

Proposal: we should wrap storm topology submission in storm-yarn

Currently to submit a storm topology:
We need:

1. storm-yarn getStormConfig --output storm.yaml
2. storm jar -Dstorm.conf.file storm.yaml job.jar

Maybe we should combine these two, and use
storm-yarn jar --appId applicationId job.jar
to simplify the job.

Since we already wraped the storm supervisor, storm ui, storm nimbus, so I think we should make storm-yarn command line interface as a complete interface, and hide the details of storm.

Launching multiple supervisors within a single host

Currently, supervisors are launched with pre-configured port #. In many use cases, we want to launch multiple supervisors with a single storm.yaml. That will only work if these supervisors were launched on separated hosts.

We should enhance storm core to allow ports to be discovered dynamically. That will empower storm-yarn to launch multiple supervisors even in a single host.

Use HDFS for app storage

Storm currently stores application jar in Nimbus' local file system.
With Storm-YARN, we should leverage HDFS for such a storage.
Supervisors/workers could then launch topologies from HDFS, instead of downloading from Nimbus.

Add two resource configuration options for each supervisor

Currently storm-yarn will just use max resource capacity yarn cluster offer to start a supervisor.

It may be a good idea to add two other option in master.yaml, so that we can config the resource limiation.

supervisor.size-mb: 16192
supervisor.vcores: 8

If yarn cluster returns max resource capatity

 vcore:32, memory: 8G 

we will pick the smaller one, to

vcores: 8, Memory: 8G.