NPE when dispatcher shutdown about kamon

[adrien-aubel]

As mentioned in #36, I tried to deploy the updated Kamon.io and got this exception at startup:

[ERROR] [06/06/2014 14:33:23.932] [threads-redis-client-akka.actor.default-dispatcher-5] [akka://threads-redis-client/system/IO-TCP/selectors] null
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1_aroundBody2(ActorCell.scala:456)
at akka.actor.ActorCell$AjcClosure3.run(ActorCell.scala:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at akka.instrumentation.ActorSystemMessagePassingTracing$$anonfun$aroundSystemMessageInvoke$1.apply(ActorSystemMessagePassingTracing.scala:61)
at kamon.trace.TraceRecorder$.withTraceContext(TraceRecorder.scala:66)
at akka.instrumentation.ActorSystemMessagePassingTracing.aroundSystemMessageInvoke(ActorSystemMessagePassingTracing.scala:61)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:1)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:349)
at akka.actor.Props.newActor(Props.scala:249)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 15 more
Caused by: java.lang.NullPointerException
at akka.actor.ActorSystemImpl.findExtension(ActorSystem.scala:697)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:706)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at kamon.metrics.Metrics$.apply(MetricsExtension.scala:97)
at kamon.Kamon$.apply(Kamon.scala:31)
at akka.instrumentation.DispatcherTracing.afterDispatcherStartup(DispatcherTracing.scala:55)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:44)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:44)
at akka.dispatch.ExecutorServiceDelegate$class.execute(ThreadPoolBuilder.scala:212)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:43)
at akka.dispatch.Dispatcher.executeTask(Dispatcher.scala:76)
at akka.dispatch.MessageDispatcher.unbatchedExecute(AbstractDispatcher.scala:145)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:113)
at akka.dispatch.MessageDispatcher.execute(AbstractDispatcher.scala:85)
at akka.util.SerializedSuspendableExecutionContext.attach(SerializedSuspendableExecutionContext.scala:75)
at akka.util.SerializedSuspendableExecutionContext.execute(SerializedSuspendableExecutionContext.scala:76)
at akka.io.SelectionHandler$ChannelRegistryImpl.<init>(SelectionHandler.scala:151)
at akka.io.SelectionHandler.<init>(SelectionHandler.scala:234)
... 24 more

The application seems to work fine, but I'd like to have your opinion on this exception before using it in production.

[dpsoft]

Adrien,
This happens always when you start the application?
El 06/06/2014 18:47, "Adrien Aubel" [email protected] escribió:

As mentioned in #36 #36, I
tried to deploy the updated Kamon.io and got this exception at startup:

[ERROR] [06/06/2014 14:33:23.932] [threads-redis-client-akka.actor.default-dispatcher-5] [akka://threads-redis-client/system/IO-TCP/selectors] null
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1_aroundBody2(ActorCell.scala:456)
at akka.actor.ActorCell$AjcClosure3.run(ActorCell.scala:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at akka.instrumentation.ActorSystemMessagePassingTracing$$anonfun$aroundSystemMessageInvoke$1.apply(ActorSystemMessagePassingTracing.scala:61)
at kamon.trace.TraceRecorder$.withTraceContext(TraceRecorder.scala:66)
at akka.instrumentation.ActorSystemMessagePassingTracing.aroundSystemMessageInvoke(ActorSystemMessagePassingTracing.scala:61)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:1)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:349)
at akka.actor.Props.newActor(Props.scala:249)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 15 more
Caused by: java.lang.NullPointerException
at akka.actor.ActorSystemImpl.findExtension(ActorSystem.scala:697)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:706)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at kamon.metrics.Metrics$.apply(MetricsExtension.scala:97)
at kamon.Kamon$.apply(Kamon.scala:31)
at akka.instrumentation.DispatcherTracing.afterDispatcherStartup(DispatcherTracing.scala:55)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:44)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:44)
at akka.dispatch.ExecutorServiceDelegate$class.execute(ThreadPoolBuilder.scala:212)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:43)
at akka.dispatch.Dispatcher.executeTask(Dispatcher.scala:76)
at akka.dispatch.MessageDispatcher.unbatchedExecute(AbstractDispatcher.scala:145)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:113)
at akka.dispatch.MessageDispatcher.execute(AbstractDispatcher.scala:85)
at akka.util.SerializedSuspendableExecutionContext.attach(SerializedSuspendableExecutionContext.scala:75)
at akka.util.SerializedSuspendableExecutionContext.execute(SerializedSuspendableExecutionContext.scala:76)
at akka.io.SelectionHandler$ChannelRegistryImpl.(SelectionHandler.scala:151)
at akka.io.SelectionHandler.(SelectionHandler.scala:234)
... 24 more

The application seems to work fine, but I'd like to have your opinion on
this exception before using it in production.

—
Reply to this email directly or view it on GitHub
#37 (comment).

[adrien-aubel]

Yes exactly. Just at startup, then it doesn't happen anymore.

[adrien-aubel]

Here it happens to one of my Redis actor systems (that uses Rediscala), and apparently the Redis doesn't work anymore in my application, it fallbacks to our SQL storage.

[dpsoft]

I think that the issue is related to the multiples ActorSystems.

how many actorsystems you are using?

best regards,

Diego

On Fri, Jun 6, 2014 at 8:25 PM, Adrien Aubel [email protected]
wrote:

Here it happens to one of my Redis actor systems, and apparently the Redis
doesn't work anymore in my application, it fallbacks to our SQL storage.

—
Reply to this email directly or view it on GitHub
#37 (comment).

[adrien-aubel]

I use 10 actor systems in this Spray application. I can detail them if you want.

[ivantopo]

Adrien, although Kamon should work properly with various actor systems in the same JVM, I'm curious on why you decided to have that many actor systems together.. could you please share a bit of insight in that?

[adrien-aubel]

Sure, so first of all we use many SBT sub-projects.
For example, the rest project (our Spray application) depends on Seq(models, db % "compile->compile;test->test", push, services, logger), these dependencies themselves usually depend on other things, like DB models generation and all.

Here are the actor systems that are used at runtime:

• db MySQL execution context (Slick 2.1)
• $serverName-redis-client Redis execution context (Rediscala 1.3.1) - 8 actor systems when all servers are used
• push Push notifications engine (uses Redis, GCM, APNS)
• rest The Spray routing processor
• services Analytics and telephony services

If there's anything that you would change here I'd be glad to hear your feedback :)

[ivantopo]

The most common practice is to have a single actor system per application and make sure everything is compartmentalized properly to prevent some parts of the app (like blocking JDBC calls) from starving other parts (like your Spray infrastructure, which is non-blocking), but that being the common case doesn't mean that what you did is wrong.. let me know if you ever raise a question regarding your architecture in the Akka mailing list, I would definitely like to know what they think about that.

Going back to the issue, we will try to isolate and solve this ASAP and get back to you, thanks again for your patience and collaboration!

[adrien-aubel]

Got it, in our case the only blocking thing that we're using is as you mentioned the JDBC call.

But we always do MySQL operations through this method:

class MySqlClient(serverName: String) extends InstrumentedBuilder {
  ...
  def async[T](f: Session => T): Future[T] = {
    Future(timer.time {
      val session = underlying.createSession()
      try {
        f(session)
      } finally {
        session.close()
      }
    })(MySqlClient.executionContext)
  }
  ...
}

object MySqlClient {
  val system = ActorSystem("db")
  implicit val executionContext = system.dispatchers.lookup("contexts.mysql")
  ...
}

So we have other projects, like our stream (WebSockets) server, that use this db project, so I thought it may be cleaner to isolate that in different actor systems.

But if you tell me that there's no interest doing so, I can also always use a single "application" actor system.

Let me know if there's anything I can do to help you guys solve this issue.

[adrien-aubel]

Btw the akka block containing this configuration:

loglevel = "INFO"
event-handlers = ["akka.event.slf4j.Slf4jEventHandler", "kamon.newrelic.NewRelicErrorLogger"]
extensions = ["kamon.newrelic.NewRelic"]
log-config-on-start = off

was initially at the root of the configuration, but that apparently caused connection issues to New Relic servers, so I've moved it nested in the rest and stream configurations (only 1 is used at a time, it depends on which application is started).

The error that I reported above occurs in both cases.