Investigate performance benchmark about exq

[j-mcnally]

can we just make this a test and then assert some anticipated performance time?

[akira]

Yeah that would be a great way to go about it. As part of #21, if we can make a test and assert performance, it would be awesome.

[j-mcnally]

Not sure how fast Timex is either, i use it to handle / parse dates. Could be a potential bottleneck?

[akira]

Not sure about Timex, did not look at that one. Even so, if we get the stats recorded async / separate process that shouldn't be an issue, since it would probably be on a different core.

[j-mcnally]

Yep yep, sounds like a plan.

[akira]

About to check in the harness - definitely notice a slowdown, this should help diagnose.

[j-mcnally]

If we want to test differences between maps / structs and poison / exjson we could try

http://www.erlang.org/doc/man/timer.html#tc-1

[akira]

OK, see #35. With old branch, it is able to get 5k in about 3 seconds. The current version only gets about 1k messages in 6 seconds. There do appear to be some intermittent failures with the Redis shutdown.

[j-mcnally]

the terminate callbacks may be getting backed up when they write to redis. Did you test with commenting out the terminate callback to manager?

[j-mcnally]

Should we create a redis connection pool? I feel like a single redis connection may cause blocking behavior

[akira]

Looking at it now, it seems like the first major culprit is the job_json method, with the Poison encoding the struct. Using Poison with Enum vs Struct definitely speeds that up.

I feel like Redis is pretty fast, and handles a large number of instructions / sec. It's definitely worth a try though. The other thing is the JSON parsing - another thing worth trying is having multiple dispatchers once the job is out of Redis. I still need to think of the process model / supervision tree to see what the best thing is - we can take a look at something like pool_boy or another library. I do feel like the stats can safely be put in another process because it is out of band (if we do that, we should captures the timestamp before sending it to the stats worker).

[j-mcnally]

is {jid, Poison.Encoder.encode(job, %Job{})} faster?

i wonder if supplying the struct type would help Poison to figure things out faster.

[akira]

Yeah looks like, this is faster it seems (same as JSEX):

    jid = UUID.uuid4
    job = Enum.into([{:queue, queue}, {:class, worker}, {:args, args}, {:jid, jid}], HashDict.new)
    {jid, Poison.encode!(job)}

[j-mcnally]

cool looks good to me.

[akira]

Seems like the other culprit is the decode (3.8 vs 0.8):

    json = Poison.decode!(state.job)
    job = %Exq.Job{
      queue: Dict.get(json, "queue"),
      class: Dict.get(json, "class"),
      jid: Dict.get(json, "jid"),
      error_message: Dict.get(json, "error_message"),
      error_class: Dict.get(json, "error_class"),
      failed_at: Dict.get(json, "failed_at"),
      retry: Dict.get(json, "retry"),
      retry_count: Dict.get(json, "retry_count"),
      processor: Dict.get(json, "processor"),
      args: Dict.get(json, "args"),
      finished_at: Dict.get(json, "finished_at"),
      enqueued_at: Dict.get(json, "enqueued_at")}

[akira]

And finally... The terminate / sending stats is the final difference. Will see if separating that out will fix it (2.5 vs 4.5 sec).

[j-mcnally]

you are doing god's work.

I know we Encode/Decode structs in there.

[akira]

OK I merged #35. There's still a few encode / decode in the stats that I didn't get a chance to fix - I wonder if we can put an encode / decode in the struct.

[j-mcnally]

yes! i was gonna recommend that. im adding more code that needs to do this, if we can do that, it would be great.

[akira]

OK, looks like that was moved into #40, closing this one.