Would be nice for tasks such as video encoding to be able to to only run one task in a specific queue at a time.

We could allow queues to be specified in the for [{"Default", 50}, {"Encoding", 1}]

Since the workers per queue would be tracked on a per instance basis, we can keep track of the number of running tasks in ETS rather than redis.

I've been trying get exq running on OSX Yosemite 10.10.5 and have run into an issue when trying to enqueue a job:

iex(7)> {:ok, jid} = Exq.enqueue(:exq, "default", Eventer, []) 
** (exit) exited in: GenServer.call(:exq, {:enqueue, "default", Eventer, []}, 5000)
    ** (EXIT) exited in: :gen_server.call(#PID<0.203.0>, :stop)
        ** (EXIT) no process
    (elixir) lib/gen_server.ex:356: GenServer.call/3
iex(7)> 
21:25:39.328 [error] GenServer :exq_enqueuer terminating
Last message: {:"$gen_cast", {:enqueue, {#PID<0.190.0>, #Reference<0.0.1.4934>}, "default", Eventer, []}}
State: %Exq.Enqueuer.Server.State{namespace: "exq", redis: #PID<0.203.0>, redis_owner: true}
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: {:error, "No timezone found for: No timezone found for: AEST"}
        lib/timezone/timezone_dst.ex:20: Timex.Timezone.Dst.is_dst?/1
        lib/dateformat/formatter.ex:139: Timex.DateFormat.Formatters.Formatter.format_token/2
        lib/dateformat/formatters/default.ex:201: Timex.DateFormat.Formatters.DefaultFormatter.do_format/3
        lib/dateformat/formatters/default.ex:194: Timex.DateFormat.Formatters.DefaultFormatter.do_format/3
        lib/dateformat/formatter.ex:36: Timex.DateFormat.Formatters.Formatter.format!/3
        (exq) lib/exq/redis/job_queue.ex:124: Exq.Redis.JobQueue.to_job_json/3
        (exq) lib/exq/redis/job_queue.ex:37: Exq.Redis.JobQueue.enqueue/5
        (exq) lib/exq/enqueuer/server.ex:43: Exq.Enqueuer.Server.handle_cast/2

So it seems my timezone is being returned as "AEST" (Australian Eastern Standard Time). This isn't parsable by Timex, so the enqueing fails.

In iex I get these results when using the version of Timex required by exq:

iex(5)> Timex.Timezone.Local.lookup
"AEST"
iex(6)> Timex.Timezone.local       
{:error, "No timezone found for: AEST"}
iex(7)> Timex.Timezone.Local.parse_tzfile("/etc/localtime")
{:ok, "AEST"}

When I setup a new project with just the latest version of Timex (0.19.5) the timezone seems to work correctly:

iex(2)> Timex.Timezone.local
%Timex.TimezoneInfo{abbreviation: "AEST",
 from: {:sunday, {{2015, 4, 5}, {2, 0, 0}}}, full_name: "Australia/Sydney",
 offset_std: 0, offset_utc: 600, until: {:sunday, {{2015, 10, 4}, {2, 0, 0}}}}
iex(3)> Timex.Timezone.Local.lookup
"Australia/Sydney"
iex(4)> Timex.Timezone.Local.parse_tzfile("/etc/localtime")
{:ok, "AEST"}

It seems to me that bumping the version of Timex should resolve the issue above. Thoughts?

Records are discouraged in elixir, convert them to structs

Should we create a job struct instead of mashalling jobs to and from maps? Might make for a nicer api?

Using this as a quick and dirty benchmark:

    {:ok, pid} = Exq.start([host: '127.0.0.1', port: 6379])
    Exq.enqueue(pid, "default", "MyWorker2", ["arg1", "arg2"])
    for n <- 1..4000, do: Exq.enqueue(pid, "default", "MyWorker", ["arg1", "arg2"])
    :timer.sleep(:infinity)

seems like there were performance differences before / after #19. Some of this may be due to Poison decode / encode of direct structs, and then stats recording. Investigate these.

Right now the chart updates but we should also be updating the stats at the top of the page with the latest stats.

The failed redis key is created as a redis set in exq but is expected to be a redis list in Resque::Failure::Redis

This causes the Resque admin overview page to error out with:

Redis::CommandError: WRONGTYPE Operation against a key holding the wrong kind of value

The stack trace points to Resque.redis.llen(:failed).to_i: https://github.com/resque/resque/blob/1-x-stable/lib/resque/failure/redis.rb#L28

Came accross this:
http://redis.io/commands/srandmember

To decide the next queues couldnt we call srandmember on namespace:queues to get the next queue to check?

I'm trying to use exq with elixir 1.1.1

I keep getting this error when I try to start the app

=INFO REPORT==== 6-Oct-2015::06:39:11 ===
application: logger
exited: stopped
type: temporary
Could not start application exq: Exq.start(:normal, []) returned an error: shutdown: failed to start child: Exq.Manager.Server
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in :eredis.start_link/6
(eredis) src/eredis.erl:48: :eredis.start_link('redis', "6379", 0, [], 100, 5000)
(exq) lib/exq/manager/server.ex:31: Exq.Manager.Server.init/1
(stdlib) gen_server.erl:328: :gen_server.init_it/6
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

mix.exs:

defmodule AW.Mixfile do
use Mix.Project

def project do
[app: :aw,
version: "0.0.1",
elixir: "~> 1.0",
#build_embedded: Mix.env == :prod,
#start_permanent: Mix.env == :prod,
escript: [main_module: AW],
deps: deps]
end

Configuration for the OTP application

Type mix help compile.app for more information

def application do
[applications: [:logger, :httpoison, :amqp, :postgrex, :exq]]
end

Dependencies can be Hex packages:

{:mydep, "~> 0.3.0"}

Or git/path repositories:

{:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}

Type mix help deps for more examples and options

defp deps do
[
{:httpoison, "> 0.7"},
{:poison, "> 1.4.0"},
{:postgrex, "> 0.8"},
{:exjsx, "> 3.2.0"},
{:amqp, "0.1.1"},
{:slugger, git: "https://github.com/triptec/slugger.git"},
{:exq, git: "https://github.com/akira/exq.git", branch: "upgrade_timex"}
]
end
end

Shouldn't these two methods simply return keys instead of {:ok, keys} 'cause of their ! ?

As an end user, I want to re-enqueue failed jobs in the UI

Acceptance Criteria

• Add "Requeue" button in failed jobs UI view for each job
• Clicking on "Requeue" button, requeues failed jobs

Hi, I managed start exq itself and it works great.
But web UI refuses to start with an error:

$ mix exq.ui --webport 4040

Started ExqUI on Port 4040
** (UndefinedFunctionError) undefined function: Exq.RouterPlug.init/1 (module Exq.RouterPlug is not available)
    Exq.RouterPlug.init([namespace: "", exqopts: [name: :exq_enq_ui, host: '127.0.0.1', webport: 4040]])
    lib/plug/adapters/cowboy.ex:155: Plug.Adapters.Cowboy.dispatch_for/2
    lib/plug/adapters/cowboy.ex:36: Plug.Adapters.Cowboy.args/4
    lib/plug/adapters/cowboy.ex:126: Plug.Adapters.Cowboy.run/4
    lib/mix/tasks/exq.ui.ex:22: Mix.Tasks.Exq.Ui.run/1
    (mix) lib/mix/cli.ex:55: Mix.CLI.run_task/2

Could you point me what is going wrong?

Provide way to modify queue subscriptions dynamically once Exq is started via subscribe(queue), unsubscribe(queue) calls.

Just curious in the syntax why GenServer calls and casts are expressed as erlang calls rather than elixir native GenServer.call and GenServer.cast?

This will make it faster to pull stat counts without retrieving the entire list.
We can have one per day and one all time.
I assume sidekiq has something like this, atleast with the sidekiq-failures addon. The addon has a reset count and clear failures, so i have to assume they track the failures stat separately.

I think as we start to look at how this can run as a standalone and a plugable component it might make sense to look at how we could have a config file rather than passing options explicity at runtime.

Sidekiq key in redis: "queue:queue_name"
Exq key: "namespace:queue:queue_name"

When feature is enabled, send job directly to workers if they are free and skip redis enqueue / dequeue.

Also add a configuration option: short_circuit_enabled

Acceptance Criteria:

• If there are free workers for the specific queue (depends on concurrency setting)
• If short_circuit_enabled is set to true in config
• Then don't enqueue job to redis, send directly to worker
• Same code path should be used except the enqueue / dequeue
• Stats should still be tracked as usual
• Should still be put in backup queue

Is there a good way to do (or plans to add such functionality) so?
Recurring Jobs is really useful concept for some maintenance jobs like periodic cleanup

maybe even cron-style, e.g. https://github.com/resque/resque-scheduler#scheduled-jobs-recurring-jobs

P.S. probably https://github.com/c-rack/quantum-elixir is the solution

We should clean up the serialize / deserialize api a bit by moving what is now repeated logic for marshaling back and forth to json, into the modules for each struct type.

I am not 100% sure what the difference is but i think there is room for improvement on how we manage workers.

We could use the dynamic supervision described at: http://learnyousomeerlang.com/supervisors

@akira if you have ideas here please share, but i think we could do better in this category.

I noticed today that job ids are currently expressed as the redis LIST index, is this really the best way. I feel like as the api matures we might want to do things like Retry jobs and delete jobs, and this could cause race conditions if the index of a job changes. Id like to look at how sidekiq handles jobs and jobids and maybe take some inspiration. Thoughts / Ideas?

If we go UUIDs i am a fan of this library: https://github.com/zyro/elixir-uuid

Failed and processed stats (stat:failed, stat:processed), as well as stats per date (stat:failed:2014-10-20).

Also potentially store active connected workers / monitored queues.

Would be nice to have an admin panel similar to sidekiq where you can see running tasks / workers etc.

We could develop it as a plug so it can be added to Phoenix and other frameworks using Plug.

Expand tests for router.ex (https://github.com/akira/exq/blob/master/web/router.ex).

Add cases for each type of endpoint along with data expected.

I was doing some perusing of erlang and found

http://www.erlang.org/doc/reference_manual/processes.html

It looks like we can

  def dispatch_work(worker_module, method, args) do
    :erlang.spawn(String.to_atom("Elixir.#{worker_module}"), method, args)
  end

instead of

  def dispatch_work(worker_module, method, args) do
    :erlang.apply(String.to_atom("Elixir.#{worker_module}"), method, args)
  end

This will actually run the worker code as its own process. Im not 100% sure what this does vs having a worker process, but wanted to throw it out there incase it could help clean up the api at all.

We should provide a first class way to auth. For now i think basic auth is fine, but rather than rely on nginx or an upstream plug we should bake it in.

Error caused by plug dependency:

Looking up alternatives for conflicting requirements on plug
  Activated version: 1.0.0
  From exq v0.2.0: >= 0.8.1 and < 1.0.0
  From phoenix_html v2.1.1: ~> 0.13 or ~> 1.0
** (Mix) Hex dependency resolution failed, relax the version requirements or unlock dependencies

Once we settle on actor / supervisor hierarchy, make sure these scenarios are handled and have specs:

Operation fails, but manager / workers should not crash if:

• Enqueue fails due to Redis being down
• Enqueue fails due to incorrect Redis datastructure type
• Invalid JSON on dequeued job
• Redis connection lost

Can we have a way to run tasks at a certain time.

Basically this would set a time to run a task at and not dequeue an item for work until that time has passed.

Basically when a task is queued we put a timestamp, by default that would be NOW but it could also be any future DateTime, so if we build an interface where you say run a task in 5 minutes the time stamp would just be 5 minutes from now.

This would also allow us to build SideTiq like functionality where we could have recurring tasks by scheduling tasks based on their recurrence in the work checking loop.

Have manager shutdown workers and wait for a timeout when a terminate is sent to it. This depends on #5 having a worker pool for workers.

• Support Redis URL for options instead of having to pass each Redis component separately (host, port, password, etc).
• Still keep the separate options for people that would like to use those.

I think we have a pretty good base for a lot of use cases.

Personally I have branches that add specific functionality that I want but might not be perfect for everyone.

I would like to explore some sort of contrib libraries to add functionality without maintaining additional forks or shoe horning them into main line.

This proposal is to add hooks into exq mainline that would allow for 3rd party libraries to add extensions.

I'd love thoughts related to how we can build a system to allow middleware similar to what sidekiq supports.

I keep running into places where i want to start Exq, but dont want it to spawn workers. The WebUI is a great example of this. I think it makes sense to be able to use Exq to connect and talk to redis, however i dont want the UI to process to actually run jobs.

If we kept Exq.start as the general queue link, we can create a new supervisor to dispatch workers. I think this relates to #11 and #6

We probably want to expire workers incase they die without us knowing. This can be achieved by using http://redis.io/commands/expire. We would set the expire when the worker initializes and just refresh the expire in a timeout in the worker.

Thoughts?

We should track failures in redis, one to show a stack trace when problem occur and two to allow failures to be retried either manually or automatically.

Various metrics on worker performance (perhaps broken out by worker type):

• Frequency per second
• Average elapsed time
• Average wait time
• etc

This can be implemented as middleware. See Sidekiq Metrics for example.

Consider using: https://github.com/pinterest/elixometer

Use the chart library sidekiq does its pretty legit

host: '127.0.0.1',

Should support

host: "127.0.0.1",

Create harness similar to http://www.mikeperham.com/2013/06/30/background-job-processing-overhead/ to time something like 20k no-op messages

Retry failed jobs up to a limit. Allow configurable retries.

Sidekiq uses a "retry" key with sorted set / score to track this information. Here is a sample entry:

{"retry":true,"queue":"default","class":"Sidekiq::Extensions::DelayedClass","args":["---\n- !ruby/class 'Array'\n- :blah\n- - 1\n  - 2\n  - 3\n"],"jid":"441fc27d25a90f0b0588a1be","enqueued_at":1414340571.833299,"error_message":"undefined method `blah' for Array:Class","error_class":"NoMethodError","failed_at":"2014-10-26T16:23:10Z","retry_count":4,"retried_at":"2014-10-26T16:28:38Z"}

Will need to figure out how to configure retry period per worker.

If job queue behaviors can be abstracted out, look into supporting Disque:

https://github.com/antirez/disque

Disque is not production quality yet, so this would be more of a long term feature.

Would be nice to have something similar to the sidekiq binary that starts the sidekiq task runner.

I think we could implement this as a separate mix task.

mix exq.run or something along those lines.