uwiger / jobs Goto Github PK
View Code? Open in Web Editor NEWJob scheduler for load regulation
License: Apache License 2.0
Job scheduler for load regulation
License: Apache License 2.0
Would it be possible to tag a release, here and in Hex.pm?
Many thanks.
If possible:
parse_trans
is deemed OTP-24 ready and updated as a dep. in jobs
Trying to add a queue with jobs:add_queue(Name, [{regulators, ...}])
throws a warning in dialyzer:
The call jobs:add_queue(Name, Props::[{'regulators',[any(),...]},...]) will never return since it differs in the 2nd argument from the success typing arguments: (any(), [{'config',atom() | [atom() | [any()] | char()]} | {'counters',[{_,_}]} | {'group_rates',[{_,_}]} | {'interval',integer()} | {'queues',[{_,_} | {_,_,_}]}])
The jobs_server:add_queue
spec is defined here:
Line 148 in a04e8d4
And the option()
type doesn't allow regulators
as a top level option:
Lines 32 to 45 in a04e8d4
From @uwiger on Slack:
the type spec for
add_queue()
is wrong. From a quick look at the code, I think it should be-spec add_queue(queue_name(), q_opts()) -> ok.
โฆ andq_opts()
could well be improved to be more precise.
Hello,
I have a project with jobs
and hackney
as a dependencies, but I can't update hackney
to latest version and apply security fixes without manually overriding parse_trans
dependency to 3.2.0
, because latest hackney
depends on certifi 2.3.1
, which depends on parse_trans 3.2.0
, but jobs
depends on parse_trans 3.0.0
.
Is there anything that prevents jobs
to be dependent on latest parse_trans
(or maybe on parse_trans 3.x
)?
Rate-limited queue group with rate-limited queues are not scheduled evenly. First queue is always suffering.
Expected behaviour is for the queues to be fairly scheduled in a group rate limit scenario.
I've created a demo project showing this behaviour: x4lldux/jobs_group_queues repo (Elixir project, sorry).
Running Aggregator.test(n_queues, n_procs)
creates a group rate limit (hardcoded 5 req/s in this demo), nqueues rate limited queues (hardcoded to 3 req/s) and nprocs processes per queue. Queues are numbered from 0..nqueues-1. Each process will ask it's respective queue for a permission, and when granted, send a message to Aggregator
gen_server.
After all process send their messages (it takes approximately nprocs seconds), a "sum up" list is returned, where each element is a tuple containing time, a tuple with numbers of grants/messages sent for each queue (up to that point in time) and a list of queues activated in that time frame (per second).
This creates 5 queues and 30 process per queue:
iex(1)> Aggregator.test(5, 30)
[
{{10, 47, 40}, {1, 0, 1, 1, 1}, [2, 4, 3, 0]},
{{10, 47, 41}, {1, 1, 2, 3, 2}, [3, 1, 2, 3, 4]},
{{10, 47, 42}, {1, 3, 4, 3, 3}, [2, 4, 1, 2, 1]},
{{10, 47, 43}, {1, 3, 6, 5, 4}, [2, 3, 2, 4, 3]},
{{10, 47, 44}, {1, 5, 7, 6, 5}, [4, 2, 1, 3, 1]},
{{10, 47, 45}, {1, 6, 9, 7, 6}, [3, 1, 2, 4, 2]},
{{10, 47, 46}, {1, 7, 11, 8, 7}, [3, 2, 4, 1, 2]},
{{10, 47, 47}, {1, 8, 13, 9, 8}, [4, 2, 1, 3, 2]},
{{10, 47, 48}, {1, 8, 15, 11, 9}, [4, 2, 3, 2, 3]},
{{10, 47, 49}, {1, 9, 16, 13, 10}, [4, 1, 3, 2, 3]},
{{10, 47, 50}, {1, 10, 17, 14, 12}, [4, 3, 4, 2, 1]},
{{10, 47, 51}, {1, 11, 18, 16, 13}, [3, 1, 4, 2, 3]},
{{10, 47, 52}, {2, 12, 20, 17, 13}, [1, 2, 3, 0, 2]},
{{10, 47, 53}, {2, 12, 21, 20, 14}, [3, 2, 3, 4, 3]},
{{10, 47, 54}, {2, 13, 21, 22, 16}, [3, 4, 1, 3, 4]},
{{10, 47, 55}, {2, 14, 23, 22, 18}, [4, 2, 4, 2, 1]},
{{10, 47, 56}, {2, 15, 25, 24, 18}, [3, 2, 3, 2, 1]},
{{10, 47, 57}, {2, 16, 27, 25, 19}, [2, 3, 1, 4, 2]},
{{10, 47, 58}, {2, 19, 27, 26, 20}, [1, 4, 1, 3, 1]},
{{10, 47, 59}, {2, 20, 29, 27, 21}, [2, 4, 1, 3, 2]},
{{10, 48, 0}, {2, 21, 30, 30, 21}, [3, 1, 3, 2, 3]},
{{10, 48, 1}, {3, 23, 30, 30, 23}, [1, 4, 1, 4, 0]},
{{10, 48, 2}, {3, 25, 30, 30, 26}, [4, 1, 4, 1, 4]},
{{10, 48, 3}, {3, 28, 30, 30, 28}, [1, 4, 1, 4, 1]},
{{10, 48, 4}, {4, 30, 30, 30, 30}, [0, 1, 4, 1, 4]},
{{10, 48, 5}, {7, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 6}, {10, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 7}, {13, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 8}, {16, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 9}, {19, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 10}, {22, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 11}, {25, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 12}, {28, 30, 30, 30, 30}, [0, 0, 0]},
{{10, 48, 13}, {30, 30, 30, 30, 30}, [0, 0]}
]
In this example first queue is rarely scheduled until most of the other queues are done. The last 10 seconds is spent solely scheduling the first queue because all others have already finished.
rebar3 xref
complains about Warning: jobs_server:options/0 calls undefined function setup:find_env_vars/1 (Xref)
.
Looking at the code, I'm not sure we're it's supposed to come from (the function, not the warning).
Hi @uwiger, the spec for jobs_server:modify_queue
missed {error, Reason}
. To avoid the dialyzer warning when other applications using jobs, we should add it.
I was trying to setup 2 queue sharing the same group_name but failed with a case_clause error while trying to use a tuple as a group_rate name. Creating the group name with a tuple is OK, but then passing this name when setting a queue fails with a case_clause
error. AFaik the group name should match an atom there:
https://github.com/uwiger/jobs/blob/master/src/jobs_server.erl#L488
I'm not sure what should be fixed there. Either fixing the code above by not matching a specific type for the name. Or fixing jobs:add_group_rate/2
and raise a badarg
error if the name is anything but atom.
Thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.