Comments (11)
Ok, I think I just found a solution for this via conditional unique indexes (learn something new every day). Give me a week to play around this technique. Hopefully will result in a 🎉
from pg-boss.
@jede Did you already try out the throttling config and this is not what you're looking for?
Oh, and thanks for 🎉! Glad to hear it's working out for you guys. 👍
from pg-boss.
Hi Tim!
I looked at throttling, and made a brief test. From what I could see a new job will start running after the throttle threshold even if there is one running?
from pg-boss.
You're right. Throttling is not as strict as what you're looking for. I do a similar workflow in my app where I use node-schedule with a cron expression to wait until the proper time to kick off a job in pg-boss. However, my interval is more forgiving than yours it would seem. :) If my task takes longer than expected, i have a lot of wiggle room before the next interval fires.
I think I could add a feature like throttling that would basically block queueing a job if a certain key is detected to be in the queue or actively running, but I want to makes sure I understand how you think it should behave from the API's perspective. Until then, are you currently working around this by querying the job table yourself?
from pg-boss.
Also, I forgot to mention my use case for throttling, because I wanted to compare it to yours. I have a task that I want to make sure doesn't run too often (not more than once per minute), but I could have several requests for that job arrive at the same time. I use throttling to make sure another job is not queued if one has already been submitted during the specified timebox.
from pg-boss.
Right now we don't really solve it in a good way :) We want to make sure that if a task takes longer then usual, we won't fire another and put even more load on the systems, so for us its better to let the first one finish instead of running them in parallel.
from pg-boss.
Do you think adding a status(jobId:uuid)
func to the API would help you out? It might perhaps return all the job properties with or without the payload. For example, if you're keeping the job id that you submitted, you could use that id to fetch the status of your job before enqueuing another one.
from pg-boss.
from pg-boss.
My first thought was in regards to a constraint so pg can be a centralized enforcer of uniqueness/correctness, especially in distributed use cases where you have multiple workers and you can't use local state to guard against this.
The status values are
created
active
complete
expired
cancelled
However, since pg-boss manages the status values internally, using status as a unique constraint may produce internal constraint violations and then we have a more complicated problem to solve. Currently, any job with status created
or expired
coupled with a retryCount < retryLimit could become active on the next request from a worker. So instead of just being concerned with active
, one would also have to be concerned with these other cases as well.
A temporary solution you could implement that would satisfy your requirements and persistence (if your process restarts or crashes, for example) would be to store this "something was submitted" state in a different table. Each time pg-boss calls your callback and you finish the work, you could reset this state along with calling done()
to indicate the job completed in pg-boss.
I think there's a way I can pull this off with a unique constraint but I haven't figured it out yet lol. I don't want to have to lock the table obviously. For exampe, if you rely on fetching, you could have 2 processes receive the "nothing was active right now" response and then both would submit 2 jobs simulaneously. That's the really nice thing about the database managing the constraint to prevent that race condition.
from pg-boss.
@jede I just pushed a beta release for this. If you don't mind, give it a try on a non-production instance and let me know your results. I've added a new option on publish called singletonKey
that implements this behavior.
The following tests show it in action:
https://github.com/timgit/pg-boss/blob/61705996e0a07f14b938247c5c19b6679a251b5d/test/singletonTest.js
from pg-boss.
Wow! Thanks! I'll give it a try 😃👏
from pg-boss.
Related Issues (20)
- Scheduling many individual jobs for the same queue HOT 3
- Document onComplete option HOT 1
- Job heartbeat support HOT 4
- Silently rejecting jobs when the db is not connected HOT 1
- New UUIDv7 increased performance and indexes HOT 2
- Add a way to limit maximum retry delay for exponential backoff
- Should expire_in_seconds set the state to expired? HOT 1
- Documentation for PgBoss constructor db option.
- sub queues or queue groups
- Specify multiple host connections for PostgreSQL / Database Fail-Over
- Use of MD5 for Advisory Locks Crashes pg-boss on FIPS-Compliant Platforms HOT 1
- useSingletonQueue missing from JobInsert type
- Error: Jest has detected the following 1 open handle potentially keeping Jest from exiting HOT 3
- Typo in validation for Postgres names HOT 3
- Update types for v10 HOT 2
- db.lock is not a function HOT 1
- Feature Request: Bull-Board support/Fork HOT 6
- Job partitions pollute public schema HOT 1
- Tests failing in CI workflow HOT 1
- Issues with queue in v10 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pg-boss.