maragudk / goqite Goto Github PK
View Code? Open in Web Editor NEWGo queue library built on SQLite and inspired by AWS SQS.
Home Page: https://www.goqite.com
License: MIT License
Go queue library built on SQLite and inspired by AWS SQS.
Home Page: https://www.goqite.com
License: MIT License
Great work Markus - exactly what I need in some side project.
Is it possible to add other database drivers, for example postgres, like in gorm?
According to schema.sql
, it currently stores message body and metadata in same table goqite
.
create table goqite (
id text primary key default ('m_' || lower(hex(randomblob(16)))),
created text not null default (strftime('%Y-%m-%dT%H:%M:%fZ')),
updated text not null default (strftime('%Y-%m-%dT%H:%M:%fZ')),
queue text not null,
body blob not null,
timeout text not null default (strftime('%Y-%m-%dT%H:%M:%fZ')),
received integer not null default 0
) strict;
We experienced (poor) performance issue with large BLOG with similar sql table, so i'd like to share our story briefly for your reference.
Our customer imported mailbox into the software which generated about 1000 records and 8GB in total (in SQLite, of course), each record is an email message about 8MB large. Our program had a bug and we had to update the SQL records manually for troubleshooting, while performing SQL command like UPDATE <table> SET received=1;
which affects all rows, SQLite command line tool toke a long time (~ 8min) to finish. This is not right and we started checking SQLite doc to figure it out.
We found the root cause (improper sql table design) and solution (separate large BLOG and metadata), hope it helps.
One thing in particular that I've found to be a little annoying to deal with is the lack of type-safety. You can put any byte slice in to jobs.Create()
or Send()
which means potential run-time errors and no IDE support. You also have to manually deal with serialization on both the caller and receiver side for every single queue. I had the same issue with asynq.
Adding type-safety is possible. I put together a very basic POC that I believe will work. Each queue would have to be declared with the type you intend to receive and process.
Is this something you would be interested in? If not, since it's a rather large refactor, I completely understand.
It would be nice if a given queue could be given one of three retention options, rather than automatically deleting the job once it's complete:
time.Duration
While the logs do provide some insight in to what's happening, having the ability to retain logs can be great for auditing and monitoring purposes. That could also provide the ability to have a dashboard of some sort, seeing how many jobs ran, how many completed, failed, etc. You could also consider storing the execution time of each job, which can be another helpful thing to audit.
I thought it would be useful to be able to specify either a specific date and time or duration to delay executing a given task.
Probably composite on queue name, creation date
For when a message is unable to be handled for whatever reason. Kinda like NAck in other queues.
I don't see why the program should panic when the database contains a job for which a runner has not been registered for. This seems like an easy state to get into when updating an application; especially since there is nothing cleaning up stale jobs as far as I can tell.
I'm interested in adding a delivery delay mechanism to your library to allow messages to be scheduled for future delivery. It would prevent messages from being delivered to runners until a specific time has been reached. After this time has passed, the message would be available to any runner to be delivered. If no time is set or the time scheduled is in the past, the behavior would be the library's existing behavior.
Please let me know if you'd be interested in receiving a PR for this improvement and if you have any guidelines you'd like me to follow. If so, feel free to assign me to the issue and I will begin work on this.
This improvement would most likely require a database migration.
Hi!
I've been using this library all morning and it is fantastic. I just wanted to pose this more as a question or usability suggestion. For most of my sqlite projects I initialize all my tables as CREATE TABLE IF NOT EXISTS
and run my ddl on setup for my apps.
Currently using the library I have to comment out goqite.Setup()
after the first run otherwise it fails since the table already exists. Would you be open to changing the default schema to CREATE TABLE IF NOT EXISTS
or is there a preferred pattern / just catching and checking the error is for the 'table already exists' error and moving on?
Thanks!
Those that have a receive count over the max.
Hi there. With reference to PR #41, it would be great if goqite could handle a particular use case I have.
I'm building a REST-ish API. Users submit jobs which are long-running (minutes/hours). Upon submission the API immediately responds with a '202 Accepted', and jobs are added to a goqite queue. According to received wisdom, the response sent to the user should include a 'Location' header with a URL where the user can check job progress, e.g. Location /task/123
where 123
is the ID of the job.
However, since the Send
method of goqite.Queue
does not return anything, I cannot reliably discern the ID of the item the user submitted. Therefore I can't build the /task/123
URL, and the user can't check progress.
I guess I could embed some identifying information into the job body, but this would need to be generated by the client whereas that's clearly a server-side responsibility. I'd also need to parse many large BLOBs in the database to look-up any item, which would be inefficient.
Consequently, I believe it would be useful if Send
could return the ID of the row it inserts. This is trivial in SQL with a returning
clause, supported in SQLite since version 3.35.0 (2021-03-12).
PR #41 provides this functionality. However, implementing it resulted in a cascade of changes throughout the code-base, which I appreciate isn't ideal. Happy to discuss alternatives! ๐
For number of messages, dead messages, maybe more.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.