shanek / matador Goto Github PK
View Code? Open in Web Editor NEWFront-end web interface for Bull Job Manager
License: MIT License
Front-end web interface for Bull Job Manager
License: MIT License
Current knockout data-bound tables are having issues where they sometimes get into infinite loops. For some reason, sometimes even if you only have one row it just keeps adding rows to the table forever. Not sure why.
I think converting the tables to react components would help with this issue. React components are also pretty fast in general.
repro:
in Failed screen, click the gear icon dropdown, then click "delete all failed jobs"
expected: the jobs disappear
actual: the jobs are still there. I can see in the browser console there is an error:
Uncaught TypeError: Cannot read property 'name' of undefined
at Object.deleteByStatus (RedisHandler.js:35)
at HTMLAnchorElement.onclick (VM1131 failed:1)
deleteByStatus @ RedisHandler.js:35
onclick @ VM1131 failed:1
using versions:
"bull": "2.2.6",
"bull-ui": "1.2.1",
here is one of the failed jobs i have:
Job ID: 4
Type: test job
Status: failed
Data: {
"num": 39906
}
Stack Trace:
["TypeError: _.rand is not a function\n at Queue.testJobBullQueue.process.job (/Users/brad/repos/nd-api/api/controllers/test-job.js:71:37)\n at Queue.tryCatcher (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/util.js:16:23)\n at Queue.handler (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/method.js:15:34)\n at /Users/brad/repos/nd-api/node_modules/bull/lib/queue.js:742:30\n at bound (domain.js:280:14)\n at runBound (domain.js:293:12)\n at tryCatcher (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/promise.js:512:31)\n at Promise._settlePromise (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/promise.js:569:18)\n at Promise._settlePromise0 (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/promise.js:614:10)\n at Promise._settlePromises (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/promise.js:693:18)\n at Async._drainQueue (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/async.js:133:16)\n at Async._drainQueues (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/async.js:143:10)\n at Immediate.Async.drainQueues (/Users/brad/repos/nd-api/node_modules/bluebird/js/release/async.js:17:14)\n at runCallback (timers.js:637:20)\n at tryOnImmediate (timers.js:610:5)\n at processImmediate [as _immediateCallback] (timers.js:582:5)",null]
Currently there is no way to have Matador listen to another redis db other than db 0, since bull
now supports using user defined user db, Matador should be able to do so as well.
Hi Shane as you know I built those 2 other components off your core... but I am just realizing there is no license in this project... Could you please add a license ?
Thank you
I would like to password protect the interface. I tried to copy the http-auth and express method but it didn't work. How can I achieve this?
I love the Matador UI. But it's slow for me.
It worked great when I was prototyping. But now in production, I have 6 queues, each with a worker, running in Redis, handling about 2-3 jobs per second each.
The workers run fine, but once they get going, maybe 15-30 mniutes later, the Matador UI gets stuck. The express path just times out at 30s. Refresh the browser and rebooting the workers doesn't seem to fix it.
It seems to get stuck after several hundred thousand jobs have completed, or when a lot of jobs are running at once. During this time, the jobs keep working hunky-dory. It's just the Madator UI
Once stuck, even if I stop all the workers, still stuck. Only clearing redis completely seems to unstick it.
Could there be some sort of redis locking happening? Or are there scaling issues with Bull or something?
Maybe Matador is trying to load hundreds and hundreds of job detaisl under the hood, eagerly? I would figure it should only lazy-load details once I drill down, and just get counts at first load.
Here's an example:
https://github.com/sayanee/Kraken_Example_SocketIO
We are using nsp
to check for vulnerabilities and bull
has 3 of them due to old version of express
. Is there any plan to upgrade to a newer version of express
?
> nsp check
(+) 3 vulnerabilities found
Regular Expression Denial of Service
Name ms
Installed 0.7.0
Vulnerable <=0.7.0
Patched >0.7.0
Path > [email protected] > [email protected] > d
More Info https://nodesecurity.io/advisories/46
Root Path Disclosure
Name send
Installed 0.10.1
Vulnerable <0.11.1
Patched >=0.11.1
Path > [email protected] > [email protected] > s
More Info https://nodesecurity.io/advisories/56
Regular Expression Denial of Service
Name ms
Installed 0.6.2
Vulnerable <=0.7.0
Patched >0.7.0
Path > [email protected] > [email protected] > s
More Info https://nodesecurity.io/advisories/46
I'm getting this error now:
knockout.min.js:60 Uncaught ReferenceError: Unable to process binding "text: function (){return pending }"
Message: pending is not defined
hi there. thanks for having built this. quick question though -- why are you doing your requires this way
updateInfo = require(process.cwd()+'/lib/updateInfo.js');
instead of, say, this way
updateInfo = require('../lib/updateInfo.js');
(depending of course where the file is)
I guess one has to choose whether do requires relative to an expected root (e.g., process.cwd()) versus relative to files. I think the latter is better.
Reason I think so, is what if I've required your Matador in another app, now all the paths are wrong.
I just want to know if there's something im unaware of as to why you did it this way?
thanks
Express updated and changed the way Kraken hooks into Express. The temporary solution is requiring Matador to use older versions of some of its dependencies (resolved issue #3), but this can't be relied on for long because eventually they'll remove the older versions of their packages (and also this is a crummy way to fix things). There is also a deprecation warning when starting Matador with this temporary solution, although Matador still works.
Need to update:
Make Matador use the latest versions of Kraken/Express
I do not think that Matador can handle URL parsing when the input is as follows:
var app = require('bull-ui/app')({
redis: {
url: 'redis://:password@host:6379'
}
});
It automatically falls back on 127.0.0.1:6379
after failing to parse the URL. I realized this after I tried to create a job from the /newjob
page.
Here are my dependencies:
"dependencies": {
"bull-ui": "1.2.3"
},
There is obviously a workaround:
var app = require('bull-ui/app')({
redis: {
host: 'pulkit-dev-3.shoppinpal.com',
port: '6379',
password: 'r3dis'
}
});
But it thought it worth mentioning the breaking behaviour above.
Currently data is just instantly refreshing itself pretty quickly. It's spamming the server and it makes it difficult to look at stuff in the networking tab.
This isn't actually too terrible, but it could be done better. I think switching to long polling or socket.io would be a better solution than what's currently in place.
2017-05-25T14:05:44.041Z - error: uncaughtException: failedJobs.keys[explodedKeys[1]].indexOf is not a function date=Thu May 25 2017 15:05:44 GMT+0100 (BST), pid=92140, uid=501, gid=20, cwd=/Users/etienne/Github/panelise/express, execPath=/usr/local/bin/node, version=v6.9.4, argv=[/usr/local/bin/node, /Users/etienne/Github/panelise/express/bin/www], rss=76759040, heapTotal=67305472, heapUsed=56975568, loadavg=[3.0810546875, 3.7080078125, 3.27099609375], uptime=1452267, trace=[column=106, file=/Users/etienne/Github/panelise/express/node_modules/bull-ui/models/redis.js, function=null, line=230, method=null, native=false, column=54, file=/Users/etienne/Github/panelise/express/node_modules/q/q.js, function=_fulfilled, line=794, method=null, native=false, column=30, file=/Users/etienne/Github/panelise/express/node_modules/q/q.js, function=self.promiseDispatch.done, line=823, method=promiseDispatch.done, native=false, column=13, file=/Users/etienne/Github/panelise/express/node_modules/q/q.js, function=Promise.promise.promiseDispatch, line=756, method=promise.promiseDispatch, native=false, column=44, file=/Users/etienne/Github/panelise/express/node_modules/q/q.js, function=null, line=564, method=null, native=false, column=17, file=/Users/etienne/Github/panelise/express/node_modules/q/q.js, function=flush, line=110, method=null, native=false, column=7, file=internal/process/next_tick.js, function=_combinedTickCallback, line=67, method=null, native=false, column=9, file=internal/process/next_tick.js, function=process._tickDomainCallback, line=122, method=_tickDomainCallback, native=false], stack=[TypeError: failedJobs.keys[explodedKeys[1]].indexOf is not a function, at /Users/etienne/Github/express/node_modules/bull-ui/models/redis.js:230:106, at _fulfilled (/Users/etienne/Github/express/node_modules/q/q.js:794:54), at self.promiseDispatch.done (/Users/etienne/Github/express/node_modules/q/q.js:823:30), at Promise.promise.promiseDispatch (/Users/etienne/Github/express/node_modules/q/q.js:756:13), at /Users/etienne/Github/express/node_modules/q/q.js:564:44, at flush (/Users/etienne/Github/express/node_modules/q/q.js:110:17), at _combinedTickCallback (internal/process/next_tick.js:67:7), at process._tickDomainCallback (internal/process/next_tick.js:122:9)]
Program node ./bin/www exited with code 1
Is there a way to gracefully shutdown Matador? An interval is still running while attempting to do a graceful shutdown of the server:
In the README it says you can use require('matador/app') which is great but the app isn't available on npm (as far as I know, matador on npm is something else). It would be a great way to include the ui as part of a worker to also make the upgrade path easy when you don't have any need for customizations anyway.
It does not show anything if I'm using bull queue in a different namespace (e.g dev:bull and prod:bull).
Regards.
There is some dependency problems on a fresh install. After a clone, I run npm install and get this:
npm ERR! peerinvalid The package express does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer [email protected] wants express@^3.16.1
npm ERR! System Linux 3.13.0-24-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /mnt/data/code/Matador
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR! code EPEERINVALID
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /mnt/data/code/Matador/npm-debug.log
npm ERR! not ok code 0
I can chase it down locally, but it is something of a barrier to adoption ;)
Example code to reproduce this.
'use strict';
var Bull = require('bull');
var queue = new Bull('test', 6379, '127.0.0.1');
queue.pause();
queue.add({ name: 'first one' });
It looks like the version of lodash included has a low severity vulnerability. After npm install bull-ui
and npm audit
:
=== npm audit security report ===
┌──────────────────────────────────────────────────────────────────────────────┐
│ Manual Review │
│ Some vulnerabilities require your attention to resolve │
│ │
│ Visit https://go.npm.me/audit-guide for additional guidance │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in │ >=4.17.5 │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ bull-ui │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ bull-ui > lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/577 │
└───────────────┴──────────────────────────────────────────────────────────────┘
I can certainly submit a pull request for this, but I haven't done anything with Matador other than install it, so don't know what kind of ramifications there could be from bumping lodash up two major versions...
Need to implement getDelayedKeys
function that looks for redis keys of the form bull:*:delayed
. Currently Matador lists delayed jobs as "stuck"
Need to add password option for people who have password protected redis databases. Making this an issue because I keep forgetting about it.
Greetings!
I'm considering redoing the front end, since the current front end is relying Knockout, which isn't particularly well known and fails in comparison to some modern solutions -- such as Anular or React.
I'm opening this issue to get an idea of what you guys think about this. I don't currently use this project anymore, but I understand that it is used, and if it would make it easier to use and contribute to in the future then I think this could be a valuable change to this project.
So, what do you think? Also, if you like the idea of the front end framework being changed, would you prefer Angular or React? I currently know most about Angular, but I'm not opposed to learning React.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.