qualityunit / resqu-server Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Describe the bug
In maintenance I found some stucked workers:
|7|[[email protected] ~]$ ps aux | grep resqu-v5-
root 3298 0.0 0.0 492948 11384 ? S 07:50 0:00 resqu-v5-11602-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 3308 0.0 0.0 492948 11436 ? S 07:50 0:00 resqu-v5-11601-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 11601 0.1 0.0 490900 5568 ? S máj03 24:12 resqu-v5-w-default_job_pool-s: Forked 3308 at 2020-05-13 00:50:14
root 11602 0.1 0.0 490900 5668 ? S máj03 24:08 resqu-v5-w-default_job_pool-s: Forked 3298 at 2020-05-13 00:50:14
It seems, that only exist and nothing to do.
Sending SIGTERM not have any effect:
|7|[[email protected] ~]$ kill 3298 3308
|7|[[email protected] ~]$ ps aux | grep resqu-v5-
root 3298 0.0 0.0 492948 11384 ? S 07:50 0:00 resqu-v5-11602-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 3308 0.0 0.0 492948 11436 ? S 07:50 0:00 resqu-v5-11601-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 11601 0.1 0.0 490900 5568 ? S máj03 24:12 resqu-v5-w-default_job_pool-s: Forked 3308 at 2020-05-13 00:50:14
root 11602 0.1 0.0 490900 5668 ? S máj03 24:08 resqu-v5-w-default_job_pool-s: Forked 3298 at 2020-05-13 00:50:14
Force killing was successfull:
|7|[[email protected] ~]$ kill 3298 3308
|7|[[email protected] ~]$ ps aux | grep resqu-v5-
root 3298 0.0 0.0 492948 11384 ? S 07:50 0:00 resqu-v5-11602-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 3308 0.0 0.0 492948 11436 ? S 07:50 0:00 resqu-v5-11601-std-proc: Processing job La_WorkReport_ProcessUserSlotReportJob
root 11601 0.1 0.0 490900 5568 ? S máj03 24:12 resqu-v5-w-default_job_pool-s: Forked 3308 at 2020-05-13 00:50:14
root 11602 0.1 0.0 490900 5668 ? S máj03 24:08 resqu-v5-w-default_job_pool-s: Forked 3298 at 2020-05-13 00:50:14
root 28504 0.0 0.0 112736 964 pts/0 S+ 09:39 0:00 grep --color=auto resqu-v5-
|7|[[email protected] ~]$ kill -9 3298 3308
|7|[[email protected] ~]$ ps aux | grep resqu-v5-
root 11601 0.1 0.0 186312 6528 ? R máj03 24:12 resqu-v5-w-default_job_pool-s: Shutting down
root 11602 0.1 0.0 359828 6660 ? R máj03 24:08 resqu-v5-w-default_job_pool-s: Shutting down
root 28576 0.0 0.0 112736 964 pts/0 S+ 09:39 0:00 grep --color=auto resqu-v5-
|7|[[email protected] ~]$ ps aux | grep resqu-v5-
root 28604 0.0 0.0 112732 964 pts/0 S+ 09:39 0:00 grep --color=auto resqu-v5-
Expected behavior
All ResQU 5 threads shutdown correctly without any killing.
Dependency version differences between LA and Resqu can cause serious issues with job execution.
The whole LA environment is directly included into resqu environment, which makes this a possibility.
There are few available solutions:
Edit: Another proposal was to merge resqu into LA and run workers as LA entry points. That approach is equivalent to the second option, because workers for every deployed version of LA would have to exist. As mentioned above, a significant effort is needed to achieve that, and at that point, there is no reason to prefer resqu over other solutions on the market.
at docker installation we were able to identify PID on worker processes which was used by different pool .
This happened at docker installation and after that, conversation indexing worker wasnt spawned, because pid was allocated by sendmails
Can we rethink allocation check in future ?
Log from Kibana
pcntl_sigtimedwait(): Interrupted system call in /opt/qu/php-resque-releases/php-resque-5.5.1/lib/Process/SignalTracker.php on line 66
Additional context
host: 4_app-q_la_ws-eu, 1_app-q_la_linode-us-nj, 2_app-q_la_linode-de
Logs from Kibana:
Uncaught Resque\Config\ConfigException: All pool names must be unique.
in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php:221 Stack trace:
#0 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php(123): Resque\Config\GlobalConfig->validatePoolNames()
#1 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php(64): Resque\Config\GlobalConfig::reload()
#2 /opt/qu/php-resque-releases/php-resque-5.6.4/scripts/startManagers.php(14): Resque\Config\GlobalConfig::initialize('/etc/resque-ser...')
#3 {main} thrown in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php on line 221
Details:
Whenever a new Resqu version is built, hash sum of generated archive should be calculated right away and placed in either separate file, release notes, whatever would be convenient or the most easy to implement. This is needed because in current workflow admins download the archive and calculating hash on their own, which means that the archive possibly could've been altered between the GH <-> admin, and there is no way to prove it wasn't.
Failure to find the file to include should result in standard job failure (Exception), as opposed to PHP process crash, so the exception handling code properly catches it.
Relevant place in code: \Resque\Job\Processor\StandardProcessor::includePath()
In order to update to PHP 8.1, compatibility should be checked and any found problems should be fixed.
Logs from Kibana:
Uncaught RedisException: socket error on read socket in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Libs/Colinmollenhour/Credis/Client.php:1172 Stack trace:
#0 [internal function]: Redis->eval('redis.call('DEL...', Array, 2)
#1 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Libs/Colinmollenhour/Credis/Client.php(1172): call_user_func_array(Array, Array) #2 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php(370): Resque\Libs\Colinmollenhour\Credis\Client->__call('eval', Array) #3 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Worker/WorkerImage.php(115): Resque\Redis->__call('eval', Array)
#4 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(88): Resque\Worker\WorkerImage->unregister()
#5 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(62): Resque\Maintenance\StaticPoolMaintainer->cleanupWorkers(10)
#6 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(45): Resque\Maintenance\StaticPoolMaintainer->maintain() #7 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(37): Resque\Init\InitProcess->recover()
#8 /opt/qu/php-resque-releases/php-resque-5.6.4/scripts/startManagers.php(21): Resque\Init\InitProcess->maintain()
#9 {main}
Next Resque\Libs\Colinmollenhour\Credis\CredisException: socket error on read socket in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Libs/Colinmollenhour/Credis/Client.php:1190 Stack trace:
#0 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php(370): Resque\Libs\Colinmollenhour\Credis\Client->__call('eval', Array) #1 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Worker/WorkerImage.php(115): Resque\Redis->__call('eval', Array)
#2 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(88): Resque\Worker\WorkerImage->unregister()
#3 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(62): Resque\Maintenance\StaticPoolMaintainer->cleanupWorkers(10)
#4 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(45): Resque\Maintenance\StaticPoolMaintainer->maintain() #5 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(37): Resque\Init\InitProcess->recover()
#6 /opt/qu/php-resque-releases/php-resque-5.6.4/scripts/startManagers.php(21): Resque\Init\InitProcess->maintain()
#7 {main}
Next Resque\RedisError: Error communicating with Redis: socket error on read socket in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php:412 Stack trace:
#0 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php(372): Resque\Redis->attemptCallRetry(Object(Resque\Libs\Colinmollenhour\Credis\CredisException), 'eval', Array)
#1 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Worker/WorkerImage.php(115): Resque\Redis->__call('eval', Array)
#2 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(88): Resque\Worker\WorkerImage->unregister()
#3 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Maintenance/StaticPoolMaintainer.php(62): Resque\Maintenance\StaticPoolMaintainer->cleanupWorkers(10)
#4 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(45): Resque\Maintenance\StaticPoolMaintainer->maintain() #5 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Init/InitProcess.php(37): Resque\Init\InitProcess->recover()
#6 /opt/qu/php-resque-releases/php-resque-5.6.4/scripts/startManagers.php(21): Resque\Init\InitProcess->maintain()
#7 {main} thrown in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php on line 412
Additional context
Level: Fatal_error
Host: 2_app-q_la_linode-us-tx
Logs from Kibana:
file_put_contents(/var/log/resque-serial-5.log): Failed to open stream: Permission denied in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Log/FileHandler.php on line 29
Details:
Today both app-q
machines in la.linode-de
were overloaded possibly by this mass action job. Both workers (1.app-q and 2.app-q)) had 100% CPU utilization that time and there were queue rising for async_rpc_jobs
which caused problems when computing filter was requested.
We were able to fix this by restarting resqu-5
service on both workers, when load on 2.app-q
lowers to normal values (around 11:42 CEST). 1.app-q
was overloaded even after restart, but I suppose that mass action need to be finished.
to be able to implement http driver and decouple the application
We found lingering queue locks without any trace of what might have happened to not remove them. The issue can be temporarily solved by simply purging resqu-v5:unique:queue_locks
.
Seeing that #11 is not going to be fixed any time soon, we should at least add a way to mitigate locks getting stuck. Since we are also slowly moving away from resqu, the proper fix might not be needed at all.
Currently, there is no actual Resqu documentation for its configuration. https://github.com/QualityUnit/resqu-server/tree/master/resources is clearly outdated and provides no context. We should have documentation with capabilities of what we may change, what is better not, and, generally, what can we do in order to adjust Resqu settings in production. Ideally, the doc's should give clear explanation of each setting that we may use, any details about the inner work of Resqu that in the opinion of author will be good to know for admins are very appreciated.
Logs from Kibana:
file_put_contents(/opt/qu/apps/accounts/u196628/accounts/default1/cache/lang/la_sk_5.37.2.18.s.php):
Failed to open stream: Permission denied in /opt/qu/apps/versions/la/5-37-2-18/include/Gpf/Lang/Storage/CacheFile.class.php on line 21
Details:
Logs from Kibana:
Array to string conversion in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Redis.php on line 404
Additional context
Level: Warning
Host: 2_app-q_la_linode-us-tx
Logs from Kibana:
Uncaught RuntimeException: Config file failed to parse. in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php:78 Stack trace:
#0 /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php(64): Resque\Config\GlobalConfig::reload()
#1 /opt/qu/php-resque-releases/php-resque-5.6.4/scripts/startManagers.php(14): Resque\Config\GlobalConfig::initialize('/etc/resque-ser...')
#2 {main} thrown in /opt/qu/php-resque-releases/php-resque-5.6.4/lib/Config/GlobalConfig.php on line 78
Details:
If I now we have mechanism what is doing removing resque locks. But customer ftmo.ladesk.com
has already lock 2 times and lock was not removed automatically. We don;t know what caused first lock but second lock seems was caused by error during execution of mass action job. Error is reported here https://github.com/QualityUnit/la-issues/issues/13439.
Whole situation caused number of emails send was just ~4 emails per hour and there was >1000 emails to be send. It seems the mostly emails instead of send was rescheduled.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.