Coder Social home page Coder Social logo

processmanager's Introduction

Pimcore - Process Manager

Requirements

  • Pimcore 11

Software License

Process Manager Plugin keeps track of all your "long running jobs". It adds a native GUI and a new portlet for your Dashboard. You can also create Executables and run them with one click. (It's planned to integrate a CRON like Syntax for recurring tasks)

Plugins using Process Manager

Getting started

  • Install via composer composer require dpfaffenbauer/process-manager:^2.0
  • Enable via command-line (or inside the pimcore extension manager): bin/console pimcore:bundle:enable ProcessManagerBundle
  • Install via command-line (or inside the pimcore extension manager): bin/console pimcore:bundle:install ProcessManagerBundle
  • Reload Pimcore
  • Open Tools -> Process Manager

Integrate to your Task

Create new Process

$processFactory = $container->get('process_manager.factory.process');
$process = $processFactory->createProcess(
    sprintf(
        'Process (%s): %s',
        $date->formatLocalized('%A %d %B %Y'),
        'Special Long Running Task'
    ),                                                  //Name
    'special_task',                                     //Type
    'Message',                                          //Message Text
    100,                                                //Total Steps
    0                                                   //Current Step
);
$process->save();                                       //Save

Advance the Progress

$process->progress();
$process->save();

Finish the Progress

$process->setProgress($process->getTotal());
$process->save();

Using the Process Logger

Process Manager also provides you with the ability to Log what exactly happens in your progress.

$logger = $container->get('process_manager.logger');

//Logs a emergency message
$logger->emergency($process, 'Total of 100 entries found');

//Logs a alert message
$logger->alert($process, 'Total of 100 entries found');

//Logs a critical message
$logger->critical($process, 'Total of 100 entries found');

//Logs a error message
$logger->error($process, 'Total of 100 entries found');

//Logs a warning message
$logger->warning($process, 'Total of 100 entries found');

//Logs a notice message
$logger->notice($process, 'Total of 100 entries found');

//Logs a info message
$logger->info($process, 'Total of 100 entries found');

//Logs a debug message
$logger->debug($process, 'Total of 100 entries found');

Reports

You can also further process the log to create a pretty report. To do that, you have to create a new service and implement the interface ProcessManagerBundle\Report\ReportInterface. Import Definitions has an example implementation of that Import Definition Report

Add a new Process Type

  • Add a new Class to your Bundle and implement ``ProcessManagerBundle\Process\ProcessInterface``` Interface
  • Add a new Form Type to your Bundle and add required fields to it
  • Add a new Service with tag process_manager.process
      import_definition.process_manager.process:
          class: Wvision\Bundle\ImportDefinitionsBundle\ProcessManager\ImportDefinitionProcess
          tags:
          - { name: 'process_manager.process', type: 'importdefinition', form-type: 'Wvision\Bundle\ImportDefinitionsBundle\Form\Type\ProcessManager\ImportDefinitionsType' }
  • Thats it, done. (You still need to handle Process creation within your Bundle yourself, there is no magic behind it)

Stoppable processes

You can implement your process to be stoppable by user via the admin panel. You need to set the stoppable flag of the process to true and it's status to ProcessManagerBundle::STATUS_RUNNING for the stop button to shop up:

$process->setStoppable(true);
$process->setStatus(ProcessManagerBundle::STATUS_RUNNING);
$process->save();

Additionally, you need to implement stop logic to your process. Track the process status and stop your process if it's set to ProcessManagerBundle::STATUS_STOPPING:

$process = $this->processRepository->find($processId);
if ($process->getStatus() == ProcessManagerBundle::STATUS_STOPPING) {
    // Here goes your process stop and cleanup logic
    ...
    
    $process->setStatus(ProcessManagerBundle::STATUS_STOPPED); // remember to set the status to stopped.
    $process->save();    
}

Cleanup command

You can execute a cleanup command from the console to delete old process entries and log files. To do this on a regular basis, you can add it as a cronjob.

# delete all process entries from the database and log files older than 604800 seconds (7 days)
$ ./bin/console process-manager:cleanup-process-data
 
# delete all process entries from the database and log files older than 86400 seconds (1 days)
$ ./bin/console process-manager:cleanup-process-data --seconds=86400

# delete only process entries from the database older than 604800 seconds (7 days) and keep the log files
$ ./bin/console process-manager:cleanup-process-data --keeplogs

Copyright and license

Copyright: lineofcode.at For licensing details please visit LICENSE.md

Interface Interface Interface

processmanager's People

Contributors

aarongerig avatar damijank avatar das-peter avatar dkarlovi avatar dpfaffenbauer avatar hethehe avatar kubaplas avatar mugge6 avatar ramundomario avatar roynilsson avatar wpeisert avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

processmanager's Issues

Add a troubleshooting note in the README: shell_exec required

On my current production install (WHM based), shell_exec is disabled by default.

This makes the plugin unusable, but it doesn't display any errors or warnings why, it behaves like everything is OK, except the process never starts.

I'd add better error checking when the process is started and a note in the README saying shell_exec is mandatory.

Add a way to attach artifacts to the completed process

If you run an export with ExportDefinitions ;) you'll end up with a file to download.

The link might point to a single file or a group of files, so called "job artifacts". Nomenclature is quite common with these job-based configurations, see for example https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html

If the process creates artifacts, the link to download the archive should be made available from the UI.

If the process is deleted, the download probably should too (?)

Duplicate XHRs

The store (or something) creates a new refresher on each process delete. If you delete 10 processes, it will do 10 reqs each iteration.

Add an index on process_manager_executables

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master
alter table `process_manager_executables` add index `idx_active_cron` (`active`,`cron`)

As suggested by Azure's MySQL performance recommendations.

Make Executables deletable

When you create a new Executable it is not possible to delete it through the ExtJS UI afterwards. I think it would be nice to be able to delete them though.

Now, if I was a good and experienced ExtJS developer like you are, then this would not be an issue, it would be a pull request. ;) Shall I try?

Throttle storing progress if lots of updates frequently

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Currently, when I'm importing a largeish Coreshop index, it will dispatch a status after each item indexed. If I also have process manager installed, this will update the database after each item. Currently this amounts to almost 20% of my runtime (indexing into Elasticsearch), which is way too much for what amount to secondary info.

Index-prod-with-dispatch-event-profile-Blackfire

The progress data is not needed in actual real time anyway since it's polled by PM in the admin, which uses a timeout based system. We could easily have the same interval here, basically not propagate the update if we had an update less than a second ago (or similar).

Process Manager CRON

Hello, Dominik.

We use your Process Manager bundle (2.5.1) and it is very helpful.
What about CRON integration? Is it enabled now?
When I run CLI or pimcore command by clicking „RUN” buton, everything works. But when try to configure CRON, nothing happens. I tried to enter * * * * *, and also without spaces *****, but it did not work.
I cannot find any instruction or tutorial for this, so I am asking you, i sit possible to setup recurring tasks or not.
pimcron

Process list scrolls to top on refresh

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? 2.6.x

Atomatic refresh of process grid causes it to scroll to top, which is pretty annoying, especially for a big lists (you cannot easily view logs for a process in the middle of the list).

I tried to fix this with preserveScrollOnRefresh: true, unfortunately didn't work for me.

Button to clear old jobs from the UI

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Simple enough: a UI button to purge ended jobs (which completed successfully or errored).

Don't run the jobs in background since they're now running on workers

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? master

Since #75 we're now running jobs on workers. Since the workers are meant to take one job at a time, work on it and then take another, they shouldn't run the actual jobs in the background, from their POV it should run in the foreground (as in, they should block until the job is done).

Currently, if you for example run two large import / export jobs, they will both be started immediately and compete for resources on the same worker, while we actually want them to be sequential and the worker is fully dedicated to the single job.

It seems the change would be in \ProcessManagerBundle\Process\Pimcore::run, correct?

Cache problems after installation

On fresh Pipmcore Installation:

When I enable processmanager, after reload I got:
Status: 404 | Not Found
URL: /admin/coreshop/resource/config
Message: No route found for "GET /admin/coreshop/resource/config"

Clear cache helps.
(The same is with ImportDefinitions)


processmanager - first dialog opening:

Timestamp: Fri Oct 19 2018 09:15:49 GMT+0200 (CEST)
Status: 404 | Not Found
URL: /admin/process_manager/executables/get-config
Method: GET
Message: No route found for "GET /admin/process_manager/executables/get-config

Cleanup Log Files

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Hi @dpfaffenbauer

We are thinking about having some clean up logic also for the log files. Currently there is an action to cleanup entries older than (default)7 days for the DB entries.

Concrete we could contribute following:

What your thougts about that?

Missing Tag for last pimcore 10 compatible version

Q A
Bug report? kinda
Feature request? not really
BC Break report? Kinda?
RFC? no
Branch? master

We use "dpfaffenbauer/process-manager": "dev-master" notation to declare the dependency to this package - this because e.g. 07553621c74367ffc5149102a3ae91972b6b4cc1 wasn't part of any official release.
However, with the recently introduced dependency changes in composer.json (9dac8d8) we can no longer use that notation.
Even a "hash-pinning" "dpfaffenbauer/process-manager": "dev-master#07553621c74367ffc5149102a3ae91972b6b4cc1", doesn't work due to limitations on how composer resolves dependencies (composer/composer#6366)

I think the best way to solve this would be to tag the last commit before the Pimcore 11 update as 3.1.1

Task stops before starting again?

Q A
Bug report? no
Feature request? no
BC Break report? no
RFC? no
Branch? master

Hey Guys,
I was just wondering, are the tasks being killed before starting again?
Some background: I have some few million products to import and have made an import task. Since there are so many products to work with, the process dies after some time (I'm assuming php memory gets full). There might be some workarounds, like manually collecting garbage, but it's not really necessary for the workflow, as the importer will run pretty much daily and check for any changes.
What I wanna do: add a cron task that starts every... say 30 min.
So what would happen if the task is still running when the cron hits the 30m mark? Does it kill the existing task and create a new one, ignores it for this run and tries to start again in 30m, or creates a new identical task?
I'm assuming it would just skip it and try again in another 30m, but figured it's better to ask you guys.

Thanks!
Alex.

Add pagination / filtering / batch delete action

When there is a big amount of processes (>100) in the database, loading them every 10 seconds starts to have an impact on performance of backend and frontend. On larger sets ExtJs scroll stops working or the process list doesn't load at all (saying communication failure).

This could be fixed by implementing at least one of, or better all of:

  • pagination to process list
  • allow filtering (using db in a request)
  • allowing batch delete

Similar things can be done to executable list, but as the list is not refreshed automatically, it's not a huge problem.

Migrate process_manager_executables.settings to JSON

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Currently, executable settings are stored as PHP serialized payloads. That means the database treats them as text and it's quite possible they become corrupted, which is exactly what happened in my case.

My suggestion is to migrate this field to MySQL's JSON data type. See pimcore/pimcore#13359

Note: the symptom of this happening is getting

Process with id 212 not found

in the logs even though the executable exists. It can't unserialize the settings payload so the exception is assumed to be a "not found" type exception, but it isn't.

Process start and completed timestamps

I would suggest adding two timestamp fields to the Process model. One for when the process was started and one for when the process completed. In our case this would help a lot when viewing historical processes.

Feature Process queue

Q A
Bug report? no
Feature request? yes/no
BC Break report? no
RFC? no
Branch? master

I have a need for a process queue. Not sure if that is something that would fit into this bundle or if I should keep it in my own bundle. Today we have executables that can be ran regularly using a cron statement. I would like to be able to create jobs and add them to a queue. A job would be an executable and optional settings data. Settings data in the job would override default settings of the executable. There would be an EventListener similar to the current CronListener that get all scheduled jobs and tries to run them. We could add a new interface SchedulableProcessInterface that extends the ProcessInterface. This interface adds a canRun method that is called before running the job. In canRun any kind of logic could be added to determine if the job in question should run at any given time. This would not affect currently implemented process types. One would need to implement custom process types to make use of it.

Allow running jobs via the Messenger

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Instead of running jobs directly during the run request, schedule the job on the Messenger queue and have a handler which runs the job on the worker instead.

Note: if using cron, the jobs might already be running via Pimcore's mainenance being moved over to the worker.

Roadmap for Pimcore X?

Q A
Feature request? yes

Hello,
currently the ProcessManager is not compatible with Pimcore X version either with PHP 8.0.
I would like to know if there are plans for upgrading the extension 😄

Regards!

Error for detail report after Pimcore update

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? master

Hi there

After updating our Pimcore to 6.8.3 we're getting an error after clicking on the Report icon.
On the Debug side we see that after the forceFind method is called there is a check if the Object "ProcessManagerBundle/Model/Process" is not an instance of a Concrete object and the method returns null.

Do you have an idea what the problem could be?

composer.json:

"require": { "php": ">=7.2", "dachcom-digital/formbuilder": "3.3.4", "dpfaffenbauer/process-manager": "dev-master", "elasticsearch/elasticsearch": "^7.9", "pimcore/customer-management-framework-bundle": "2.5.7", "pimcore/pimcore": "6.8.3", "pimlab/website-settings": "1.0.5", "wikimedia/composer-merge-plugin": "^1.4" },

Stack Trace:

Timestamp: Wed Oct 21 2020 16:07:01 GMT+0200 (Mitteleuropäische Sommerzeit)
Status: 404 | Not Found
URL: /admin/process_manager/processes/log-report
Message: The "59" has not been found
Trace:
in /var/www/pimcore/vendor/coreshop/core-shop/src/CoreShop/Bundle/ResourceBundle/Controller/ResourceController.php:178 #0 /var/www/pimcore/vendor/dpfaffenbauer/process-manager/src/ProcessManagerBundle/Controller/ProcessController.php(91): CoreShop\Bundle\ResourceBundle\Controller\ResourceController->findOr404('59') #1 /var/www/pimcore/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php(158): ProcessManagerBundle\Controller\ProcessController->logReportAction(Object(Symfony\Component\HttpFoundation\Request)) #2 /var/www/pimcore/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php(80): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #3 /var/www/pimcore/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php(201): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #4 /var/www/pimcore/web/app.php(36): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #5 {main}

Clear old processes should support clearing *all* processes too

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Branch? master

Currently the button "Clear old processes" clears processes older than 7 days. It's an OK idea, but in many cases you might have many many processes running daily, in which leaving 7 days of them is just too many (it might be hundreds).

There should be an option to clear all processes, or maybe some more intervals (older than 1h, 1d, 1w).

Better handle processes which exited with error

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? yes
Branch? master

Currently, the processes which fail are not marked as such in the UI, they are just kept around and the impression (from users) is the process "hanged".

A better approach would be to keep track of the process execution and exit codes, marking the processes which exited unexpectedly in some way (maybe red-ish background or something?), signaling to the user this went badly.

ProcessManager doesn't handle all processes which terminated with an error correctly

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? master

Currently, when a process fails with an error, process manager will claim it's still running. From the users' POV, this seems like the process "stalled", but it actually died.

There's several ways we could do this:

  1. check when was the last update from the process, if it wasn't updated in some timeout, we assume it died and mark it as such, this only works for Data Definitions AFAIK
  2. actually write down the PID and monitor the process' STDOUT, STDERR and if it still exists, this is much more robust, but is more complex

Dao::prepareQueryBuilderForTotalCount() changed parameters in Pimcore >=10.5

Q A
Bug report? yes
Feature request? no
BC Break report? yes
RFC? no
Branch? master

Error is:
Too few arguments to function ProcessManagerBundle\Model\Process\Listing\Dao::prepareQueryBuilderForTotalCount(), 1 passed in /var/www/html/vendor/dpfaffenbauer/process-manager/src/ProcessManagerBundle/Model/Process/Listing/Dao.php on line 116 and exactly 2 expected

Since Pimcore 10.5 the method Dao::prepareQueryBuilderForTotalCount() needs 2 parameters:
protected function prepareQueryBuilderForTotalCount(QueryBuilder $queryBuilder, string $identifierColumn): void

Would be an easy fix (just add 'id' as second parameter), but then you have to set the minimum required Pimcore Version to 10.5

Otherwiese you could change the getTotalCount() method to something like:

public function getTotalCount()
    {
        $queryBuilder = $this->getQueryBuilder('id');
        return $queryBuilder->execute()->rowCount();
    }

What do you think? Should I create a PR for that (which solution do you prefer)?

Multiple Frontend/Backend Servers issue

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? master

We are running PimCore in a multi docker-container setup with multiple frontend-nodes and backend-nodes (for administration).
Maintenance Jobs are run by one of the backend-nodes.

In this kind of setup the locking/release does not work (preventing multiple processes to run).
It would be nice to include besides the pid the server id (e.g. hostname) and not start a process if another server reports the process as running.

Expected behaviour:

  • Do not start a process if another server reports it as running
  • Remove dead pids only when on the running server via maintenance task (to be run on any server of course)

process manager issue when upgraded to 5.0.0 version

Q A
Bug report? yes
Feature request? no

I have updated process manger version to 5.0.0, used pimcore 11, but got below error while migrating , type error

i tried installing 5.0.3 and 5.0.4 version but no luck still same error exists.

please help me to solve this issue, im stuck from few days in this issue.

monitor1Error

Log functionality not working after Pimcore maintenance compress logs

Q A
Bug report? yes
Feature request? no
BC Break report? no
RFC? no
Branch? master

Method Pimcore\Log\Maintenance::cleanupLogFiles() which seems to be run during Pimcore maintenance is compressing the log files into gz and keeping them for 7 days. So as your library is storing the logs into PIMCORE_LOG_DIRECTORY those are affected as well.

Your code ProcessManagerBundle\Logger\DefaultHandlerFactory is expecting for getLog method the file name is process_manager_%s.log however the name is process_manager_%s.log.gz. That is the reason that method will fail to load compressed log files and this lead to not working functionality to show or download log for log entries which were compressed. This is affecting than the controller method ProcessManagerBundle\Controller\ProcessController::getLog which is unable to return the log data.

I think it may be fine to amend DefaultHandlerFactory::getLog and DefaultHandlerFactory::cleanup to try to load+unzip / unlink the file with .gz if file with .log does not exist. Do you think there is some another place which needs to be changed? I can do the PR then.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.