gyselroth / tubee Goto Github PK

4.0 3.0 4.0 3.2 MB

Data management engine with synchronization support

License: GNU General Public License v3.0

PHP 99.14% Makefile 0.43% Dockerfile 0.44%

sync ldap mongodb objects active-directory mysql csv xml balloon rest ucs

tubee's Introduction

tubee

tubee is a data management engine with proxy capabilities for other services and its core feature is the possibility to synchronize data between multiple services (endpoints) such as databases, ldap server, file formats, web applications and more. Everything can be fully automated using tubee. You may specify different synchronization workflows and defined custom attribute mappings. Create scripted attributes, conditions, synchronization jobs and more. tubee can be used to automatically synchronize your objecs between multiple endpoints. This can be everything in its nature, for example synchronize user accounts from an XML file to Active Directory and MongoDB. Do whatever you have to do.

Features

Namespace support
Supports Import/Export to and from various different technologies
Resource versioning
Full asynchronous sync jobs
Time triggered sync jobs
RBAC
Proxy for supported endpoints (Access endpoints via the tubee layer)
Query rewriting for different endpoints (Query data from endpoints with the same query language)
Attribute mapping between tubee and endpoints
Attribtue scripting, rewriting and more
Attribute map workflows
Full featured OpenAPI v2 REST API
SDK's for 3rt party software
Full support for a cloud native deployment like on Kubernetes
Perfectly scalable for your needs
Console client for Linux, Windows and OSX

Endpoints

Endpoints
- LDAP (OpenLDAP, ActiveDirectory and other LDAP server)
- Various SQL Databases (PDO, All relational SQL database engines)
- Native MySQL/MariaDB
- MongoDB
- Moodle
- balloon
- ODataRest (Like Microsoft online (Office365 and more))
- XML (via different storage backends, see Storage drivers)
- CSV (via different storage backends, see Storage drivers)
- JSON (via different storage backends, see Storage drivers)
- Images (via different storage backends, see Storage drivers)
- Ucs (Univention Corporate Server)
Storage drivers for data formats:
- LocalFilesystem
- balloon cloud server
- SMB (Windows/Samba share via smb)
- Stream (HTTP,FTP and more)

Documentation

Visit the docs to get started!

Changelog

A changelog is available here.

Contribute

We are glad that you would like to contribute to this project. Please follow the given terms.

tubee's People

Contributors

Stargazers

Watchers

Forkers

fossabot s-aebischer lulzzz ewaldkleefstra

tubee's Issues

Add attribute type

Describe the change

Specify attribute mapping type.

Current situation

Currently there are three attribute mapping types:

value
from
script

Should

Add a new kind which can be set to static, from or script.
And a new value which holds the value.

This will avoid mistakes since at the current behaviour all three may be set.

Additional context

Add any other context about the problem here.

V8JsException: Array index exceeds maximum supported bound

Describe the bug

2019-06-05T07:11:36+00:00 DEBUG parse csv line [array{"Class":"Class","2105":"2355","0-LP":"W6m","68":"71","500":"13","NULL":"NULL","-10":"11"}]
2019-06-05T07:11:36+00:00 DEBUG process object [404] import for object [5cf76b28682f38001a6b7893] into data type [kzn::classes]
2019-06-05T07:11:36+00:00 DEBUG start workflow [kzn::classes::new-class-cn::create-update] [ensure=last] for the current object
2019-06-05T07:11:36+00:00 DEBUG no workflow condition set for workflow [kzn::classes::new-class-cn::create-update]
2019-06-05T07:11:36+00:00 ERROR failed import data object from source endpoint [kzn::classes::new-class-cn]
V8JsException: Array index exceeds maximum supported bound in /usr/share/tubee/src/lib/AttributeMap.php:78 stacktrace: #0 /usr/share/tubee/src/lib/Workflow/ImportWorkflow.php(70): Tubee\AttributeMap->map(Array, Object(MongoDB\BSON\UTCDateTime))
#1 /usr/share/tubee/src/lib/Async/Sync.php(372): Tubee\Workflow\ImportWorkflow->import(Object(Tubee\Collection), Array, Object(MongoDB\BSON\UTCDateTime), false)
#2 /usr/share/tubee/src/lib/Async/Sync.php(192): Tubee\Async\Sync->import(Object(Tubee\Collection), Array, Object(Generator), false, true)
#3 /usr/share/tubee/src/lib/Async/Sync.php(172): Tubee\Async\Sync->execute(Object(Tubee\Collection), Object(Tubee\Endpoint\Csv))
#4 /usr/share/tubee/src/lib/Async/Sync.php(140): Tubee\Async\Sync->loopEndpoints(Object(Tubee\Collection), Array, Array, Array)
#5 /usr/share/tubee/src/lib/Async/Sync.php(125): Tubee\Async\Sync->loopCollections(Array, Array)
#6 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(574): Tubee\Async\Sync->start()
#7 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(482): TaskScheduler\Worker->executeJob(Array)
#8 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(333): TaskScheduler\Worker->processJob(Array)
#9 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(227): TaskScheduler\Worker->queueJob(Array)
#10 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(287): TaskScheduler\Worker->processAll()
#11 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(473): TaskScheduler\WorkerManager->spawnWorker()
#12 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(332): TaskScheduler\WorkerManager->handleJob(Array)
#13 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(176): TaskScheduler\WorkerManager->main()
#14 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Queue.php(148): TaskScheduler\WorkerManager->process()
#15 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Queue.php(93): TaskScheduler\Queue->initWorkerManager()
#16 /usr/share/tubee/src/lib/Console/Jobs.php(69): TaskScheduler\Queue->process()
#17 /usr/share/tubee/src/lib/Bootstrap/Cli.php(106): Tubee\Console\Jobs->__invoke()
#18 /usr/share/tubee/src/lib/Bootstrap/Cli.php(130): Tubee\Bootstrap\Cli->executeCommand(Object(Tubee\Console\Jobs))
#19 /usr/share/tubee/src/lib/Bootstrap/Cli.php(91): Tubee\Bootstrap\Cli->routeCommand()
#20 /usr/share/tubee/bin/console/tubeecli(24): Tubee\Bootstrap\Cli->process()
#21 {main}

To Reproduce

Import from CsvEndpoint with a csv file without a headerline.
And workflow mappings to specific header names.

Expected behavior

A clear and concise description of what you expected to happen.

Environment

tubee server version: [e.g. v1.0.0]
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

UCS-endpoint does not find endpoint object while rename and move

Describe the bug

UCS-endpoint does not find endpoint-object when the object-dn and username should be changed

Long running processes with MicrosoftGraphEndpoint ends in multiple 401 errors

Describe the bug

Process log:

message: "Client error: `POST https://graph.microsoft.com/v1.0/users` resulted in a `401 Unauthorized` response:\n{\r\n  \"error\": {\r\n    \"code\": \"InvalidAuthenticationToken\",\r\n    \"message\": \"Access token has expired.\",\r\n    \"innerError (truncated...)\n"
        code: 401

To Reproduce

Sync an endpoint of type MicrosoftGraphEndpoint which takes longer than 3600s.
The oauth2 token will be invalid after that time and all further objects fail.

Expected behavior

Renew token if a 401 exception of that type occurs.

Environment

tubee server version: v1.0.0-beta63

Ucs endpoint must check search response for equality

Describe the change

Ucs can only handle wildcard search request.

Current situation

If a search for 'foobar' is done, and Ucs has two records named 'foobar' and 'foobar2' both will get returned.

Should

The Ucs endpoint must search all found records and match a single one.

Additional context

Add any other context about the problem here.

job attributes fall back to default values

Describe the bug

When creating a new job the attributes timeout and interval_reference are not saved.

To Reproduce

Create a new job and define timeout and interval_reference. When you open the job again the default values are shown (timeout = 0; interval_reference = end)

Allow DataObject to be named manually

Is:
Currently name can not be set manually for a DataObject.
A DataObject takes its name from the autogenerated resource id.

Should:
Allow the name to be set manually. If no name was submitted take the auto generated id as name.

Add default timeout if 5s to endpoint resources definitions

Description

Is:
Related #28

Should:

tubee server version

tubee cli client version (If related)

PHP version

Operating system

Webserver

Relation GarbageWorflow deletes all DataObjects at once

Current situation

When a relation GarbageWorflow is configured with ensure: absent all relations are deleted at once. In this case it's not possible to configure a condition before the relation gets deleted.

Should

When in a GarbageWorkflow (relation) ensure is set to "absent" the Worfklow should delete the relation object in progress and not all at once.

Do not throw an exception during query for an non existing mssql field

Describe the bug

Query an mssql endpoint should not end in an exception if a non existing field was given:

[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "2019-22-05 08:21:10 [Tubee\Rest\Middlewares\ExceptionHandler,ERROR]: uncaught exception failed to execute sql query with error [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid column name 'PersonID'. (42S22)]  [object] (Tubee\Endpoint\Pdo\Exception\InvalidQuery(code: 0): failed to execute sql query with error [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid column name 'PersonID'. (42S22) at /usr/share/tubee/src/lib/Endpoint/Pdo/Wrapper.php:89)"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "[stacktrace]"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#0 /usr/share/tubee/src/lib/Endpoint/Pdo.php(53): Tubee\Endpoint\Pdo\Wrapper->select('SELECT * FROM v...')"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#1 /usr/share/tubee/src/lib/Acl.php(143): Tubee\Endpoint\Pdo->getAll(Array, 0, 20, Array)"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#2 /usr/share/tubee/src/lib/Rest/Pager.php(34): Tubee\Acl->filterOutput(Object(Zend\Diactoros\ServerRequest), Object(Micro\Auth\Identity), Object(Generator))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#3 /usr/share/tubee/src/lib/Rest/Helper.php(42): Tubee\Rest\Pager::fromRequest(Object(Generator), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#4 /usr/share/tubee/src/lib/Rest/v1/Endpoints.php(171): Tubee\Rest\Helper::getAll(Object(Zend\Diactoros\ServerRequest), Object(Micro\Auth\Identity), Object(Tubee\Acl), Object(Generator))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#5 [internal function]: Tubee\Rest\v1\Endpoints->getAllObjects(Object(Zend\Diactoros\ServerRequest), Object(Micro\Auth\Identity), 'kzn', 'accounts', Object(Tubee\Endpoint\Pdo))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#6 /usr/share/tubee/src/lib/Rest/Middlewares/RequestHandler.php(77): call_user_func_array(Array, Array)"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#7 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Tubee\Rest\Middlewares\RequestHandler->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#8 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#9 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#10 /usr/share/tubee/vendor/middlewares/trailing-slash/src/TrailingSlash.php(54): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#11 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Middlewares\TrailingSlash->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#12 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#13 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#14 /usr/share/tubee/src/lib/Rest/Middlewares/Acl.php(51): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#15 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Tubee\Rest\Middlewares\Acl->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#16 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#17 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#18 /usr/share/tubee/vendor/gyselroth/micro-auth/src/Middleware/Auth.php(59): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#19 /usr/share/tubee/src/lib/Rest/Middlewares/Auth.php(32): Micro\Auth\Middleware\Auth->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#20 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Tubee\Rest\Middlewares\Auth->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#21 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#22 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#23 /usr/share/tubee/vendor/middlewares/fast-route/src/FastRoute.php(64): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#24 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Middlewares\FastRoute->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#25 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#26 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#27 /usr/share/tubee/src/lib/Rest/Middlewares/QueryDecoder.php(72): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#28 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Tubee\Rest\Middlewares\QueryDecoder->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#29 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#30 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#31 /usr/share/tubee/vendor/middlewares/payload/src/Payload.php(75): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#32 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Middlewares\Payload->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#33 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#34 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#35 /usr/share/tubee/src/lib/Rest/Middlewares/ExceptionHandler.php(57): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#36 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Tubee\Rest\Middlewares\ExceptionHandler->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#37 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#38 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#39 /usr/share/tubee/vendor/middlewares/access-log/src/AccessLog.php(115): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#40 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Middlewares\AccessLog->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#41 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#42 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#43 /usr/share/tubee/vendor/middlewares/negotiation/src/ContentType.php(107): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#44 /usr/share/tubee/vendor/lcobucci/content-negotiation-middleware/src/ContentTypeMiddleware.php(73): Middlewares\ContentType->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#45 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(94): Lcobucci\ContentNegotiation\ContentTypeMiddleware->process(Object(Zend\Diactoros\ServerRequest), Object(mindplay\middleman\Delegate))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#46 [internal function]: mindplay\middleman\Dispatcher->mindplay\middleman\{closure}(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#47 /usr/share/tubee/vendor/mindplay/middleman/src/Delegate.php(38): call_user_func(Object(Closure), Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#48 /usr/share/tubee/vendor/mindplay/middleman/src/Dispatcher.php(59): mindplay\middleman\Delegate->handle(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#49 /usr/share/tubee/bin/httpdocs/index.php(44): mindplay\middleman\Dispatcher->dispatch(Object(Zend\Diactoros\ServerRequest))"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: "#50 {main}"
[22-May-2019 08:21:10] WARNING: [pool www] child 213 said into stdout: ""

To Reproduce

Create mssql endpoint
Do a query with a non existing field

Expected behavior

Log exception and return an empty generator.

Environment

tubee server version: [e.g. v1.0.0]
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

Implement php-scheduler 4.0.13

Is your feature request related to a problem? Please describe

Implement new php-scheduler version 4.0.13 which offers a custom notification method.

Describe the solution you'd like

Implement a custom notification method in the SyncClass to inform customers when a job changed its state (e.g. done, failed, canceled...).
Also implement an identifier which can be defined in the job to include dataobject specific information in the notification mail.

Support for EndpointObject paging

Description

Is:

Should:

tubee server version

tubee cli client version (If related)

PHP version

Operating system

Webserver

Wrong debug log: total counter is less than the current DataObject

Describe the bug

Wrong debug log: total counter is less than the current DataObject:
2019-05-22T08:34:58+00:00 DEBUG process [3324] import for object [3323] into data type [kzn::roles]

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Correct total number.

Environment

tubee server version: [e.g. v1.0.0]
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

Process fails and includes wrong exception stacktrace

Describe the bug

sync ep accounts -n myns -l debug -f --trace -q "data.username=username" test-ad:

Single object sync fails with exception (Not for every object but for this one):

2019-22-05 07:35:57 [TaskScheduler\Worker,ERROR]: failed execute job [5ce4fbda0c894400c366ffac] of type [Tubee\Async\Sync] on worker [5ce4f20d6f93e500060fc90d]  [object] (Error(code: 0): Wrong parameters for PDOException([string $message [, long $code [, Throwable $previous = NULL]]]) at /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Process.php:133)
[stacktrace]
#0 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Process.php(133): Exception->__construct('SQLSTATE[42000]...', '42000')
#1 /usr/share/tubee/src/lib/Async/Sync.php(147): TaskScheduler\Process->wait()
#2 /usr/share/tubee/src/lib/Async/Sync.php(124): Tubee\Async\Sync->loopCollections(Array, Array)
#3 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(574): Tubee\Async\Sync->start()
#4 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(482): TaskScheduler\Worker->executeJob(Array)
#5 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(333): TaskScheduler\Worker->processJob(Array)
#6 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Worker.php(227): TaskScheduler\Worker->queueJob(Array)
#7 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(287): TaskScheduler\Worker->processAll()
#8 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(259): TaskScheduler\WorkerManager->spawnWorker()
#9 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(200): TaskScheduler\WorkerManager->spawnMinimumWorkers()
#10 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(325): TaskScheduler\WorkerManager->exitWorker(17, Array)
#11 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/WorkerManager.php(176): TaskScheduler\WorkerManager->main()
#12 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Queue.php(148): TaskScheduler\WorkerManager->process()
#13 /usr/share/tubee/vendor/gyselroth/mongodb-php-task-scheduler/src/Queue.php(93): TaskScheduler\Queue->initWorkerManager()
#14 /usr/share/tubee/src/lib/Console/Jobs.php(69): TaskScheduler\Queue->process()
#15 /usr/share/tubee/src/lib/Bootstrap/Cli.php(106): Tubee\Console\Jobs->__invoke()
#16 /usr/share/tubee/src/lib/Bootstrap/Cli.php(130): Tubee\Bootstrap\Cli->executeCommand(Object(Tubee\Console\Jobs))
#17 /usr/share/tubee/src/lib/Bootstrap/Cli.php(91): Tubee\Bootstrap\Cli->routeCommand()
#18 /usr/share/tubee/bin/console/tubeecli(24): Tubee\Bootstrap\Cli->process()
#19 {main}

To Reproduce

Expected behavior

The exception stacktrace is not much helpful since it shows the stacktrace from the process handler.

Environment

tubee server version: [e.g. v1.0.0]
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

Do not drop fields if skip is true

Describe the bug

tubee drops an existing field from a DataObject if one sets skip: true in a matching workflow later.

Expected behavior

Ignore field, do not drop it.

Environment

tubee server version: v1.0.0-beta39
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

New resource ConfigMap

Is your feature request related to a problem? Please describe

We would like to see a possibility to have a config which can be injected into any kind of other resources like secrets.

Describe the solution you'd like

A ConfigMap would be quite similar to a secret, the only difference is that secrets are specially encrypted and encoded.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Include exception for failed processes

Is your feature request related to a problem? Please describe

I'm missing the exception in a failed process.

Describe the solution you'd like

Include the exception for for failed processes.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Add driver options of each endpoint to openapi specs.

Description

Is:
Currently anything can be specified in driver options of each endpoint.

Should:
The specs should declare what can be configured.

tubee server version

v1.0.0-beta1

tubee cli client version (If related)

PHP version

Operating system

Webserver

Uncatched exception when endpoint is unavailable

Describe the bug

When some endpoints are unavailable, tubee throws an uncatched exception. For example, when the post request to a MicrosoftGraphEndpoint fails, an uncatched exception is thrown.

V2 rewrite this thing in golang

SqlSrvUsers: delete SQL-user when deleting Login

Describe the change

When deleting a SQL-Login all associated SLQ-users should be deleted.

Current situation

When deleting a SQL-Login only the SQL-users within the same database are deleted.

Should

All SQL-Users with the same SID should be deleted.

interval_reference feature of php-task-scheduler

Describe the change

Integrate new interval_reference feature of gyselroth/php-task-scheduler (gyselroth/php-task-scheduler#38).

Cross namespace secrets and config mounts

Is your feature request related to a problem? Please describe

Currently its required to create two secrets if it is required in two namespaces.

Describe the solution you'd like

I'd like to import a secret or an upcoming config from another namespace.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

MicrosoftGraph-Endpoint ensure: merge does not work

Describe the bug

ensure: merge does not work properly on member and owner attribute when synchronizing Office365 groups.
the existing members/owners and the new ones are not merged properly. each time some members are removed. when ensure: merge is configured new members should be added and the existing ones should remain a member of the group.

pdo endpoint (mssql) generates wrong filter

Describe the bug

[22-May-2019 08:25:09] WARNING: [pool www] child 231 said into stdout: "2019-22-05 08:25:09 [Tubee\Endpoint\Pdo,DEBUG]: find all resources from endpoint [kzn::roles::im-mssql-relations] with query [(AND type='Person-Department'AND person_id='4046677')]  "
[22-May-2019 08:25:09] WARNING: [pool www] child 231 said into stdout: "2019-22-05 08:25:09 [Tubee\Endpoint\Pdo\Wrapper,DEBUG]: execute sql query [SELECT * FROM view_TAMWEB_export_relations WHERE (AND type='Person-Department'AND person_id='4046677')]  "
[22-May-2019 08:25:09] WARNING: [pool www] child 231 said into stdout: "2019-22-05 08:25:09 [Tubee\Rest\Middlewares\ExceptionHandler,ERROR]: uncaught exception failed to execute sql query with error [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near the keyword 'AND'. (42000)]  [object] (Tubee\Endpoint\Pdo\Exception\InvalidQuery(code: 0): failed to execute sql query with error [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near the keyword 'AND'. (42000) at /usr/share/tubee/src/lib/Endpoint/Pdo/Wrapper.php:89)"
[22-May-2019 08:25:09] WARNING: [pool www] child 231 said into stdout: "[stacktrace]"

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Should be (AND type='Person-Department'AND person_id='4046677') without leading AND.

Environment

tubee server version: [e.g. v1.0.0]
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

Replace PdoEndpoint with native SqlServerEndpoint

Description

Is:
Currently there is a PdoEndpoint which has several disadvantages.
For example since multiple drivers are supported they still have different sql implementations.
For instance chunking response is different from MySQL to an SQL Server. There is no LIMIT in the SqlServer world.

Should:
Implement native MS SqlServer endpoint. Drop PDO support.

tubee server version

v1.0.0-beta1

tubee cli client version (If related)

PHP version

Operating system

Webserver

New endpoint ucs (univention corporate server)

https://www.univention.com/products/ucs/

Get error when use -w to get logfiles

Describe the bug

When I run "tubectl get ps 5d038a6a8cb66f015c1479f3 -n krr --logs --trace -w" I get the follwing error. The use of --stream does work. The problem seems to be -w

internal/streams/legacy.js:59
throw er; // Unhandled stream error in pipe.
^

Error: Invalid JSON (Unexpected "h" at position 1 in state STOP)
at Parser.proto.charError (/Users/sandro.aebischer/node_modules/jsonparse/jsonparse.js:90:16)
at Parser.proto.write (/Users/sandro.aebischer/node_modules/jsonparse/jsonparse.js:267:27)
at Stream. (/Users/sandro.aebischer/node_modules/JSONStream/index.js:23:12)
at Stream.stream.write (/Users/sandro.aebischer/node_modules/through/index.js:26:11)
at Request.ondata (internal/streams/legacy.js:16:26)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at IncomingMessage. (/Users/sandro.aebischer/node_modules/request/request.js:1080:12)
at emitOne (events.js:116:13)
at IncomingMessage.emit (events.js:211:7)
at IncomingMessage.Readable.read (_stream_readable.js:475:10)
at flow (stream_readable.js:846:34)
at resume (_stream_readable.js:828:3)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)

case sensitive in LdapEndpoint

Describe the bug

LdapEndpoint: attributes should be case sensitive

last_sync/last_successful_sync have an old timestamp

Describe the bug

After syncing an object the timestamps have an older value.

The bug occurs while the timestamp is initialized in the constructor https://github.com/gyselroth/tubee/blob/master/src/lib/Async/Sync.php#L109 and \Taskscheduler fetches Tubee\Async\Sync from the tubee PSR-11 container while that service is declared as a singleton (micro-container).

To Reproduce

Sync single object, the wont get the current timestamp.

Expected behavior

Set timestamp from current process.

Environment

tubee server version: 1.0.0-beta30
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.

Maintain old processes on status when deleting job

Describe the bug

When a job gets canceled every process from the past of that job gets canceled too. Even when the process has the status done.

To Reproduce

Cancel a job which has already done processes in the past.

Expected behavior

Only the canceled job and the direct processes should be updated. All processes from the past which are already done should not be touched

SqlSrvUsers-Endpoint: transaction during a change even without differences

Describe the bug

Within the SqlSrvUsers-Endpoint a transaction to the server is opened (and closed again) even if no changes need to be made on the endpoint dataobject.

To Reproduce

To reproduce this error, an attribute must be configured that is not delivered by the endpoint by default. The correct way would be to configure this attribute as “writeonly”. However, if this is not done, the endpoint should not start a transaction without a change being made to the endpoint.

LdapEndpoint value is not a string

Describe the bug

When in a LdapEndpoint an attribut is not a string the only message the user gets is "2019-06-14T09:26:58+00:00 WARNING ldap_modify_batch(): Each element of a 'values' array must be a string in /usr/share/tubee/vendor/dreamscapes/ldap-core/Dreamscapes/Ldap/Core/Ldap.php:533"
The workflow does not run correctly, so an additional warning/message would be good

Update endpoint status of DataObject

Describe the bug

Update result of endpoint status when the value has changed.

MongodbEndpoint: Error - Cannot unpack array with string keys

Describe the bug

When creating a new MongodbEndpoint source-EP the following Error pops up:
#Error: Error - Cannot unpack array with string keys

Add support for modify/add/delete EndpointObject resources

Is your feature request related to a problem? Please describe

No.

Describe the solution you'd like

It should be possible to directly operate on EndpointObject resources.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

add mongodb v4 compatibility

Describe the solution you'd like

add mongodb v4 compatibility

Add history for each resource type

Is your feature request related to a problem? Please describe

Currently history is stored for every resource type but may only be requested from data object resources.

Describe the solution you'd like

Add history api endpoints for every resource.

Exception: Ignore history for secrets. There shall be no history for secrets. Also there must not be any api endpoints to retrieve secret history.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Add workflow with no data ends in Undefined index exception

Describe the bug

[07-Jun-2019 06:18:26] WARNING: [pool www] child 465 said into stdout: "2019-07-06 06:18:26 [Tubee\Bootstrap\ContainerBuilder,DEBUG]: Undefined index: data in /usr/share/tubee/src/lib/Workflow/Factory.php:179  "
[07-Jun-2019 06:18:26] WARNING: [pool www] child 465 said into stdout: "2019-07-06 06:18:26 [Tubee\Rest\Middlewares\ExceptionHandler,ERROR]: uncaught exception Undefined index: data]  [object] (ErrorException(code: 0): Undefined index: data at /usr/share/tubee/src/lib/Workflow/Factory.php:179)"

To Reproduce

Add workflow without data field:

{
	"_id" : ObjectId("5cf927fa651dfc01da7ffc92"),
	"name" : "create-update",
	"secrets" : [ ],
	"kind" : "Workflow",
	"namespace" : "ns",
	"collection" : "accounts",
	"endpoint" : "im-mssql",
	"created" : ISODate("2019-06-06T14:49:30.296Z"),
	"changed" : ISODate("2019-06-06T14:49:30.296Z"),
	"version" : 1
}

Expected behavior

The default should be used for data. Or maybe data should be required since an empty data does not really make sense.

Environment

tubee server version: v1.0.0-beta20
tubectl version: [e.g. v1.0.0]
MongoDB Version: [e.g. v3.4.18]
PHP Version [e.g. v7.2.1]

Additional context

Add any other context about the problem here.