datafibers / df_data_service Goto Github PK

"topic.for.query" only supports single topic. Make it supports list of the topics.
Are we able to support list of the topics as "topic.for.result"?

Need schema visualization and modification

CLI need to support run single DF application meaning both rest and ui are in the same host and same port with different url
after #21

curl -X GET -H "Content-Type: application/json" --data http://localhost:8083/connectors

(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $ ./UUPSMT06
2016-11-17.15:57:01 - Oracle authentication file found - will be used for SQLPlus Login
0 <-- Characters spit out by SQLplus

Status code = 0

Status code = 0

Status code = 0

Status code = 0
0 <-- Characters spit out by SQL*plus
MDM Logger factory being used is com.dwl.base.logging.DWLLog4jLoggerFactory
ERROR - [Exception_CustomerDelegate_InvokeEJBObject:] CDKBC2027E:The system is unable to connect to the server. Additional error message = CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - com.dwl.base.exception.DWLResponseException: CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
at com.dwl.tcrm.delegate.CustomerDelegate.getServiceHandler(CustomerDelegate.java:146)
at com.dwl.tcrm.delegate.CustomerDelegate.processRequest(CustomerDelegate.java:113)
at com.dwl.batchframework.delegate.JTAEnbledCustomer.processRequest(JTAEnbledCustomer.java:61)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.addTasks(TaskChainCreatorHelper.java:203)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.createTaskChainFromTemplates(TaskChainCreatorHelper.java:128)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreator.createTaskChain(TaskChainCreator.java:122)
at com.dwl.batchframework.BatchController.runBatch(BatchController.java:306)
at com.dwl.batchframework.BatchControllerStarter.runBatch(BatchControllerStarter.java:150)
at com.dwl.batchframework.BatchControllerStarter.main(BatchControllerStarter.java:106)
at com.dwl.batchframework.BatchController.main(BatchController.java:205)

Failed to create the task chain. CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
0 <-- Characters spit out by SQL*plus
(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $

Thank you,

Lyudmila Malko | QA Analyst | TSS | Email:Lyudmila.Malko@tdcom

From: Malko, Lyudmila
Sent: November-17-16 3:56 PM
To: Zhu, Schubert
Subject: FW: error

(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $ ./UUPSMT06
2016-11-17.15:51:04 - Oracle authentication file found - will be used for SQLPlus Login
0 <-- Characters spit out by SQLplus

Status code = 0

Status code = 0

Status code = 0

Status code = 0
0 <-- Characters spit out by SQL*plus
MDM Logger factory being used is com.dwl.base.logging.DWLLog4jLoggerFactory
ERROR - [Exception_CustomerDelegate_InvokeEJBObject:] CDKBC2027E:The system is unable to connect to the server. Additional error message = CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - com.dwl.base.exception.DWLResponseException: CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
at com.dwl.tcrm.delegate.CustomerDelegate.getServiceHandler(CustomerDelegate.java:146)
at com.dwl.tcrm.delegate.CustomerDelegate.processRequest(CustomerDelegate.java:113)
at com.dwl.batchframework.delegate.JTAEnbledCustomer.processRequest(JTAEnbledCustomer.java:61)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.addTasks(TaskChainCreatorHelper.java:203)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.createTaskChainFromTemplates(TaskChainCreatorHelper.java:128)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreator.createTaskChain(TaskChainCreator.java:122)
at com.dwl.batchframework.BatchController.runBatch(BatchController.java:306)
at com.dwl.batchframework.BatchControllerStarter.runBatch(BatchControllerStarter.java:150)
at com.dwl.batchframework.BatchControllerStarter.main(BatchControllerStarter.java:106)
at com.dwl.batchframework.BatchController.main(BatchController.java:205)

Failed to create the task chain. CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
0 <-- Characters spit out by SQL*plus
(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $

Thank you,

Lyudmila Malko | QA Analyst | TSS | Email:Lyudmila.Malko@tdcom

From: Malko, Lyudmila
Sent: November-17-16 3:41 PM
To: Zhu, Schubert
Subject: error

Status code = 0
0 <-- Characters spit out by SQL*plus
MDM Logger factory being used is com.dwl.base.logging.DWLLog4jLoggerFactory
ERROR - [Exception_CustomerDelegate_InvokeEJBObject:] CDKBC2027E:The system is unable to connect to the se rver. Additional error message = CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
ERROR - com.dwl.base.exception.DWLResponseException: CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested except ion is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
at com.dwl.tcrm.delegate.CustomerDelegate.getServiceHandler(CustomerDelegate.java:146)
at com.dwl.tcrm.delegate.CustomerDelegate.processRequest(CustomerDelegate.java:113)
at com.dwl.batchframework.delegate.JTAEnbledCustomer.processRequest(JTAEnbledCustomer.java:61)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.addTasks(TaskChainCreator Helper.java:203)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreatorHelper.createTaskChainFromTempla tes(TaskChainCreatorHelper.java:128)
at com.ibm.mdm.batchframework.bulkprocessing.contexts.TaskChainCreator.createTaskChain(TaskChainCreato r.java:122)
at com.dwl.batchframework.BatchController.runBatch(BatchController.java:306)
at com.dwl.batchframework.BatchControllerStarter.runBatch(BatchControllerStarter.java:150)
at com.dwl.batchframework.BatchControllerStarter.main(BatchControllerStarter.java:106)
at com.dwl.batchframework.BatchController.main(BatchController.java:205)

Failed to create the task chain. CORBA TRANSACTION_ROLLEDBACK 0x0 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x0 minor code: 0 completed: Maybe
0 <-- Characters spit out by SQLplus
(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $
(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $
(DEV) malkol2@webspherevm-70e1d:/td/UPS_ST_36114/job $ ./UUPSMT05
2016-11-17.15:37:38 - Oracle authentication file found - will be used for SQLPlus Login
0 <-- Characters spit out by SQL*plus

Thank you,

Lyudmila Malko | QA Analyst | TSS | Email:Lyudmila.Malko@tdcom

New integration test case needed for demo/developer environment

Support Flink savepoints, which are manually triggered checkpoints, enabled in Flink transforms.

Use docker composite to build DF developer environment

Evaluate Gradle and move to it from maven

DF needs a shortcut visualization approach, such as MongoDB, and/or HBase, Zeppelin

Delete Jar/UDF uploaded with transforms when you want to delete a transform. Default is to delete.

1. Evaluate existing Kafka Connects
2. Setup plan for additional connects needed
3. Consider the logging and schedule requirement in connector

Support delete kafka topic associated when you delete a task
This can also be exposed as API for Flink UDF

1. The webpage is not refreshed after adding records from UI

DF needs a Kafka topic view for better management

Connector Name for FileStreaming should be same to the name in the connect config. Need to check this or automatically set this.

We need more wiser way to read command line parameters,
jar -cluster s

Unit Test case as part of code bases
Travis file update for CI

Right now there is no data from output topic from Flink streaming. Because the Kafka FIle Stream Connect Source did not serialize json. It converts it to String which cannot read by Flink Json Table

After version 1.2.0, Apache Flink will offer REST API for monitoring purpose. Then, DF Processor can leverage this new service to synchronize the active job status like what we did for Kafka connects
Also, we can fetch the job log information to repo.

Able to launch Flink UDF Jar from DF

1. NG-Admin (using)
2. SB Admin (more features)
3. Other

Need to support to deal with Avro type of data if Kafka Connect enable
key.converter.schemas.enable=true
key.converter.schemas.enable=true