data-integrations / multi-table-plugins

Pipeline plugins that allow reading from multiple DB tables with the same source

Multiple Table Plugins

CDAP Pipeline plugins that allow reading from multiple DB tables with the same source

• Multiple Relational Database Tables Batch Source
• Dynamic Multiple Filesets Batch Sink

Build

To build this plugin:

   mvn clean package

The build will create a .jar and .json file under the target directory. These files can be used to deploy your plugins.

Deployment

You can deploy your plugins using the CDAP CLI:

> load artifact <target/dynamic-spark-<version>.jar config-file <target/dynamic-spark-<version>.json>

Mailing Lists

CDAP User Group and Development Discussions:

• [email protected] <https://groups.google.com/d/forum/cdap-user>

The cdap-user mailing list is primarily for users using the product to develop applications or building plugins for appplications. You can expect questions from users, release announcements, and any other discussions that we think will be helpful to the users.

IRC Channel

CDAP IRC Channel: #cdap on irc.freenode.net

License and Trademarks

Copyright © 2017-2019 Cask Data, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Cask is a trademark of Cask Data, Inc. All rights reserved.

Apache, Apache HBase, and HBase are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.

multi-table-plugins's Issues

Building error

I got following errors by building. The error looks same as the #22 error.

First, Could anyone fix version like Albertshau fixed on #22 .

Second, actually, I want to use this utf8 plugin, too.
But I got same errors.
Could anyone fix or tell me how to fix like Albertshau fixed on #22 .

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13:38 min
[INFO] Finished at: 2020-06-25T13:26:34+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project multi-table-plugins: Could not resolve dependencies for project io.cdap.plugin:multi-table-plugins:jar:1.2.0-SNAPSHOT: Could not find artifact jdk.tools:jdk.tools:jar:1.6 at specified path C:\Program Files\java\jdk-14.0.1/../lib/tools.jar -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Support Dynamic Multi GCS Sink

Same as the ADLS class: DynamicMultiADLSSink

Update doco to explain typical implementation of tableNamePattern and schemaNamePattern in JDBC drivers

Would be great to note in the doco that JDBC drivers typically implement .getTables such that schemaNamePattern and tableNamePattern with _ (any single character) and % (any sequence) as wildcard characters.

eg. tableNamePattern = myTableSeq%

this plugin doesn't work for mysql database

The mysql driver doesn't return the database name. I have a branch ready

Add DynamicMultiGcpAvroSink

Could not resolve dependencies io.cdap.plugin:multi-table-plugins:jar

Steps I did

sudo apt-get install openjdk-8-jdk
sudo apt install maven

Cloned the repo and built

git clone https://github.com/data-integrations/multi-table-plugins.git
cd multi-table-plugin/
mvn  clean install

ERROR

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  31.757 s
[INFO] Finished at: 2020-04-22T13:51:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project multi-table-plugins: Could not resolve dependencies for project io.cdap.plugin:multi-table-plugins:jar:1.2.0-SNAPSHOT: The following artifacts could not be resolved: io.cdap.cdap:cdap-etl-api:jar:6.0.0-SNAPSHOT, io.cdap.plugin:hydrator-common:jar:2.2.0-SNAPSHOT, io.cdap.cdap:cdap-data-pipeline:jar:6.0.0-SNAPSHOT, io.cdap.cdap:hydrator-test:jar:6.0.0-SNAPSHOT, jdk.tools:jdk.tools:jar:1.6: Could not find artifact io.cdap.cdap:cdap-etl-api:jar:6.0.0-SNAPSHOT in sonatype (https://oss.sonatype.org/content/groups/public) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

I don't for most of the plugins from this repository, Im getting these kinds of error messages like Could not resolve dependencies

Please add tags when changing versions in pom xml

Hi,

It is recommended to use the tags/releases in github when changing code, especially when changing versions in pom or adding new functionality.

Netezza multiple table source not reading views

Hi,

I'm using multiple table source plugin to read multiple views from a Netezza database but only tables or materialized views seems to be retrived by the plugin.

It's an intended behavior?

Thanks