Coder Social home page Coder Social logo

hbnking / flume-ng-database Goto Github PK

View Code? Open in Web Editor NEW
9.0 2.0 7.0 11.45 MB

Flume-ng sqlsource support data comes from database

Java 14.58% HTML 84.20% JavaScript 0.07% CSS 1.15%
flume-sqlsource database flume-ng flume-source-plugin flume-sink-plugin json mongodb

flume-ng-database's People

Contributors

hbnking avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

rabkool clks2001 abacook gerrylfc cclovezbf xubingyc jekbon

flume-ng-database's Issues

内存溢出

无数据的情况下轮询几秒后轮询速度会加快，然后内存溢出
20/02/12 04:27:34 ERROR source.SQLSource: RDB process error ,error is java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PollableSourceRunner-SQLSource-r1" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.StringBuffer.toString(StringBuffer.java:669)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "PollableSourceRunner-SQLSource-r1"

flume 请求一次 获取的数据量过大 内存溢出的问题 

2019-02-13 17:06:04,032 (PollableSourceRunner-SQLSource-r1) [INFO - com.hbn.rdb.common.DriverQuery.executeQuery(DriverQuery.java:67)] select * from  newtable  where id > 189606791   ;
Exception in thread "PollableSourceRunner-SQLSource-r1" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:832)
        at com.mysql.cj.core.conf.ReadableIntegerProperty.getValue(ReadableIntegerProperty.java:41)
        at com.mysql.cj.core.conf.ReadableIntegerProperty.getValue(ReadableIntegerProperty.java:31)
        at com.mysql.cj.mysqla.io.TextRowFactory.createFromPacketPayload(TextRowFactory.java:64)
        at com.mysql.cj.mysqla.io.TextRowFactory.createFromPacketPayload(TextRowFactory.java:40)
        at com.mysql.cj.mysqla.io.ResultsetRowReader.read(ResultsetRowReader.java:83)
        at com.mysql.cj.mysqla.io.ResultsetRowReader.read(ResultsetRowReader.java:38)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.read(MysqlaProtocol.java:1660)
        at com.mysql.cj.mysqla.io.TextResultsetReader.read(TextResultsetReader.java:82)
        at com.mysql.cj.mysqla.io.TextResultsetReader.read(TextResultsetReader.java:43)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.read(MysqlaProtocol.java:1672)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.readAllResults(MysqlaProtocol.java:1718)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.sqlQueryDirect(MysqlaProtocol.java:1006)
        at com.mysql.cj.mysqla.MysqlaSession.sqlQueryDirect(MysqlaSession.java:431)
        at com.mysql.cj.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:1974)
        at com.mysql.cj.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:1936)
        at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1422)
        at com.hbn.rdb.common.DriverQuery.executeQuery(DriverQuery.java:70)
        at com.hbn.rdb.source.SQLSourceHelper.executeQuery(SQLSourceHelper.java:245)
        at com.hbn.rdb.source.SQLSource.process(SQLSource.java:79)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133)
        at java.lang.Thread.run(Thread.java:748)

编译失败

抱歉，您有编译好的包吗，我们一直编译不过
[ERROR] Failed to execute goal on project RDB: Could not resolve dependencies for project com.hbn:RDB:jar:0.0.1: Failure to find com.oracle:ojdbc6:jar:11.2.0.3 in http://repo1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of mirrorId has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

打包失败 /RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[482,60] 找不到符号

[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java: /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java使用或覆盖了已过时的 API。
[WARNING] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
[WARNING] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/common/FileStatus.java: /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/common/FileStatus.java使用了未经检查或不安全的操作。
[WARNING] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/common/FileStatus.java: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
[INFO] 4 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[477,57] 找不到符号
符号: 类 SQLType
位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[482,60] 找不到符号
符号: 类 SQLType
位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[487,57] 找不到符号
符号: 类 SQLType
位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[492,60] 找不到符号
符号: 类 SQLType
位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[476,5] 方法不会覆盖或实现超类型的方法
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[481,5] 方法不会覆盖或实现超类型的方法
[INFO] 6 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.161 s
[INFO] Finished at: 2019-06-09T11:57:41+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project RDB: Compilation failure: Compilation failure:
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[477,57] 找不到符号
[ERROR] 符号: 类 SQLType
[ERROR] 位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[482,60] 找不到符号
[ERROR] 符号: 类 SQLType
[ERROR] 位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[487,57] 找不到符号
[ERROR] 符号: 类 SQLType
[ERROR] 位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[492,60] 找不到符号
[ERROR] 符号: 类 SQLType
[ERROR] 位置: 类 com.hbn.rdb.page.PageableResultSet
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[476,5] 方法不会覆盖或实现超类型的方法
[ERROR] /home/test/github/Flume-ng-Database/RDB/src/main/java/com/hbn/rdb/page/PageableResultSet.java:[481,5] 方法不会覆盖或实现超类型的方法
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

郁闷，java懂一点，mvn打包刚开始是odbc加载不上，后来又报这个错误

join 后 字段名称重复，应当改为设置项。

join 后 多表 如果有 相同的字段名称 转为 json 时 会出现问题[因为他们的key 字段名称相同，会覆盖掉]，会丢数据 ，虽然本身程序 设计了一些小参数来避免这样的问题（比如自增字段。 但是还不够人性化），将会在后续版本 修改为 自定义字段名项。

存在问题 ，多表join 的场景 如何 控制输出

单表直接抽取 可以用直接使用 where + limit 模型 这样效率比较高 ，
但是 如果是 join 的 场景 就会有一些问题 ，join 后 返回的数据量 太大， 会导致内存溢出 ，这个方面问题 有待修复 。应该把 分页模型单独拿出来 。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.