Comments (11)
现在是默认读取1000行,现在还不支持配置
from chunjun.
你的mysql是什么版本
from chunjun.
谢谢回答我的问题。
我用的版本信息:5.7.10
我之前用release里面的1.5 只能local
现在用了branch里面的 1.5 能用standalone 但是还是不支持 yarn
使用yarn 会提示:
16:01:06.836 [main] DEBUG org.apache.hadoop.security.authentication.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: No flink session found on yarn cluster.
at com.dtstack.flinkx.launcher.ClusterClientFactory.createYarnClient(ClusterClientFactory.java:126)
at com.dtstack.flinkx.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:57)
at com.dtstack.flinkx.launcher.Launcher.main(Launcher.java:87)
Caused by: java.lang.RuntimeException: No flink session found on yarn cluster.
at com.dtstack.flinkx.launcher.ClusterClientFactory.createYarnClient(ClusterClientFactory.java:111)
... 2 more
这个kerbose 是默认开启的 还是说我哪里配置不对?
我并没有配置这个kerbose
from chunjun.
java.io.IOException: Couldn't read data - Application was streaming results when the connection failed. Consider raising value of 'net_write_timeout' on the server.
at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.nextRecordInternal(JdbcInputFormat.java:275)
at com.dtstack.flinkx.inputformat.RichInputFormat.nextRecord(RichInputFormat.java:82)
at com.dtstack.flinkx.inputformat.RichInputFormat.nextRecord(RichInputFormat.java:44)
at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:91)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:94)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Application was streaming results when the connection failed. Consider raising value of 'net_write_timeout' on the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3903)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1999)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:374)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:354)
at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6312)
at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.nextRecordInternal(JdbcInputFormat.java:272)
... 9 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:101)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:144)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:174)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3011)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3522)
... 17 more
然后就抛异常了。
我们现在想做的是把mysql的大表 有好几亿的数据同步到hdfs上,目前还没有成功
from chunjun.
5.7版本的mysql还没有适配过,现在我们这边测试的都是5.6
from chunjun.
okay, 谢谢。
我们再找别的办法 做这个全量同步(数据量 6-7亿),你们这边有什么建议吗?
from chunjun.
同步开启多并发,是要设置切分键的,要不然是没效果的
from chunjun.
我看你截的图,虽然设置了4个并发,其实只有一个通道在读
from chunjun.
https://github.com/DTStack/flinkx/blob/master/docs/mysqlreader.md 看这个里面的splitpk 介绍
from chunjun.
好的,我先看看这个切分,非常感谢
from chunjun.
这个我有设置的,只是好像没有起效
from chunjun.
Related Issues (20)
- [Feature][ftp] 支持读取中文路径、优化读取execl
- [Bug] [ddl] maven package failed
- Release v1.12.8-cross join unnest使用报错
- [mongodb-x]-mongo维表连接失败
- [Feature][hbase] sql模式下新增特性
- 本地执行mysqlcdcreader到streamwriter,GenericRowData类型无法支持rowSize! HOT 1
- [Feature][ftp] sql模式下,支持指定字段的列索引,支持指定sheetNo工作表
- [Feature][Module Name] support hbase incremental read
- [Question][chunjun-oracle] oracle的clob类型,使用transformer报错
- [Bug] [chunjun-connector-binlog] Skipping unrecognized binlog event Unknown type:41 from: mysql-bin.xxx
- [Bug] [Module Name] Bug title mysql-mysql中加入transformer操作,sql可能有点长报错
- [Feature][gbase8s] 增加支持gbase8s版本connector HOT 7
- [Feature][connector][mysql] The connector supports MySQL Driver 8.x
- [Feature][chunjun-core] Supports capturing dirty data from the source and when the source sends it downstream
- [Bug] [sqlserver] 在 yarn-pre-job 模式找不到 sqlserver 相关驱动
- [Bug] [hbase] The HBase unit test failed.
- [Question][chunjun-connector-starrocks] Data size check HOT 1
- [Bug] [StarRocks] streamLoad报错502时,统计指标错误统计了未成功写入的记录数
- [Bug] [chunjun-connector-jdbc-base] clickhouse jdbc connection not supported setAutoCommit(false) HOT 1
- [Improve][chunjun-connector-clickhouse] Added support for map<string, unit>data types
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chunjun.