Comments (7)
现在支持flink1.4和flink1.5,master是flink1.4编译的
from chunjun.
现在支持flink1.4和flink1.5,master是flink1.4编译的
第一个问题:
StreamSource<OUT, ?> sourceOperator;
if (function instanceof StoppableFunction) {
sourceOperator = new StoppableStreamSource<>(cast2StoppableSourceFunction(function));
} else {
sourceOperator = new StreamSource<>(function);
}
return new DataStreamSource<>(this, typeInfo, sourceOperator, isParallel, sourceName);
}
源码下载下来:
sourceOperator = new StoppableStreamSource<>(cast2StoppableSourceFunction(function));
这行会有报错?
第二个问题:flink是1.5,standalone模式,没有使用HA,flinkx也是下载flinkx-r1.5.0版本的,执行命令:
[root@bd129106 flink-data-transfer]# bin/flinkx -mode standalone -job /tmp/zyl/flink-data-transfer/jobs/mysql_to_mysql.json -plugin /tmp/zyl/flink-data-transfer/plugins -flinkconf /tmp/zyl/flink-data-transfer/flinkconf/
会报如下错误:
16:59:53.539 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
16:59:53.550 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
16:59:53.550 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
16:59:53.559 [main] INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
Exception in thread "main" org.apache.flink.client.deployment.ClusterRetrieveException: Couldn't retrieve standalone cluster
at org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:51)
at com.dtstack.flinkx.launcher.ClusterClientFactory.createStandaloneClient(ClusterClientFactory.java:66)
at com.dtstack.flinkx.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:55)
at com.dtstack.flinkx.launcher.Launcher.main(Launcher.java:87)
Caused by: java.lang.NullPointerException
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:58)
at org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:194)
at org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:157)
at org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:49)
... 3 more
from chunjun.
拉下1.5 分支,第二个问题改了,你再试下
from chunjun.
拉下1.5 分支,第二个问题改了,你再试下
拉了1.5分支,运行报错信息变了
18:10:11.824 [flink-akka.actor.default-dispatcher-4] INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@bd129106:51191]
18:10:11.831 [main] INFO org.apache.flink.client.program.StandaloneClusterClient - Actor system started at akka.tcp://flink@bd129106:51191
18:10:12.000 [flink-akka.actor.default-dispatcher-4] DEBUG akka.serialization.Serialization(akka://flink) - Using serializer [akka.serialization.JavaSerializer] for message [akka.actor.Identify]
18:10:12.028 [flink-akka.actor.default-dispatcher-4] DEBUG akka.remote.EndpointWriter - Drained buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: 0, noBackoffCount: 0 , adaptiveBackoff: 1000
18:10:12.047 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.util.LeaderRetrievalUtils - Could not retrieve the leader for address akka.tcp://[email protected]:6123/user/jobmanager.
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://[email protected]:6123/), Path(/user/jobmanager)]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:68)
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:66)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:76)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:120)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:75)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Exception in thread "main" org.apache.flink.client.program.ProgramInvocationException: Failed to retrieve the JobManager gateway.
at org.apache.flink.client.program.ClusterClient.runDetached(ClusterClient.java:519)
at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:113)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
at org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:77)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:410)
at com.dtstack.flinkx.launcher.Launcher.main(Launcher.java:97)
Caused by: org.apache.flink.util.FlinkException: Could not connect to the leading JobManager. Please check that the JobManager is running.
at org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:932)
at org.apache.flink.client.program.ClusterClient.runDetached(ClusterClient.java:517)
... 5 more
Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Could not retrieve the leader gateway.
at org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:82)
at org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:927)
... 6 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at scala.concurrent.Await.result(package.scala)
at org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:80)
... 7 more
from chunjun.
你先用yarn模式,yarn模式是没问题的
from chunjun.
我之前也遇到同样的问题,V1.4版本,FLink集群按照HA方式部署后就执行成功了
from chunjun.
我之前也遇到同样的问题,V1.4版本,FLink集群按照HA方式部署后就执行成功了
你源码下载的V1.4版本,flink集群版本也是1.4,standlone 的HA方式部署的,能成功?
from chunjun.
Related Issues (20)
- [Bug] [jdbc base] 使用Collections.singletonList返回不可修改list,但代码中进行了修改 HOT 1
- StarRocksOutputFormat.writeMultipleRecordsInternal()中if判断导致数据积压问题bug
- [Bug][chunjun-connectors] jdbc's properties not work where jdbc's reader has set properties
- 纯钧如何使用Kubernetes Operator提交
- [Bug] [chunjun-client] 本地local模式提交执行任务情况下connector 的加载错误
- [Feature][ftp] 支持读取中文路径、优化读取execl
- [Bug] [ddl] maven package failed
- Release v1.12.8-cross join unnest使用报错
- [mongodb-x]-mongo维表连接失败
- [Feature][hbase] sql模式下新增特性
- 本地执行mysqlcdcreader到streamwriter,GenericRowData类型无法支持rowSize! HOT 1
- [Feature][ftp] sql模式下,支持指定字段的列索引,支持指定sheetNo工作表
- [Feature][Module Name] support hbase incremental read
- [Question][chunjun-oracle] oracle的clob类型,使用transformer报错
- [Bug] [chunjun-connector-binlog] Skipping unrecognized binlog event Unknown type:41 from: mysql-bin.xxx
- [Bug] [Module Name] Bug title mysql-mysql中加入transformer操作,sql可能有点长报错
- [Feature][gbase8s] 增加支持gbase8s版本connector HOT 7
- [Feature][connector][mysql] The connector supports MySQL Driver 8.x
- [Feature][chunjun-core] Supports capturing dirty data from the source and when the source sends it downstream
- [Bug] [sqlserver] 在 yarn-pre-job 模式找不到 sqlserver 相关驱动
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chunjun.