0xqq / flink_in_action

streaming as u know

flink on yarn部署

flink on yarn需要的组件与版本如下

1. Zookeeper 3.4.9 用于做Flink的JobManager的HA服务
2. hadoop 2.6.5 搭建HDFS和Yarn
3. flink 1.4.2版本（scala 2.11）

Zookeeper, HDFS 和 Yarn 的组件的安装可以参照网上的教程。

在zookeeper，HDFS 和Yarn的组件的安装好的前提下，在客户机上提交Flink任务，具体流程如下：

• 在启动Yarn-Session 之前， 设置好HADOOP_HOME,YARN_CONF_DIR ， HADOOP_CONF_DIR环境变量中三者的一个。如下所示， 根据具体的hadoop 路径来设置

   $ export HADOOP_HOME=/usr/local/hadoop-current

• 配置flink 目录下的flink-conf.yaml, 如下所示

jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 256
taskmanager.heap.mb: 512
taskmanager.numberOfTaskSlots: 1
taskmanager.memory.preallocate: false
parallelism.default: 1
jobmanager.web.port: 8081

# yarn
yarn.maximum-failed-containers: 99999

#akka config
akka.watch.heartbeat.interval: 5 s
akka.watch.heartbeat.pause: 20 s
akka.ask.timeout: 60 s
akka.framesize: 20971520b

#high-avaliability
high-availability: zookeeper
## 根据安装的zookeeper信息填写
high-availability.zookeeper.quorum: 10.141.61.226:2181,10.141.53.244:2181,10.141.18.219:2181
high-availability.zookeeper.path.root: /flink
## HA 信息存储到HDFS的目录，根据各自的Hdfs情况修改
high-availability.zookeeper.storageDir: hdfs:///flink/recovery/

#checkpoint config
state.backend: rocksdb
## checkpoint到HDFS的目录 根据各自安装的HDFS情况修改
state.backend.fs.checkpointdir: hdfs:///flink/checkpoint
## 对外checkpoint到HDFS的目录
state.checkpoints.dir: hdfs:///flink/savepoint

#memory config
env.java.opts: -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -XX:+HeapDumpOnOutOfMemoryError
yarn.heap-cutoff-ratio: 0.2
taskmanager.memory.off-heap: true

• 提交Yarn-Session，切换到flink的bin 目录下,提交命令如下

   $ bin/yarn-session.sh -n 2 -s 6 -jm 3072 -tm 6144 -nm test -d

启动yarn-session的参数解释如下

参数	参数解释	设置推荐
-n(--container)	taskmanager的数量
-s(--slots)	用启动应用所需的slot数量/ -s 的值向上取整，有时可以多一些taskmanager，做冗余 每个taskmanager的slot数量，默认一个slot一个core，默认每个taskmanager的slot的个数为1	6～10
-jm	jobmanager的内存（单位MB)	3072
-tm	每个taskmanager的内存（单位MB)	根据core 与内存的比例来设置，-s的值＊ （core与内存的比）来算
-nm	yarn 的appName(现在yarn的ui上的名字)｜
-d	后台执行

• 提交yarn－session 后，可以在yarn的ui上看到一个应用（应用有一个appId）, 切换到flink的bin目录下，提交flink 应用。命令如下

 $ ./flink -run file:///home/yarn/test.jar -a 1 -p 12 -yid appId -nm flink-test -d

启动flink 应用的参数解释如下

参数	参数解释
-j	运行flink 应用的jar所在的目录
-a	运行flink 应用的主方法的参数
-p	运行flink应用的并行度
-c	运行flink应用的主类, 可以通过在打包设置主类
-nm	flink 应用名字，在flink-ui 上面展示
-d	后台执行
--fromsavepoint	flink 应用启动的状态恢复点

• 启动flink应用成功，即可在yarn ui 点击对应应用的ApplicationMaster链接,既可以查看flink-ui ，并查看flink 应用运行情况。

15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=73, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 0, elements= [18460, 18490, 18520, 18500, 18530, 18560, 18510, 18540, 18570, 18550, 18580, 18610, 18590, 18620, 18650, 18600, 18630, 18660, 18640, 18670, 18700, 18680, 18710, 18740, 18690, 18720, 18730, 18760, 18790, 18770, 18800, 18750, 18780, 18810, 18820, 18850, 18830, 18860, 18890, 18840, 18870, 18900, 18880, 18910, 18940, 18920, 18950, 18930, 18960, 18990, 18970, 19000, 19030, 18980, 19010, 19040, 19070, 19020, 19050, 19080, 19060, 19090, 19120, 19100, 19130, 19160, 19110, 19140, 19170, 19150, 19180, 19210, 19190, 19220, 19250, 19200, 19230, 19260, 19240, 19270, 19300, 19280, 19310, 19340, 19290, 19320, 19350] 15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=73, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 1, elements= [18481, 18511, 18541, 18491, 18521, 18551, 18501, 18531, 18561, 18571, 18601, 18631, 18581, 18611, 18641, 18591, 18621, 18651, 18661, 18691, 18671, 18701, 18731, 18681, 18711, 18741, 18721, 18751, 18781, 18761, 18791, 18771, 18801, 18811, 18841, 18821, 18851, 18881, 18831, 18861, 18891, 18871, 18901, 18931, 18911, 18941, 18971, 18921, 18951, 18981, 18961, 18991, 19021, 19001, 19031, 19061, 19011, 19041, 19071, 19051, 19081, 19111, 19091, 19121, 19151, 19101, 19131, 19161, 19141, 19171, 19201, 19181, 19211, 19241, 19191, 19221, 19251, 19231, 19261, 19291, 19321, 19271, 19301, 19331, 19281, 19311, 19341] 15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=74, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 7, elements= [18457, 18487, 18517, 18497, 18527, 18557, 18507, 18537, 18567, 18547, 18577, 18607, 18587, 18617, 18647, 18597, 18627, 18657, 18637, 18667, 18697, 18677, 18707, 18737, 18687, 18717, 18747, 18727, 18757, 18787, 18767, 18797, 18777, 18807, 18817, 18847, 18827, 18857, 18887, 18837, 18867, 18897, 18877, 18907, 18937, 18917, 18947, 18927, 18957, 18987, 18967, 18997, 19027, 18977, 19007, 19037, 19067, 19017, 19047, 19077, 19057, 19087, 19117, 19097, 19127, 19157, 19107, 19137, 19167, 19147, 19177, 19207, 19187, 19217, 19247, 19197, 19227, 19257, 19237, 19267, 19297, 19277, 19307, 19337, 19287, 19317, 19347] 15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=74, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 5, elements= [18475, 18505, 18535, 18485, 18515, 18545, 18575, 18495, 18525, 18555, 18585, 18565, 18595, 18625, 18605, 18635, 18665, 18615, 18645, 18675, 18655, 18685, 18715, 18695, 18725, 18705, 18735, 18745, 18775, 18755, 18785, 18765, 18795, 18805, 18835, 18865, 18815, 18845, 18875, 18825, 18855, 18885, 18915, 18895, 18925, 18905, 18935, 18965, 18945, 18975, 18955, 18985, 19015, 19045, 18995, 19025, 19055, 19005, 19035, 19065, 19075, 19105, 19135, 19085, 19115, 19145, 19095, 19125, 19155, 19165, 19195, 19225, 19175, 19205, 19235, 19265, 19185, 19215, 19245, 19275, 19255, 19285, 19315, 19295, 19325, 19355, 19305, 19335, 19365] 15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=72, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 9, elements= [18469, 18499, 18529, 18509, 18539, 18569, 18519, 18549, 18579, 18559, 18589, 18619, 18599, 18629, 18659, 18609, 18639, 18669, 18649, 18679, 18709, 18689, 18719, 18699, 18729, 18739, 18769, 18749, 18779, 18759, 18789, 18799, 18829, 18859, 18809, 18839, 18869, 18899, 18819, 18849, 18879, 18909, 18889, 18919, 18929, 18959, 18939, 18969, 18949, 18979, 19009, 19039, 18989, 19019, 19049, 19079, 18999, 19029, 19059, 19089, 19069, 19099, 19129, 19109, 19139, 19119, 19149, 19159, 19189, 19219, 19169, 19199, 19229, 19259, 19179, 19209, 19239, 19269, 19249, 19279, 19309, 19289, 19319, 19349, 19299, 19329, 19359] 15:36:21,000 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=74, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 2, elements= [18472, 18502, 18532, 18512, 18542, 18572, 18522, 18552, 18582, 18562, 18592, 18622, 18602, 18632, 18662, 18612, 18642, 18672, 18652, 18682, 18712, 18692, 18722, 18702, 18732, 18742, 18772, 18752, 18782, 18762, 18792, 18802, 18832, 18862, 18812, 18842, 18872, 18822, 18852, 18882, 18912, 18892, 18922, 18902, 18932, 18962, 18942, 18972, 18952, 18982, 19012, 19042, 18992, 19022, 19052, 19082, 19002, 19032, 19062, 19092, 19072, 19102, 19132, 19112, 19142, 19122, 19152, 19162, 19192, 19222, 19172, 19202, 19232, 19262, 19182, 19212, 19242, 19272, 19252, 19282, 19312, 19292, 19322, 19352, 19302, 19332, 19362] 15:36:21,003 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=74, processtime= 1533800181003,window= TimeWindow{start=1533800180000, end=1533800181000},key= 3, elements= [18463, 18493, 18523, 18503, 18533, 18563, 18513, 18543, 18573, 18553, 18583, 18613, 18593, 18623, 18653, 18603, 18633, 18663, 18643, 18673, 18703, 18683, 18713, 18693, 18723, 18733, 18763, 18743, 18773, 18803, 18753, 18783, 18813, 18793, 18823, 18853, 18833, 18863, 18893, 18843, 18873, 18903, 18883, 18913, 18943, 18923, 18953, 18933, 18963, 18973, 19003, 19033, 18983, 19013, 19043, 19073, 18993, 19023, 19053, 19083, 19063, 19093, 19123, 19103, 19133, 19163, 19113, 19143, 19173, 19153, 19183, 19213, 19193, 19223, 19253, 19203, 19233, 19263, 19243, 19273, 19303, 19283, 19313, 19343, 19293, 19323, 19353] 15:36:21,001 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=75, processtime= 1533800181000,window= TimeWindow{start=1533800180000, end=1533800181000},key= 4, elements= [18454, 18484, 18514, 18494, 18524, 18554, 18504, 18534, 18564, 18544, 18574, 18604, 18584, 18614, 18644, 18594, 18624, 18654, 18634, 18664, 18694, 18674, 18704, 18734, 18684, 18714, 18744, 18724, 18754, 18784, 18764, 18794, 18774, 18804, 18814, 18844, 18824, 18854, 18884, 18834, 18864, 18894, 18874, 18904, 18934, 18914, 18944, 18974, 18924, 18954, 18984, 18964, 18994, 19024, 19004, 19034, 19064, 19014, 19044, 19074, 19054, 19084, 19114, 19094, 19124, 19154, 19104, 19134, 19164, 19144, 19174, 19204, 19184, 19214, 19244, 19194, 19224, 19254, 19234, 19264, 19294, 19324, 19274, 19304, 19334, 19284, 19314, 19344] 15:36:21,003 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=75, processtime= 1533800181003,window= TimeWindow{start=1533800180000, end=1533800181000},key= 8, elements= [18478, 18508, 18538, 18488, 18518, 18548, 18578, 18498, 18528, 18558, 18588, 18568, 18598, 18628, 18608, 18638, 18668, 18618, 18648, 18678, 18658, 18688, 18698, 18728, 18708, 18738, 18718, 18748, 18778, 18758, 18788, 18768, 18798, 18808, 18838, 18818, 18848, 18878, 18828, 18858, 18888, 18868, 18898, 18928, 18908, 18938, 18968, 18918, 18948, 18978, 18958, 18988, 19018, 19048, 18998, 19028, 19058, 19008, 19038, 19068, 19078, 19108, 19088, 19118, 19148, 19098, 19128, 19158, 19138, 19168, 19198, 19228, 19178, 19208, 19238, 19188, 19218, 19248, 19278, 19258, 19288, 19318, 19268, 19298, 19328, 19358, 19308, 19338] 15:36:21,003 ERROR org.spafka.streaming.flink.Eos.FlinkKafkaBackPressure - t=75, processtime= 1533800181003,window= TimeWindow{start=1533800180000, end=1533800181000},key= 6, elements= [18466, 18496, 18526, 18506, 18536, 18566, 18516, 18546, 18576, 18556, 18586, 18616, 18596, 18626, 18656, 18606, 18636, 18666, 18646, 18676, 18706, 18686, 18716, 18696, 18726, 18736, 18766, 18746, 18776, 18756, 18786, 18796, 18826, 18856, 18806, 18836, 18866, 18896, 18816, 18846, 18876, 18906, 18886, 18916, 18946, 18926, 18956, 18936, 18966, 18976, 19006, 19036, 18986, 19016, 19046, 19076, 18996, 19026, 19056, 19086, 19066, 19096, 19126, 19106, 19136, 19116, 19146, 19156, 19186, 19216, 19166, 19196, 19226, 19256, 19176, 19206, 19236, 19266, 19246, 19276, 19306, 19286, 19316, 19346, 19296, 19326, 19356]