sink组故障转移
有的时候需要设置故障转移,就是比如说现在有一个channel,如果只设置一个的话,一旦该sink端崩掉的话就会造成输出数据中断,所以需要就是配置两个sink,并且设置优先级,优先发送哪个,如果优先级高的sink没有问题的话,channel的数据就会一直发送到它,但是一旦它崩掉,channel就会发送到sink组中另外一个优先级高的sink,如果过了阈值时间,优先级最高的起来的话,channel会继续选择它进行发送数据。
下面是flume的配置,flume1会配置两个sink,并设置优先级。
Flume1
a1.sources = r1 a1.sinks = k1 k2 a1.channels = c1 # 配置sink组 a1.sinkgroups=g1 a1.sinkgroups.g1.sinks = k1 k2 # 配置故障转移 a1.sinkgroups.g1.processor.type = failover # 配置优先级 a1.sinkgroups.g1.processor.priority.k1 = 5 a1.sinkgroups.g1.processor.priority.k2 = 10 a1.sinkgroups.g1.processor.maxpenalty = 10000 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 a1.sinks.k1.type = avro a1.sinks.k1.hostname=hadoop102 a1.sinks.k1.port=4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname=hadoop102 a1.sinks.k2.port=4142 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c1
Flume2
a2.sources = r1 a2.sinks = k1 a2.channels = c1 a2.sources.r1.type = avro a2.sources.r1.bind=hadoop102 a2.sources.r1.port=4141 a2.sinks.k1.type = logger a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1
Flume3
a3.sources = r1 a3.sinks = k1 a3.channels = c1 a3.sources.r1.type = avro a3.sources.r1.bind=hadoop102 a3.sources.r1.port=4142 a3.sinks.k1.type = logger a3.channels.c1.type = memory a3.channels.c1.capacity = 1000 a3.channels.c1.transactionCapacity = 100 a3.sources.r1.channels = c1 a3.sinks.k1.channel = c1