Flume抽取日志

监控节点dt02上的日志文件，并将日志传输到节点dt03

agent结构图

在dt02上配置agent

exec-menory-avro.conf

##将dt02的数据发送给dt03

exec-menory-avro.sources = exec-source
exec-menory-avro.sinks = avro-sink
exec-menory-avro.channels = menory-channel

exec-menory-avro.sources.exec-source.type = exec
exec-menory-avro.sources.exec-source.command = tail -F /root/data-liwenchi/data.log
exec-menory-avro.sources.exec-source.shell = /bin/sh -c

exec-menory-avro.sinks.avro-sink.type = avro
exec-menory-avro.sinks.avro-sink.hostname = dt03
exec-menory-avro.sinks.avro-sink.port = 44444

exec-menory-avro.channels.menory-channel.type = memory
exec-menory-avro.channels.menory-channel.capacity = 1000
exec-menory-avro.channels.menory-channel.transactionCapacity = 100

exec-menory-avro.sources.exec-source.channels = menory-channel        
exec-menory-avro.sinks.avro-sink.channel = menory-channel

在dt03上配置agent

#接收从dt02发送过来的数据

avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = menory-channel

avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = dt03
avro-memory-logger.sources.avro-source.port = 44444

avro-memory-logger.sinks.logger-sink.type = logger

avro-memory-logger.channels.menory-channel.type = memory
avro-memory-logger.channels.menory-channel.capacity = 1000
avro-memory-logger.channels.menory-channel.transactionCapacity = 100

avro-memory-logger.sources.avro-source.channels = menory-channel        
avro-memory-logger.sinks.logger-sink.channel = menory-channel

启动脚本

run.sh

#!/bin/bash
flume-ng agent \
--name avro-memory-logger \
--conf /root/liwenchi/flume \
--conf-file avro-memory-logger.conf \
-Dflume.root.logger=INFO,console

注意事项

启动时，先启动dt03上的agent再启动dt02上的agent

日志收集过程

1. 节点dt01上监控一个文件，当用户访问时记录行为日志到dt01上的access.log中
2. avro sink将新产生的日志输出到对应的avro source指定的hostname和port上
3. 通过avro source对应的agent将日志输出到控制台(kafka)