监控节点dt02上的日志文件，并将日志传输到节点dt03
agent结构图
在dt02上配置agent
exec-menory-avro.conf
##将dt02的数据发送给dt03
exec-menory-avro.sources = exec-source
exec-menory-avro.sinks = avro-sink
exec-menory-avro.channels = menory-channel
exec-menory-avro.sources.exec-source.type = exec
exec-menory-avro.sources.exec-source.command = tail -F /root/data-liwenchi/data.log
exec-menory-avro.sources.exec-source.shell = /bin/sh -c
exec-menory-avro.sinks.avro-sink.type = avro
exec-menory-avro.sinks.avro-sink.hostname = dt03
exec-menory-avro.sinks.avro-sink.port = 44444
exec-menory-avro.channels.menory-channel.type = memory
exec-menory-avro.channels.menory-channel.capacity = 1000
exec-menory-avro.channels.menory-channel.transactionCapacity = 100
exec-menory-avro.sources.exec-source.channels = menory-channel
exec-menory-avro.sinks.avro-sink.channel = menory-channel在dt03上配置agent
#接收从dt02发送过来的数据
avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = menory-channel
avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = dt03
avro-memory-logger.sources.avro-source.port = 44444
avro-memory-logger.sinks.logger-sink.type = logger
avro-memory-logger.channels.menory-channel.type = memory
avro-memory-logger.channels.menory-channel.capacity = 1000
avro-memory-logger.channels.menory-channel.transactionCapacity = 100
avro-memory-logger.sources.avro-source.channels = menory-channel
avro-memory-logger.sinks.logger-sink.channel = menory-channel启动脚本
run.sh
#!/bin/bash
flume-ng agent \
--name avro-memory-logger \
--conf /root/liwenchi/flume \
--conf-file avro-memory-logger.conf \
-Dflume.root.logger=INFO,console注意事项
启动时，先启动dt03上的agent再启动dt02上的agent
日志收集过程
- 节点dt01上监控一个文件，当用户访问时记录行为日志到dt01上的access.log中
- avro sink将新产生的日志输出到对应的avro source指定的hostname和port上
- 通过avro source对应的agent将日志输出到控制台(kafka)
