canal 1.1.2推送kafka非常慢

环境信息

canal version 1.1.2 mysql version 5.7.22

问题描述

看kafka监控, message in/s 很低

看canal监控blocking 100%一直

canal.properties

canal.mq.retries = 0 canal.mq.batchSize = 1048576 canal.mq.maxRequestSize = 16777216 canal.mq.lingerMs = 200 canal.mq.bufferMemory = 33554432 canal.mq.canalBatchSize = 1048576 canal.mq.canalGetTimeout = 100 canal.mq.flatMessage = true canal.mq.compressionType = none canal.mq.acks = all

instance.properties开了tsdb 其他配置都是默认值

压测了一下kafka, 感觉不慢啊

[2019-03-06 21:44:19]#bin/kafka-producer-perf-test.sh --topic test1 --num-records 500000 --record-size 10000 --throughput -1 --producer-props bootstrap.servers=x.x.x.x:9092 acks=-1 batch.size=1048576 [2019-03-06 21:44:25]21090 records sent, 4207.1 records/sec (40.12 MB/sec), 681.9 ms avg latency, 793.0 max latency. [2019-03-06 21:44:30]22672 records sent, 4525.3 records/sec (43.16 MB/sec), 739.1 ms avg latency, 805.0 max latency. [2019-03-06 21:44:35]23192 records sent, 4635.6 records/sec (44.21 MB/sec), 717.1 ms avg latency, 758.0 max latency. [2019-03-06 21:44:40]23192 records sent, 4620.8 records/sec (44.07 MB/sec), 720.6 ms avg latency, 754.0 max latency. [2019-03-06 21:44:45]23192 records sent, 4619.0 records/sec (44.05 MB/sec), 718.9 ms avg latency, 757.0 max latency. [2019-03-06 21:44:50]21736 records sent, 4342.0 records/sec (41.41 MB/sec), 767.4 ms avg latency, 1026.0 max latency. [2019-03-06 21:44:55]22152 records sent, 4412.7 records/sec (42.08 MB/sec), 754.0 ms avg latency, 996.0 max latency. [2019-03-06 21:45:00]22880 records sent, 4576.0 records/sec (43.64 MB/sec), 728.2 ms avg latency, 769.0 max latency. [2019-03-06 21:45:05]22048 records sent, 4403.4 records/sec (41.99 MB/sec), 754.5 ms avg latency, 996.0 max latency. [2019-03-06 21:45:10]22360 records sent, 4466.6 records/sec (42.60 MB/sec), 745.8 ms avg latency, 950.0 max latency. [2019-03-06 21:45:15]23296 records sent, 4659.2 records/sec (44.43 MB/sec), 714.7 ms avg latency, 754.0 max latency. [2019-03-06 21:45:20]23504 records sent, 4684.9 records/sec (44.68 MB/sec), 709.8 ms avg latency, 739.0 max latency. [2019-03-06 21:45:25]23608 records sent, 4713.1 records/sec (44.95 MB/sec), 706.6 ms avg latency, 748.0 max latency. [2019-03-06 21:45:30]23504 records sent, 4691.4 records/sec (44.74 MB/sec), 708.0 ms avg latency, 742.0 max latency. [2019-03-06 21:45:35]23192 records sent, 4621.8 records/sec (44.08 MB/sec), 719.5 ms avg latency, 766.0 max latency. [2019-03-06 21:45:40]23296 records sent, 4648.0 records/sec (44.33 MB/sec), 715.8 ms avg latency, 752.0 max latency. [2019-03-06 21:45:45]23296 records sent, 4651.8 records/sec (44.36 MB/sec), 715.2 ms avg latency, 752.0 max latency. [2019-03-06 21:45:50]23712 records sent, 4732.9 records/sec (45.14 MB/sec), 704.1 ms avg latency, 739.0 max latency. [2019-03-06 21:45:55]23608 records sent, 4712.2 records/sec (44.94 MB/sec), 705.7 ms avg latency, 745.0 max latency. [2019-03-06 21:46:00]23608 records sent, 4710.3 records/sec (44.92 MB/sec), 706.4 ms avg latency, 732.0 max latency. [2019-03-06 21:46:05]23608 records sent, 4711.2 records/sec (44.93 MB/sec), 706.7 ms avg latency, 746.0 max latency. [2019-03-06 21:46:09]500000 records sent, 4588.839941 records/sec (43.76 MB/sec), 720.54 ms avg latency, 1026.00 ms max latency, 715 ms 50th, 751 ms 95th, 970 ms 99th, 995 ms 99.9th.

请问flatMessage是啥意思啊, 本人初级运维,不懂开发,求大佬们解释一下

原提问者GitHub用户Fanduzi