简介

BPF编译器集合（BPF Compiler Collection，简称BCC）。项目地址https://github.com/iovisor/bcc，是一个用于创建高效内核跟踪和操作程序的工具包，包括几个有用的工具和示例。它利用了扩展的 BPF（伯克利包过滤器），正式名称为 eBPF。
BCC 使 BPF 程序更易于编写，使用 C 中的内核工具（并包括围绕 LLVM 的 C 包装器），以及 Python 和 lua 中的前端。它适用于许多任务，包括性能分析和网络流量控制。
BCC还包含多个可以直接使用的BPF性能分析和故障定位工具。

BCC结构

BCCde目录结构如图

# ls
CMakeLists.txt           FAQ.txt      LINKS.md       SPECS   docker    images         man      src
CODEOWNERS               INSTALL.md   QUICKSTART.md  cmake   docs      introspection  scripts  tests
CONTRIBUTING-SCRIPTS.md  LICENSE.txt  README.md      debian  examples  libbpf-tools   snap     tools

• tools：包含BCC提供的工具及其示例。
• man：包含工具的帮助文档。
• src：包含Python、C++、Lua的BCC库

安装BCC

内核要求

BCC推荐使用最新的内核，并且内核需要开启选项

• CONFIG_BPF=y
• CONFIG_BPF_SYSCALL=y
• CONFIG_BPF_EVENTS=y
• CONFIG_BPF_JIT=y
• CONFIG_HAVE_EBPF_JIT=y

Ubuntu

BCC已经被打包到Ubuntu的仓库中，包名为bpfcc-tools，可以直接使用命令安装

apt-get install bpfcc-tools linux-headers-$(uname -r)

执行完成后BCC工具会安装到/sbin目录下，并带有-bpfcc后缀

ls /sbin/*-bpfcc
/sbin/argdist-bpfcc       /sbin/fileslower-bpfcc      /sbin/perlcalls-bpfcc    /sbin/syncsnoop-bpfcc
/sbin/bashreadline-bpfcc  /sbin/filetop-bpfcc         /sbin/perlflow-bpfcc     /sbin/syscount-bpfcc
/sbin/bindsnoop-bpfcc     /sbin/funccount-bpfcc       /sbin/perlstat-bpfcc     /sbin/tclcalls-bpfcc
/sbin/biolatency-bpfcc    /sbin/funcinterval-bpfcc    /sbin/phpcalls-bpfcc     /sbin/tclflow-bpfcc
/sbin/biolatpcts-bpfcc    /sbin/funclatency-bpfcc     /sbin/phpflow-bpfcc      /sbin/tclobjnew-bpfcc
/sbin/biosnoop-bpfcc      /sbin/funcslower-bpfcc      /sbin/phpstat-bpfcc      /sbin/tclstat-bpfcc
/sbin/biotop-bpfcc        /sbin/gethostlatency-bpfcc  /sbin/pidpersec-bpfcc    /sbin/tcpaccept-bpfcc
/sbin/bitesize-bpfcc      /sbin/hardirqs-bpfcc        /sbin/profile-bpfcc      /sbin/tcpconnect-bpfcc
/sbin/bpflist-bpfcc       /sbin/inject-bpfcc          /sbin/pythoncalls-bpfcc  /sbin/tcpconnlat-bpfcc
/sbin/btrfsdist-bpfcc     /sbin/javacalls-bpfcc       /sbin/pythonflow-bpfcc   /sbin/tcpdrop-bpfcc
/sbin/btrfsslower-bpfcc   /sbin/javaflow-bpfcc        /sbin/pythongc-bpfcc     /sbin/tcplife-bpfcc
/sbin/cachestat-bpfcc     /sbin/javagc-bpfcc          /sbin/pythonstat-bpfcc   /sbin/tcpretrans-bpfcc
/sbin/cachetop-bpfcc      /sbin/javaobjnew-bpfcc      /sbin/readahead-bpfcc    /sbin/tcprtt-bpfcc
/sbin/capable-bpfcc       /sbin/javastat-bpfcc        /sbin/reset-trace-bpfcc  /sbin/tcpstates-bpfcc
/sbin/cobjnew-bpfcc       /sbin/javathreads-bpfcc     /sbin/rubycalls-bpfcc    /sbin/tcpsubnet-bpfcc
/sbin/compactsnoop-bpfcc  /sbin/killsnoop-bpfcc       /sbin/rubyflow-bpfcc     /sbin/tcpsynbl-bpfcc
/sbin/cpudist-bpfcc       /sbin/klockstat-bpfcc       /sbin/rubygc-bpfcc       /sbin/tcptop-bpfcc
/sbin/cpuunclaimed-bpfcc  /sbin/llcstat-bpfcc         /sbin/rubyobjnew-bpfcc   /sbin/tcptracer-bpfcc
/sbin/criticalstat-bpfcc  /sbin/mdflush-bpfcc         /sbin/rubystat-bpfcc     /sbin/threadsnoop-bpfcc
/sbin/dbslower-bpfcc      /sbin/memleak-bpfcc         /sbin/runqlat-bpfcc      /sbin/tplist-bpfcc
/sbin/dbstat-bpfcc        /sbin/mountsnoop-bpfcc      /sbin/runqlen-bpfcc      /sbin/trace-bpfcc
/sbin/dcsnoop-bpfcc       /sbin/mysqld_qslower-bpfcc  /sbin/runqslower-bpfcc   /sbin/ttysnoop-bpfcc
/sbin/dcstat-bpfcc        /sbin/netqtop-bpfcc         /sbin/shmsnoop-bpfcc     /sbin/vfscount-bpfcc
/sbin/deadlock-bpfcc      /sbin/nfsdist-bpfcc         /sbin/slabratetop-bpfcc  /sbin/vfsstat-bpfcc
/sbin/dirtop-bpfcc        /sbin/nfsslower-bpfcc       /sbin/sofdsnoop-bpfcc    /sbin/wakeuptime-bpfcc
/sbin/drsnoop-bpfcc       /sbin/nodegc-bpfcc          /sbin/softirqs-bpfcc     /sbin/xfsdist-bpfcc
/sbin/execsnoop-bpfcc     /sbin/nodestat-bpfcc        /sbin/solisten-bpfcc     /sbin/xfsslower-bpfcc
/sbin/exitsnoop-bpfcc     /sbin/offcputime-bpfcc      /sbin/sslsniff-bpfcc     /sbin/zfsdist-bpfcc
/sbin/ext4dist-bpfcc      /sbin/offwaketime-bpfcc     /sbin/stackcount-bpfcc   /sbin/zfsslower-bpfcc
/sbin/ext4slower-bpfcc    /sbin/oomkill-bpfcc         /sbin/statsnoop-bpfcc
/sbin/filelife-bpfcc      /sbin/opensnoop-bpfcc       /sbin/swapin-bpfcc

或者你也可以从仓库拉取最新的包

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

Ubuntu 从源码构建

要从源代码构建工具链，需要：

• LLVM 3.7.1 或更高版本，编译时支持 BPF（默认=on）
• Clang，由与 LLVM 相同的树构建
• cmake (>=3.1)、gcc (>=4.7)、flex、bison
• LuaJIT，如果你想要 Lua 支持

安装依赖项

# For Bionic (18.04 LTS)
sudo apt-get -y install bison build-essential cmake flex git libedit-dev \
  libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

# For Eoan (19.10) or Focal (20.04.1 LTS)
sudo apt install -y bison build-essential cmake flex git libedit-dev \
  libllvm7 llvm-7-dev libclang-7-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils
  
# For Hirsute (21.04)  or Impish (21.10)
sudo apt install -y bison build-essential cmake flex git libedit-dev   libllvm11 llvm-11-dev libclang-11-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

# For other versions
sudo apt-get -y install bison build-essential cmake flex git libedit-dev \
  libllvm3.7 llvm-3.7-dev libclang-3.7-dev python zlib1g-dev libelf-dev python3-distutils

# For Lua support
sudo apt-get -y install luajit luajit-5.1-dev

编译和安装BCC

git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd

BCC工具

下表列出了一些BCC工具及其用途

funccount

简述

funccount对事件，特别是函数调用进行计数，可以使它获取

• 某个内核态或用户态函数是否被调用过
• 该函数每秒被调用多少次

funccount在内核中使用一个BPF映射表数据结构维护事件计数，它只讲总数给用户态。

语法

funccoount的命令行参数包括可以用来改变的选型和事件字符串：

funccount [options] eventname    

eventname的语法是：

• name或p:name：对内核函数name()进行插装。
• lib:name或p:lib:name：对用户态lib库中的函数name进行插桩。
• path:name：对于path路径下文件的用户态函数name进行插桩。
• t:system:name：对名为system:name的内核跟踪点进行插装。
• u:lib:name：对lib库中名为name的USDT探针进行插桩。
• *：用来匹配任意字符的通配符。

选项：

• -h、--help：显示帮助信息
• -p PID、--pid PID：只对该PID进行跟踪
• -i INTERVAL、--interval INTERVAL：统计总数，每INTERVAL秒
• -d DURATION、--duration DURATION：总共跟踪的秒数
• -T、--timestamp：输出时间
• -r、--regexp：使用正则表达式
• -D、--debug：启动前打印BPF程序信息

示例

对虚拟文件系统(VFS)内核函数进行统计：

funccount-bpfcc 'vfs_*'

对TCP内核函数进行计数：

funccount-bpfcc 'tcp_*'

统计每秒TCP发送函数的调用次数：

funccount-bpfcc -i 1 'tcp_send*'

统计每秒块I/O时间的数量：

funccount-bpfcc -i 1 't:block:*'

展示每秒新创建的进程数量：

funccount-bpfcc -i 1 t:sched:sched_process_fork

展示每秒libc中的getaddrinfo函数的调用次数：

funccount-bpfcc -i c:getaddrinfo

对libgo中全部的os.*调用进行计数：

funccount-bpfcc 'go:os.*'

源码

# @lint-avoid-python-3-compatibility-imports
#
# funccount Count functions, tracepoints, and USDT probes.
#           For Linux, uses BCC, eBPF.
#
# USAGE: funccount [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-r]
#                  [-c CPU] pattern
#
# The pattern is a string with optional '*' wildcards, similar to file
# globbing. If you'd prefer to use regular expressions, use the -r option.
#
# Copyright (c) 2015 Brendan Gregg.
# Licensed under the Apache License, Version 2.0 (the "License")
#
# 09-Sep-2015   Brendan Gregg       Created this.
# 18-Oct-2016   Sasha Goldshtein    Generalized for uprobes, tracepoints, USDT.

from __future__ import print_function
from bcc import ArgString, BPF, USDT
from time import sleep, strftime
import argparse
import re
import signal
import sys
import traceback

debug = False

def verify_limit(num):
    probe_limit = BPF.get_probe_limit()
    if num > probe_limit:
        raise Exception("maximum of %d probes allowed, attempted %d" %
                        (probe_limit, num))

class Probe(object):
    def __init__(self, pattern, use_regex=False, pid=None, cpu=None):
        """Init a new probe.

        Init the probe from the pattern provided by the user. The supported
        patterns mimic the 'trace' and 'argdist' tools, but are simpler because
        we don't have to distinguish between probes and retprobes.

            func            -- probe a kernel function
            lib:func        -- probe a user-space function in the library 'lib'
            /path:func      -- probe a user-space function in binary '/path'
            p::func         -- same thing as 'func'
            p:lib:func      -- same thing as 'lib:func'
            t:cat:event     -- probe a kernel tracepoint
            u:lib:probe     -- probe a USDT tracepoint
        """
        parts = bytes(pattern).split(b':')
        if len(parts) == 1:
            parts = [b"p", b"", parts[0]]
        elif len(parts) == 2:
            parts = [b"p", parts[0], parts[1]]
        elif len(parts) == 3:
            if parts[0] == b"t":
                parts = [b"t", b"", b"%s:%s" % tuple(parts[1:])]
            if parts[0] not in [b"p", b"t", b"u"]:
                raise Exception("Type must be 'p', 't', or 'u', but got %s" %
                                parts[0])
        else:
            raise Exception("Too many ':'-separated components in pattern %s" %
                            pattern)

        (self.type, self.library, self.pattern) = parts
        if not use_regex:
            self.pattern = self.pattern.replace(b'*', b'.*')
            self.pattern = b'^' + self.pattern + b'$'

        if (self.type == b"p" and self.library) or self.type == b"u":
            libpath = BPF.find_library(self.library)
            if libpath is None:
                # This might be an executable (e.g. 'bash')
                libpath = BPF.find_exe(str(self.library))
            if libpath is None or len(libpath) == 0:
                raise Exception("unable to find library %s" % self.library)
            self.library = libpath

        self.pid = pid
        self.cpu = cpu
        self.matched = 0
        self.trace_functions = {}   # map location number to function name

    def is_kernel_probe(self):
        return self.type == b"t" or (self.type == b"p" and self.library == b"")
    # 加载到内核
    def attach(self):
        if self.type == b"p" and not self.library:
            for index, function in self.trace_functions.items():
                self.bpf.attach_kprobe(
                        event=function,
                        fn_name="trace_count_%d" % index)
        elif self.type == b"p" and self.library:
            for index, function in self.trace_functions.items():
                self.bpf.attach_uprobe(
                        name=self.library,
                        sym=function,
                        fn_name="trace_count_%d" % index,
                        pid=self.pid or -1)
        elif self.type == b"t":
            for index, function in self.trace_functions.items():
                self.bpf.attach_tracepoint(
                        tp=function,
                        fn_name="trace_count_%d" % index)
        elif self.type == b"u":
            pass    # Nothing to do -- attach already happened in `load`

    def _add_function(self, template, probe_name):
        new_func = b"trace_count_%d" % self.matched
        text = template.replace(b"PROBE_FUNCTION", new_func)
        text = text.replace(b"LOCATION", b"%d" % self.matched)
        self.trace_functions[self.matched] = probe_name
        self.matched += 1
        return text

    def _generate_functions(self, template):
        self.usdt = None
        text = b""
        if self.type == b"p" and not self.library:
            functions = BPF.get_kprobe_functions(self.pattern)
            verify_limit(len(functions))
            for function in functions:
                text += self._add_function(template, function)
        elif self.type == b"p" and self.library:
            # uprobes are tricky because the same function may have multiple
            # addresses, and the same address may be mapped to multiple
            # functions. We aren't allowed to create more than one uprobe
            # per address, so track unique addresses and ignore functions that
            # map to an address that we've already seen. Also ignore functions
            # that may repeat multiple times with different addresses.
            addresses, functions = (set(), set())
            functions_and_addresses = BPF.get_user_functions_and_addresses(
                                        self.library, self.pattern)
            verify_limit(len(functions_and_addresses))
            for function, address in functions_and_addresses:
                if address in addresses or function in functions:
                    continue
                addresses.add(address)
                functions.add(function)
                text += self._add_function(template, function)
        elif self.type == b"t":
            tracepoints = BPF.get_tracepoints(self.pattern)
            verify_limit(len(tracepoints))
            for tracepoint in tracepoints:
                text += self._add_function(template, tracepoint)
        elif self.type == b"u":
            self.usdt = USDT(path=str(self.library), pid=self.pid)
            matches = []
            for probe in self.usdt.enumerate_probes():
                if not self.pid and (probe.bin_path != self.library):
                    continue
                if re.match(self.pattern, probe.name):
                    matches.append(probe.name)
            verify_limit(len(matches))
            for match in matches:
                new_func = b"trace_count_%d" % self.matched
                text += self._add_function(template, match)
                self.usdt.enable_probe(match, new_func)
            if debug:
                print(self.usdt.get_text())
        return text

    def load(self):
        trace_count_text = b"""
int PROBE_FUNCTION(void *ctx) {
    FILTERPID
    FILTERCPU
    int loc = LOCATION;
    counts.atomic_increment(loc);
    return 0;
}
        """
        bpf_text = b"""#include <uapi/linux/ptrace.h>

BPF_ARRAY(counts, u64, NUMLOCATIONS);
        """

        # We really mean the tgid from the kernel's perspective, which is in
        # the top 32 bits of bpf_get_current_pid_tgid().
        if self.pid:
            trace_count_text = trace_count_text.replace(b'FILTERPID',
                b"""u32 pid = bpf_get_current_pid_tgid() >> 32;
                   if (pid != %d) { return 0; }""" % self.pid)
        else:
            trace_count_text = trace_count_text.replace(b'FILTERPID', b'')

        if self.cpu:
            trace_count_text = trace_count_text.replace(b'FILTERCPU',
                b"""u32 cpu = bpf_get_smp_processor_id();
                   if (cpu != %d) { return 0; }""" % int(self.cpu))
        else:
            trace_count_text = trace_count_text.replace(b'FILTERCPU', b'')

        bpf_text += self._generate_functions(trace_count_text)
        bpf_text = bpf_text.replace(b"NUMLOCATIONS",
                                    b"%d" % len(self.trace_functions))
        if debug:
            print(bpf_text)

        if self.matched == 0:
            raise Exception("No functions matched by pattern %s" %
                            self.pattern)

        self.bpf = BPF(text=bpf_text,
                       usdt_contexts=[self.usdt] if self.usdt else [])
        self.clear()    # Initialize all array items to zero

    def counts(self):
        return self.bpf["counts"]

    def clear(self):
        counts = self.bpf["counts"]
        for location, _ in list(self.trace_functions.items()):
            counts[counts.Key(location)] = counts.Leaf()

class Tool(object):
    def __init__(self):
        examples = """examples:
    ./funccount 'vfs_*'             # count kernel fns starting with "vfs"
    ./funccount -r '^vfs.*'         # same as above, using regular expressions
    ./funccount -Ti 5 'vfs_*'       # output every 5 seconds, with timestamps
    ./funccount -d 10 'vfs_*'       # trace for 10 seconds only
    ./funccount -p 185 'vfs_*'      # count vfs calls for PID 181 only
    ./funccount t:sched:sched_fork  # count calls to the sched_fork tracepoint
    ./funccount -p 185 u:node:gc*   # count all GC USDT probes in node, PID 185
    ./funccount c:malloc            # count all malloc() calls in libc
    ./funccount go:os.*             # count all "os.*" calls in libgo
    ./funccount -p 185 go:os.*      # count all "os.*" calls in libgo, PID 185
    ./funccount ./test:read*        # count "read*" calls in the ./test binary
    ./funccount -c 1 'vfs_*'        # count vfs calls on CPU 1 only
    """
        parser = argparse.ArgumentParser(
            description="Count functions, tracepoints, and USDT probes",
            formatter_class=argparse.RawDescriptionHelpFormatter,
            epilog=examples)
        parser.add_argument("-p", "--pid", type=int,
            help="trace this PID only")
        parser.add_argument("-i", "--interval",
            help="summary interval, seconds")
        parser.add_argument("-d", "--duration",
            help="total duration of trace, seconds")
        parser.add_argument("-T", "--timestamp", action="store_true",
            help="include timestamp on output")
        parser.add_argument("-r", "--regexp", action="store_true",
            help="use regular expressions. Default is \"*\" wildcards only.")
        parser.add_argument("-D", "--debug", action="store_true",
            help="print BPF program before starting (for debugging purposes)")
        parser.add_argument("-c", "--cpu",
            help="trace this CPU only")
        parser.add_argument("pattern",
            type=ArgString,
            help="search expression for events")
        self.args = parser.parse_args()
        global debug
        debug = self.args.debug
        self.probe = Probe(self.args.pattern, self.args.regexp, self.args.pid,
                           self.args.cpu)
        if self.args.duration and not self.args.interval:
            self.args.interval = self.args.duration
        if not self.args.interval:
            self.args.interval = 99999999

    @staticmethod
    def _signal_ignore(signal, frame):
        print()

    def run(self):
        # 加载BPF程序
        self.probe.load()
        self.probe.attach()
        print("Tracing %d functions for \"%s\"... Hit Ctrl-C to end." %
              (self.probe.matched, bytes(self.args.pattern)))
        exiting = 0 if self.args.interval else 1
        seconds = 0
        while True:
            try:
                sleep(int(self.args.interval))
                seconds += int(self.args.interval)
            except KeyboardInterrupt:
                exiting = 1
                # as cleanup can take many seconds, trap Ctrl-C:
                signal.signal(signal.SIGINT, Tool._signal_ignore)
            if self.args.duration and seconds >= int(self.args.duration):
                exiting = 1

            print()
            if self.args.timestamp:
                print("%-8s\n" % strftime("%H:%M:%S"), end="")

            print("%-36s %8s" % ("FUNC", "COUNT"))
            counts = self.probe.counts()
            for k, v in sorted(counts.items(),
                               key=lambda counts: counts[1].value):
                if v.value == 0:
                    continue
                print("%-36s %8d" %
                      (self.probe.trace_functions[k.value], v.value))

            if exiting:
                print("Detaching...")
                exit()
            else:
                self.probe.clear()

if __name__ == "__main__":
    try:
        Tool().run()
    except Exception:
        if debug:
            traceback.print_exc()
        elif sys.exc_info()[0] is not SystemExit:
            print(sys.exc_info()[1])