MOT_Challenge官方代码TrackEval介绍-阿里云开发者社区

MOT_Challenge官方代码TrackEval介绍

作者声明

原文链接：凤⭐尘》》https://www.cnblogs.com/phoenixash/

MOTChallenge 介绍：

MOTChallenge是多目标跟踪领域最为常用的benchmark

CLEAR MOT Metrics认为一个好的多目标跟踪器应该有如下三点特性：

1.所有出现的目标都要能够及时找到（检测的性能）

2.找到目标位置要尽可能可真实目标位置一致（检测的性能）

3.保持追踪一致性，避免跟踪目标的跳变（匹配的性能）

MOTChallenge的评价指标一共有十一个，分别是：

Measure	Better	Perfect	Description
MOTA	higher	100%	跟踪的准确度，和出现FN，FP，IDs的数量负相关，可能出现负值。
MOTP	higher	100%	跟踪的精度，GT和检测的bbox的匹配交叠
IDF1	higher	100%	引入track ID的F1
FAF	lower	0	每帧的平均误报警数
MT	higher	100%	命中的轨迹占总轨迹的占比，定义命中的轨迹为长度小于ground truth 80%的轨迹
ML	lower	0	丢失的轨迹占总轨迹的占比，定义丢失轨迹为长度小于ground truth 20%的轨迹
FP	lower	0	FP的总数量，false positives也就是误检
FN	lower	0	FN的总数量，false negatives也就是漏检
IDs	lower	0	ID改变的总数量
Frag	lower	0	轨迹被打断的总数量
Hz	higher	Inf	处理速度，不包括检测器的耗时，而且这个指标由作者提供，MOTChallenge是计算不出来的，因为递交的是offline文件。

TrackEval 介绍:

provides code for a number of different tracking evaluation metrics.

**源码地址：**https://github.com/JonathonLuiten/TrackEval

**安装环境：**常见的数据处理环境

The code is written 100% in python with only numpy and scipy as minimum requirements.

**数据格式：**官网上下载data样例，放在TrackEval/下，从样例中可以看到评估支持多种常用的跟踪数据集

TrackEval 使用 - MOTA评估

以MOTA指标为例，通过自己的验证集，介绍TrackEval的使用。

首先data文件夹下，gt包含每个数据集的标注，gt/mot_challenge中包含训练集、验证集的标注，测试集没有标注。

data
|-- gt
|   |-- bdd100k
|   |-- mot_challenge
| |-- ......
|   `-- youtube_vis
`-- trackers
    |-- bdd100k
    |-- mot_challenge
    |-- ......
    `-- youtube_vis

data/gt/mot_challenge/seqmaps里面存储的是每个数据需要测试的视频序列名称，因为mot17包含3种检测器的视频序列，通常我们只使用一个检测器的视频序列，因为gt的标注都一样。这里如果只使用SDP检测的视频序列，当评估mot17验证集时，需要把其他检测器的视频序列名称删除。

1、修改需要评估的视频序列名称

例如：

修改源文件MOT17-train内容：data/gt/mot_challenge/seqmaps/MOT17-train.txt

name
MOT17-02-SDP
MOT17-04-SDP
MOT17-05-SDP
MOT17-09-SDP
MOT17-10-SDP
MOT17-11-SDP
MOT17-13-SDP

我的是：

name
001
002
003
004
...
011

2、添加验证集标签

因为不想过多修改原代码，这里删除MOT16-val下的文件，放入自己的验证集真值

data
|-- gt
  |-- mot_challenge
    |-- MOT16-val
      |-- 001
      | |-- gt.txt
      | `-- seqinfo.ini
      |-- 002
      | |-- ......
      |...
      |-- 011
        |-- gt.txt
        `--seqinfo.ini

3、添加跟踪的结果

在data/trackers/mot_challenge/MOT16-val文件夹下创建yourtrack/data文件夹保存每个视频跟踪的结果。例如：验证集的结果放入data/trackers/mot_challenge/MOT16-val/ICPR2022/data

trackers
    |-- mot_challenge
    |-- MOT16-val
      |-- ICPR2022
        |-- data
          |-- 001.txt
          |-- ...
          |-- 010.txt
          `-- 011.txt

4、运行检测脚本

进入TrackEval/scripts

python run_mot_challenge.py --BENCHMARK MOT17 --TRACKERS_TO_EVAL yourtracker --METRICS CLEAR Identity --USE_PARALLEL True --GT_LOC_FORMAT {gt_folder}/{seq}/gt/gt_val_half.txt

5、修改检测脚本

也可以自己根据自己的数据集格式修改检测脚本：

TrackEval/scripts的每个文件头部都有Command Line Arguments: Defaults介绍参数含义。

在TrackEval/trackeval/datasets/文件夹下复制一个自己的数据集对应配置文件如sat_challenge_2d_box.py并修改：

1、默认配置函数：

CLASSES_TO_EVAL，BENCHMARK，SPLIT_TO_EVAL, GT_LOC_FORMAT等

@staticmethod
    def get_default_dataset_config():
        """Default class config values"""
        code_path = utils.get_code_path()
        default_config = {
            'GT_FOLDER': os.path.join(code_path, 'data/gt/mot_challenge/'),  # Location of GT data
            'TRACKERS_FOLDER': os.path.join(code_path, 'data/trackers/mot_challenge/'),  # Trackers location
            'OUTPUT_FOLDER': None,  # Where to save eval results (if None, same as TRACKERS_FOLDER)
            'TRACKERS_TO_EVAL': None,  # Filenames of trackers to eval (if None, all in folder)
            'CLASSES_TO_EVAL': ['car'],  # Valid: ['pedestrian']
            'BENCHMARK': 'MOT16',  # Valid: 'MOT17', 'MOT16', 'MOT20', 'MOT15'
            'SPLIT_TO_EVAL': 'val',  # Valid: 'train', 'test', 'all'
            'INPUT_AS_ZIP': False,  # Whether tracker input files are zipped
            'PRINT_CONFIG': True,  # Whether to print current config
            'DO_PREPROC': True,  # Whether to perform preprocessing (never done for MOT15)
            'TRACKER_SUB_FOLDER': 'data',  # Tracker files are in TRACKER_FOLDER/tracker_name/TRACKER_SUB_FOLDER
            'OUTPUT_SUB_FOLDER': '',  # Output files are saved in OUTPUT_FOLDER/tracker_name/OUTPUT_SUB_FOLDER
            'TRACKER_DISPLAY_NAMES': None,  # Names of trackers to display, if None: TRACKERS_TO_EVAL
            'SEQMAP_FOLDER': None,  # Where seqmaps are found (if None, GT_FOLDER/seqmaps)
            'SEQMAP_FILE': None,  # Directly specify seqmap file (if none use seqmap_folder/benchmark-split_to_eval)
            'SEQ_INFO': None,  # If not None, directly specify sequences to eval and their number of timesteps
            'GT_LOC_FORMAT': '{gt_folder}/{seq}/gt/gt.txt',  # '{gt_folder}/{seq}/gt/gt.txt'
            'SKIP_SPLIT_FOL': False,  # If False, data is in GT_FOLDER/BENCHMARK-SPLIT_TO_EVAL/ and in
                                      # TRACKERS_FOLDER/BENCHMARK-SPLIT_TO_EVAL/tracker/
                                      # If True, then the middle 'benchmark-split' folder is skipped for both.
        }
        return default_config

2、初始化函数：

self.valid_classes = [‘car’,‘airplane’,‘ship’,‘train’]

self.class_name_to_class_id = {‘car’: 1, ‘airplane’: 2, ‘ship’: 3, ‘train’: 4}

def __init__(self, config=None):
        """Initialise dataset, checking that all required files are present"""
        super().__init__()
        # Fill non-given config values with defaults
        self.config = utils.init_config(config, self.get_default_dataset_config(), self.get_name())
        self.benchmark = self.config['BENCHMARK']
        gt_set = self.config['BENCHMARK'] + '-' + self.config['SPLIT_TO_EVAL']
        self.gt_set = gt_set
        if not self.config['SKIP_SPLIT_FOL']:
            split_fol = gt_set
        else:
            split_fol = ''
        self.gt_fol = os.path.join(self.config['GT_FOLDER'], split_fol)
        self.tracker_fol = os.path.join(self.config['TRACKERS_FOLDER'], split_fol)
        self.should_classes_combine = False
        self.use_super_categories = False
        self.data_is_zipped = self.config['INPUT_AS_ZIP']
        self.do_preproc = self.config['DO_PREPROC']
        self.output_fol = self.config['OUTPUT_FOLDER']
        if self.output_fol is None:
            self.output_fol = self.tracker_fol
        self.tracker_sub_fol = self.config['TRACKER_SUB_FOLDER']
        self.output_sub_fol = self.config['OUTPUT_SUB_FOLDER']
        # Get classes to eval
        self.valid_classes = ['pedestrian']
        self.class_list = [cls.lower() if cls.lower() in self.valid_classes else None
                           for cls in self.config['CLASSES_TO_EVAL']]
        if not all(self.class_list):
            raise TrackEvalException('Attempted to evaluate an invalid class. Only pedestrian class is valid.')
        self.class_name_to_class_id = {'pedestrian': 1, 'person_on_vehicle': 2, 'car': 3, 'bicycle': 4, 'motorbike': 5,
                                       'non_mot_vehicle': 6, 'static_person': 7, 'distractor': 8, 'occluder': 9,
                                       'occluder_on_ground': 10, 'occluder_full': 11, 'reflection': 12, 'crowd': 13}
        self.valid_class_numbers = list(self.class_name_to_class_id.values())
        # Get sequences to eval and check gt files exist
        self.seq_list, self.seq_lengths = self._get_seq_info()
        if len(self.seq_list) < 1:
            raise TrackEvalException('No sequences are selected to be evaluated.')
        # Check gt files exist
        for seq in self.seq_list:
            if not self.data_is_zipped:
                curr_file = self.config["GT_LOC_FORMAT"].format(gt_folder=self.gt_fol, seq=seq)
                if not os.path.isfile(curr_file):
                    print('GT file not found ' + curr_file)
                    raise TrackEvalException('GT file not found for sequence: ' + seq)
        if self.data_is_zipped:
            curr_file = os.path.join(self.gt_fol, 'data.zip')
            if not os.path.isfile(curr_file):
                print('GT file not found ' + curr_file)
                raise TrackEvalException('GT file not found: ' + os.path.basename(curr_file))
        # Get trackers to eval
        if self.config['TRACKERS_TO_EVAL'] is None:
            self.tracker_list = os.listdir(self.tracker_fol)
        else:
            self.tracker_list = self.config['TRACKERS_TO_EVAL']
        if self.config['TRACKER_DISPLAY_NAMES'] is None:
            self.tracker_to_disp = dict(zip(self.tracker_list, self.tracker_list))
        elif (self.config['TRACKERS_TO_EVAL'] is not None) and (
                len(self.config['TRACKER_DISPLAY_NAMES']) == len(self.tracker_list)):
            self.tracker_to_disp = dict(zip(self.tracker_list, self.config['TRACKER_DISPLAY_NAMES']))
        else:
            raise TrackEvalException('List of tracker files and tracker display names do not match.')
        for tracker in self.tracker_list:
            if self.data_is_zipped:
                curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol + '.zip')
                if not os.path.isfile(curr_file):
                    print('Tracker file not found: ' + curr_file)
                    raise TrackEvalException('Tracker file not found: ' + tracker + '/' + os.path.basename(curr_file))
            else:
                for seq in self.seq_list:
                    curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol, seq + '.txt')
                    if not os.path.isfile(curr_file):
                        print('Tracker file not found: ' + curr_file)
                        raise TrackEvalException(
                            'Tracker file not found: ' + tracker + '/' + self.tracker_sub_fol + '/' + os.path.basename(
                                curr_file))

并在__init__.py里添加from .sat_challenge_2d_box import SatChallenge2DBox

3、修改运行脚本

复制一份TrackEval/scripts/run_mot_challenge.py为run_sat_challenge.py修改：

dataset_list = [trackeval.datasets.SatChallenge2DBox(dataset_config)]

freeze_support()
    # Command line interface:
    default_eval_config = trackeval.Evaluator.get_default_eval_config()
    default_eval_config['DISPLAY_LESS_PROGRESS'] = False
    default_dataset_config = trackeval.datasets.MotChallenge2DBox.get_default_dataset_config()
    default_metrics_config = {'METRICS': ['HOTA', 'CLEAR', 'Identity'], 'THRESHOLD': 0.5}
    config = {**default_eval_config, **default_dataset_config, **default_metrics_config}  # Merge default configs
    parser = argparse.ArgumentParser()
    for setting in config.keys():
        if type(config[setting]) == list or type(config[setting]) == type(None):
            parser.add_argument("--" + setting, nargs='+')
        else:
            parser.add_argument("--" + setting)
    args = parser.parse_args().__dict__
    for setting in args.keys():
        if args[setting] is not None:
            if type(config[setting]) == type(True):
                if args[setting] == 'True':
                    x = True
                elif args[setting] == 'False':
                    x = False
                else:
                    raise Exception('Command line parameter ' + setting + 'must be True or False')
            elif type(config[setting]) == type(1):
                x = int(args[setting])
            elif type(args[setting]) == type(None):
                x = None
            elif setting == 'SEQ_INFO':
                x = dict(zip(args[setting], [None]*len(args[setting])))
            else:
                x = args[setting]
            config[setting] = x
    eval_config = {k: v for k, v in config.items() if k in default_eval_config.keys()}
    dataset_config = {k: v for k, v in config.items() if k in default_dataset_config.keys()}
    metrics_config = {k: v for k, v in config.items() if k in default_metrics_config.keys()}
    # Run code
    evaluator = trackeval.Evaluator(eval_config)
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
    metrics_list = []
    for metric in [trackeval.metrics.HOTA, trackeval.metrics.CLEAR, trackeval.metrics.Identity, trackeval.metrics.VACE]:
        if metric.get_name() in metrics_config['METRICS']:
            metrics_list.append(metric(metrics_config))
    if len(metrics_list) == 0:
        raise Exception('No metrics selected for evaluation')
    evaluator.evaluate(dataset_list, metrics_list)

制作自己的MOT

https://blog.csdn.net/gubeiqing/article/details/123648141

References:

https://cloud.tencent.com/developer/article/1902308

https://zhuanlan.zhihu.com/p/391396206

AIEarth是一个由众多领域内专家博主共同打造的学术平台，旨在建设一个拥抱智慧未来的学术殿堂！【平台地址：https://devpress.csdn.net/aiearth】很高兴认识你！加入我们共同进步！

MOT_Challenge官方代码TrackEval介绍