机器学习PAI在使用dino训练自己数据时，显存在不停的增加，，请问如何排查呢？

使用的训练命令： python tools/train.py configs/detection/dino/dino_5sc_swinl_center_iou_memoryreduce_26e_obj365.py --pretrained checkpoints/epoch_22.pth --work_dir work_dir/

日志如下： 2023-06-02 15:50:02,481 - easycv - INFO - Epoch [1][420/13236] lr: 1.000e-04, eta: 3 days, 2:49:58, time: 0.690, data_time: 0.012, memory: 8668, loss_ce: 1.9952, class_error: 85.8944, loss_bbox: 0.9517, loss_giou: 1.7226, loss_center: 0.9545, loss_iouaware: 1.1281, loss_ce_0: 1.8327, loss_bbox_0: 0.9776, loss_giou_0: 1.7397, loss_center_0: 0.9458, loss_iouaware_0: 1.1103, loss_ce_1: 1.9569, loss_bbox_1: 0.9610, loss_giou_1: 1.7214, loss_center_1: 0.9410, loss_iouaware_1: 1.1258, loss_ce_2: 2.0271, loss_bbox_2: 0.9531, loss_giou_2: 1.7204, loss_center_2: 0.9518, loss_iouaware_2: 1.1275, loss_ce_3: 2.0033, loss_bbox_3: 0.9523, loss_giou_3: 1.7213, loss_center_3: 0.9589, loss_iouaware_3: 1.1285, loss_ce_4: 2.0029, loss_bbox_4: 0.9524, loss_giou_4: 1.7215, loss_center_4: 0.9594, loss_iouaware_4: 1.1272, loss_ce_interm: 1.9905, loss_bbox_interm: 0.9592, loss_giou_interm: 1.7260, loss_center_interm: 0.9149, loss_iouaware_interm: 1.1416, loss_ce_dn: 1.1557, loss_bbox_dn: 0.4540, loss_giou_dn: 1.3770, loss_center_dn: 1.3841, loss_iouaware_dn: 1.3356, loss_ce_dn_0: 0.9538, loss_bbox_dn_0: 0.4562, loss_giou_dn_0: 1.3734, loss_center_dn_0: 1.3862, loss_iouaware_dn_0: 1.3380, loss_ce_dn_1: 0.9745, loss_bbox_dn_1: 0.4554, loss_giou_dn_1: 1.3742, loss_center_dn_1: 1.3856, loss_iouaware_dn_1: 1.3368, loss_ce_dn_2: 1.0250, loss_bbox_dn_2: 0.4549, loss_giou_dn_2: 1.3748, loss_center_dn_2: 1.3849, loss_iouaware_dn_2: 1.3365, loss_ce_dn_3: 1.0980, loss_bbox_dn_3: 0.4545, loss_giou_dn_3: 1.3756, loss_center_dn_3: 1.3842, loss_iouaware_dn_3: 1.3360, loss_ce_dn_4: 1.1567, loss_bbox_dn_4: 0.4542, loss_giou_dn_4: 1.3763, loss_center_dn_4: 1.3843, loss_iouaware_dn_4: 1.3356, loss: 80.7759 Traceback (most recent call last): File "/home/EasyCV/tools/train.py", line 322, in main() File "/home/EasyCV/tools/train.py", line 310, in main train_model( File "/home/EasyCV/package/easycv/apis/train.py", line 312, in train_model runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/usr/local/python3.9/lib/python3.9/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run epoch_runner(data_loaders[i], **kwargs) File "/home/EasyCV/package/easycv/runner/ev_runner.py", line 107, in train self.run_iter(data_batch, train_mode=True) File "/home/EasyCV/package/easycv/runner/ev_runner.py", line 72, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "/usr/local/python3.9/lib/python3.9/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step return self.module.train_step(*inputs[0], **kwargs[0]) File "/home/EasyCV/package/easycv/models/base.py", line 105, in train_step losses = self(**data, mode='train') File "/usr/local/python3.9/lib/python3.9/site-packages/torch/nn/modules/modul 机器学习PAI在使用dino训练自己数据时，显存在不停的增加，出现CUDA out of memory 的问题，修改了batch、尺寸都没有解决，请问如何排查呢？