报错FloatingPointError: Loss became infinite or NaN at iteration=88!

简介: 报错FloatingPointError: Loss became infinite or NaN at iteration=88!

项目场景:


Traceback (most recent call last):
  File "/home/yuan/桌面/shenchunhua/CondInst-master/train_net.py", line 255, in <module>
    args=(args,),
  File "/home/yuan/anaconda3/envs/AdelaiNet/lib/python3.7/site-packages/detectron2/engine/launch.py", line 62, in launch
    main_func(*args)
  File "/home/yuan/桌面/shenchunhua/CondInst-master/train_net.py", line 235, in main
    return trainer.train()
  File "/home/yuan/桌面/shenchunhua/CondInst-master/train_net.py", line 118, in train
    self.train_loop(self.start_iter, self.max_iter)
  File "/home/yuan/桌面/shenchunhua/CondInst-master/train_net.py", line 107, in train_loop
    self.run_step()
  File "/home/yuan/anaconda3/envs/AdelaiNet/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 232, in run_step
    self._detect_anomaly(losses, loss_dict)
  File "/home/yuan/anaconda3/envs/AdelaiNet/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 245, in _detect_anomaly
    self.iter, loss_dict
FloatingPointError: Loss became infinite or NaN at iteration=88!
loss_dict = {'loss_fcos_cls': tensor(nan, device='cuda:0', grad_fn=<DivBackward0>), 'loss_fcos_loc': tensor(0.5552, device='cuda:0', grad_fn=<DivBackward0>), 'loss_fcos_ctr': tensor(0.7676, device='cuda:0', grad_fn=<DivBackward0>), 'loss_mask': tensor(0.8649, device='cuda:0', grad_fn=<DivBackward0>), 'data_time': 0.0022056670004531043}


20200805075812593.png


原因分析:


学习率的问题,导致损失爆炸了,可以把学习调整一下!

目录
相关文章
|
3月前
|
机器学习/深度学习
Epoch、Batch 和 Iteration 的区别详解
【8月更文挑战第23天】
418 0
成功解决ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
成功解决ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
成功解决but is 0 and 2 (computed from start 0 and end 9223372 over shape with rank 2 and stride-1)
成功解决but is 0 and 2 (computed from start 0 and end 9223372 over shape with rank 2 and stride-1)
|
机器学习/深度学习 算法框架/工具
【问题记录与解决】KeyError: ‘acc‘ plt.plot(N[150:], H.history[“acc“][150:], label=“train_acc“) # KeyError: ‘
【问题记录与解决】KeyError: ‘acc‘ plt.plot(N[150:], H.history[“acc“][150:], label=“train_acc“) # KeyError: ‘
【问题记录与解决】KeyError: ‘acc‘ plt.plot(N[150:], H.history[“acc“][150:], label=“train_acc“) # KeyError: ‘
LeetCode 367. Valid Perfect Square
给定一个正整数 num,编写一个函数,如果 num 是一个完全平方数,则返回 True,否则返回 False。
95 0
LeetCode 367. Valid Perfect Square
成功解决linear_model\stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been ad
成功解决linear_model\stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been ad
|
算法
P, NP, NP-complete, NP-hard问题对比
图片来源于维基百科 左图在假设P≠NP的情况下有效,右图在假设P=NP的情况下有效 在假定P≠NP的情况下, 有 NP问题:可以在多项式时间内被验证的问题。
1588 0
|
算法
Transition matrix
**Transition matrix** 中文名:转移矩阵;转换矩阵;跃迁矩阵;状态转移矩阵
2636 0
|
Windows
1029. Median (25)
#include #include #include #include using namespace std; int main(){ long long n, t; cin >> n; ...
896 0