2、CONVOLUTIONAL NEURAL NETWORK
Using DenseNet-121 (Huang et al., 2017) and ResNet-34 (He et al., 2016), we then consider the task of image classification on the standard CIFAR-10 dataset. In this experiment, we employ the fixed budget of 200 epochs and reduce the learning rates by 10 after 150 epochs.
DenseNet ：We first run a DenseNet-121 model on CIFAR-10 and our results are shown in Figure 3. We can see that adaptive methods such as ADAGRAD, ADAM and AMSGRAD appear to perform better than the non-adaptive ones early in training. But by epoch 150 when the learning rates are decayed, SGDM begins to outperform those adaptive methods. As for our methods, ADABOUND and AMSBOUND, they converge as fast as adaptive ones and achieve a bit higher accuracy than SGDM on the test set at the end of training. In addition, compared with their prototypes, their performances are enhanced evidently with approximately 2% improvement in the test accuracy.
ResNet ：Results for this experiment are reported in Figure 3. As is expected, the overall performance of each algorithm on ResNet-34 is similar to that on DenseNet-121. ADABOUND and AMSBOUND even surpass SGDM by 1%. Despite the relative bad generalization ability of adaptive methods, our proposed methods overcome this drawback by allocating bounds for their learning rates and obtain almost the best accuracy on the test set for both DenseNet and ResNet on CIFAR-10.
然后利用DenseNet-121 (Huang et al.2017)和ResNet-34 (He et al.2016)对CIFAR-10标准数据集进行图像分类。在这个实验中，我们使用200个epoch的固定预算，在150个epoch后将学习率降低10个。
DenseNet：我们首先在CIFAR-10上运行DenseNet-121模型，结果如图3所示。我们可以看到，ADAGRAD、ADAM和AMSGRAD等自适应方法在早期训练中表现得比非自适应方法更好。但是到了历元150，当学习速率衰减时，SGDM开始优于那些自适应方法。对于我们的方法ADABOUND和AMSBOUND，它们收敛速度和自适应方法一样快，并且在训练结束时的测试集上达到比SGDM稍高的精度。此外，与原型机相比，其性能得到了显著提高，测试精度提高了约2%。
ResNet：实验结果如图3所示。正如预期的那样，ResNet-34上的每个算法的总体性能与DenseNet-121上的相似。ADABOUND和AMSBOUND甚至超过SGDM 1%。尽管自适应方法的泛化能力相对较差，但我们提出的方法克服了这一缺点，为其学习速率分配了界限，在CIFAR-10上对DenseNet和ResNet的测试集都获得了几乎最佳的准确率。
3、RECURRENT NEURAL NETWORK
Finally, we conduct an experiment on the language modeling task with Long Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber, 1997). From two experiments above, we observe that our methods show much more improvement in deep convolutional neural networks than in perceptrons. Therefore, we suppose that the enhancement is related to the complexity of the architecture and run three models with (L1) 1-layer, (L2) 2-layer and (L3) 3-layer LSTM respectively. We train them on Penn Treebank, running for a fixed budget of 200 epochs. We use perplexity as the metric to evaluate the performance and report results in Figure 4.
We find that in all models, ADAM has the fastest initial progress but stagnates in worse performance than SGD and our methods. Different from phenomena in previous experiments on the image classification tasks, ADABOUND and AMSBOUND does not display rapid speed at the early training stage but the curves are smoother than that of SGD.
我们发现，在所有模型中，ADAM的初始进展最快，但在性能上停滞不前，不如SGD和我们的方法。与以往在图像分类任务实验中出现的现象不同，ADABOUND和AMSBOUND在训练初期的速度并不快，但曲线比SGD平滑。
Comparing L1, L2 and L3, we can easily notice a distinct difference of the improvement degree. In L1, the simplest model, our methods perform slightly 1.1% better than ADAM while in L3, the most complex model, they show evident improvement over 2.8% in terms of perplexity. It serves as evidence for the relationship between the model’s complexity and the improvement degree.
对比L1、L2和L3，我们可以很容易地发现改善程度的显著差异。在最简单的模型L1中，我们的方法比ADAM的方法略好1.1%，而在最复杂的模型L3中，我们的方法在复杂的方面明显优于2.8%。为模型的复杂性与改进程度之间的关系提供了依据。
实验结果分析
To investigate the efficacy of our proposed algorithms, we select popular tasks from computer vision and natural language processing. Based on results shown above, it is easy to find that ADAM and AMSGRAD usually perform similarly and the latter does not show much improvement for most cases. Their variants, ADABOUND and AMSBOUND, on the other hand, demonstrate a fast speed of convergence compared with SGD while they also exceed two original methods greatly with respect to test accuracy at the end of training. This phenomenon exactly confirms our view mentioned in Section 3 that both large and small learning rates can influence the convergence.
Besides, we implement our experiments on models with different complexities, consisting of a per- ceptron, two deep convolutional neural networks and a recurrent neural network. The perceptron used on the MNIST is the simplest and our methods perform slightly better than others. As for DenseNet and ResNet, obvious increases in test accuracy can be observed. We attribute this differ- ence to the complexity of the model. Specifically, for deep CNN models, convolutional and fully connected layers play different parts in the task. Also, different convolutional layers are likely to be responsible for different roles (Lee et al., 2009), which may lead to a distinct variation of gradients of parameters. In other words, extreme learning rates (huge or tiny) may appear more frequently in complex models such as ResNet. As our algorithms are proposed to avoid them, the greater enhance- ment of performance in complex architectures can be explained intuitively. The higher improvement degree on LSTM with more layers on language modeling task also consists with the above analysis.
为了研究我们提出的算法的有效性，我们从计算机视觉和自然语言处理中选择流行的任务。根据上面显示的结果，不难发现ADAM和AMSGRAD的表现通常是相似的，而AMSGRAD在大多数情况下并没有太大的改善。另一方面，它们的变体ADABOUND和AMSBOUND与SGD相比具有较快的收敛速度，同时在训练结束时的测试精度也大大超过了两种原始方法。这一现象正好印证了我们在第3节中提到的观点，学习速率的大小都会影响收敛。
此外，我们还对不同复杂度的模型进行了实验，包括一个per- ceptron模型、两个深度卷积神经网络模型和一个递归神经网络模型。MNIST上使用的感知器是最简单的，我们的方法比其他方法稍好一些。DenseNet和ResNet的测试精度明显提高。我们把这种不同归因于模型的复杂性。具体来说，对于深度CNN模型，卷积层和全连通层在任务中扮演不同的角色。此外，不同的卷积层可能负责不同的角色(Lee et al.2009)，这可能导致参数梯度的明显变化。换句话说，极端的学习速率(巨大或微小)可能在ResNet等复杂模型中出现得更频繁。由于我们的算法是为了避免这些问题而提出的，因此可以直观地解释在复杂体系结构中性能的提高。LSTM在语言建模任务上的层次越多，改进程度越高，也与上述分析一致。
PS：因为时间比较紧，博主翻译的不是特别尽善尽美，如有错误，请指出，谢谢！
