5.1 循环神经网络（时间序列）
循环神经网络（RNN）用于建模带有时间关系的数据。它的架构是这样的。
在最基本的 RNN 中，单元（方框）中的操作和全连接层没什么区别，都是线性变换和激活。它完全可以看做多个全连接层的横向扩展。
但是操作数量多了之后，就会有梯度消失和爆炸的问题，于是人们改良了 RNN 单元，添加了精巧的结构来避免这样问题。这是 RNN 的几种改良结构：
操作步骤
导入所需的包。
import tensorflow as tf import numpy as np import pandas as pd import matplotlib as mpl import matplotlib.pyplot as plt
导入数据，并进行预处理。我们使用国际航班乘客数据集，由于它不存在于任何现有库中，我们需要先下载它。
ts = pd.read_csv('international-airline-passengers.csv', usecols=[1], header=0).dropna().values.ravel()
之后，我们需要将其转换为 RNN 单元可以接受的格式。可接受的格式是三维的，维度分别表示样本、时间和特征。我们需要一个窗口大小，表示几个历史值与当前值有关，然后我们按照它来切分时间序列，就能得到样本。
我仅仅使用原始特征，也就是乘客数量。我并不是表明 RNN 预测得有多好，只是告诉大家 RNN 怎么用。以后有了更好的特征再补充。
wnd_sz = 5 ds = [] for i in range(0, len(ts) - wnd_sz + 1): ds.append(ts[i:i + wnd_sz]) ds = np.asarray(ds) x_ = ds[:, 0:wnd_sz - 1] y_ = ds[:, [wnd_sz - 1]] x_ = np.expand_dims(x_, 2)
之后是训练集和测试集的划分。同样要注意绝对不能打乱。
train_size = int(len(x_) * 0.7) x_train = x_[:train_size] y_train = y_[:train_size] x_test = x_[train_size:] y_test = y_[train_size:]
由于 RNN 单元里面使用了tanh激活函数，它对于数据的尺度非常敏感，我们需要将数据标准化。但是，如果我们遵循传统方法，就发现效果不是很好，RNN 甚至不能预测趋势。观察数据集就会发现，后面的数据明显大于前面的数据。如果我们使用训练集的均值和标准差来标准化，仍旧是如此。换句话说，对于训练集来说，测试集是“新的分布”。所以我们分别计算训练集和测试集的均值和标准差，并使用它们自己的均值和标准差，将训练集和测试集标准化。
x_mu_train = x_train.mean(0) x_sigma_train = x_train.std(0) y_mu_train = y_train.mean(0) y_sigma_train = y_train.std(0) x_train = (x_train - x_mu_train) / x_sigma_train y_train = (y_train - y_mu_train) / y_sigma_train x_mu_test = x_test.mean(0) x_sigma_test = x_test.std(0) y_mu_test = y_test.mean(0) y_sigma_test = y_test.std(0) x_test = (x_test - x_mu_test) / x_sigma_test y_test = (y_test - y_mu_test) / y_sigma_test
定义超参数。
| 变量 | 含义 |
n_step |
时间步长 |
n_input |
样本特征数 |
n_epoch |
迭代数 |
n_hidden |
循环层的单元数 |
n_output |
输出层单元数 |
lr |
学习率 |
n_step = wnd_sz - 1 n_input = 1 n_hidden = 4 n_output = 1 n_epoch = 10000 lr = 0.1
搭建模型。循环层之后添加了一个输出层，目的是把循环层输出的四个特征压缩为一个特征，与标签匹配。
| 变量 | 含义 |
x |
输入 |
y |
真实标签 |
cell |
循环层 |
w_l2 |
输出层的权重 |
b_l2 |
输出层的偏置 |
h_l1 |
循环层的输出 |
h_l2 |
模型的输出 |
x = tf.placeholder(tf.float64, [None, n_step, n_input]) y = tf.placeholder(tf.float64, [None, n_output]) cell = tf.nn.rnn_cell.GRUCell(n_hidden) w_l2 = tf.Variable(np.random.rand(n_hidden, n_output)) b_l2 = tf.Variable(np.random.rand(1, n_output)) h_l1, _ = tf.nn.dynamic_rnn(cell, x, dtype=tf.float64) h_l2 = h_l1[:, -1] @ w_l2 + b_l2
定义 MSE 损失、优化操作、和 R 方度量指标。
| 变量 | 含义 |
loss |
损失 |
op |
优化操作 |
r_sqr |
R 方 |
loss = tf.reduce_mean((h_l2 - y) ** 2) op = tf.train.AdamOptimizer(lr).minimize(loss) y_mean = tf.reduce_mean(y) r_sqr = 1 - tf.reduce_sum((y - h_l2) ** 2) / tf.reduce_sum((y - y_mean) ** 2)
使用训练集训练模型。
losses = [] r_sqrs = [] with tf.Session() as sess: sess.run(tf.global_variables_initializer()) for e in range(n_epoch): _, loss_ = sess.run([op, loss], feed_dict={x: x_train, y: y_train}) losses.append(loss_)
使用测试集计算 R 方。
r_sqr_ = sess.run(r_sqr, feed_dict={x: x_test, y: y_test}) r_sqrs.append(r_sqr_)
每一百步打印损失和度量值。
if e % 100 == 0: print(f'epoch: {e}, loss: {loss_}, r_sqr: {r_sqr_}')
得到模型对训练特征和测试特征的预测值。
y_train_pred = sess.run(h_l2, feed_dict={x: x_train}) y_test_pred = sess.run(h_l2, feed_dict={x: x_test}) y_train_pred = y_train_pred * y_sigma_train + y_mu_train y_test_pred = y_test_pred * y_sigma_test + y_mu_test
输出：
epoch: 0, loss: 1.172066950288726, r_sqr: 0.08181895235767012 epoch: 100, loss: 0.06762743481295605, r_sqr: 0.74114029701611 epoch: 200, loss: 0.04335752219908887, r_sqr: 0.6723681622904742 epoch: 300, loss: 0.030765678607308333, r_sqr: 0.6122519542608709 epoch: 400, loss: 0.0255667258649117, r_sqr: 0.5849438991657725 epoch: 500, loss: 0.027757059089794138, r_sqr: 0.5547220968537527 epoch: 600, loss: 0.02280867896340781, r_sqr: 0.5770103850696798 epoch: 700, loss: 0.021203888157209534, r_sqr: 0.5588744061563752 epoch: 800, loss: 0.020631124908383643, r_sqr: 0.5340900532988054 epoch: 900, loss: 0.01848138445620317, r_sqr: 0.4793500483558242 epoch: 1000, loss: 0.01750574268333934, r_sqr: 0.4363987472239601 epoch: 1100, loss: 0.019063501795946507, r_sqr: 0.39722141072773787 epoch: 1200, loss: 0.0162686638744368, r_sqr: 0.3656020052876743 epoch: 1300, loss: 0.01700178459519973, r_sqr: 0.34215312925023267 epoch: 1400, loss: 0.01680496271967236, r_sqr: 0.32958240891225643 epoch: 1500, loss: 0.022387361152892183, r_sqr: 0.311875062539985 epoch: 1600, loss: 0.01692835079726568, r_sqr: 0.31973211945726576 epoch: 1700, loss: 0.014981184565736732, r_sqr: 0.34878737027421214 epoch: 1800, loss: 0.015259222044438638, r_sqr: 0.34762914874016715 epoch: 1900, loss: 0.013233242381529384, r_sqr: 0.33428957917112656 epoch: 2000, loss: 0.013406887787008756, r_sqr: 0.33925822033716413 epoch: 2100, loss: 0.014663732662643054, r_sqr: 0.347748811545596 epoch: 2200, loss: 0.013725021637547831, r_sqr: 0.3537942857251979 epoch: 2300, loss: 0.012930256601236268, r_sqr: 0.34525246614475136 epoch: 2400, loss: 0.013634394369979042, r_sqr: 0.31935143022735935 epoch: 2500, loss: 0.012295407249325577, r_sqr: 0.32843801172553166 epoch: 2600, loss: 0.012086876034542369, r_sqr: 0.2944659162449187 epoch: 2700, loss: 0.011431132027934963, r_sqr: 0.3320304352651986 epoch: 2800, loss: 0.026766484877065972, r_sqr: 0.3516603080525481 epoch: 2900, loss: 0.013249484588141427, r_sqr: 0.32412481149383066 epoch: 3000, loss: 0.01105305279694339, r_sqr: 0.3292310540476926 epoch: 3100, loss: 0.011470806104939772, r_sqr: 0.3297084772513311 epoch: 3200, loss: 0.011529391738337445, r_sqr: 0.32604568271503975 epoch: 3300, loss: 0.016529603402840765, r_sqr: 0.3043582054899223 epoch: 3400, loss: 0.013990350362048133, r_sqr: 0.3199655109188301 epoch: 3500, loss: 0.01153881610786597, r_sqr: 0.3133999574164319 epoch: 3600, loss: 0.012797147535482193, r_sqr: 0.31258610293795597 epoch: 3700, loss: 0.011476855518680756, r_sqr: 0.34671798591970837 epoch: 3800, loss: 0.010467519750448642, r_sqr: 0.3104500031552391 epoch: 3900, loss: 0.010726962231936567, r_sqr: 0.3138785388392976 epoch: 4000, loss: 0.011354515890079096, r_sqr: 0.3439112763768726 epoch: 4100, loss: 0.010956935944379312, r_sqr: 0.33047301867737333 epoch: 4200, loss: 0.010516217542845503, r_sqr: 0.3169866902021138 epoch: 4300, loss: 0.017644784078390447, r_sqr: 0.30790429205962355 epoch: 4400, loss: 0.010311798410482015, r_sqr: 0.33750001847417244 epoch: 4500, loss: 0.012096887824821146, r_sqr: 0.3483536401639381 epoch: 4600, loss: 0.010336452329588753, r_sqr: 0.333357260461554 epoch: 4700, loss: 0.010768585264597813, r_sqr: 0.3286332805909209 epoch: 4800, loss: 0.010187296826748472, r_sqr: 0.3508737658734946 epoch: 4900, loss: 0.012830964380652418, r_sqr: 0.38883425472040245 epoch: 5000, loss: 0.010034723893693883, r_sqr: 0.34865494006975606 epoch: 5100, loss: 0.018500185229121037, r_sqr: 0.37866142043477613 epoch: 5200, loss: 0.010063261243586386, r_sqr: 0.34191028337439755 epoch: 5300, loss: 0.011189870387229503, r_sqr: 0.34997022979225245 epoch: 5400, loss: 0.011487597291485683, r_sqr: 0.31996396612360156 epoch: 5500, loss: 0.018130474919110774, r_sqr: 0.40163834751360294 epoch: 5600, loss: 0.010631864046823009, r_sqr: 0.31888613030550417 epoch: 5700, loss: 0.010235333409754856, r_sqr: 0.33371577618904724 epoch: 5800, loss: 0.017054583875343695, r_sqr: 0.4573069550292207 epoch: 5900, loss: 0.010375495082076958, r_sqr: 0.34263745746177443 epoch: 6000, loss: 0.02899888987961715, r_sqr: 0.4731489130962029 epoch: 6100, loss: 0.009635932593390793, r_sqr: 0.3377314247566131 epoch: 6200, loss: 0.009756606958491803, r_sqr: 0.3433460486517317 epoch: 6300, loss: 0.015849799410205617, r_sqr: 0.38774652539473653 epoch: 6400, loss: 0.00952936061724416, r_sqr: 0.3316576525214012 epoch: 6500, loss: 0.01726899798192304, r_sqr: 0.3273077681611758 epoch: 6600, loss: 0.009523356787685604, r_sqr: 0.3307726070509249 epoch: 6700, loss: 0.012766831260869482, r_sqr: 0.3307548009627649 epoch: 6800, loss: 0.009289520442372325, r_sqr: 0.32373411787154416 epoch: 6900, loss: 0.011158193836457308, r_sqr: 0.32997251352858914 epoch: 7000, loss: 0.02869441623350593, r_sqr: 0.3338995201916093 epoch: 7100, loss: 0.018485258063551675, r_sqr: 0.2938792287466744 epoch: 7200, loss: 0.008861181401821593, r_sqr: 0.32004037549190045 epoch: 7300, loss: 0.00867730190276807, r_sqr: 0.31307907559266057 epoch: 7400, loss: 0.010568595192759116, r_sqr: 0.3007627324642683 epoch: 7500, loss: 0.008931675261205094, r_sqr: 0.3355236299841886 epoch: 7600, loss: 0.01437912261193081, r_sqr: 0.3118478993546139 epoch: 7700, loss: 0.013553350799683056, r_sqr: 0.3492071104037119 epoch: 7800, loss: 0.030876872968019105, r_sqr: 0.31738784411969434 epoch: 7900, loss: 0.010089701030161817, r_sqr: 0.3301184791799566 epoch: 8000, loss: 0.01147013359703656, r_sqr: 0.30306917655113186 epoch: 8100, loss: 0.009855414853123119, r_sqr: 0.34582101555677947 epoch: 8200, loss: 0.014423066317111677, r_sqr: 0.3470667522167451 epoch: 8300, loss: 0.011664752107448714, r_sqr: 0.32157739869359203 epoch: 8400, loss: 0.009646714124369984, r_sqr: 0.3355179490543708 epoch: 8500, loss: 0.008793324000994002, r_sqr: 0.2839973244304941 epoch: 8600, loss: 0.009740176108560508, r_sqr: 0.324717537310496 epoch: 8700, loss: 0.008509925688666948, r_sqr: 0.29162404590044055 epoch: 8800, loss: 0.00935785011763191, r_sqr: 0.2925641966682784 epoch: 8900, loss: 0.010778117882046828, r_sqr: 0.2969009246659946 epoch: 9000, loss: 0.008386197722733947, r_sqr: 0.31203296266480274 epoch: 9100, loss: 0.01152050463672165, r_sqr: 0.32619935954711543 epoch: 9200, loss: 0.008279818958514217, r_sqr: 0.3172741922458261 epoch: 9300, loss: 0.00767664017767626, r_sqr: 0.33894394589120425 epoch: 9400, loss: 0.0093665602478405, r_sqr: 0.3409878486332283 epoch: 9500, loss: 0.014677227625938132, r_sqr: 0.3553809732953209 epoch: 9600, loss: 0.008036392158721397, r_sqr: 0.33729133931823474 epoch: 9700, loss: 0.011275509041766035, r_sqr: 0.30956775357004673 epoch: 9800, loss: 0.00885348178403466, r_sqr: 0.3402377958439502 epoch: 9900, loss: 0.009747783923937898, r_sqr: 0.3843754214763523
绘制时间序列及其预测值。
plt.figure() plt.plot(ts, label='Original') y_train_pred = np.concatenate([ [np.nan] * n_input, y_train_pred.ravel() ]) y_test_pred = np.concatenate([ [np.nan] * (n_input + train_size), y_test_pred.ravel() ]) plt.plot(y_train_pred, label='y_train_pred') plt.plot(y_test_pred, label='y_test_pred') plt.legend() plt.show()
绘制训练集上的损失。
plt.figure() plt.plot(losses) plt.title('Loss on Training Set') plt.xlabel('#epoch') plt.ylabel('MSE') plt.show()
绘制测试集上的 R 方。
plt.figure() plt.plot(r_sqrs) plt.title('$R^2$ on Testing Set') plt.xlabel('#epoch') plt.ylabel('$R^2$') plt.show()
