一、简单回顾

全连接被称为Dense层或者Deep层。输入数据样本的不同特征。

CNN用了权重共享的概念，而全连接层的参数量是巨大的。我们可以使用RNN解决如下图（天气预报预测）这种带有序列模式的数据（如NLP、天气、股市金融数据等），并且使用权重共享的概念来减少参数量。

下图栗子简述：已知前三天的天气，并且每个样本有3个特征（天数、温度、气压），label是是否下雨。

如果需要用图像生成文本，可以用CNN+FC层后的结果输入RNN。如果没有先验前置信息h0，就设置和h1一样的全0向量即可（维度要匹配）。

二、RNN算法

2.1 RNN Cell

RNN Cell本质上为一个线性层（共享权重的线性层，如上图），在t时刻的N维向量，经过RNN Cell后变为一个M维的向量h t h_th

。

W i h \mathrm{W}_{\mathrm{ih}}W

：和输入向量x t x_tx

相乘的权重矩阵，维度大小为h i d d e n _ s i z e × i n p u t _ s i z e hidden\_size \times input\_sizehidden_size×input_size。

W h h \mathrm{W}_{\mathrm{hh}}W

：和隐层向量h t − 1 h_{t-1}h

t−1

相乘的权重矩阵，维度大小为h i d d e n _ s i z e × h i d d e n _ s i z e hidden\_size \times hidden\_sizehidden_size×hidden_size。

几点注意：

（1）通过RNN Cell的维度和上一个hidden_size的维度相同。

（2）也可以将两个线性层的运算合并：

严谨写的矩阵运算形式是：

（3）维度的要求：

seqLen = 3指序列长度为3，每个样本里有x1，x2，x3。

（4）一个batch中，各个元素之间是并行计算；输入数据是按批次来的，每一批3个。

（5）注意输入的hidden有参数numLayers（如上图，每种颜色是一个线性层），指RNN的层数。可以发现输入和输出的2个参数之间，不同的只有input_size变为hidden_size。上图中上方和右方两排是output，而下方和左方两排是input。

2.2 文本转为向量

（1）将单词转成one-hot编码，注意下面的input_size为4。

2.3 注意维度

注意训练部分的内层for循环的input和inputs的维度：

另外label和labels的维度：

# 训练模型
for epoch in range(15):
    loss = 0
    optimizer.zero_grad()
    # 初始化h0
    hidden = net.init_hidden()
    print('Predicted string:', end = '')
    # input是(seq × batch × inputsize) 依次拿x1,x2..x5
    for input, label in zip(inputs, labels):
        # zip函数是沿着第一个维度拼接
        hidden = net(input, hidden)
        # 没用item,因为整个序列的loss之和才是损失(要构建计算图)
        loss += criterion(input, hidden)
        _, idx = hidden.max(dim = 1)
       print(idx2char[idx.item()], end = '')
    loss.backward()
    optimizer.step()
    print(', Epoch [%d/15] loss = %.4f' % (epoch + 1, loss.item()))

2.4 输出是预测值

三、RNN小栗子

3.1 如何使用RNNCell

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 23 09:07:58 2021
@author: 86493
"""
import torch 
import torch.nn as nn
batch_size = 1 
# x1, x2, x3，即样本数为3
seq_len = 3  
# 每个xi都是一个4维向量
input_size = 4
hidden_size = 2
cell = torch.nn.RNNCell(input_size = input_size,
                        hidden_size = hidden_size)
# (seq, batch, features)
dataset = torch.randn(seq_len,
                      batch_size,
                      input_size)
hidden = torch.zeros(batch_size,
                     hidden_size)
# 分别读x1, x2, x3
for idx, input in enumerate(dataset):
    print('=' * 20, idx, '=' * 20)
    # Input size: torch.Size([1, 4])
    print('Input size:', input.shape)
    hidden = cell(input, hidden)
    # outputs size: torch.Size([1, 2])
    print('outputs size:', hidden.shape)
    print(hidden)

结果为：

==================== 0 ====================
Input size: torch.Size([1, 4])
outputs size: torch.Size([1, 2])
tensor([[ 0.1293, -0.9119]], grad_fn=<TanhBackward>)
==================== 1 ====================
Input size: torch.Size([1, 4])
outputs size: torch.Size([1, 2])
tensor([[ 0.1533, -0.7704]], grad_fn=<TanhBackward>)
==================== 2 ====================
Input size: torch.Size([1, 4])
outputs size: torch.Size([1, 2])
tensor([[ 0.3880, -0.4625]], grad_fn=<TanhBackward>)

3.2 如何使用RNN

如果直接使用nn.RNN，就不用像3.1一样自己写循环了。

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 23 09:07:58 2021
@author: 86493
"""
import torch 
import torch.nn as nn
batch_size = 1    
input_size = 4
hidden_size = 2
num_layers = 1
cell = torch.nn.RNN(input_size = input_size,
                    hidden_size = hidden_size,
                    num_layers = num_layers)
# (seqLen, batchSize, inputSize)
inputs = torch.randn(seq_len,
                     batch_size,
                     input_size)
hidden = torch.zeros(num_layers,
                     batch_size,
                     hidden_size)
out, hidden = cell(inputs, hidden)
print('Output size:', out.shape)
print('Output:', out)
print('Hidden size:', hidden.shape)
print('Hidden:', hidden)

结果为：

Output size: torch.Size([3, 1, 2])
Output: tensor([[[-0.2704, -0.7284]],
        [[-0.4312,  0.0836]],
        [[ 0.6894, -0.9946]]], grad_fn=<StackBackward>)
Hidden size: torch.Size([1, 1, 2])
Hidden: tensor([[[ 0.6894, -0.9946]]], grad_fn=<StackBackward>)

四、RNNCell训练

在（四）中我们是先实现RNN Cell，再手动写循环调用训练等逻辑；在（五）中我们可以直接调用RNN网络（代码会少一丢丢）。后面的代码也可参考这次代码的注释（注意维度）。

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 23 09:17:10 2021
@author: 86493
"""
import torch 
import torch.nn as nn
import matplotlib.pyplot as plt
input_size = 4
hidden_size = 4
batch_size = 1
losslst = []
# 准备数据
idx2char = ['e', 'h', 'l', 'o']
# input数据是字符串hello
x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]
one_hot_lookup = [[1, 0, 0, 0],
                  [0, 1, 0, 0],
                  [0, 0, 1, 0],
                  [0, 0, 0, 1]]
# 构造独热向量，其维度为(SeqLen * InputSize)
x_one_hot = [one_hot_lookup[x] for x in x_data]
# view(-1,..)保留原始SeqLen，并添加batch_size,input_size两个维度
inputs = torch.Tensor(x_one_hot).view(-1,
                                      batch_size,
                                      input_size)
# 将labels转换为(SeqLen * 1)的维度,即矩阵形式
labels = torch.LongTensor(y_data).view(-1, 1)
# 模型设计
class Model(nn.Module):
    def __init__(self, input_size, hidden_size, batch_size):
        super(Model, self).__init__()
        self.batch_size = batch_size
        self.input_size = input_size 
        self.hidden_size = hidden_size 
        self.rnncell = torch.nn.RNNCell(input_size = self.input_size,
                                        hidden_size = self.hidden_size)
    def forward(self, input, hidden):
        # RNNCell input = (batchsize * inputsize)
        # RNNCell hidden = (batchsize * hiddensize)
        hidden = self.rnncell(input, hidden)
        return hidden
    # 生成默认的初始隐层h0，batch_size也仅为了构造h0
    def init_hidden(self):
        return torch.zeros(self.batch_size, self.hidden_size)
net = Model(input_size,
            hidden_size,
            batch_size)
# loss函数和优化器
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(),
                             lr = 0.1)
# 训练模型
for epoch in range(15):
    loss = 0
    optimizer.zero_grad()
    # 初始化h0
    hidden = net.init_hidden()
    print('Predicted string:', end = '')
    # input是(seqLen × batchsize × inputsize) 依次拿x1,x2..x5
    for input, label in zip(inputs, labels):
        # zip函数是沿着第一个维度拼接
        hidden = net(input, hidden)
        # 序列的每一项损失都需要累加
        # 没用item,因为整个序列的loss之和才是损失(要构建计算图)
        loss += criterion(hidden, label)
        # hidden是四维的(e h l o),找出概率值最大的数的下标
        _, idx = hidden.max(dim = 1)   
        # 每一轮训练能输出的预测字符串
        print(idx2char[idx.item()], end = '')
    loss.backward()
    optimizer.step()
    losslst.append(loss.item())
    print(', Epoch [%d/15] loss = %.4f' % (epoch + 1, loss.item()))
plt.plot(range(15), losslst)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

同时从预测的字符串结果看，当趋于收敛时，字符串是ohlol：

Predicted string:hhhhh, Epoch [1/15] loss = 6.2508
Predicted string:ohlol, Epoch [2/15] loss = 4.9792
Predicted string:ohlol, Epoch [3/15] loss = 4.2028
Predicted string:ohlol, Epoch [4/15] loss = 3.7331
Predicted string:ohlol, Epoch [5/15] loss = 3.3555
Predicted string:ohlol, Epoch [6/15] loss = 3.0020
Predicted string:ohlol, Epoch [7/15] loss = 2.6944  
Predicted string:ohlol, Epoch [8/15] loss = 2.4562
Predicted string:ohlol, Epoch [9/15] loss = 2.2846
Predicted string:ohlol, Epoch [10/15] loss = 2.1613
Predicted string:ohlol, Epoch [11/15] loss = 2.0679
Predicted string:ohlol, Epoch [12/15] loss = 1.9959
Predicted string:ohlol, Epoch [13/15] loss = 1.9450
Predicted string:ohlol, Epoch [14/15] loss = 1.9128
Predicted string:ohlol, Epoch [15/15] loss = 1.8900

五、用RNN模块训练

用RNN模块训练就简化很多：

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 23 09:17:10 2021
@author: 86493
"""
import torch 
import torch.nn as nn
import matplotlib.pyplot as plt
input_size = 4
hidden_size = 4
num_layers = 1
batch_size = 1
seq_len = 5
# 准备数据
idx2char = ['e', 'h', 'l', 'o']
x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]
one_hot_lookup = [[1, 0, 0, 0],
                  [0, 1, 0, 0],
                  [0, 0, 1, 0],
                  [0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data]
inputs = torch.Tensor(x_one_hot).view(seq_len,
                                      batch_size,
                                      input_size)
# labels(seqLen * batchsize, 1)，为了后面进行矩阵运算，计算交叉熵
labels = torch.LongTensor(y_data)
losslst = []
# 模型设计
class Model(nn.Module):
    def __init__(self, input_size, hidden_size, batch_size):
        super(Model, self).__init__()
        self.num_layers = num_layers
        self.batch_size = batch_size
        self.input_size = input_size 
        self.hidden_size = hidden_size 
        self.rnn = torch.nn.RNN(input_size = self.input_size,
                                hidden_size = self.hidden_size,
                                num_layers = num_layers)
    def forward(self, input):
        hidden = torch.zeros(self.num_layers,
                             self.batch_size,
                             self.hidden_size)
        out, _ = self.rnn(input, hidden)
        # 输出要变成两维的，用交叉熵的时候变成一个矩阵
        # 即维度为(SeqLen * batchsize, hiddensize)
        return out.view(-1, self.hidden_size)
net = Model(input_size,
            hidden_size,
            batch_size)
# loss函数和优化器
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(),
                             lr = 0.05)
# RNN中输入(SeqLen * batchsize * inputsize)
# RNN中输出(SeqLen * batchsize * hiddensize)
# labels的维度为 (hiddensize * 1)
for epoch in range(15):
    optimizer.zero_grad()
    outputs = net(inputs)
    # labels的shape是seq × B × 1
    # outputs的shape是seq × B × H
    loss = criterion(outputs, labels)
    losslst.append(loss.item())
    loss.backward()
    optimizer.step()
    _, idx = outputs.max(dim = 1)
    idx = idx.data.numpy()
    print('Predicted:', ''.join([idx2char[x] for x in idx]), end = '')
    print(', Epoch [%d/15] loss = %.3f' % (epoch + 1, loss.item()))
plt.plot(range(15), losslst)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

结果为：

Predicted: lhhhh, Epoch [1/15] loss = 1.481
Predicted: lhlhh, Epoch [2/15] loss = 1.360
Predicted: lhlll, Epoch [3/15] loss = 1.244
Predicted: lhlll, Epoch [4/15] loss = 1.132
Predicted: lhlll, Epoch [5/15] loss = 1.026
Predicted: ohlll, Epoch [6/15] loss = 0.931
Predicted: ohlll, Epoch [7/15] loss = 0.852
Predicted: ohlol, Epoch [8/15] loss = 0.791
Predicted: ohlol, Epoch [9/15] loss = 0.744
Predicted: ohlol, Epoch [10/15] loss = 0.706
Predicted: ohlol, Epoch [11/15] loss = 0.675
Predicted: ohlol, Epoch [12/15] loss = 0.649
Predicted: ohlol, Epoch [13/15] loss = 0.626
Predicted: ohlol, Epoch [14/15] loss = 0.605
Predicted: ohlol, Epoch [15/15] loss = 0.588

六、优化：Embedding

6.1 通过embedding降维

独热编码向量：维度会太高、向量系数、硬编码。

通过embedding将向量编码为低维、稠密的向量（从data中学习）。

nn.Embedding的shape，注意是会多添加一个embedding_dim维度：

可以通过pytorch官网的embedding小栗子来熟悉其用法（具体看注释）：

# -*- coding: utf-8 -*-
"""
Created on Sun Oct 24 08:51:55 2021
@author: 86493
"""
import torch
import torch.nn as nn
from torch.autograd import Variable
# 10个size为3的tensor
# 嵌入字典的大小为10(即有10个词)，每个词向量的维度为3
embedding = nn.Embedding(10, 3)
# a batch of 2 samples of 4 indices each
# 该LongTensor的数字范围只能在0~9(因为设置了10)
input1 = torch.LongTensor([[1, 2, 4, 5], 
                           [4, 3, 2, 9]])
emb1 = embedding(input1) # tensor([[1, 0, 2, 2, 3]])
print(emb1)
print(emb1.shape)
# torch.Size([2, 4, 3]) # 即每个词都被映射为一个3维向量
print('-' * 60)
# 带有padding_idx的栗子
embedding = nn.Embedding(10, 3, padding_idx = 0)
input2 = Variable(torch.LongTensor([[0, 2, 0, 5]]))
emb2 = embedding(input2)
print(emb2)
print(emb2.shape)
# torch.Size([1, 4, 3])

tensor([[[ 0.3004, -0.7126,  0.8605],
         [ 0.1484, -0.9476,  1.0352],
         [ 2.2382, -0.3619, -1.6866],
         [-0.2713,  0.3627,  0.4067]],
        [[ 2.2382, -0.3619, -1.6866],
         [ 1.2409,  0.6028,  0.0371],
         [ 0.1484, -0.9476,  1.0352],
         [-0.5018,  0.3566, -0.6405]]], grad_fn=<EmbeddingBackward>)
torch.Size([2, 4, 3])
------------------------------------------------------------
tensor([[[ 0.0000,  0.0000,  0.0000],
         [ 2.4998,  0.4584,  0.0595],
         [ 0.0000,  0.0000,  0.0000],
         [ 1.0930, -1.0224,  0.8367]]], grad_fn=<EmbeddingBackward>)
torch.Size([1, 4, 3])

6.2 embedding改进的代码

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 23 19:12:40 2021
@author: 86493
"""
import torch 
import torch.nn as nn
import matplotlib.pyplot as plt
num_class = 4
input_size = 4
hidden_size = 8
embedding_size = 10
num_layers = 2
batch_size = 1
seq_len = 5
# 准备数据
idx2char = ['e', 'h', 'l', 'o']
# (batch, seq_len)
x_data = [[1, 0, 2, 2, 3]]
# (batch * seq_len)
y_data = [3, 1, 2, 3, 2]
one_hot_lookup = [[1, 0, 0, 0],
                  [0, 1, 0, 0],
                  [0, 0, 1, 0],
                  [0, 0, 0, 1]]
# inputs 作为交叉熵中的Inputs,维度为(batchsize, Seqlen)
inputs = torch.LongTensor(x_data)
# labels 作为交叉熵中的target,维度为(batchsize, Seqlen)
labels = torch.LongTensor(y_data)
losslst = []
# 模型设计
class Model(nn.Module):
    def __init__(self, input_size, hidden_size, batch_size):
        super(Model, self).__init__()
        # 字典大小为input_size,映射后的词向量大小为embedding_size
        self.emb = torch.nn.Embedding(input_size, embedding_size)
        self.rnn = torch.nn.RNN(input_size = embedding_size,
                                hidden_size = hidden_size,
                                num_layers = num_layers,
                                batch_first = True)
        # 增加线性层使得在处理输入输出维度不同时更加稳定
        self.fc = nn.Linear(hidden_size, num_class)
    def forward(self, x):
        hidden = torch.zeros(num_layers, x.size(0), hidden_size)
        # (batch, seqLen, embeddingSize)
        x = self.emb(x)
        x, _ = self.rnn(x, hidden)
        x = self.fc(x)
        return x.view(-1, num_class)
net = Model(input_size,
            hidden_size,
            batch_size)
# loss函数和优化器
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(),
                             lr = 0.05)
for epoch in range(15):
    optimizer.zero_grad()
    outputs = net(inputs)
    # labels的shape是seq×B×1
    # outputs的shape是seq×B×H
    loss = criterion(outputs, labels)
    losslst.append(loss.item())
    loss.backward()
    optimizer.step()
    _, idx = outputs.max(dim = 1)
    idx = idx.data.numpy()
    print('Predicted:', ''.join([idx2char[x] for x in idx]), end = '')
    print(', Epoch [%d/15] loss = %.3f' % (epoch + 1, loss.item()))
plt.plot(range(15), losslst)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

这次可以看到第6个epoch就收敛到ohlol了，上次是第8个epoch才收敛到这个单词。

Predicted: oeeol, Epoch [1/15] loss = 1.371
Predicted: ollll, Epoch [2/15] loss = 1.122
Predicted: ollll, Epoch [3/15] loss = 0.980
Predicted: ollll, Epoch [4/15] loss = 0.849
Predicted: ohlll, Epoch [5/15] loss = 0.703
Predicted: ohlol, Epoch [6/15] loss = 0.543
Predicted: ohlol, Epoch [7/15] loss = 0.386
Predicted: ohlol, Epoch [8/15] loss = 0.269
Predicted: ohlol, Epoch [9/15] loss = 0.180
Predicted: ohlol, Epoch [10/15] loss = 0.113
Predicted: ohlol, Epoch [11/15] loss = 0.075
Predicted: ohlol, Epoch [12/15] loss = 0.051
Predicted: ohlol, Epoch [13/15] loss = 0.036
Predicted: ohlol, Epoch [14/15] loss = 0.026
Predicted: ohlol, Epoch [15/15] loss = 0.019

七、LSTM网络

初学LSTM可以先不理很多教程说的××门（可解释性问题）。

可以参考nn.LSTM官方文档：https://pytorch.org/docs/stable/nn.html#lstm

八、介于RNN和LSTM：GRU

可以参考nn.LSTM官方文档：https://pytorch.org/docs/stable/nn.html#gru

【PyTorch基础教程15】循环神经网络RNN

2.4 输出是预测值

三、RNN小栗子

3.1 如何使用RNNCell

3.2 如何使用RNN

四、RNNCell训练

五、用RNN模块训练

6.2 embedding改进的代码

热门文章

最新文章

相关课程

相关电子书

推荐镜像

探索云世界

热门

云计算

大数据

云原生

人工智能

数据库

开发与运维

活动广场

任务中心

训练营

直播

乘风者计划

下载

镜像站

技术资料

【PyTorch基础教程15】循环神经网络RNN

2.4 输出是预测值

三、RNN小栗子

3.1 如何使用RNNCell

3.2 如何使用RNN

四、RNNCell训练

五、用RNN模块训练

6.2 embedding改进的代码

热门文章

最新文章

相关课程

相关电子书

推荐镜像