开发者社区> 隐士2018> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

利用卷积神经网络(cnn)实现文本分类

简介:
+关注继续查看

卷积神经网络在情感分析中取得了很好的成果,相比于之前浅层的机器学习方法如NB、SVM效果更好,特别实在数据集较大的情况下,并且CNN不用我们手动去提取特征,原浅层ML是需要进行文本特征提取、文本特征表示、归一化、最后进行文本分类,文本特征提取主要可以分为四步:(1):对全部训练文档进行分词,由这些词作为向量的维数来表示文本;(2):统计每一类文档中所有出现的词语及其频率,然后过滤,剔除停用词和单字词;(3):统计每一类内出现词语的总词频,并取若干个频率更高的词汇作为这一类的特征词集;(4):去除每一类别中都出现的词,合并所有类别的特征词集,形成总特征词集,最后得到的特征词集是我们用到的特征集合,再用该集合去筛选测试集中的特征。文本的特征表示是利用TF-IDF公式来计算词的权值,这也充分利用的是特征提取时提取的特征来计算特征权值大小的,归一化处理需要处理的数据,经过处理后限制在一定范围内,经过处理后,我们原来的文本信息已经抽象成一个向量化的样本集,然后将样本集和训练好的模板进行相似度计算,若属于该类别,则与其他类别的模板文件进行计算,直到分进相应的类别,这是浅层ML进行文本分类的方式;

CNN进行文本分类相对简单一些,我结合最近做的一些实验总结了一下:

在利用CNN进行文本分类的时候,首先要将原始文本进行预处理,主要还是分词、去除停用词等,然后对预处理后的文本进行向量化利用word2vec,我利用的时word2vec中的skip-gram模型,将搜狗数据集表示为了200维的词向量形式;转化为词向量后就可以将每一句话转化为一个矩阵的形式,这样就跟利用CNN处理图像分类很相似;

说一下实验,我的实验环境:

tensorflow1.2、gpu1050Ti、Ubuntu16.04、pycharm、python2.7

# encoding=utf-8
from __future__ import unicode_literals

import tensorflow as tf
import numpy as np


class TextCNN(object):
    """
    使用CNN用于情感分析
    整个CNN架构包括词嵌入层,卷积层,max-pooling层和softmax
    """
    def __init__(
      self, sequence_length, num_classes,vocab_size,embedding_size, embedding_table,
            filter_sizes, num_filters, l2_reg_lambda=0.0):

        # 输入,输出,dropoutplaceholder
        self.input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x")
        self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
        self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")

        # Keeping track of l2 regularization loss (optional)
        l2_loss = tf.constant(0.0)

        # 词嵌入层
        with tf.device('/cpu:0'), tf.name_scope("embedding"):
            W = tf.Variable(embedding_table,name="W")
            self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
            self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

        # 生成卷积层和max-pooling        pooled_outputs = []
        for i, filter_size in enumerate(filter_sizes):
            with tf.name_scope("conv-maxpool-%s" % filter_size):
                # Convolution Layer
                filter_shape = [filter_size, embedding_size, 1, num_filters]
                W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
                b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
                conv = tf.nn.conv2d(
                    self.embedded_chars_expanded,
                    W,
                    strides=[1, 1, 1, 1],
                    padding="VALID",
                    name="conv")
                # Apply nonlinearity
                # h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
                h=tf.nn.relu6(tf.nn.bias_add(conv,b),name="relu")
                # Maxpooling over the outputs
                # pooled = tf.nn.max_pool(
                #     h,
                #     ksize=[1, sequence_length - filter_size + 1, 1, 1],
                #     strides=[1, 1, 1, 1],
                #     padding='VALID',
                #     name="pool")
                # pooled_outputs.append(pooled)
                pooled = tf.nn.avg_pool(
                    h,
                    ksize=[1, sequence_length - filter_size + 1, 1, 1],
                    strides=[1, 1, 1, 1],
                    padding='VALID',
                    name="pool")
                pooled_outputs.append(pooled)

        # max-pooling层的各种特征整合在一起
        num_filters_total = num_filters * len(filter_sizes)
        self.h_pool = tf.concat(pooled_outputs,3)
        self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total])

        # 添加全连接层,用于分类
        with tf.name_scope("full-connection"):
            W_fc1 = tf.Variable(tf.truncated_normal([num_filters_total,500], stddev=0.1))
            b_fc1 = tf.Variable(tf.constant(0.1,shape=[500]))
            self.h_fc1 = tf.nn.relu6(tf.matmul(self.h_pool_flat, W_fc1) + b_fc1)

        # 添加dropout层用于缓和过拟化
        with tf.name_scope("dropout"):
            # self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)
            self.h_drop = tf.nn.dropout(self.h_fc1, self.dropout_keep_prob)

        # 产生最后的输出和预测
        with tf.name_scope("output"):
            # W = tf.get_variable(
            #     "W",
            #     shape=[num_filters_total, num_classes],
            #     initializer=tf.contrib.layers.xavier_initializer())
            W = tf.get_variable(
                "W",
                shape=[500, num_classes],
                initializer=tf.contrib.layers.xavier_initializer())
            b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
            l2_loss += tf.nn.l2_loss(W)
            l2_loss += tf.nn.l2_loss(b)
            self.scores = tf.nn.xw_plus_b(self.h_drop, W, b, name="scores")
            self.predictions = tf.argmax(self.scores, 1, name="predictions")

        # 定义模型的损失函数
        with tf.name_scope("loss"):
            losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y)
            self.loss = tf.reduce_mean(losses) + l2_reg_lambda * l2_loss

        # 定义模型的准确率
        with tf.name_scope("accuracy"):
            correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))
            self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"), name="accuracy")

以上时TextCNN的模型结构代码,然后开始进行train,并利用summary和checkpoints来记录模型和训练时的参数等等,利用十折交叉验证来产生准确率,最后利用tensorboard查看accuracy、loss、w、b等等变化图;训练py的代码:
 
#! /usr/bin/env python
# encoding=utf-8
import tensorflow as tf
import numpy as np
import os
import time
import datetime
import data_loader
from cnn_graph import TextCNN
from tensorflow.contrib import learn
from sklearn import cross_validation
import preprocessing
# tf.global_variables
# 伴随tensorflowsummarycheckout
# ==================================================

# Model Hyperparameters
tf.flags.DEFINE_integer("embedding_dim", 200, "Dimensionality of character embedding (default: 128)")
tf.flags.DEFINE_string("filter_sizes", "3,4,5", "Comma-separated filter sizes (default: '3,4,5')")
tf.flags.DEFINE_integer("num_filters", 40, "Number of filters per filter size (default: 128)")
tf.flags.DEFINE_float("dropout_keep_prob", 0.5, "Dropout keep probability (default: 0.5)")
tf.flags.DEFINE_float("l2_reg_lambda", 3.0, "L2 regularizaion lambda (default: 0.0)")

# Training parameters
tf.flags.DEFINE_integer("batch_size", 50, "Batch Size (default: 64)")
tf.flags.DEFINE_integer("num_epochs", 100, "Number of training epochs (default: 200)")
tf.flags.DEFINE_integer("evaluate_every", 100, "Evaluate model on dev set after this many steps (default: 100)")
tf.flags.DEFINE_integer("checkpoint_every", 100, "Save model after this many steps (default: 100)")
# Misc Parameters
tf.flags.DEFINE_boolean("allow_soft_placement", True, "Allow device soft device placement")
tf.flags.DEFINE_boolean("log_device_placement", False, "Log placement of ops on devices")

# w2v文件路径
tf.flags.DEFINE_string("w2v_path", "./w2v_model/retrain_vectors_100.bin", "w2v file")
tf.flags.DEFINE_string("file_dir","./data_process/jd","train/test dataSet")

FLAGS = tf.flags.FLAGS
FLAGS._parse_flags()
print("\nParameters:")
for attr, value in sorted(FLAGS.__flags.items()):
    print("{}={}".format(attr.upper(), value))
print("")


# Data Preparatopn
# ==================================================

# Load data
print("Loading data...")
files = ["reviews.neg","reviews.pos"]
# 加载所有的未切分的数据
x_text, y_labels,neg_examples,pos_examples = data_loader.\
    load_data_and_labels(data_dir=FLAGS.file_dir,files=files,splitable=False)

# 获取消极数据的2/3,得到的评论的长度离散度更低
neg_accept_length = preprocessing.freq_factor(neg_examples,
                                         percentage=0.8, drawable=False)
neg_accept_length = [item[0] for item in neg_accept_length]
neg_examples = data_loader.load_data_by_length(neg_examples,neg_accept_length)

# 获取积极数据的2/3,得到的评论的长度离散度更低
pos_accept_length = preprocessing.freq_factor(pos_examples,
                                         percentage=0.8, drawable=False)
pos_accept_length = [item[0] for item in pos_accept_length]
pos_examples = data_loader.load_data_by_length(pos_examples,pos_accept_length)

x_text = neg_examples + pos_examples
neg_labels = [[1,0] for _ in neg_examples]
pos_labels = [[0,1] for _ in pos_examples]
y_labels = np.concatenate([neg_labels,pos_labels], axis=0)
print("Loading data finish")

# Build vocabulary
max_document_length = max([len(x.split(" ")) for x in x_text]) # 最长的句子的长度
print(max_document_length)
vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)
x = np.array(list(vocab_processor.fit_transform(x_text)))

# 加载提前训练的w2v数据集
word_vecs = data_loader.load_bin_vec(fname=FLAGS.w2v_path,
                         vocab=list(vocab_processor.vocabulary_._mapping),
                                     ksize=FLAGS.embedding_dim)
# 加载嵌入层的table
W = data_loader.get_W(word_vecs=word_vecs,
                  vocab_ids_map=vocab_processor.vocabulary_._mapping,
                  k=FLAGS.embedding_dim,is_rand=False)

# 随机化数据
np.random.seed(10)
shuffle_indices = np.random.permutation(np.arange(len(y_labels)))
x_shuffled = x[shuffle_indices]
y_shuffled = y_labels[shuffle_indices]

out_path = os.path.abspath(os.path.join(os.path.curdir, "runs","parameters"))
parameters = "新全连接+jd数据+10\n" \
             "embedding_dim: {},\n" \
             "filter_sizes:{},\n" \
             "num_filters:{},\n" \
             "dropout_keep_prob:{},\n" \
             "l2_reg_lambda:{},\n" \
             "num_epochs:{},\n" \
             "batch_size:{}".format(FLAGS.embedding_dim,FLAGS.filter_sizes,FLAGS.num_filters,
                                    FLAGS.dropout_keep_prob,FLAGS.l2_reg_lambda,FLAGS.num_epochs,
                                    FLAGS.batch_size)
open(out_path, 'w').write(parameters)

# Training
# ==================================================
def train(X_train, X_dev, x_test, y_train, y_dev, y_test):
    with tf.Graph().as_default():
        session_conf = tf.ConfigProto(
          allow_soft_placement=FLAGS.allow_soft_placement,
          log_device_placement=FLAGS.log_device_placement)
        sess = tf.Session(config=session_conf)
        with sess.as_default():
            cnn = TextCNN(
                sequence_length=max_document_length,
                num_classes=2,
                vocab_size=len(vocab_processor.vocabulary_),
                embedding_size=FLAGS.embedding_dim,
                embedding_table=W,
                filter_sizes=list(map(int, FLAGS.filter_sizes.split(","))),
                num_filters=FLAGS.num_filters,
                l2_reg_lambda=FLAGS.l2_reg_lambda)

            # Define Training procedure
            global_step = tf.Variable(0, name="global_step", trainable=False)
            optimizer = tf.train.AdamOptimizer(1e-3)
            grads_and_vars = optimizer.compute_gradients(cnn.loss)
            train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)

            # Keep track of gradient values and sparsity (optional)
            grad_summaries = []
            for g, v in grads_and_vars:
                if g is not None:
                    grad_hist_summary = tf.summary.histogram("{}/grad/hist".format(v.name), g)
                    sparsity_summary = tf.summary.scalar("{}/grad/sparsity".format(v.name), tf.nn.zero_fraction(g))
                    grad_summaries.append(grad_hist_summary)
                    grad_summaries.append(sparsity_summary)
            grad_summaries_merged = tf.summary.merge(grad_summaries)

            # Output directory for models and summaries
            timestamp = str(int(time.time()))
            out_dir = os.path.abspath(os.path.join(os.path.curdir, "runs", timestamp))
            print("Writing to {}\n".format(out_dir))

            # Summaries for loss and accuracy
            loss_summary = tf.summary.scalar("loss", cnn.loss)
            acc_summary = tf.summary.scalar("accuracy", cnn.accuracy)

            # Train Summaries
            train_summary_op = tf.summary.merge([loss_summary, acc_summary, grad_summaries_merged])
            train_summary_dir = os.path.join(out_dir, "summaries", "train")
            train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph)

            # Dev summaries
            dev_summary_op = tf.summary.merge([loss_summary, acc_summary])
            dev_summary_dir = os.path.join(out_dir, "summaries", "dev")
            dev_summary_writer = tf.summary.FileWriter(dev_summary_dir, sess.graph)


            # Checkpoint directory. Tensorflow assumes this directory already exists so we need to create it
            checkpoint_dir = os.path.abspath(os.path.join(out_dir, "checkpoints"))
            checkpoint_prefix = os.path.join(checkpoint_dir, "model")
            if not os.path.exists(checkpoint_dir):
                os.makedirs(checkpoint_dir)
            saver = tf.train.Saver(tf.global_variables())

            # Write vocabulary
            vocab_processor.save(os.path.join(out_dir, "vocab"))

            # Initialize all variables
            # sess.run(tf.initialize_all_variables())
            sess.run(tf.global_variables_initializer())

            def train_step(x_batch, y_batch):
                """
                A single training step
                """
                feed_dict = {
                  cnn.input_x: x_batch,
                  cnn.input_y: y_batch,
                  cnn.dropout_keep_prob: FLAGS.dropout_keep_prob
                }
                _, step, summaries, loss, accuracy = sess.run(
                    [train_op, global_step, train_summary_op, cnn.loss, cnn.accuracy],
                    feed_dict)
                # _, step, loss, accuracy = sess.run(
                #     [train_op, global_step, cnn.loss, cnn.accuracy],
                #     feed_dict)
                time_str = datetime.datetime.now().isoformat()
                print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))
                train_summary_writer.add_summary(summaries, step)

            def dev_step(x_batch, y_batch, writer=None):
                """
                Evaluates model on a dev set
                """
                feed_dict = {
                  cnn.input_x: x_batch,
                  cnn.input_y: y_batch,
                  cnn.dropout_keep_prob: 1.0
                }
                step, summaries, loss, accuracy = sess.run(
                    [global_step, dev_summary_op, cnn.loss, cnn.accuracy],
                    feed_dict)
                # step, loss, accuracy = sess.run(
                #     [global_step, cnn.loss, cnn.accuracy],
                #     feed_dict)
                time_str = datetime.datetime.now().isoformat()
                print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))
                if writer:
                    writer.add_summary(summaries, step)



            # Generate batches
            batches = data_loader.batch_iter(
                list(zip(X_train, y_train)), FLAGS.batch_size, FLAGS.num_epochs)
            # Training loop. For each batch...
            for batch in batches:
                x_batch, y_batch = zip(*batch)
                train_step(x_batch, y_batch)
                current_step = tf.train.global_step(sess, global_step)
                if current_step % FLAGS.evaluate_every == 0:
                    print("\nEvaluation:")
                    dev_step(X_dev, y_dev, writer=dev_summary_writer)
                    # dev_step(X_dev, y_dev, writer=None)
                    print("")
                if current_step % FLAGS.checkpoint_every == 0:
                    path = saver.save(sess, checkpoint_prefix, global_step=current_step)
                    print("Saved model checkpoint to {}\n".format(path))

            # Test loop
            # Generate batches for one epoch
            batches = data_loader.batch_iter(list(x_test), FLAGS.batch_size, 1, shuffle=False)
            # Collect the predictions here
            all_predictions = []
            for x_test_batch in batches:
                batch_predictions = sess.run(cnn.predictions, {cnn.input_x: x_test_batch, cnn.dropout_keep_prob: 1.0})
                all_predictions = np.concatenate([all_predictions, batch_predictions])

            correct_predictions = float(sum(
                all_predictions == np.argmax(y_test,axis=1)))

            print("Total number of test examples: {}".format(len(y_test)))
            print("Accuracy: {:g}".format(correct_predictions / float(len(y_test))))
            # open(os.path.join(out_dir,"test"),'a').write("Accuracy: {:g}".format(correct_predictions / float(len(y_test))))
            out_path = os.path.abspath(os.path.join(os.path.curdir, "runs","test"))
            open(out_path,'a').write("{:g},".format(correct_predictions / float(len(y_test))))
            print("\n写入成功!\n")


# cross-validation
kf = cross_validation.KFold(len(x_shuffled), n_folds=3)
for train_index, test_index in kf:
    X_train_total = x_shuffled[train_index]
    y_train_total = y_shuffled[train_index]
    x_test = x_shuffled[test_index]
    y_test = y_shuffled[test_index]

    # 分割训练集与验证集
    X_train, X_dev, y_train, y_dev = cross_validation.train_test_split(
        X_train_total, y_train_total, test_size=0.2, random_state=0)

    print("Vocabulary Size: {:d}".format(len(vocab_processor.vocabulary_)))
    print("Train/Dev split: {:d}/{:d}".format(len(y_train), len(y_dev)))
训练完成后准确率83%左右,还需要在进一步进行改进来提升正确率,比如利用chunk max-pooling方法代替max-pooling,利用集成的方法,因为word embedding词忽略了当前上下文的含义,潜在认为相同词在不同文本中的含义相同,所以可以利用词义消歧来提升其正确率等等;
训练模型保存在checkpoints中,由model-4000.index,model-4000.meta,model-4000.data等;
最后tensorboard --logdir  /home/yang/PycharmProjects/cnn-text-classification-master/runs/1515468832
/home/yang/PycharmProjects/cnn-text-classification-master/runs/1515468832/checkpoints/model-4300

Total number of test examples: 1333

Accuracy: 0.825956


转自:http://blog.csdn.net/gentelyang/article/details/79011194

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
水很深的深度学习(四)——卷积神经网络CNN
上世纪60年代,Hubel等人通过对猫视觉皮层细胞的研究,提出了感受野这个概念,到80年代,Fukushima在感受野概念的基础之上提出了神经认知机的概念,可以看作是卷积神经网络的第一个实现网络,神经认知机将一个视觉模式分解成许多子模式(特征),然后进入分层递阶式相连的特征平面进行处理,它试图将视觉系统模型化,使其能够在即使物体有位移或轻微变形的时候,也能完成识别。
23 0
【ELM分类】基于粒子群优化卷积神经网络CNN结合极限学习机ELM实现数据分类附matlab代码
【ELM分类】基于粒子群优化卷积神经网络CNN结合极限学习机ELM实现数据分类附matlab代码
73 0
9 大主题卷积神经网络(CNN)的 PyTorch 实现
9 大主题卷积神经网络(CNN)的 PyTorch 实现
221 0
卷积神经网络(CNN)详解
卷积神经网络(CNN)详解
114 0
实时卷积神经网络实现人脸检测和情感/性别分类
实时卷积神经网络实现人脸检测和情感/性别分类
67 0
DL之CNN:利用卷积神经网络算法(2→2,基于Keras的API-Sequential)利用MNIST(手写数字图片识别)数据集实现多分类预测
DL之CNN:利用卷积神经网络算法(2→2,基于Keras的API-Sequential)利用MNIST(手写数字图片识别)数据集实现多分类预测
137 0
CNN卷积神经网络
一、BP神经网络回顾 人工全连接神经网络 (1)每相邻两层之间的每个神经元之间都是有边相连的 (2)当输入层的特征维度变得很高时,这时全连接网络需要训练               的参数就会增大很多,计算速度就会变得...
1656 0
(CS231-2017)卷积神经网络视觉识别:线性分类(1)
本文为CS231n-2017 Convolutional Neural Networks for Visual Recognition中Note:Linear Classification的中文翻译 线性分类 在上一节中我们介绍了图像分类的问题,这个问题中讲述的任务是从一系列固定的图像目录中选取一个单独的样本给一副特定的图像进行分配。
1202 0
cdh5.1.0 yum安装说明
<div class="markdown_views"> <p>接触过hadoop的人都知道hadoop的搭建过程非常麻烦,尤其是对于初学者,幸运的是有cloudera这样的公司为我们做了一些发行版,我们可以直接使用apt-get,yum等包管理工具进行安装。 <br> 下面我要说得是,在centos6.5 64位下,通过yum安装cdh5.1.0,需要注意的是5.1.0默认是
1905 0
+关注
隐士2018
蹭热度,自学AI
文章
问答
文章排行榜
最热
最新
相关电子书
更多
ImageNet:VGGNet,ResNet,Incepti
立即下载
WordRank embedding:"crowned"is
立即下载
概率图模型
立即下载