前言

本文主要介绍通过python实现数据聚类、脚本开发、办公自动化。读取voc数据，聚类voc数据。

一、业务逻辑

• 读取voc数据采集的数据
• 批处理，使用jieba进行分词，去除停用词
• LDA模型计算词汇和每个词的频率
• 将可视化结果保存到HTML文件中

  二、具体产出

  

三、执行脚本

python lda.py

四、关键代码

# LDA主题分析模型
import pandas as pd
import jieba
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
import pyLDAvis

fileName = "100005785591" # 文件名

# 加载停用词
with open('stopwordsfull', 'r', encoding='utf-8') as f:
    stopwords = set([line.strip() for line in f])

# 加载业务域名词
with open('luyouqi.txt', 'r', encoding='utf-8') as f:
    business_terms = set([line.strip() for line in f])

# 为jieba分词库增加业务名词
for term in business_terms:
    jieba.add_word(term)

# 对评论进行分词
def tokenize(text):
    words = jieba.cut(text)
    filtered_words = [word for word in words if word not in stopwords]
    return ' '.join(filtered_words)

# 从xlsx文件加载评论数据
data = pd.read_excel('clean/cleaned_voc'+fileName+'.xlsx')
comments = data['content'].tolist()

# 对每个评论进行分词并且形成新的评论列表
tokenized_comments = [tokenize(comment) for comment in comments]

# 使用CountVectorizer来获取词频
vectorizer = CountVectorizer(max_df=0.85, min_df=2, max_features=1000)
X = vectorizer.fit_transform(tokenized_comments)

# LDA模型
lda = LatentDirichletAllocation(n_components=5, random_state=42)
lda.fit(X)

# 计算词汇和每个词的频率
vocab = vectorizer.get_feature_names_out()
term_frequency = X.sum(axis=0).tolist()[0]

# 获取文档-主题分布和文档长度
doc_topic_dists = lda.transform(X)
doc_lengths = [len(doc.split()) for doc in comments]

# 使用pyLDAvis.prepare方法进行可视化
lda_display = pyLDAvis.prepare(
    topic_term_dists=lda.components_,
    doc_topic_dists=doc_topic_dists,
    doc_lengths=doc_lengths,
    vocab=vocab,
    term_frequency=term_frequency
)

# 将可视化结果保存到HTML文件中
output_file_path = 'lda/'+fileName+'.html'
pyLDAvis.save_html(lda_display, output_file_path)

# 读取生成的HTML文件并替换CDN链接为本地路径
with open(output_file_path, 'r', encoding='utf-8') as file:
    file_contents = file.read()

file_contents = file_contents.replace(
    'https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.4.0/pyLDAvis/js/ldavis.v1.0.0.js',
    'ldavis.v1.0.0.js'
)
file_contents = file_contents.replace(
    'https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.4.0/pyLDAvis/js/ldavis.v1.0.0.css',
    'ldavis.v1.0.0.css'
)

# 保存修改后的HTML文件
with open(output_file_path, 'w', encoding='utf-8') as file:
    file.write(file_contents)

五、关键文件

luyouqi.text 分词字典（片段）

2.4G
2.5G口
软路由
2.5G
WiFi
WiFi5
WiFi6
WiFi4

stopwordsfull 停用词（片段）

客户
层面
菜鸟
滑丝
换货
三思
固记
厂商
吸引力
体会

六、LDA话题权重优先级参考

https://www.bilibili.com/video/BV1Sr4y1C7Xc/?spm_id_from=333.337.search-card.all.click