AI 助理
备案控制台
开发者社区 大数据 文章 正文

Spark机器学习6·聚类模型(spark-shell)

2016-12-21 3935
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： ![](http://img3.douban.com/lpic/s28277325.jpg) [Spark机器学习](http://book.douban.com/subject/26593179/) - K-均值(K-mean)聚类 目的：最小化所有类簇中的方差之和 - 类簇内方差和...


Spark机器学习

  • K-均值(K-mean)聚类 目的：最小化所有类簇中的方差之和

    • 类簇内方差和(WCSS,within cluster sum of squared errors)
    • fuzzy K-means

  • 层次聚类(hierarchical culstering)

    • 凝聚聚类(agglomerative clustering)
    • 分列式聚类(divisive clustering)

0 运行环境

cd $SPARK_HOME
bin/spark-shell --name my_mlib --packages org.jblas:jblas:1.2.4 --driver-memory 4G --executor-memory 4G --driver-cores 2
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.Rating
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.clustering.KMeans
import breeze.linalg._
import breeze.numerics.pow

1 提取特征

val PATH = "/Users/erichan/sourcecode/book/Spark机器学习"
val movies = sc.textFile(PATH+"/ml-100k/u.item")

println(movies.first)

1|Toy Story (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Toy%20Story%20(1995)|0|0|0|1|1|1|0|0|0|0|0|0|0|0|0|0|0|0|0

提取标签

val genres = sc.textFile(PATH+"/ml-100k/u.genre")
genres.take(5).foreach(println)

unknown|0
Action|1
Adventure|2
Animation|3
Children's|4

val genreMap = genres.filter(!_.isEmpty).map(line => line.split("\\|")).map(array => (array(1), array(0))).collectAsMap
println(genreMap)

Map(2 -> Adventure, 5 -> Comedy, 12 -> Musical, 15 -> Sci-Fi, 8 -> Drama, 18 -> Western, 7 -> Documentary, 17 -> War, 1 -> Action, 4 -> Children's, 11 -> Horror, 14 -> Romance, 6 -> Crime, 0 -> unknown, 9 -> Fantasy, 16 -> Thriller, 3 -> Animation, 10 -> Film-Noir, 13 -> Mystery)

val titlesAndGenres = movies.map(_.split("\\|")).map { array =>
    val genres = array.toSeq.slice(5, array.size)
    val genresAssigned = genres.zipWithIndex.filter { case (g, idx) =>
        g == "1"
    }.map { case (g, idx) =>
        genreMap(idx.toString)
    }
    (array(0).toInt, (array(1), genresAssigned))
}
println(titlesAndGenres.first)

(1,(Toy Story (1995),ArrayBuffer(Animation, Children's, Comedy)))

训练推荐模型

val rawData = sc.textFile(PATH+"/ml-100k/u.data")
val rawRatings = rawData.map(_.split("\t").take(3))
val ratings = rawRatings.map{ case Array(user, movie, rating) => Rating(user.toInt, movie.toInt, rating.toDouble) }
ratings.cache
val alsModel = ALS.train(ratings, 50, 10, 0.1)

val movieFactors = alsModel.productFeatures.map { case (id, factor) => (id, Vectors.dense(factor)) }
val movieVectors = movieFactors.map(_._2)
val userFactors = alsModel.userFeatures.map { case (id, factor) => (id, Vectors.dense(factor)) }
val userVectors = userFactors.map(_._2)

归一化

val movieMatrix = new RowMatrix(movieVectors)
val movieMatrixSummary = movieMatrix.computeColumnSummaryStatistics()
val userMatrix = new RowMatrix(userVectors)
val userMatrixSummary = userMatrix.computeColumnSummaryStatistics()

println("Movie factors mean: " + movieMatrixSummary.mean)
println("Movie factors variance: " + movieMatrixSummary.variance)
println("User factors mean: " + userMatrixSummary.mean)
println("User factors variance: " + userMatrixSummary.variance)

Movie factors mean: [-0.2955253575969453,-0.2894017158566661,-0.2319822953560126,0.002917648331681182,0.16553261386128745,-0.21992534888550966,-0.03380127825698873,-0.20603088790834398,-0.15619861138444532,-0.028497688228493936,0.16963530616257805,0.14388067884376599,-0.0017092576491059591,-0.09626837303920982,-0.06064127207772349,-0.06045518556672421,0.18751923345914,0.2399624456229244,0.26532560070303446,0.05541910564428427,-0.015674971004015527,0.011168436718639107,-0.04741294377492476,0.11574693735017375,0.1289987655696671,0.44134038441588025,-0.5900688729554584,-0.03768358034266212,0.008887881298347921,0.20425041421871237,-0.20022602485759528,0.2697605004694663,0.10361325058554109,0.210277185021123,-0.22259636797095098,0.1174637780755839,-0.13688720440722232,0.03767713022869551,-0.0558405163043045,-0.12431407617904076,-0.046046222769634326,-0.20808223343120555,-0.3272035383525689,-0.2069514616509938,-0.0754149005227642,0.0856900404902959,0.06164062157888312,-0.06518672356795488,-0.32742867325628294,0.20285276122166002]

Movie factors variance: [0.04665858952268178,0.0348699710379137,0.03579479789217705,0.029901233287017822,0.03584001448747631,0.030266373892482327,0.03305718879524182,0.02686945635500392,0.025320493042287756,0.024438753389466123,0.02575390293411112,0.028972744965903668,0.02827972104601086,0.033613911928577246,0.033662480558315735,0.02833746789842838,0.02994391891556463,0.04394221701123749,0.03435422169469965,0.03561654218653061,0.02682492748548697,0.029604704664741063,0.024673702738648908,0.030823597518982247,0.028442111141151614,0.03613743157595084,0.05449475590178096,0.042012621165520236,0.028896894307921802,0.033241681676696215,0.02633619851965672,0.035711481477364235,0.025481764774248593,0.028764828375131987,0.0272530758482775,0.029673397451581197,0.03148963131813852,0.03622387708462999,0.02816170323774573,0.033017372289155716,0.028641152942670445,0.02904189221086495,0.030234076747492195,0.04509970117296679,0.029449883713724593,0.02756067270740738,0.04139144468263727,0.030245838006703486,0.03131689738936245,0.03378427027186054]

User factors mean: [-0.49867551877683824,-0.44459915531291566,-0.36051481893169574,0.017151848798776233,0.3213603826583396,-0.3196901675619378,-0.07224328943358119,-0.29041744434669287,-0.22727507102332345,-0.03178720415880569,0.25862894293461186,0.22888402894019788,0.012327199821030293,-0.14990885838046697,-0.1281515413333295,-0.09431829455241085,0.2679196025618735,0.38335691552119355,0.34604572069945905,0.11174974992685119,-0.02180706957147866,0.005610012764397019,-0.08491397316018835,0.20231176194774866,0.17161396689284497,0.6398163397864598,-0.8673987425745228,-0.10283010351171737,0.028330477167842844,0.30443187793692406,-0.301912604145753,0.4138735923728453,0.2256847560401456,0.285848070636566,-0.2605794171061,0.22449121780469036,-0.23998269543836812,0.036814175516996,-0.0679476059994798,-0.14427917258340417,-0.0833994179810923,-0.3582564875155623,-0.4564982359022274,-0.358039104184582,-0.12317214145750788,0.15037235678650748,0.06053431528961892,-0.06831269426575506,-0.5051800522709825,0.3151860279443428]

User factors variance: [0.04443021235777847,0.03057049227554263,0.03697914530468325,0.037294292401115044,0.035107560835514376,0.03456306589890253,0.03582898189508532,0.028049150877918694,0.032265557628909904,0.033678972590911474,0.03107771568048393,0.03456737756860466,0.035184013404102404,0.04264936219513472,0.04120372326054623,0.03364277736735525,0.040292531435941865,0.04006060147670186,0.03950365342886879,0.04560154697337463,0.030231562691714647,0.041120732342916626,0.03118953330313852,0.03508187607535198,0.03228272297984499,0.03603017959168009,0.04917534366846078,0.059425007832722164,0.03161224197770566,0.04211986001194535,0.02891350391303218,0.05259534335774597,0.03483271651803892,0.040489027307905476,0.03125884956067426,0.0379774604293261,0.035875980098136084,0.043509576391072786,0.03338290356822281,0.03675372599031079,0.03379511912889908,0.02951817116168268,0.0430380317818896,0.04214608566562065,0.03376833767379957,0.0314188022932176,0.048481326691437995,0.03724671278315033,0.034103714500646594,0.046064657833824844]

2 训练

val numClusters = 5
val numIterations = 10
val numRuns = 3
val movieClusterModel = KMeans.train(movieVectors, numClusters, numIterations, numRuns)
val movieClusterModelConverged = KMeans.train(movieVectors, numClusters, 100)
val userClusterModel = KMeans.train(userVectors, numClusters, numIterations, numRuns)

3 预测

val movie1 = movieVectors.first
val movieCluster = movieClusterModel.predict(movie1)
println(movieCluster)
val predictions = movieClusterModel.predict(movieVectors)
println(predictions.take(10).mkString(","))
def computeDistance(v1: DenseVector[Double], v2: DenseVector[Double]): Double = pow(v1 - v2, 2).sum
val titlesWithFactors = titlesAndGenres.join(movieFactors)
val moviesAssigned = titlesWithFactors.map { case (id, ((title, genres), vector)) =>
    val pred = movieClusterModel.predict(vector)
    val clusterCentre = movieClusterModel.clusterCenters(pred)
    val dist = computeDistance(DenseVector(clusterCentre.toArray), DenseVector(vector.toArray))
    (id, title, genres.mkString(" "), pred, dist)
}
val clusterAssignments = moviesAssigned.groupBy { case (id, title, genres, cluster, dist) => cluster }.collectAsMap

for ( (k, v) <- clusterAssignments.toSeq.sortBy(_._1)) {
    println(s"Cluster $k:")
    val m = v.toSeq.sortBy(_._5)
    println(m.take(20).map { case (_, title, genres, _, d) => (title, genres, d) }.mkString("\n"))
    println("=====\n")
}

4 评估性能

内部评价指标

  • WCSS
  • Davies-Bouldin指数
  • Dunn指数
  • 轮廓系数(silhouetee coefficient)

外不评价指标

  • Rand measure
  • F-measure
  • 雅卡尔系数(Jaccard index)

MLlib的computeCost

val movieCost = movieClusterModel.computeCost(movieVectors)
val userCost = userClusterModel.computeCost(userVectors)
println("WCSS for movies: " + movieCost)
println("WCSS for users: " + userCost)

5 调优

通过交叉验证选择K

val trainTestSplitMovies = movieVectors.randomSplit(Array(0.6, 0.4), 123)
val trainMovies = trainTestSplitMovies(0)
val testMovies = trainTestSplitMovies(1)
val costsMovies = Seq(2, 3, 4, 5, 10, 20).map { k => (k, KMeans.train(trainMovies, numIterations, k, numRuns).computeCost(testMovies)) }
println("Movie clustering cross-validation:")
costsMovies.foreach { case (k, cost) => println(f"WCSS for K=$k id $cost%2.2f") }

WCSS for K=2 id 870.36
WCSS for K=3 id 858.28
WCSS for K=4 id 847.40
WCSS for K=5 id 840.71
WCSS for K=10 id 842.58
WCSS for K=20 id 843.24

val trainTestSplitUsers = userVectors.randomSplit(Array(0.6, 0.4), 123)
val trainUsers = trainTestSplitUsers(0)
val testUsers = trainTestSplitUsers(1)
val costsUsers = Seq(2, 3, 4, 5, 10, 20).map { k => (k, KMeans.train(trainUsers, numIterations, k, numRuns).computeCost(testUsers)) }
println("User clustering cross-validation:")
costsUsers.foreach { case (k, cost) => println(f"WCSS for K=$k id $cost%2.2f") }

WCSS for K=2 id 573.50
WCSS for K=3 id 580.33
WCSS for K=4 id 574.84
WCSS for K=5 id 575.61
WCSS for K=10 id 586.05
WCSS for K=20 id 577.01

文章标签：
Apache
Scala
数据挖掘
机器学习/深度学习
分布式计算
关键词：
人工智能平台 PAI模型
人工智能平台 PAI聚类
apache spark机器学习
spark人工智能平台 PAI
apache spark模型
六翁
目录
相关文章
肥猪lele01
|
1月前
|
机器学习/深度学习 数据挖掘
机器学习之聚类——模糊聚类FCM
机器学习之聚类——模糊聚类FCM
肥猪lele01
28 4
小森ai小小贾
|
1月前
|
机器学习/深度学习 算法 搜索推荐
机器学习中的聚类
**文章摘要：** 本文介绍了聚类算法的基本概念、应用、实现流程和评估方法。聚类是一种无监督学习技术，用于将数据分为相似的组，如K-means、层次聚类、DBSCAN和谱聚类。K-means算法通过迭代优化质心，将数据点分配到最近的簇，直至质心不再变化。模型评估包括误差平方和(SSE)、肘部方法（确定最佳簇数）和轮廓系数法（Silhouette Coefficient），以量化聚类的紧密度和分离度。应用场景涵盖用户画像、广告推荐和图像分割等。在Python的sklearn库中，可以使用KMeans API进行聚类操作。
小森ai小小贾
43 1
孜倦与shine
|
1月前
|
机器学习/深度学习 分布式计算 算法
基于Spark中随机森林模型的天气预测系统
基于Spark中随机森林模型的天气预测系统
孜倦与shine
38 1
龙大吉
|
1月前
|
机器学习/深度学习 算法 数据挖掘
机器学习——DBSCAN 聚类算法
【6月更文挑战第8天】DBSCAN是一种基于密度的无监督聚类算法，能处理不规则形状的簇和噪声数据，无需预设簇数量。其优点包括自动发现簇结构和对噪声的鲁棒性。示例代码展示了其基本用法。然而，DBSCAN对参数选择敏感，计算效率受大规模数据影响。为改善这些问题，研究方向包括参数自适应和并行化实现。DBSCAN在图像分析、数据分析等领域有广泛应用，通过持续改进，将在未来保持重要地位。
龙大吉
46 2
东方睿赢
|
29天前
|
机器学习/深度学习 分布式计算 算法
在机器学习项目中，选择算法涉及问题类型识别（如回归、分类、聚类、强化学习）
【6月更文挑战第28天】在机器学习项目中，选择算法涉及问题类型识别（如回归、分类、聚类、强化学习）、数据规模与特性（大数据可能适合分布式算法或深度学习）、性能需求（准确性、速度、可解释性）、资源限制（计算与内存）、领域知识应用以及实验验证（交叉验证、模型比较）。迭代过程包括数据探索、模型构建、评估和优化，结合业务需求进行决策。
东方睿赢
30 0
jjczqyjxgdegi
|
29天前
|
机器学习/深度学习 算法 数据可视化
技术心得记录：机器学习笔记之聚类算法层次聚类HierarchicalClustering
技术心得记录：机器学习笔记之聚类算法层次聚类HierarchicalClustering
jjczqyjxgdegi
26 0
jjczqyjxgdegi
|
29天前
|
机器学习/深度学习 分布式计算 API
技术好文：Spark机器学习笔记一
技术好文：Spark机器学习笔记一
jjczqyjxgdegi
19 0
月亮给我抄代码
|
2月前
|
机器学习/深度学习 数据采集 分布式计算
【机器学习】Spark ML 对数据进行规范化预处理 StandardScaler 与向量拆分
标准化Scaler是数据预处理技术，用于将特征值映射到均值0、方差1的标准正态分布，以消除不同尺度特征的影响，提升模型稳定性和精度。Spark ML中的StandardScaler实现此功能，通过`.setInputCol`、`.setOutputCol`等方法配置并应用到DataFrame数据。示例展示了如何在Spark中使用StandardScaler进行数据规范化，包括创建SparkSession，构建DataFrame，使用VectorAssembler和StandardScaler，以及将向量拆分为列。规范化有助于降低特征重要性，提高模型训练速度和计算效率。
月亮给我抄代码
51 6
月亮给我抄代码
|
2月前
|
机器学习/深度学习 分布式计算 算法
【机器学习】Spark ML 对数据特征进行 One-Hot 编码
One-Hot 编码是机器学习中将离散特征转换为数值表示的方法，每个取值映射为一个二进制向量，常用于避免特征间大小关系影响模型。Spark ML 提供 OneHotEncoder 进行编码，输入输出列可通过 `inputCol` 和 `outputCol` 参数设置。在示例中，先用 StringIndexer 对类别特征编码，再用 OneHotEncoder 转换，最后展示编码结果。注意 One-Hot 编码可能导致高维问题，可结合实际情况选择编码方式。
月亮给我抄代码
51 6
小森ai小小贾
|
1月前
|
机器学习/深度学习 算法 搜索推荐
机器学习聚类算法
聚类算法是无监督学习技术，用于发现数据集中的自然群体，如用户画像、广告推荐等。常见的聚类算法包括K-Means，它基于距离分配样本至簇，适合球形分布；层次聚类则通过合并或分裂形成簇，能发现任意形状的簇；DBSCAN依据密度来聚类，对噪声鲁棒。KMeans API中`sklearn.cluster.KMeans(n_clusters=8)`用于指定簇的数量。评估聚类效果可使用轮廓系数、SSE等指标，Elbow方法帮助选择合适的K值。
小森ai小小贾
22 0

热门文章

最新文章

  • 1
    基于区块链的机器学习模型创建方案
  • 2
    机器学习PAI-DSW交互式建模个人版有奖评测活动
  • 3
    机器学习实战二：波士顿房价预测 Boston Housing（上）
  • 4
    机器学习识别乳腺癌
  • 5
    《阿里云AI产品必知必会系列电子书》——机器学习平台PAI——DesignerQuickStart使用教程（3）
  • 6
    谷歌对其机器学习系统信心倍增 已应用到所有搜索结果中
  • 7
    机器学习技法--学习笔记04--Soft SVM
  • 8
    面向机器学习的特征工程 三、文本数据: 展开、过滤和分块
  • 9
    机器学习基础
  • 10
    《机器学习与R语言（原书第2版）》一2.4　总结
  • 1
    使用shell 脚本打印图形3
    39
  • 2
    使用shell 脚本打印图形2
    35
  • 3
    使用shell 脚本打印图形1
    38
  • 4
    调整虚拟机内存参数的shell 脚本
    35
  • 5
    从shell脚本发送邮件
    35
  • 6
    使用 shell 脚本打印图形
    47
  • 7
    调整虚拟机内存参数的 shell 脚本
    42
  • 8
    备份 MySQL 的 shell 脚本(mysqldump版本)
    36
  • 9
    每天解析一个shell脚本（82）
    25
  • 10
    每天解析一个shell脚本（68）
    20

    • 相关课程

    更多
  • 场景实践 - 机器学习PAI实现精细化营销
  • 场景实践 - 基于阿里云PAI机器学习平台使用时间序列分解模型预测商品销量
  • 场景实践 - 基于机器学习进行收入预测分析
  • 机器学习概览及常见算法
  • 机器学习入门-概念原理及常用算法
  • 大数据实时计算框架Spark快速入门

    • 相关电子书

    更多
  • Hybrid Cloud and Apache Spark
  • Scalable Deep Learning on Spark
  • Comparison of Spark SQL with Hive

    • 相关实验场景

    更多
  • Alibaba Cloud Linux操作系统Shell程序
  • 基于Hologres+PAI+计算巢，5分钟搭建企业级AI问答知识库
  • 使用PAI-快速开始，低代码实现大语言模型微调和部署
    • 下一篇
    通义千问API入门教程