备案控制台
开发者社区 大数据 文章 正文

Spark机器学习4·分类模型(spark-shell)

2016-12-21 2812
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： ![](http://img3.douban.com/lpic/s28277325.jpg) [Spark机器学习](http://book.douban.com/subject/26593179/) - 线性模型 - 逻辑回归--逻辑损失(logistic loss) - 线性...


Spark机器学习

  • 线性模型

    • 逻辑回归--逻辑损失(logistic loss)
    • 线性支持向量机（Support Vector Machine, SVM）--合页损失(hinge loss)
  • 朴素贝叶斯(Naive Bayes)
  • 决策树

0 准备数据

kaggle2.blob.core.windows.net/competitions-data/kaggle/3526/train.tsv

sed 1d train.tsv > train_noheader.tsv

0 运行环境

cd /Users/erichan/Garden/spark-1.5.1-bin-cdh4

bin/spark-shell --name my_mlib --packages org.jblas:jblas:1.2.4-SNAPSHOT --driver-memory 4G --executor-memory 4G --driver-cores 2

import org.apache.spark.mllib.feature._
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.optimization._
import org.apache.spark.mllib.classification._
import org.apache.spark.mllib.evaluation._
import org.apache.spark.mllib.classification._
import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.configuration.Algo
import org.apache.spark.mllib.tree.impurity._

1 提取特征

val PATH = "/Users/erichan/sourcecode/book/Spark机器学习"
val rawData = sc.textFile(PATH+"/train_noheader.tsv")
val records = rawData.map(line => line.split("\t"))
records.first

Array[String] = Array("http://www.bloomberg.com/news/2010-12-23/ibm-predicts-holographic-calls-air-breathing-batteries-by-2015.html", "4042", "{""title"":""IBM Sees Holographic Calls Air Breathing Batteries ibm sees holographic calls, air-breathing batteries"",""body"":""A sign stands outside the International Business Machines Corp IBM Almaden Research Center campus in San Jose California Photographer Tony Avelar Bloomberg Buildings stand at the International Business Machines Corp IBM Almaden Research Center campus in the Santa Teresa Hills of San Jose California Photographer Tony Avelar Bloomberg By 2015 your mobile phone will project a 3 D image of anyone who calls and your laptop will be powered by kinetic energy At least that s what International Business Machines Corp sees ...

val data = records.map { r =>
    val trimmed = r.map(_.replaceAll("\"", ""))
    val label = trimmed(r.size - 1).toInt
    val features = trimmed.slice(4, r.size - 1).map(d => if (d == "?") 0.0 else d.toDouble)
    LabeledPoint(label, Vectors.dense(features))
}
data.cache
val numData = data.count

numData: Long = 7395

// note that some of our data contains negative feature vaues. For naive Bayes we convert these to zeros
val nbData = records.map { r =>
    val trimmed = r.map(_.replaceAll("\"", ""))
    val label = trimmed(r.size - 1).toInt
    val features = trimmed.slice(4, r.size - 1).map(d => if (d == "?") 0.0 else d.toDouble).map(d => if (d < 0) 0.0 else d)
    LabeledPoint(label, Vectors.dense(features))
}

2 训练分类模型

2.1 逻辑回归模型

// train a Logistic Regression model
val numIterations = 10
val maxTreeDepth = 5
val lrModel = LogisticRegressionWithSGD.train(data, numIterations)

2.2 SVM模型

val svmModel = SVMWithSGD.train(data, numIterations)

2.3 朴素贝叶斯

val nbModel = NaiveBayes.train(nbData)

2.4 决策树

val dtModel = DecisionTree.train(data, Algo.Classification, Entropy, maxTreeDepth)

3 使用分类模型

3.1 预测

以逻辑回归模型为例

val dataPoint = data.first
val prediction = lrModel.predict(dataPoint.features)

prediction: Double = 1.0

val trueLabel = dataPoint.label

trueLabel: Double = 0.0

val predictions = lrModel.predict(data.map(lp => lp.features))
predictions.take(5)

Array[Double] = Array(1.0, 1.0, 1.0, 1.0, 1.0)

4 评估性能

4.1 逻辑回归模型的正确率

val lrTotalCorrect = data.map { point =>
  if (lrModel.predict(point.features) == point.label) 1 else 0
}.sum
val lrAccuracy = lrTotalCorrect / numData

lrAccuracy: Double = 0.5146720757268425

4.2 SVM模型的正确率

val svmTotalCorrect = data.map { point =>
  if (svmModel.predict(point.features) == point.label) 1 else 0
}.sum
val svmAccuracy = svmTotalCorrect / numData

svmAccuracy: Double = 0.5146720757268425

4.3 朴素贝叶斯的正确率

val nbTotalCorrect = nbData.map { point =>
  if (nbModel.predict(point.features) == point.label) 1 else 0
}.sum
val nbAccuracy = nbTotalCorrect / numData

nbAccuracy: Double = 0.5803921568627451

4.4 决策树的正确率

// decision tree threshold needs to be specified
val dtTotalCorrect = data.map { point =>
  val score = dtModel.predict(point.features)
  val predicted = if (score > 0.5) 1 else 0
  if (predicted == point.label) 1 else 0
}.sum
val dtAccuracy = dtTotalCorrect / numData

dtAccuracy: Double = 0.6482758620689655

4.5 ROC曲线和AUC

val metrics = Seq(lrModel, svmModel).map { model =>
    val scoreAndLabels = data.map { point =>
          (model.predict(point.features), point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (model.getClass.getSimpleName, metrics.areaUnderPR, metrics.areaUnderROC)
}
val nbMetrics = Seq(nbModel).map{ model =>
    val scoreAndLabels = nbData.map { point =>
          val score = model.predict(point.features)
          (if (score > 0.5) 1.0 else 0.0, point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (model.getClass.getSimpleName, metrics.areaUnderPR, metrics.areaUnderROC)
}
val dtMetrics = Seq(dtModel).map{ model =>
    val scoreAndLabels = data.map { point =>
          val score = model.predict(point.features)
          (if (score > 0.5) 1.0 else 0.0, point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (model.getClass.getSimpleName, metrics.areaUnderPR, metrics.areaUnderROC)
}
val allMetrics = metrics ++ nbMetrics ++ dtMetrics
allMetrics.foreach{ case (m, pr, roc) =>
    println(f"$m, Area under PR: ${pr * 100.0}%2.4f%%, Area under ROC: ${roc * 100.0}%2.4f%%")
}

LogisticRegressionModel, Area under PR: 75.6759%, Area under ROC: 50.1418%
SVMModel, Area under PR: 75.6759%, Area under ROC: 50.1418%
NaiveBayesModel, Area under PR: 68.0851%, Area under ROC: 58.3559%
DecisionTreeModel, Area under PR: 74.3081%, Area under ROC: 64.8837%

5 改进和调优

5.1 特征标准化

val vectors = data.map(lp => lp.features)
val matrix = new RowMatrix(vectors)
val matrixSummary = matrix.computeColumnSummaryStatistics()

println(matrixSummary.mean)
println(matrixSummary.min)
println(matrixSummary.max)
println(matrixSummary.variance)
println(matrixSummary.numNonzeros)

[0.4122580529952672,2.761823191986608,0.4682304732861389,0.21407992638350232,0.09206236071899916,0.04926216043908053,2.255103452212041,-0.10375042752143335,0.0,0.05642274498417851,0.02123056118999324,0.23377817665490194,0.2757090373659236,0.615551048005409,0.6603110209601082,30.07707910750513,0.03975659229208925,5716.598242055447,178.75456389452353,4.960649087221096,0.17286405047031742,0.10122079189276552]

[0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.0,0.0,0.0,0.0,0.045564223,-1.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0]

[0.999426,363.0,1.0,1.0,0.980392157,0.980392157,21.0,0.25,0.0,0.444444444,1.0,0.716883117,113.3333333,1.0,1.0,100.0,1.0,207952.0,4997.0,22.0,1.0,1.0]

[0.10974244167559001,74.30082476809639,0.04126316989120241,0.02153343633200108,0.009211817450882448,0.005274933469767946,32.53918714591821,0.09396988697611545,0.0,0.0017177410346628928,0.020782634824610638,0.0027548394224293036,3.683788919674426,0.2366799607085986,0.22433071201674218,415.8785589543846,0.03818116876739597,7.877330081138463E7,32208.116247426184,10.45300904576431,0.03359363403832393,0.006277532884214705]

[5053.0,7354.0,7172.0,6821.0,6160.0,5128.0,7350.0,1257.0,0.0,7362.0,157.0,7395.0,7355.0,4552.0,4883.0,7347.0,294.0,7378.0,7395.0,6782.0,6868.0,7235.0]

val scaler = new StandardScaler(withMean = true, withStd = true).fit(vectors)
val scaledData = data.map(lp => LabeledPoint(lp.label, scaler.transform(lp.features)))

println(data.first.features)
println(scaledData.first.features)
println((0.789131 - 0.41225805299526636)/math.sqrt(0.1097424416755897))

[0.789131,2.055555556,0.676470588,0.205882353,0.047058824,0.023529412,0.443783175,0.0,0.0,0.09077381,0.0,0.245831182,0.003883495,1.0,1.0,24.0,0.0,5424.0,170.0,8.0,0.152941176,0.079129575]

[1.137647336497678,-0.08193557169294771,1.0251398128933331,-0.05586356442541689,-0.4688932531289357,-0.3543053263079386,-0.3175352172363148,0.3384507982396541,0.0,0.828822173315322,-0.14726894334628504,0.22963982357813484,-0.14162596909880876,0.7902380499177364,0.7171947294529865,-0.29799681649642257,-0.2034625779299476,-0.03296720969690391,-0.04878112975579913,0.9400699751165439,-0.10869848852526258,-0.2788207823137022]

1.137647336497682

使用标准化重新训练

val lrModelScaled = LogisticRegressionWithSGD.train(scaledData, numIterations)
val lrTotalCorrectScaled = scaledData.map { point =>
  if (lrModelScaled.predict(point.features) == point.label) 1 else 0
}.sum
val lrAccuracyScaled = lrTotalCorrectScaled / numData
// lrAccuracyScaled: Double = 0.6204192021636241
val lrPredictionsVsTrue = scaledData.map { point =>
    (lrModelScaled.predict(point.features), point.label)
}
val lrMetricsScaled = new BinaryClassificationMetrics(lrPredictionsVsTrue)
val lrPr = lrMetricsScaled.areaUnderPR
val lrRoc = lrMetricsScaled.areaUnderROC

println(f"${lrModelScaled.getClass.getSimpleName}\nAccuracy: ${lrAccuracyScaled * 100}%2.4f%%\nArea under PR: ${lrPr * 100.0}%2.4f%%\nArea under ROC: ${lrRoc * 100.0}%2.4f%%")

LogisticRegressionModel
Accuracy: 62.0419%
Area under PR: 72.7254%
Area under ROC: 61.9663%

5.2 其他特征

val categories = records.map(r => r(3)).distinct.collect.zipWithIndex.toMap
val numCategories = categories.size
val dataCategories = records.map { r =>
    val trimmed = r.map(_.replaceAll("\"", ""))
    val label = trimmed(r.size - 1).toInt
    val categoryIdx = categories(r(3))
    val categoryFeatures = Array.ofDim[Double](numCategories)
    categoryFeatures(categoryIdx) = 1.0
    val otherFeatures = trimmed.slice(4, r.size - 1).map(d => if (d == "?") 0.0 else d.toDouble)
    val features = categoryFeatures ++ otherFeatures
    LabeledPoint(label, Vectors.dense(features))
}

println(dataCategories.first)

(0.0,[0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.789131,2.055555556,0.676470588,0.205882353,0.047058824,0.023529412,0.443783175,0.0,0.0,0.09077381,0.0,0.245831182,0.003883495,1.0,1.0,24.0,0.0,5424.0,170.0,8.0,0.152941176,0.079129575])

标准化转换

// standardize the feature vectors
val scalerCats = new StandardScaler(withMean = true, withStd = true).fit(dataCategories.map(lp => lp.features))
val scaledDataCats = dataCategories.map(lp => LabeledPoint(lp.label, scalerCats.transform(lp.features)))

println(dataCategories.first.features)
println(scaledDataCats.first.features)

[0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.789131,2.055555556,0.676470588,0.205882353,0.047058824,0.023529412,0.443783175,0.0,0.0,0.09077381,0.0,0.245831182,0.003883495,1.0,1.0,24.0,0.0,5424.0,170.0,8.0,0.152941176,0.079129575]

[-0.02326210589837061,2.7207366564548514,-0.4464212047941535,-0.22052688457880879,-0.028494000387023734,-0.2709990696925828,-0.23272797709480803,-0.2016540523193296,-0.09914991930875496,-0.38181322324318134,-0.06487757239262681,-0.6807527904251456,-0.20418221057887365,-0.10189469097220732,1.137647336497678,-0.08193557169294771,1.0251398128933331,-0.05586356442541689,-0.4688932531289357,-0.3543053263079386,-0.3175352172363148,0.3384507982396541,0.0,0.828822173315322,-0.14726894334628504,0.22963982357813484,-0.14162596909880876,0.7902380499177364,0.7171947294529865,-0.29799681649642257,-0.2034625779299476,-0.03296720969690391,-0.04878112975579913,0.9400699751165439,-0.10869848852526258,-0.2788207823137022]

使用扩展后的特征训练逻辑回归模型

val lrModelScaledCats = LogisticRegressionWithSGD.train(scaledDataCats, numIterations)
val lrTotalCorrectScaledCats = scaledDataCats.map { point =>
  if (lrModelScaledCats.predict(point.features) == point.label) 1 else 0
}.sum
val lrAccuracyScaledCats = lrTotalCorrectScaledCats / numData
val lrPredictionsVsTrueCats = scaledDataCats.map { point =>
    (lrModelScaledCats.predict(point.features), point.label)
}
val lrMetricsScaledCats = new BinaryClassificationMetrics(lrPredictionsVsTrueCats)
val lrPrCats = lrMetricsScaledCats.areaUnderPR
val lrRocCats = lrMetricsScaledCats.areaUnderROC

println(f"${lrModelScaledCats.getClass.getSimpleName}\nAccuracy: ${lrAccuracyScaledCats * 100}%2.4f%%\nArea under PR: ${lrPrCats * 100.0}%2.4f%%\nArea under ROC: ${lrRocCats * 100.0}%2.4f%%")

LogisticRegressionModel
Accuracy: 66.5720%
Area under PR: 75.7964%
Area under ROC: 66.5483%

5.3 使用正确的数据格式

使用1-of-k便民店类型特征构建数据集

val dataNB = records.map { r =>
    val trimmed = r.map(_.replaceAll("\"", ""))
    val label = trimmed(r.size - 1).toInt
    val categoryIdx = categories(r(3))
    val categoryFeatures = Array.ofDim[Double](numCategories)
    categoryFeatures(categoryIdx) = 1.0
    LabeledPoint(label, Vectors.dense(categoryFeatures))
}

重新训练贝叶斯模型，并评估性能

val nbModelCats = NaiveBayes.train(dataNB)
val nbTotalCorrectCats = dataNB.map { point =>
  if (nbModelCats.predict(point.features) == point.label) 1 else 0
}.sum
val nbAccuracyCats = nbTotalCorrectCats / numData
val nbPredictionsVsTrueCats = dataNB.map { point =>
    (nbModelCats.predict(point.features), point.label)
}
val nbMetricsCats = new BinaryClassificationMetrics(nbPredictionsVsTrueCats)
val nbPrCats = nbMetricsCats.areaUnderPR
val nbRocCats = nbMetricsCats.areaUnderROC

println(f"${nbModelCats.getClass.getSimpleName}\nAccuracy: ${nbAccuracyCats * 100}%2.4f%%\nArea under PR: ${nbPrCats * 100.0}%2.4f%%\nArea under ROC: ${nbRocCats * 100.0}%2.4f%%")

NaiveBayesModel
Accuracy: 60.9601%
Area under PR: 74.0522%
Area under ROC: 60.5138%

5.4 模型参数调优

5.4.1 线性模型

基础优化技术：随机梯度下降（SGD）

辅助函数：根据输入，训练模型

// helper function to train a logistic regresson model
def trainWithParams(input: RDD[LabeledPoint], regParam: Double, numIterations: Int, updater: Updater, stepSize: Double) = {
    val lr = new LogisticRegressionWithSGD
    lr.optimizer.setNumIterations(numIterations).setUpdater(updater).setRegParam(regParam).setStepSize(stepSize)
    lr.run(input)
}

辅助函数：根据输入数据和分类模型，计算AUC

// helper function to create AUC metric
def createMetrics(label: String, data: RDD[LabeledPoint], model: ClassificationModel) = {
    val scoreAndLabels = data.map { point =>
          (model.predict(point.features), point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (label, metrics.areaUnderROC)
}

迭代次数调优

// cache the data to increase speed of multiple runs agains the dataset
scaledDataCats.cache

// num iterations
val iterResults = Seq(1, 5, 10, 50).map { param =>
    val model = trainWithParams(scaledDataCats, 0.0, param, new SimpleUpdater, 1.0)
    createMetrics(s"$param iterations", scaledDataCats, model)
}
iterResults.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

1 iterations, AUC = 64.97%
5 iterations, AUC = 66.62%
10 iterations, AUC = 66.55%
50 iterations, AUC = 66.81%

步长调优

// step size
val numIterations = 10
val stepResults = Seq(0.001, 0.01, 0.1, 1.0, 10.0).map { param =>
    val model = trainWithParams(scaledDataCats, 0.0, numIterations, new SimpleUpdater, param)
    createMetrics(s"$param step size", scaledDataCats, model)
}
stepResults.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

0.001 step size, AUC = 64.95%
0.01 step size, AUC = 65.00%
0.1 step size, AUC = 65.52%
1.0 step size, AUC = 66.55%
10.0 step size, AUC = 61.92%

使用SquaredL2Updater研究正则化参数

// regularization
val regResults = Seq(0.001, 0.01, 0.1, 1.0, 10.0).map { param =>
    val model = trainWithParams(scaledDataCats, param, numIterations, new SquaredL2Updater, 1.0)
    createMetrics(s"$param L2 regularization parameter", scaledDataCats, model)
}
regResults.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

0.001 L2 regularization parameter, AUC = 66.55%
0.01 L2 regularization parameter, AUC = 66.55%
0.1 L2 regularization parameter, AUC = 66.63%
1.0 L2 regularization parameter, AUC = 66.04%
10.0 L2 regularization parameter, AUC = 35.33%

5.4.2 决策树

辅助函数：接收树的深度和不纯度

// investigate decision tree
def trainDTWithParams(input: RDD[LabeledPoint], maxDepth: Int, impurity: Impurity) = {
    DecisionTree.train(input, Algo.Classification, impurity, maxDepth)
}

使用Entropy不纯度

// investigate tree depth impact for Entropy impurity
val dtResultsEntropy = Seq(1, 2, 3, 4, 5, 10, 20).map { param =>
    val model = trainDTWithParams(data, param, Entropy)
    val scoreAndLabels = data.map { point =>
        val score = model.predict(point.features)
          (if (score > 0.5) 1.0 else 0.0, point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (s"$param tree depth", metrics.areaUnderROC)
}
dtResultsEntropy.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

1 tree depth, AUC = 59.33%
2 tree depth, AUC = 61.68%
3 tree depth, AUC = 62.61%
4 tree depth, AUC = 63.63%
5 tree depth, AUC = 64.88%
10 tree depth, AUC = 76.26%
20 tree depth, AUC = 98.45%

使用Gini不纯度

// investigate tree depth impact for Gini impurity
val dtResultsGini = Seq(1, 2, 3, 4, 5, 10, 20).map { param =>
    val model = trainDTWithParams(data, param, Gini)
    val scoreAndLabels = data.map { point =>
        val score = model.predict(point.features)
          (if (score > 0.5) 1.0 else 0.0, point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (s"$param tree depth", metrics.areaUnderROC)
}
dtResultsGini.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

1 tree depth, AUC = 59.33%
2 tree depth, AUC = 61.68%
3 tree depth, AUC = 62.61%
4 tree depth, AUC = 63.63%
5 tree depth, AUC = 64.89%
10 tree depth, AUC = 78.37%
20 tree depth, AUC = 98.87%

5.4.3 朴素贝叶斯

辅助函数：接收lamda参数

// investigate Naive Bayes parameters
def trainNBWithParams(input: RDD[LabeledPoint], lambda: Double) = {
    val nb = new NaiveBayes
    nb.setLambda(lambda)
    nb.run(input)
}
val nbResults = Seq(0.001, 0.01, 0.1, 1.0, 10.0).map { param =>
    val model = trainNBWithParams(dataNB, param)
    val scoreAndLabels = dataNB.map { point =>
          (model.predict(point.features), point.label)
    }
    val metrics = new BinaryClassificationMetrics(scoreAndLabels)
    (s"$param lambda", metrics.areaUnderROC)
}
nbResults.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.2f%%") }

0.001 lambda, AUC = 60.51%
0.01 lambda, AUC = 60.51%
0.1 lambda, AUC = 60.51%
1.0 lambda, AUC = 60.51%
10.0 lambda, AUC = 60.51%

5.4.4 交叉验证

划分训练集和测试集

// illustrate cross-validation
// create a 60% / 40% train/test data split
val trainTestSplit = scaledDataCats.randomSplit(Array(0.6, 0.4), 123)
val train = trainTestSplit(0)
val test = trainTestSplit(1)

测试集的模型性能

val regResultsTest = Seq(0.0, 0.001, 0.0025, 0.005, 0.01).map { param =>
    val model = trainWithParams(train, param, numIterations, new SquaredL2Updater, 1.0)
    createMetrics(s"$param L2 regularization parameter", test, model)
}
regResultsTest.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.6f%%") }

0.0 L2 regularization parameter, AUC = 66.480874%
0.001 L2 regularization parameter, AUC = 66.480874%
0.0025 L2 regularization parameter, AUC = 66.515027%
0.005 L2 regularization parameter, AUC = 66.515027%
0.01 L2 regularization parameter, AUC = 66.549180%

训练集的模型性能

// training set results
val regResultsTrain = Seq(0.0, 0.001, 0.0025, 0.005, 0.01).map { param =>
    val model = trainWithParams(train, param, numIterations, new SquaredL2Updater, 1.0)
    createMetrics(s"$param L2 regularization parameter", train, model)
}
regResultsTrain.foreach { case (param, auc) => println(f"$param, AUC = ${auc * 100}%2.6f%%") }

0.0 L2 regularization parameter, AUC = 66.260311%
0.001 L2 regularization parameter, AUC = 66.260311%
0.0025 L2 regularization parameter, AUC = 66.260311%
0.005 L2 regularization parameter, AUC = 66.238294%
0.01 L2 regularization parameter, AUC = 66.238294%

文章标签：
Scala
机器学习/深度学习
Apache
分布式计算
Spark
关键词：
人工智能平台 PAI分类模型
人工智能平台 PAI模型
人工智能平台 PAI分类
spark人工智能平台 PAI
apache spark机器学习
六翁
目录
相关文章
LabVIEW开发
|
1天前
|
机器学习/深度学习 监控 算法
LabVIEW使用机器学习分类模型探索基于技能课程的学习
LabVIEW使用机器学习分类模型探索基于技能课程的学习
LabVIEW开发
7 1
shuj
|
2天前
|
机器学习/深度学习 数据采集
构建高效机器学习模型的最佳实践
【5月更文挑战第11天】 在数据驱动的时代背景下，机器学习已经成为企业与研究者解决复杂问题的重要工具。本文将探讨构建高效机器学习模型的关键步骤，包括数据预处理、特征工程、模型选择与调参、以及性能评估。我们将深入分析这些步骤的重要性，并提供实用的技巧和最佳实践，以助读者提高模型的预测能力与泛化性能。通过案例分析和经验总结，本文旨在为从业者提供一套系统的方法论，帮助他们在面对各种机器学习项目时能够更有效地设计和实现解决方案。
shuj
4 0
爱吃糖的范同学
|
2天前
|
机器学习/深度学习 存储 算法
【机器学习】使用贝叶斯模型做分类时，可能会碰到什么问题？怎么解决？
【5月更文挑战第11天】【机器学习】使用贝叶斯模型做分类时，可能会碰到什么问题？怎么解决？
爱吃糖的范同学
2 0
爱吃糖的范同学
|
2天前
|
机器学习/深度学习
【机器学习】噪声数据对贝叶斯模型有什么样的影响？
【5月更文挑战第10天】【机器学习】噪声数据对贝叶斯模型有什么样的影响？
爱吃糖的范同学
29 17
爱吃糖的范同学
|
2天前
|
机器学习/深度学习 数据处理
【机器学习】生成式模型与判别式模型有什么区别？
【5月更文挑战第10天】【机器学习】生成式模型与判别式模型有什么区别？
爱吃糖的范同学
14 5
微生信
|
3天前
|
机器学习/深度学习 人工智能 算法
高性价比发文典范——101种机器学习算法组合革新骨肉瘤预后模型
随着高通量测序技术的飞速发展和多组学分析的广泛应用，科研人员在探索生物学奥秘时经常遇到一个令人又爱又恼的问题：如何从浩如烟海的数据中挖掘出潜在的疾病关联靶点？又如何构建一个全面而有效的诊断或预后模型？只有通过优雅的数据挖掘、精致的结果展示、深入的讨论分析，并且辅以充分的湿实验验证，我们才能锻造出一篇兼具深度与广度的“干湿结合”佳作。
微生信
14 0
高性价比发文典范——101种机器学习算法组合革新骨肉瘤预后模型
aliyun8599273441-30642
|
3天前
|
机器学习/深度学习 数据采集 监控
构建高效机器学习模型的最佳实践
【5月更文挑战第10天】 在面对海量数据和复杂问题时，构建一个既高效又准确的机器学习模型显得至关重要。本文将探讨一系列实用的技术和策略，旨在帮助数据科学家和工程师优化他们的机器学习工作流程。从数据预处理到模型训练，再到最终的评估与部署，我们将深入讨论如何通过最佳实践提升模型性能，同时确保过程的可复现性和可扩展性。
aliyun8599273441-30642
8 4
请看我回答~
|
3天前
|
机器学习/深度学习 数据采集 监控
构建高效机器学习模型的五大技巧
【5月更文挑战第10天】 在数据驱动的时代，机器学习模型的性能往往决定了一个项目的成败。本文将深入探讨如何通过五个关键步骤——数据预处理、特征工程、模型选择、超参数调优以及模型评估与部署，来构建一个高效的机器学习模型。我们将提供实用的技术细节和策略，帮助读者避免常见的陷阱，并提升模型的准确性和泛化能力。无论你是机器学习新手还是有经验的开发者，本文的技巧都将对你构建更健壮、高效的模型大有裨益。
请看我回答~
5 0
游客uul6wusao3sds
|
4天前
|
机器学习/深度学习 算法 异构计算
构建高效机器学习模型的策略与实践
【5月更文挑战第8天】 随着数据科学领域的不断进步，机器学习（ML）已成为解决复杂问题的重要工具。然而，构建一个既高效又准确的ML模型并非易事。本文将详细探讨在设计和训练机器学习模型时可以采用的一系列策略，以优化其性能和效率。我们将讨论特征工程的重要性、选择合适的算法、调整参数以及评估模型的有效性。通过这些策略，读者将能够更好地理解如何提升模型的预测能力并避免常见的陷阱。
游客uul6wusao3sds
27 9
库库的里昂
|
5天前
|
机器学习/深度学习 算法 数据挖掘
【机器学习】各大模型原理简介
【机器学习】各大模型原理简介
库库的里昂
9 2

热门文章

最新文章

  • 1
    图解大数据 | 使用Spark分析新冠肺炎疫情数据@综合案例
  • 2
    BR-MLP基于spark+Hadoop分布式数据挖掘解决方案功能剖析
  • 3
    Spark 源码分析 -- BlockStore
  • 4
    地铁译：Spark for python developers --- 搭建Spark虚拟环境3
  • 5
    Spark RDD类源码阅读
  • 6
    Spark中分布式使用HanLP（1.7.0)分词示例
  • 7
    7月24日晚Spark社区直播：【Apache Spark 基于 Apache Arrow 的列式存储优化】
  • 8
    Spark 分析网站排名热度
  • 9
    MapReduce Shuffle原理 与 Spark Shuffle原理
  • 10
    AnalyticDB MySQL — Spark 助力在OSS上构建低成本数据湖
  • 1
    【AAAI 2024】再创佳绩！阿里云人工智能平台PAI多篇论文入选
    470
  • 2
    阿里云人工智能平台PAI多篇论文入选EMNLP 2023
    453
  • 3
    阿里云人工智能平台 PAI 扩散模型加速采样算法论文入选 CIKM 2023
    36846
  • 4
    构建高效机器学习模型：从数据预处理到模型调优
    10
  • 5
    机器学习的魔法（三）解析无监督学习的黑科技，揭秘新闻话题背后的神奇算法
    26
  • 6
    机器学习的魔法（二）超越预测的界限-揭秘机器学习的黑科技-探索监督学习中的回归和分类问题
    138
  • 7
    机器学习的魔法（一）从零开始理解吴恩达的精炼笔记
    37
  • 8
    MAC系统机器学习环境配置常见问题
    33
  • 9
    构建高效机器学习模型：从数据预处理到模型优化
    34
  • 10
    构建高效机器学习模型：从数据预处理到模型调优
    41

    • 相关课程

    更多
  • 场景实践 - 机器学习PAI实现精细化营销
  • 场景实践 - 基于阿里云PAI机器学习平台使用时间序列分解模型预测商品销量
  • 场景实践 - 基于机器学习进行收入预测分析
  • 机器学习概览及常见算法
  • 机器学习入门-概念原理及常用算法
  • 大数据实时计算框架Spark快速入门

    • 相关电子书

    更多
  • Hybrid Cloud and Apache Spark
  • Scalable Deep Learning on Spark
  • Comparison of Spark SQL with Hive

    • 相关实验场景

    更多
  • 基于Hologres+PAI+计算巢，5分钟搭建企业级AI问答知识库
  • Alibaba Cloud Linux操作系统Shell程序
    • 下一篇
    部署LAMP环境（Alibaba Cloud Linux 3）