spark rdd median 中位数求解

+关注继续查看

lookup(key)

Return the list of values in the RDD for key key. This operation is done efficiently if the RDD has a known partitioner by only searching the partition that the key maps to.

>>> l = range(1000)
>>> rdd = sc.parallelize(zip(l, l), 10)
>>> rdd.lookup(42)  # slow
[42]
>>> sorted = rdd.sortByKey()
>>> sorted.lookup(42)  # fast
[42]
>>> sorted.lookup(1024)
[]
>>> rdd2 = sc.parallelize([(('a', 'b'), 'c')]).groupByKey()
>>> list(rdd2.lookup(('a', 'b'))[0])
['c']

You need to sort RDD and take element in the middle or average of two elements. Here is example with RDD[Int]:

  import org.apache.spark.SparkContext._

val rdd: RDD[Int] = ???

val sorted = rdd.sortBy(identity).zipWithIndex().map {
case (v, idx) => (idx, v)
}

val count = sorted.count()

val median: Double = if (count % 2 == 0) {
val l = count / 2 - 1
val r = l + 1

all_data = sc.parallelize([25,1,2,3,4,5,6,7,8,100])
all_data.sortBy(lambda x:x).zipWithIndex().map(lambda x: (x[1],x[0])).collect
[(0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 25), (9, 100)]

10092 0

12077 0

13893 0

11893 0

4510 0
+关注
4269

0

《2021云上架构与运维峰会演讲合集》

《零基础CSS入门教程》

《零基础HTML入门教程》