>>> rdd = sc.parallelize([("a", "1"), ("b", 1), ("a", 1), ("a", 1)]) >>> rdd.distinct().countByKey().items() [('a', 2), ('b', 1)] OR: from operator import add
rdd.distinct().map(lambda x: (x[0], 1)).reduceByKey(add)
rdd.distinct().keys().map(lambda x: (x, 1)).reduceByKey(add)
distinct(numPartitions=None)
Return a new RDD containing the distinct elements in this RDD.
>>> sorted(sc.parallelize([1, 1, 2, 3]).distinct().collect()) [1, 2, 3]
countByKey()
Count the number of elements for each key, and return the result to the master as a dictionary.
>>> rdd = sc.parallelize([("a", 1), ("b", 1), ("a", 1)]) >>> sorted(rdd.countByKey().items()) [('a', 2), ('b', 1)]
本文转自张昺华-sky博客园博客，原文链接：http://www.cnblogs.com/bonelee/p/7155153.html，如需转载请自行联系原作者
