pyspark-session-2021-05-05-join.txt

PySpark Documentation: Join function in PySpark:
http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.RDD.join.html

$ ./bin/pyspark
Python 3.7.2 (v3.7.2:9a3ffc0492, Dec 24 2018, 02:44:43)

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.1
      /_/

Using Python version 3.7.2 (v3.7.2:9a3ffc0492, Dec 24 2018 02:44:43)
Spark context Web UI available at http://10.0.0.93:4040
Spark context available as 'sc' (master = local[*], app id = local-1620269740798).
SparkSession available as 'spark'.
>>>
>>> x = spark.sparkContext.parallelize([("spark", 1), ("hadoop", 4)])
>>> x.collect()
[
 ('spark', 1), 
 ('hadoop', 4)
]
>>>
>>> y = spark.sparkContext.parallelize([("spark", 2), ("hadoop", 5)])
>>> y.collect()
[
 ('spark', 2), 
 ('hadoop', 5)
]
>>>
>>> joined = x.join(y)
>>> joined.collect()
[
 ('spark', (1, 2)), 
 ('hadoop', (4, 5))
]
>>>
>>>
>>> x = spark.sparkContext.parallelize([("a", 1), ("b", 4), ("c", 4)])
>>> x.collect()
[('a', 1), ('b', 4), ('c', 4)]
>>> y = spark.sparkContext.parallelize([("a", 2), ("a", 3), ("a", 7), ("d", 8)])
>>> y.collect()
[('a', 2), ('a', 3), ('a', 7), ('d', 8)]
>>>
>>> joined = x.join(y)
>>> joined.collect()
[('a', (1, 2)), ('a', (1, 3)), ('a', (1, 7))]
>>>
>>>
>>> joined.count()
3
>>> x = spark.sparkContext.parallelize([("a", 1), ("b", 4), ("b", 5), ("c", 4)]);
>>> x.collect()
[('a', 1), ('b', 4), ('b', 5), ('c', 4)]
>>>
>>> y = spark.sparkContext.parallelize([("a", 2), ("a", 3), ("a", 7), ("b", 61), ("b", 71), ("d", 8)])
>>> y.collect()
[('a', 2), ('a', 3), ('a', 7), ('b', 61), ('b', 71), ('d', 8)]
>>> joined = x.join(y)
>>> joined.collect()
[
 ('b', (4, 61)), 
 ('b', (4, 71)), 
 ('b', (5, 61)), 
 ('b', (5, 71)), 
 ('a', (1, 2)), 
 ('a', (1, 3)), 
 ('a', (1, 7))
]
>>>
>>>#pyspark.RDD.cartesian
>>>#RDD.cartesian(other)
>>>#Return the Cartesian product of this RDD and another one, 
>>>#that is, the RDD of all pairs of elements (a, b) where a is 
>>>#in self and b is in other.
>>># Examples

>>>
>>> rdd = spark.sparkContext.parallelize([1, 2])
>>> sorted(rdd.cartesian(rdd).collect())
[(1, 1), (1, 2), (2, 1), (2, 2)]