[FEA] Support collect_list

[beckernick]

Similar to #2973 , I'd like to be able to collect_list on grouped objects like in Spark-sql or pandas with a lambda function. Spark API doc.

Groupby examples:

Pandas:

import pandas as pd
​
df = pd.DataFrame()
df['col1'] = [0,1,0,1,0]
df['col2'] = [12,42,12,93,38]
​
res = df.groupby("col1").agg({
    "col2": lambda x: list(x)
})
print(res)
              col2
col1              
0     [12, 12, 38]
1         [42, 93]

Pyspark:

import pandas as pd
​
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import functions as F
​
sc = SparkContext()
sqlContext = SQLContext(sc)
​
df = pd.DataFrame()
df['col1'] = [0,1,0,1,0]
df['col2'] = [12,42,12,93,38]
​
sdf = sqlContext.createDataFrame(df)
res = sdf.groupby("col1").agg(F.collect_list("col2"))
print(res.show())
+----+------------------+
|col1|collect_list(col2)|
+----+------------------+
|   0|      [12, 12, 38]|
|   1|          [42, 93]|
+----+------------------+

Non-Groupby

Pyspark:

import pandas as pd
​
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import functions as F
​
sc = SparkContext()
sqlContext = SQLContext(sc)
​
df = pd.DataFrame()
df['col1'] = [0,1,0,1,0]
df['col2'] = [12,42,12,93,38]
​
sdf = sqlContext.createDataFrame(df)
res = sdf.agg(F.collect_list("col2"))
print(res.show())
+--------------------+
|  collect_list(col2)|
+--------------------+
|[12, 42, 12, 93, 38]|
+--------------------+

[beckernick]

@shwina can we now close this issue after #5874 . I believe #2973 cannot be closed yet

[shwina]

Yes - we support collecting into list via list but not collecting into sets. That will require libcudf support. cc: @harrism @kkraus14

[kkraus14]

Yup, closing this as this is implemented.