Pivot Row To Column Level

I have a spark dataframe t which is the result of a spark.sql('...') query. Here is the first few rows from t: | yyyy_mm_dd | x_id | x_name | b_app | status | has_pol

Solution 1:

Assuming df is your dataframe. pivot is quite straight forward to use when you read the doc.

df.groupBy(
    "yyyy_mm_dd", "x_id", "x_name", "b_app", "status"
).pivot("has_policy", [0, 1]).sum("count")