Using Pandas To Add List Elements Together

I have the following array of dicts: items = [ { 'FirstName': 'David', 'Language': ['en',] }, { 'FirstName': 'David', 'Language': ['fr',] }, { 'FirstName': 'D

Solution 1:

Aggregate all with sum, transform values to set and then to_dict()

>>> df.groupby('FirstName').sum()["Language"].transform(set).reset_index().to_dict(orient='records')

[{'FirstName': 'Bob', 'Language': {'en'}},
 {'FirstName': 'David', 'Language': {'en', 'fr'}}]

Solution 2:

You can use:

df.groupby('FirstName').sum().applymap(lambda s: list(set(s)))

applymap will apply a function to every element, mapping a list to a de-duplicated list.

In [185]: df = pd.DataFrame(items)

In [186]: df
Out[186]: 
  FirstName Language
0     David     [en]
1     David     [fr]
2     David     [en]
3       Bob     [en]

In [187]: df.groupby('FirstName').sum().applymap(lambda s: list(set(s)))
Out[187]: 
           Language
FirstName          
Bob            [en]
David      [en, fr]

Solution 3:

Use:

df.groupby('FirstName')['Language']\
  .apply(lambda x: list(set(x.str[0])))\
  .reset_index().to_dict('r')

Output:

[{'FirstName': 'Bob', 'Language': ['en']},
 {'FirstName': 'David', 'Language': ['fr', 'en']}]

Create a set our of the first element of the list in each group.

Timings

%timeit df.groupby('FirstName').sum().applymap(lambda s: list(set(s)))

7.31 ms ± 534 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df.groupby('FirstName')['Language'].apply(lambda x: list(set(x.str[0]))).to_frame()

2.09 ms ± 207 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Solution 4:

from more_itertools import flatten
df.groupby('FirstName')['Language'].apply(lambda x: list(set(flatten([i for i in x]))))
.reset_index()
.to_dict(orient='records')

[{'FirstName': 'Bob', 'Language': ['en']},
 {'FirstName': 'David', 'Language': ['en', 'fr']}]

I assume you kept getting unhashable type: 'list' so I simply use a list comprehension to get the value from it and then flatten it since it ended up looking like this after just a list comprehension:

FirstName
Bob                  [[en]]
David    [[en], [fr], [en]]