[BUG] Unable to update array column for subset of rows

[dagardner-nv]

Describe the bug
Unable to update values of a dataframe for a column of type array for a subset of rows via .loc with a list[list].

Tested with versions: 24.02.02 & nightly 24.04.00a508

Updating the values for all rows works:

df['a'] = [[0,0,0], [9, 10, 11], [20, 21, 22]]

however updating a subset of rows with a bool mask fails:

df.loc[mask, 'a'] = new_values

I believe at least part of the problem is (python/cudf/cudf/core/column/lists.py) ~line 90:

    def __setitem__(self, key, value):
        if isinstance(value, list):
            value = cudf.Scalar(value)
        if isinstance(value, cudf.Scalar):
            if value.dtype != self.dtype:
                raise TypeError("list nesting level mismatch")

When the column is ListDtype(int64) and the incoming values look like [[9, 10, 11], [20, 21, 22]]
It will create a cudf.Scalar which will have a dtype of ListDtype(ListDtype(int64)) .

Causing the value.dtype != self.dtype to fail.

Steps/Code to reproduce bug

import os

import cupy
import numpy
import pandas as pd
import cudf

data = {'apple': ['pie', 'cake', 'candy'], 'a': [[3,2,1], [4,5,6], [8,7,9]], 'b': [10, 20, 30]}


if os.environ.get('USE_PANDAS') != None:
    df = pd.DataFrame(data)
else:
    df = cudf.DataFrame(data)

mask = [False, True, True]
new_values = [[9, 10, 11], [20, 21, 22]]

# Other datatypes work
df.loc[mask, 'b'] = [25, 35]
df['b'].loc[mask] = [26, 36]
print(df)

print(f"array column type: {repr(df['a'].dtype)}", flush=True)

try:
    df.loc[mask, 'a'] = new_values
except ValueError as e:
    print(f"Encoundered error setting new values ({e}) trying another way")

    try:
        df['a'].loc[mask] = new_values
    except TypeError as e:
        print(f"Encoundered error setting new values ({e})")

print(df)

Expected behavior
Update the values

Environment overview (please complete the following information)

• Environment location: [Bare-metal]
• Method of cuDF install: [conda]

[wence-]

Related #11944 and #13291. I think the workaround I posted in #11944 (comment) should work, but I don't like it

[dagardner-nv]

@wence- Thanks, the work-around in #11944 works for me. Feel free to close this issue as a duplicate.