BUG: `assign` of `subfunctions` do not work as expected before op2.par_loop when run in parallel about firedrake

[dham]

What does n.dat.data_ro look like in each case?

[lrtfm]

The output of print(n.dat.data_ro) are same

(array([0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([[ 0.30901699, -0.80901699,  0.5       ],
       [-0.30901699, -0.80901699,  0.5       ],
       [ 0.        , -0.52573111,  0.85065081],
       [ 0.5       , -0.30901699,  0.80901699],
       [-0.5       , -0.30901699,  0.80901699],
       [-0.80901699, -0.5       ,  0.30901699],
       [-0.85065081,  0.        ,  0.52573111],
       [ 0.5       ,  0.30901699,  0.80901699],
       [ 0.        ,  0.        ,  1.        ],
       [ 0.        ,  0.52573111,  0.85065081],
       [-0.5       ,  0.30901699,  0.80901699],
       [-1.        ,  0.        ,  0.        ],
       [ 0.30901699,  0.80901699,  0.5       ],
       [-0.80901699,  0.5       ,  0.30901699],
       [-0.30901699,  0.80901699,  0.5       ],
       [ 0.        , -1.        ,  0.        ],
       [ 0.52573111, -0.85065081,  0.        ],
       [-0.52573111, -0.85065081,  0.        ],
       [ 0.80901699, -0.5       ,  0.30901699],
       [ 0.85065081,  0.        ,  0.52573111],
       [-0.80901699, -0.5       , -0.30901699],
       [-0.85065081,  0.        , -0.52573111],
       [ 0.80901699,  0.5       ,  0.30901699],
       [ 0.52573111,  0.85065081,  0.        ],
       [-0.52573111,  0.85065081,  0.        ],
       [-0.80901699,  0.5       , -0.30901699],
       [ 0.        ,  1.        ,  0.        ]]))
(array([0., 0., 0.]), array([[ 0.30901699,  0.80901699, -0.5       ],
       [-0.30901699,  0.80901699, -0.5       ],
       [ 0.        ,  0.52573111, -0.85065081],
       [-0.5       ,  0.30901699, -0.80901699],
       [ 0.80901699,  0.5       , -0.30901699],
       [ 0.5       ,  0.30901699, -0.80901699],
       [ 0.85065081,  0.        , -0.52573111],
       [ 0.5       , -0.30901699, -0.80901699],
       [ 0.        ,  0.        , -1.        ],
       [ 0.        , -0.52573111, -0.85065081],
       [-0.5       , -0.30901699, -0.80901699],
       [ 1.        ,  0.        ,  0.        ],
       [ 0.30901699, -0.80901699, -0.5       ],
       [ 0.80901699, -0.5       , -0.30901699],
       [-0.30901699, -0.80901699, -0.5       ]]))

[connorjward]

What about n.subfunctions[1].dat.halo_valid in both cases? This could be an issue with some of the tricks I used in assign to avoid halo exchanges. I may not have given full enough consideration to halo validity for mixed dats.

[ksagiyam]

I think this feature in undertested as Firedrake assembles block-by-block. For instance, do we not need MixedDatKernelArg here https://github.com/OP2/PyOP2/blob/da14715cca2d2174d6f5554444e84f1a5d92cdf9/pyop2/parloop.py#L317 to have halo exchanges happen?

[lrtfm]

What about n.subfunctions[1].dat.halo_valid in both cases? This could be an issue with some of the tricks I used in assign to avoid halo exchanges. I may not have given full enough consideration to halo validity for mixed dats.

The values are all False in both cases.

[connorjward]

I think this feature in undertested as Firedrake assembles block-by-block. For instance, do we not need MixedDatKernelArg here https://github.com/OP2/PyOP2/blob/da14715cca2d2174d6f5554444e84f1a5d92cdf9/pyop2/parloop.py#L317 to have halo exchanges happen?

Great spot! This does look like it could be responsible.

[connorjward]

I think this feature in undertested as Firedrake assembles block-by-block. For instance, do we not need MixedDatKernelArg here https://github.com/OP2/PyOP2/blob/da14715cca2d2174d6f5554444e84f1a5d92cdf9/pyop2/parloop.py#L317 to have halo exchanges happen?

Great spot! This does look like it could be responsible.

Turns out that this is a little trickier to change than I thought it would be. Would it be OK to mark this issue as "wontfix"? The issue appears to stem from the fact that MixedDat and Dat have slightly different implementations and in pyop3 I make no distinction between the two, resulting in a single code path, so the issue should go away.

On second thought, I don't think that would be the right thing to do. We should at least find out why this is happening (and perhaps submit an xfailing test).

[connorjward]

This is fixed by OP2/PyOP2#710