i.sentinel.mask: returncode error about grass-addons

[veroandreo]

Sounds good to me! Thanks for such a prompt response and all the explanations :)

[RobiFag]

H vero,
I have just tested i.sentinel.mask using your image and the process ended without errors.

Steps:

1. Download the image using i.sentinel.download
2. Import into a WGS84/UTM21 location using r.import
3. Apply cloud mask

i.sentinel.mask -s 
blue=T21JYM_20200126T134201_B02_10m@test_vero 
green=T21JYM_20200126T134201_B03_10m@test_vero 
red=T21JYM_20200126T134201_B04_10m@test_vero 
nir=T21JYM_20200126T134201_B08_10m@test_vero 
nir8a=T21JYM_20200126T134201_B8A_20m@test_vero 
swir11=T21JYM_20200126T134201_B11_20m@test_vero 
swir12=T21JYM_20200126T134201_B12_20m@test_vero 
cloud_mask=v_cloud 
cloud_raster=r_cloud 
shadow_mask=v_shadow 
shadow_raster=r_shadow 
cloud_threshold=50000 
shadow_threshold=10000 
mtd_file=S2A_MSIL2A_20200126T134201_N0213_R124_T21JYM_20200126T154908\S2A_MSIL2A_20200126T134201_N0213_R124_T21JYM_20200126T154908.SAFE\GRANULE\L2A_T21JYM_A024002_20200126T134204\MTD_TL.xml

System description:

• Operating System: Windows8
• GRASS GIS version 7.8.2 (2019) - installed using OSGeo4W 64bit installer
• PROJ: 6.2.1
• GDAL/OGR: 3.0.2
• GEOS: 3.8.0
• SQLite: 3.29.0
• Python: 3.7.0

[veroandreo]

Yes, I forgot to mention (actually I didn't think it was relevant then) that the error arises when setting the computational region to a subset of the full scene. If I, like you, run the process for the full scene, then it finishes without problems.

Do you think the reported problem is just something particular to that region subset or something more general for the module that won't work for scene subsets?

[ninsbl]

Just a wild guess: The problem might be that there are no clouds or shaddows in the smaller region?

@veroandreo : could you try to delete , stderr=subprocess.DEVNULL from https://github.com/OSGeo/grass-addons/blob/master/grass7/imagery/i.sentinel/i.sentinel.mask/i.sentinel.mask.py#L561 ?
Then you should get the full error output.

[RobiFag]

I have just tested the module setting as region and it works fine. Is it possible that you choose a region in which the module does not detect clouds or more probably shadows?
My region is:

projection: 1 (UTM)
zone:       -21
datum:      wgs84
ellipsoid:  wgs84
north:      7159480
south:      7119380
west:       708540
east:       773140
nsres:      20
ewres:      20
rows:       2005
cols:       3230
cells:      6476150

[veroandreo]

Just a wild guess: The problem might be that there are no clouds or shadows in the smaller region?

Actually, the other way around... Within the subset there was only one shadow (red) identified and a couple of clouds (blue) with their respective shadows that were missed (area in the square).

This is the region setting I was using (Iguazú falls area in case you wonder :))

g.region -p n=7169584.87 w=739426.81 e=761177.37 s=7152969.62 align=`g.list type=raster pattern="*B02_1*"`

@RobiFag could such cases be considered somehow easily within the module?

[RobiFag]

So only a shadow is detected in your computational region and no clouds.
If I remember correctly, I had already added some check about the area of clouds and shadows detected but you should get a clearer message (e.g 'No clouds have been detected').
Maybe something has changed in the code @ninsbl ? (e.g. a82876e#diff-86a1994a46407a9b6bd9d1b786d7869fL378-R389 )

[RobiFag]

Maybe I have understood..in Vero's case some clouds are detected but are then removed by the cleaning geometries procedure (remove small areas). Actually there was a check in the code about the detected clouds but it considered only the temporary vector cloud map and not the cleaned one. Moreover the if statement regarding the cloud check was removed (a82876e#diff-86a1994a46407a9b6bd9d1b786d7869fL378-R389)

This is a possible code side solution:

else:
        gscript.message(_('--- Cleaning geometries ---'))
        gscript.run_command('v.clean',
            input=tmp["cloud_v"],
            output=cloud_mask,
            tool='rmarea',
            threshold=cloud_threshold)
        info_c_clean = gscript.parse_command('v.info',
            map=cloud_mask,
            flags='t')
        if info_c_clean['areas'] == '0':
            gscript.warning(_('No clouds have been detected'))
            check_cloud = 0
        else:
            check_cloud = 1
    gscript.message(_('--- Finish cloud detection procedure ---'))
    # End of Clouds detection

    if (options['shadow_mask'] or options['shadow_raster']) and check_cloud == 1:

In this way Vero's case ends without error but with the warning message 'no clouds detected

WARNING: All subsequent operations will be limited to the current computational region
WARNING: Any rescale factor has been applied
--- Start computing maximum values of bands ---
T21JYM_20200126T134201_B02_10m_double@test_vero
T21JYM_20200126T134201_B03_10m_double@test_vero
T21JYM_20200126T134201_B04_10m_double@test_vero
T21JYM_20200126T134201_B08_10m_double@test_vero
T21JYM_20200126T134201_B8A_20m_double@test_vero
T21JYM_20200126T134201_B11_20m_double@test_vero
T21JYM_20200126T134201_B12_20m_double@test_vero
--- Computed maximum value: dict_values([0.7832, 0.8584, 0.8288, 1.0008, 0.7023, 0.9381, 1.3597]) ---
--- Statistics have been computed! ---
--- Start clouds detection procedure ---
--- Computing cloud mask... ---
--- Converting raster cloud mask into vector map ---
Estrazione delle aree in corso ...
Writing areas...
Si sta ricreando la topologia per il vettore <cloud_v_6924@test_vero>...
Registrando primitive...
Costruendo aree...
Isole allegate...
Centroidi allegati...
r.to.vect completo.
--- Cleaning geometries ---
--------------------------------------------------
Tool: Threshold
Remove small areas: 50000
--------------------------------------------------
Copying features...
Sto ricostruendo parti della topologia...
Si sta ricreando la topologia per il vettore <v_cloud_new3@test_vero>...
Registrando primitive...
Costruendo aree...
Isole allegate...
Centroidi allegati...
--------------------------------------------------
Tool: Remove small areas
654 areas of total size 314723 removed
--------------------------------------------------
Sto ricostruendo la topologia per la mappa vettoriale di output...
Si sta ricreando la topologia per il vettore <v_cloud_new3@test_vero>...
Registrando primitive...
WARNING: No clouds have been detected
--- Finish cloud detection procedure ---
WARNING: No shadow mask will be computed

Maybe you can try with smaller cloud and shadow thresholds!

[veroandreo]

[skipping the rest]

WARNING: No clouds have been detected
--- Finish cloud detection procedure ---
WARNING: No shadow mask will be computed

Why no shadow mask computed? What if, like in this case, only cloud shadow is present in the scene or region subset? Are shadows dependent on finding clouds first then?

Maybe you can try with smaller cloud and shadow thresholds!

Right! I have not modified them. Which is the unit of these thresholds? Square meters? If yes, we could add that in the description of the options to make it clearer

[RobiFag]

The shadow mask, in this case, is not computed since the cloud mask is mandatory for shadow mask creation. In fact, the cloud map is used during the cleaning phase of the shadow mask in order to remove misclassifications (see the manual for further details about the algorithm of cloud and shadow detection).

Regarding the thresholds, the unit is square meters (v.clean - tool=rmarea).

[RobiFag]

I have already made some changes to the code (see #86 ). Now the computation of the shadow mask is independent of the clouds mask. If no clouds are detected, the shadow mask is computed but the procedure of cleaning misclassifications from shadow is not performed.

[ninsbl]

Can the ticket be closed with this PR merged:
#86
?

[veroandreo]

Can the ticket be closed with this PR merged:
#86
?

Yes, I think so. Thanks again both of you 😊