GTiff/GeoTIFF driver uses wrong metada from files with simillar names about gdal

[BishopGIS]

This is not normal GeoTIFF file, but DIMAP. You need to open not tiff file in gdalinfo, but XML (in attached archive some files are absent).

$ gdalinfo DIM_PHR1B_PMS-N_201708210848215_SEN_2471996101-001.XML
ERROR 3: Cannot open file 'LINEAGE/STRIP_DS_PHR1B_201708210848215_FR1_PX_E037N55_0811_04356_DIM.XML'
ERROR 4: IMG_PHR1B_PMS-N_201708210848215_SEN_2471996101-001_R1C1.JP2: No such file or directory
gdalinfo failed - unable to open 'DIM_PHR1B_PMS-N_201708210848215_SEN_2471996101-001.XML'.

You can see the example of correct DIMAP dataset in autotests: https://github.com/OSGeo/gdal/tree/master/autotest/gdrivers/data/dimap2

[rouault]

I've just improved the documentation of GDALDriver::Delete() to be

 * The driver will attempt to delete the named dataset in a driver specific
 * fashion.  Full featured drivers will delete all associated files,
 * database objects, or whatever is appropriate.  The default behaviour when
 * no driver specific behaviour is provided is to attempt to delete all the
 * files that are returned by GDALGetFileList() on the dataset handle.

In that instance the XML files are seen as 'side-car' files of the GeoTIFF handle and thus destroyed at the same time. If you just want to delete the .tif file, you can use plain VSIUnlink()

[an-ivanov]

I know, that there is DIMAP.
That is the issue! The tif and the xml files from DIMAP are different datasets and do not have common data except some common part in names. But the names are different. The tif file has an extra suffix _ql16 in comparison with xml file. And the issue is that GDAL links the xml files (even with different names) to the tif dataset!

[rouault]

@an-ivanov Can you confirm that the DIM_PHR1B_PMS-N_201708210848215_SEN_2471996101-001_ql16.tif file is part of the original dataset ?
@BishopGIS The driver uses the sidecar files (per gcore/mdreader/reader_pleiades.cpp), but perhaps that's inappropriate for a ql16 file ? I see this file is in a Krassovsky Transverse Mercator, but also exposes the RPC from the RPC.xml file, which is indeed inappropriate (the file cannot be 'raw' and orthorectified at the same time)

[BishopGIS]

It seems to me that mixing DIMAP dataset files and other raster files is undefined behaviour. This is user responsibility not to "damage" datasets such way.
There are satellite or UAV imagery products with predefined file structure. In that case metadata reader works. When this structure is modified by user it's hard to split which files belong to dataset and which not.
@rouault What about RPC - for normal dataset I agree - this is inappropriate, but if dataset was modified or damaged - anything may be.

[an-ivanov]

The main question, why the DIMAP files, with names differing from tif file, are used as auxiliary for the tif?
I just manually generated a quicklook with additional suffix _ql16. So I didn't replace or remove anything from the dataset folder. But when I tried remove the quicklook with GDALDeleteDataset, the XML files from the original dataset were removed as well and the DIMAP dataset was corrupted .
Don't you think, that it's strange behavior of the GDAL library?
More strange that for example suffixes -ql16 or .ql16 for the tif file do not lead to this.
I guess there is a kind of bug in filename comparison algo for searching auxiliary files.
Moreover it's strange, why GeoTIFF driver tries to read DIMAP dataset files.

[rouault]

@BishopGIS Would there be a way to restrict the Pleiades MDReader to only trigger on .tif files that are part of the original dataset, or is there just too much possible suffixes ?
Otherwise I agree we can't do much about that, and as @an-ivanov found there are naming strategies that prevent the Pleiades MDReader to believe that the custom file is an official DIMAP file.

[an-ivanov]

I guess all rasters of a DIMAP dataset could be found in DIM under the following section:
<Raster_Data> <Data_Access> <Data_Files>

[an-ivanov]

At least suffixes for original names could be checked more precisely with pattern similar to _R%dC%d

[BishopGIS]

I only found for Pléiades (DIMAP v2) that tiled raster has name addition for it tiles with this pattern: IMG_<ProductMOS_ID>_RjCj.<jp2|tiff>
http://www.cscrs.itu.edu.tr/assets/downloads/PleiadesUserGuide.pdf (see page 51).

So it can be fixed following way: look for metadata file DIM_<ProductMOS_ID>.XML for IMG_<ProductMOS_ID>.<jp2|tiff> and IMG_<ProductMOS_ID>_RjCj.<jp2|tiff> files only.
This is not guarantee that user not create such GeoTiff image ended R0C0.tiff manually in product folder but solve this case.

Another fix may be full file name compare. I think this also fixes the case there the whole RPC applied to a single tile.

What option is more suitable? I like the second.

[an-ivanov]

Of course full name comparison is more preferable.

[rouault]

Another fix may be full file name compare

Not sure what you mean exactly here

[BishopGIS]

@rouault I mean to search metadata file (DIM_<ProductMOS_ID>.XML) only for such image IMG_<ProductMOS_ID>.<jp2|tiff> without any additions like _<4 any chars>..
Drop this part: https://github.com/OSGeo/gdal/blob/master/gdal/gcore/mdreader/reader_pleiades.cpp#L68-L78 and connected checks.

[rouault]

Drop this part: https://github.com/OSGeo/gdal/blob/master/gdal/gcore/mdreader/reader_pleiades.cpp#L68-L78 and connected checks.

That would be restrictive for tiled products. GDALMDReaderPleiades::LoadRPCXmlFile() is currently able to deal with _RxCy files to adjust the RPC values to be correct for each GeoTIFF tile.

[BishopGIS]

Ok, so the option one - check only _RjCj pattern instead of _<4 any chars>.