data loss on getmap requests when vector data is reprojected EPSG:2056 to EPSG:4326 about mapserver

[geographika]

@ltclm - thanks for providing a fully working test case. I can confirm the issue still exists in the MapServer main branch.
The issue is that the Map extent is modified at various points in the WMS code, and then the map drawing code. At some point the search rectangle used for selecting features is moved from the extent passed in the original request.

This can be seen in the image below. The orange area is the original request extent, and the pink is the extent used to select the underlying features. As you can see in the image only the lower two features intersect, and so are the only features drawn.

It is harder to pin down which of the various modifications to the extent is the cause of the shift, and why this may be necessary.

[geographika]

This appears to be a projection issue, and possibly related to #6479 and #6478.

Setting int apply_over = MS_FALSE; in the following code brings back the correct results:

MapServer/src/mapproject.c

Line 1902 in da769fc

int apply_over = MS_TRUE;

[ltclm]

thank you @geographika for your response, the insights and the link to the related issues 👍
i can confirm that changing the projection of the layer from

"init=epsg:2056"

to

"+proj=somerc +lat_0=46.9524055555556 +lon_0=7.43958333333333 +k_0=1 +x_0=2600000 +y_0=1200000 +ellps=bessel +towgs84=674.374,15.056,405.346,0,0,0,0 +units=m +no_defs +type=crs"

as proposed here #6478 (comment) is fixing the projection issue for the reprojected getmap requests.

But with the new projection settings at layer level some getmap requests in the native data projection are not working anymore and an empty map is returned:

service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: CGI Request 4 on process 10107
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): rendering using outputformat named jpeg (AGG/JPEG).
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): WMS/WFS set-up and query, 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): Layer 0 (test), 0.013s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): Drawing Label Cache, 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap() total time: 0.013s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msSaveImage(stdout) total time: 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: mapserv request processing time (msLoadMap not incl.): 0.014s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msFreeMap(): freeing map at 0x560b568291d0.
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: freeLayer(): freeing layer at 0x560b56831e30.
service-wms_1  | 172.24.0.1 - - [12/Feb/2024:16:26:55 +0000] "GET /local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image/jpeg&TRANSPARENT=false&LAYERS=test&WIDTH=256&HEIGHT=256&CRS=EPSG:2056&STYLES=&TIME=current&BBOX=2690771.2,1285872.0,2690796.8,1285897.6 HTTP/1.1" 200 993

[mizzPpY]

Hi @ltclm ,
this comment is not related to the original issue, but I would like to know how does your mapserver dockerfile and deployment file looks like?

I was using camptocamp/mapserver:7.4 image deploy on kubernetes cluster, it was working fine but encounter issue updating to 8.0. When curl http://localhost it is returning status 500 instead of the usual response "No query information to decode. QUERY_STRING is set, but empty.".

Had been looking everywhere what is causing the upgrade failure but couldn't find any answer, since I saw you using the same image and able to at least get something, I hope you can provide some insight for me.
Thank you.

[oshawa-connection-jacobs]

Just to hopefully add another data point, this is happening to me as well for datasets in web mercator and other projected coordinate systems, requesting the map and getFeatureInfo in web mercator. Mapserver 8.0.1, PROJ 9.4.0.

Expanding the Projection from "EPSG:3857" to ""+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs +type=crs" doesn't seem to have an effect.

@geographika Do you know if anyone is actively working on this/ it has already been fixed? If not, I would be very motivated to take a look myself.

[geographika]

@oshawa-connection-jacobs - I'm not aware of anyone actively looking at this, and there are no relevant commits to mapproject.c that aren't already in the 8.0.1 release (if this is where the issue lies). It would be great if you can provide any more insight into the issue.

[ltclm]

a fix of this would be highliy appreciated by us too. we are currently running a custom built of mapserver 7.6.5 where we have set int apply_over = MS_FALSE (as proposed as workaround in this issue).

This seems to solve the disappearing features in reprojected getmap requests. But requests lead from time to time to core dumps, especially under load. So the requests or the pattern is hard to reproduce and share. we had to disable the core dumps in our k8s setup to prevent the ephemeral storage to be completely consumed.

[oshawa-connection]

I've dug a bit into my problem I was having with mapserver 8.0.1 with proj version 9.

When I take the code from the gzipped 8.0.1 release, build that with proj v9, I get a similar (though not the same) problem as above. When I clone from main branch, it works perfectly.

When I step through the msProjectRect function with the 8.0.1 release + proj v9, it doesn't change the bbox at all, so I assume my seperate problem must be elsewhere and fixed by some commit to a different file that was merged since November.

@geographika when you replicated this problem, were you using the code directly from main branch, or from the gzipped mapserver release, and which proj version?

Screenshots of my example, now fixed by cloning mapserver main branch and using proj V9. That should also prevent others from having to do workarounds described above too, so long as you don't mind using non-official released code.

[geographika]

@oshawa-connection - I would have been using the main branch and PROJ9, but I don't know what changes could have affected this since February when I could recreate the original issue.

[oshawa-connection]

I think the issue I was having was similar, but unrelated to this issue - sorry for the confusion. If I take that swiss data example I do indeed get the same behavior using main branch.

Mapserver 8.0.1 release, proj 9 -> I get my issue.
Mapserver 8.0.1 release proj 8 -> Fixed my issue.
Mapserver main branch proj 9 -> Also fixed my issue.
Mapserver main branch proj 9 -> Still get ITCLM's issue.

[oshawa-connection]

Sorry for the message spam, I think I have solved @ltclm 's original problem too, I have created a PR for it here: #7074

The source of this problem was a workaround in mapserver for a bug in proj prior to version 9.4.1. The workaround checks for proj major versions between 6 and 9, and then applies a fix. However, versions 9.0.0 through 9.4.0 suffer from the original proj bug, and do not have the workaround applied.

Proj 9.4.0 is the version that gets installed when using the latest GDAL docker image, which hopefully explains why you were having the problem.

The long term solution is to upgrade to proj 9.4.1, which unfortunately atm means installing proj from source, hopefully that's not too painful for you. Or alternatively, downgrade to any proj version 8.

Here is that 4 polygon layer under proj 9.4.1 when really zoomed in, using proj 9.4.1