utf_8_char test is failing on Windows about editorconfig-core-c

[ppalaga]

utf_8_char also fails when we run it on Window against our Java parser lib. Our debug output seems to suggest that the the ctest process is using an improper encoding when passing the Asian file path to our executable. This is the relevant section of Testing/Temporary/LastTest.log. Note that ctest lists the command properly with 中文.txt, but our executable sees ??.txt. Is there any cmake option to fix this?

89/179 Testing: utf_8_char
89/179 Test: utf_8_char
Command: "C:/Program Files/Java/jdk1.7.0/bin/java.exe" "-cp" "C:/projects/ec4j/core/target/org.eclipse.ec4j.core.jar" "org.eclipse.ec4j.core.cli.Cli" "-f" "utf8char.in" "C:/projects/ec4j/core/editorconfig-core-test/glob/中文.txt"
Directory: C:/projects/ec4j/core/editorconfig-core-test/glob
"utf_8_char" start time: Nov 12 18:48 Coordinated Universal Time
Output:
----------------------------------------------------------
Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <?> at index 50: C:/projects/ec4j/core/editorconfig-core-test/glob/??.txt
	at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
	at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
	at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
	at java.nio.file.Paths.get(Paths.java:84)
	at org.eclipse.ec4j.core.cli.Cli.main(Cli.java:117)
<end of output>
Test time =   0.09 sec
----------------------------------------------------------
Test Fail Reason:
Required regular expression not found.Regex=[^key=value[ 	
]*$
]
"utf_8_char" end time: Nov 12 18:48 Coordinated Universal Time
"utf_8_char" time elapsed: 00:00:00
----------------------------------------------------------

[xuhdev]

Hmm, have you reported to cmake? If so, can you point out the link?

[xuhdev]

I am sort of curious about this: I successfully ran all tests on Windows before. Do you have any idea why this is happening?

[ppalaga]

have you reported to cmake?

No.

Do you have any idea why this is happening?

No, no idea. I am very new to cmake.

Well, as I said in my previous post, I actually have a suspection that the problem is somewhere between cmake and the executable. Either cmake or windows shell or whatever else there in between uses an improper encoding when passing the path to the executable.

[xuhdev]

Where did you obtain cmake?

[ppalaga]

Where did you obtain cmake?

It is available by default on AppVeyor. No idea where they have it from. In any case, the version is 3.9.6 - see https://ci.appveyor.com/project/xuhdev/editorconfig-core-c/build/1.0.3#L5

[xuhdev]

@ppalaga I had one thought: maybe it's the PCRE library? We seem to have built the 8-bit version.

[ppalaga]

8-bit version

Are you sure it is possible to build an 8 bit in 2017? (Maybe a stupid question. I am really new to cmake)

[xuhdev]

It's in the AppVeyor output:

-- PCRE configuration summary:
-- 
--   Install prefix .................. : C:/projects/bin/x64-static/build
--   C compiler ...................... : C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/bin/Hostx86/x64/cl.exe
--   C++ compiler .................... : C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/bin/Hostx86/x64/cl.exe
--   C compiler flags ................ : /DWIN32 /D_WINDOWS /W3 
--   C++ compiler flags .............. : /DWIN32 /D_WINDOWS /W3 /GR /EHsc 
-- 
--   Build 8 bit PCRE library ........ : ON
--   Build 16 bit PCRE library ....... : OFF
--   Build 32 bit PCRE library ....... : OFF
--   Build C++ library ............... : OFF

[ppalaga]

Indeed, 8 bit in 2017!

[xuhdev]

I tried to look this up again: I don't think it's the 8-bit issue. From its readme, it looks like the 8-bit library supports UTF-8. Instead, I suspect these two lines are the culprit:

--   Enable UTF support .............. : OFF
--   Unicode properties .............. : OFF

[chcg]

@xuhdev @ppalaga See https://ci.appveyor.com/project/chcg/editorconfig-core-c/build/1.0.12
, also with PCRE_SUPPORT_UTF ON the windows test for utf8 support is failing. This is probably due to the issue that windows is using UTF16 for the unicode support, see
https://stackoverflow.com/questions/2050973/what-encoding-are-filenames-in-ntfs-stored-as.
So this seems to be an issue not easily resolvable on windows with the normal c library file accesses.

[ppalaga]

@chcg I do not think UTF16@NTFS is the problem. core-c lib is not opening the file. The path is just a string that gets matched against some glob patterns. From what I saw debugging ec4j java impl. The problem was somewhere between ctest, windows shell and our ec4j executable. Somebody more experienced in c programming than me should try to add some debug output to the core-c to see what bytes is it getting for the asian file name. I assume one will see some broken string like ??.txt like we see with ec4j.

[xuhdev]

@ppalaga Have you tried to run the tests on your own Windows system? Or at least run the executable on some Unicode file names? I've never had this issue on my system before. But I'm not sure whether appveyor configures something differently.

[ppalaga]

Have you tried to run the tests on your own Windows system?

Sorry, appveyor is the only Windows system at my disposal.

[xuhdev]

If we don't know the reason for now, perhaps the best compromise is to disable this test on Windows on AppVeyor for EditorConfig C Core only. Do you have any idea how to do this?

[chcg]

@xuhdev See editorconfig/editorconfig-core-test#18

[ppalaga]

As I mentioned in the PR, you can use the -E option when calling ctest much like we did in ec4j ec4j/ec4j@1c84965#diff-180360612c6b8c4ed830919bbb4dd459R31

[xuhdev]

@ppalaga Many thanks! I followed your method and disabled the test case utf_8_char successfully.