gulrak / filesystem Goto Github PK

Currently , fs::rename() will return with error 183, if destination file already exists. The standard indicate that it must overwrite file , here is the patch to use MoveFileExW instead.

GHC_INLINE void rename(const path& from, const path& to, std::error_code& ec) noexcept
{
    ec.clear();
#ifdef GHC_OS_WINDOWS
    if (from != to) {
        if (!MoveFileExW(detail::fromUtf8<std::wstring>(from.u8string()).c_str(), detail::fromUtf8<std::wstring>(to.u8string()).c_str(), (DWORD) MOVEFILE_REPLACE_EXISTING)) {
            ec = detail::make_system_error();
        }
    }
#else
    if (from != to) {
        if (::rename(from.c_str(), to.c_str()) != 0) {
            ec = detail::make_system_error();
        }
    }
#endif
}

The C++17 standard demands for std::filesystem::path: "For Windows-based operating systems, value_type is wchar_t and preferred_separator is the backslash character (L’\\’)." (30.10.8-2)

While I still don't really like the decision, I will try to help those in need of a more std::filesystem conforming implementation for C++11/14 by implementing an option to build ghc::filesystem
with ghc::filesystem::path::value_type as wchar_t and ghc::filesystem::path::string_type as std::basic_string<wchar_t> on Windows.

It might be a good idea to actually activate that option from the helper header files that try to detect std::filesystem and include ghc::filesystem only when no standard version is available, as in these situations the resulting fs::path should have the same interface.

Is your feature request related to a problem? Please describe.
Issue of an app I was building jpd002/Play-#825
It includes a ghc::filesystem as dependency. If I run make install, it will install files in /usr/share/include and /usr/lib64/cmake.

Describe the solution you'd like
Here can be an option to not install these files if the package is included as a dependency of other apps.

Describe alternatives you've considered

Additional context

fs::canonical(...) must handle a non-existent path as an error. As fs::exists("") is false, it is wrong to return the current directory for fs::canonical("").

Here's the errors I get when compiling a project with filesystem as a submodule:

D:\jonas\Code\WololoKingdoms\libwololokingdoms\third_party\filesystem\include\ghc\filesystem.hpp:1455:
Error: C2672: 'ghc::filesystem::detail::toUtf8': no matching overloaded function found
Error: C2784: 'std::string ghc::filesystem::detail::toUtf8(const charT *)': could not deduce template argument for 'const charT *' from 'const Source'
Error: C2784: 'std::string ghc::filesystem::detail::toUtf8(const std::basic_string<_Elem,_Traits,_Alloc> &)': could not deduce template argument for 'const std::basic_string<_Elem,_Traits,_Alloc> &' from 'const Source'

This error has occasionally appeared and disappeared before, so it's quite possible that it's a problem on my end, but this time I can't seem to make it go away with updating/cleaning as was the case before. I'm not sure how to handle this, so any tips/pointers would be appreciated if someone has an idea what the issue might be.

Describe the bug

With ../.. only 1 dir is unrolled.

To Reproduce

    const char *path = "ab/cd/ef/../../qw";
    ghc::filesystem::path srcPath = ghc::filesystem::u8path(path);
    std::string destPath = srcPath.lexically_normal().u8string();

Expected behavior
Should produce ab/qw, produces ab/cd/qw.

Attached is a test project.

filesystem_test.zip

UPD:

More test cases:

"\\/\\///\\/" produces //////, expected /.

"a/b/..\\//..///\\/../c\\\\/" produces a/b/c///, expected ../c/.

"a/b/../../../c" produces a/c, expected ../c.

"..a/b/..\\//..///\\/../c\\\\/" produces ..a/b/c///, expected ../c/.

I know your tests are currently not being run on 2019. I gave it a shot and tried to compile anyways. I got a bunch of warnings and an error in one of the examples. Error was easy enough to fix (missing include) but the warnings are mostly about template magic in some system header. I don't get it exactly but seems to be originating from the utf8 conversion somehow.
..so VS2019 build would be awesome to have. Thanks for sharing this great lib.

Describe the bug
For the methods filesystem::create_directories, the document N4687 says (see 30.10.15.6)

Returns: true if a new directory was created, otherwise false. The signature with argument ec returns false if an error occurs.

However, the current implementation in v1.3.0 returns true everytime.

To Reproduce

// should return false as the folder already exists
std::cout << boolalpha << std::filesystem::create_directories(std::filesystem::current_path()) << std::endl

Expected behavior
In case the path already exists and this is a directory, the returned value should be false. This is the behaviour implemented within VS2019 (16.4) and Xcode 11.

Additional context
A way to fix this issue could be to test if the path already exists. For example

// Around line 3400
GHC_INLINE bool create_directories(const path& p, std::error_code& ec) noexcept
{
    path current;
    ec.clear();
    // BEGIN_PROPOSITION
    std::error_code tec;
    auto fs = status(p, tec);
    if (status_known(fs) && exists(fs) && is_directory(fs)) {
        return false;
    }
    // END_PROPOSITION
    for (path::string_type part : p) {
        current /= part;
//...

Thank you for this library!

When using it with mingw, preferred_separator does not get defined:

undefined reference to `ghc::filesystem::path::preferred_separator'

https://ci.appveyor.com/project/sass/libsass/builds/26929504/job/jl88r6p7ci6bq9ad

This is with filesystem.hpp from master, mingw x86_64-4.9.2-release-win32-seh-rt_v4-rev3

Describe the bug
I get the following warning when building with Xcode 11.2:

Loop variable 's' has type 'const ghc::filesystem::path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy

Inside path::parent_path implementation, this line:
for (const string_type& s : input_iterator_range<iterator>(begin(), --end())) {

Loop variable 's' has type 'const ghc::filesystem::path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy

Inside path::lexically_normal implementation, this line:
for (const string_type& s : *this) {

Loop variable 'part' has type 'const path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy
Inside create_directories implementation, this line:
for (const path::string_type& part : p) {

Would it be possible to remove them?

Describe the bug
fs::remove_all should allow deletion of a file.

To Reproduce

assert(fs::remove_all("test.txt") == 1)
assert(fs::remove_all("test.txt") == 0)

Expected behavior
But now it will throw an exception.

Additional context

Describe the bug
warning in gcc 9.2 on MSYS2 environment

C:/msys64/home/phlpt/HELICS/build_gcc/include/helics_cxx/helics/external/filesystem.hpp: In function 'void ghc::filesystem::detail::create_symlink(const ghc::filesystem::path&, const ghc::filesystem::path&, bool, std::error_code&)':
C:/msys64/home/phlpt/HELICS/build_gcc/include/helics_cxx/helics/external/filesystem.hpp:1605:159: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateSymbolicLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, long unsigned int)'} [-Wcast-function-type]
 1605 |     static CreateSymbolicLinkW_fp api_call = reinterpret_cast<CreateSymbolicLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateSymbolicLinkW"));
      |                                                                                                                                                               ^

To Reproduce
Detailed steps to reproduce the behavior.

Expected behavior
It would be nice if the code were warning free

Additional context
MSYS2, gcc 9.2 compiler --std=c++14, most warnings turned on.
Tried with the latest release, but that didn't clear up the warning.

Is your feature request related to a problem? Please describe.
I'm currently trying to use this library in a codebase that should be compiled without exceptions.

Describe the solution you'd like
Replace throwing exceptions with std::error_code. Since exceptions seem to be only used in the detail:: namespace this should not break compatibility with the std:: implementation.

Describe alternatives you've considered
Write my own?

Describe the bug
As the implementation and declaration stays in one file, i can not include this file in different source files
if these files would be linked together. Because multiple definition exist in object files.

You may separate the implementation into .cc file ?

filesystem.hpp: In function 'void ghc::filesystem::detail::create_symlink(const ghc::filesystem::path&, const ghc::filesystem::path&, bool, std::error_code&)':
filesystem.hpp:1605:159: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateSymbolicLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, long unsigned int)'} [-Wcast-function-type]
 1605 |     static CreateSymbolicLinkW_fp api_call = reinterpret_cast<CreateSymbolicLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateSymbolicLinkW"));
      |                                                                                                                                                               ^
filesystem.hpp: In function 'void ghc::filesystem::detail::create_hardlink(const ghc::filesystem::path&, const ghc::filesystem::path&, std::error_code&)':
filesystem.hpp:1622:147: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateHardLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, _SECURITY_ATTRIBUTES*)'} [-Wcast-function-type]
 1622 |     static CreateHardLinkW_fp api_call = reinterpret_cast<CreateHardLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateHardLinkW"));

Should probably use static_cast<> instead

Hi,

We are currently checking for alternatives to boost filesystem.
As of now, unless mistaken, std::filesystem is still not supported on iOS < 13, and after searching a bit, I stumbled on your project's page.

I did not see any mention of iOS in the supported platforms. Is this just a lack of time to test on iOS, or would this require extra work for this platform?

Thanks!

Describe the bug
Using musl instead of glibc results in the #else branch in systemErrorText getting used; musl follows the POSIX standard.

To Reproduce
Compiling a program using ghc::filesystem on Alpine Linux (uses musl as its libc by default) as part of a CI build is how I encountered this issue. Otherwise, setup another compiler to use musl instead of glibc.

Expected behavior
The variant of strerror_r that returns an int should be used.

Additional context
Line 208 of the log in https://cloud.drone.io/nightlark/HELICS-src/22/2/4 is where I first saw the problem.

As you know, MacOS support -std=c++20 even in 10.14 but does not support <filesystem> until 10.15.

char8_t, std::u8string are coming soon. std::filesystem::path::string() will returns std::u8string instead of std::string.

Hello,
I'm compiling with MinGW gcc 9.2.0 on Windows 10 and
the constant ERROR_FILE_TOO_LARGE (line 4228) is not defined.

OS: Windows 10

Describe the bug
The base path to which a sub path is to be concatenated, is not considered at all.

To Reproduce

using filesystem = ghc::filesystem;
auto base_path = filesystem::path("/binaries");
auto path = base_path / filesystem::path("./sub_path");

path is now "/sub_path"

Expected behavior
path should be "/binaries/sub_path".

OpenRCT2 is a project that uses your library and we've recently received a bug report regarding inability to build on FreeBSD. I saw the support for this platform was added across #49 and #51, but it's only to be included in the upcoming, v1.3.1 release. According to https://github.com/gulrak/filesystem/milestone/17 that seems all done and 15 days overdue already and I wanted to check with you what's the ETA of this release before we start breakingapplying local changes to the filesystem library?

Is your feature request related to a problem? Please describe.
Not a problem, just some compiler warnings I got in the latest macOS with the latest Xcode.

Describe the solution you'd like
Comment and possibly act on these:

filesystem.hpp:3561:44: warning: zero as null pointer constant [-Wzero-as-null-pointer-constant]
if (::getcwd(buffer.get(), pathlen) == NULL) {
^~~~
nullptr

filesystem.hpp:2559:12: note: call 'std::move' explicitly to avoid copying on older compilers
return fn;
^~
std::move(fn)

filesystem.hpp:2559:12: warning: prior to the resolution of a defect report against ISO C++11, local variable 'fn' would have been copied despite being returned by name, due to its not matching the function return type ('ghc::filesystem::path' vs 'ghc::filesystem::path::impl_string_type' (aka 'basic_string')) [-Wreturn-std-move-in-c++11]
return fn;
^~

Windows case-sensitivity for drive letters is not always interpreted correctly.

To Reproduce:

fs::path path = fs::relative("c:\\dev\\working\\directory\\file.txt");

Assuming the current working directory is C:\dev\working\directory, one would expect path to be file.txt. However, because the second (optional) argument to relative is evaluated as C:\dev\working\directory (capital 'C'), lexically_relative will exit early on its first check and simply return a default-constructed path.

A fix for this specific issue could be to force drive letters to be capital, say, in weakly_canonical. Now, even Windows can't guarantee case-insensitivity for all file names, but I do believe that drive letters are case-insensitive -- for example, I don't think you can have a d: and D: drive mounted simultaneously. Additionally, under MSVC, it appears that path equality checks are completely case-insensitive when using the standard template library's file system implementation. This does not seem to be the case when compiling under any other compiler.

There should be an option to select if unicode errors (invalid byte sequence or bad codepoint) should lead to errors or be replaced by the unicode replacement character (U+FFFD), like it is currently done.

First of all - finally one working impl. Great job!

Describe the bug
copy and remove_all with error_code throws exceptions due to the error code not being propagated to the directory_iterator's constructor when iterating over the files.

        for (const directory_entry& x : directory_iterator(from)) {

    for (const directory_entry& de : directory_iterator(p)) {

shoud be

        for (const directory_entry& x : directory_iterator(from, ec)) {

    for (const directory_entry& de : directory_iterator(p, ec)) {

Expected behavior
These should not throw when using the error_code api

Describe the bug
While testing the POSIX implementation of the recursive_directory_iterator on macOS, I found the skip_permission_deniedoption to not work when iterating over my complete home directory hitting the ~/Library/Application Support/MobileSync folder. The increment still threw a filesystem_error. The reason is not wrong permissions, but "System Integrety Protection". The opendir results in an EPERM and not EACESS, and while EPERM was added to the increment handling of directory_iterator it is missing from the opendir error handling.

To Reproduce
Just try to iterate over a macOS SIP protected folder.

Expected behavior
The iterator should not enter the protected folder but continue to scan next to it.

Seems like std::vector is being used but not included. This is with forward declaration by including ghc/fs_std_fwd.hpp and happens only on Windows.

Check in ghc/filesystem.hpp:1009

Solved by adding #include just before #endif // GHC_EXPAND_IMPL

Is your feature request related to a problem? Please describe.
The BSD-3-Clause license type complicates the filesystem library usage.

Describe the solution you'd like
The MIT license is more permissive, so it would be nice to apply it instead.

Describe alternatives you've considered
Any other permissive licenses are welcome.

Additional context
Our team is going to open source some library (under MIT license) and having a dependency on the BSD license is not okay then. So if the license type is not changed, we'll have to find another filesystem implementation.

ghc::filesystem is slower than std::filesystem on Windows when working with directories that contain many files.
I tested it in directory which contains about 16600 files.
Enumerating files via ghc::filesystem takes about 50-60 seconds on my PC (NTFS, HDD).
Enumerating files via std::filesystem used from example code takes 2.5-5 seconds (same directory).

  fs::path object_path = fs::u8path(start_path);

  for (auto &entry_path : fs::directory_iterator(object_path)) {
    fs::file_status entry_status = fs::status(entry_path);
    // some simple operations like push names to list
  }

I think the problem corresponds to non-optimized iterator operators and 'status' operations.
Please look on call stack on image. Simple iterator increment operation requires call for 'status', which processed all data gathering, strings assigning, etc. So, for single pass for one file, 'status' operation called so many times with gathering a lot of not necessary information.

Maybe caching of file names, extensions, root paths, etc. instead calculation on each call, will make library faster.

Describe the bug

exists() returns false when given a path to an NTFS reparse point that points to a volume.
Additionally, status() returns not_found for such path.

To Reproduce

In Disk Manager, format a new volume. Instead of assigning a drive-letter, assign a mount-point at an empty directory on an NTFS volume.

Call exists() and pass in the path to the mount point.

Observe it returning false, even though the volume exists and is mounted.

Expected behavior

exists() is expected to return true.

Additional context

The problem arises from resolving a "symlink", in status_ex. The symlink being an IO_REPARSE_TAG_MOUNT_POINT is handled here.

The symlink resolves to something of this form: "\??\Volume{<GUID>}". This path is correct. However, it's assigned into a path object (result = ). The path assign() member function will strip the \??\ prefix, which makes the path incorrect.

Recursing down into another call to status_ex() (here) will then fail saying the path doesn't exist (because it doesn't).

Removing the code that's stripping the prefix fixes the problem:

--- a/filesystem.hpp
+++ b/filesystem.hpp
@@ -1663,9 +1663,9 @@ GHC_INLINE void path::postprocess_path_with_format(path::impl_string_type& p, pa
                         p[0] = '\\';
                     }
                 }
-                else if (detail::startsWith(p, std::string("\\??\\"))) {
-                    p.erase(0, 4);
-                }
+//                else if (detail::startsWith(p, std::string("\\??\\"))) {
+//                    p.erase(0, 4);
+//                }
             }
             for (auto& c : p) {
                 if (c == '\\') {

However, that is likely introducing other issues. Perhaps the best solution would be to have a resolveLink return a plain native sting (instead of path) and also make status_ex() accept a plain string, to avoid round-tripping via path in this case.

Describe the bug
The template version of ghc::filesystem::path::generic_string doesn't return a result, leading to a compilation error when being used.

To Reproduce
Just use it.

The test suite does not seem to build under centos7 with devtoolset-9. This seems to be due to a used TestAllocator not implementing rebind (deprecated in C++17, removed in C++20), e.g.,

/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h: In instantiation of ‘class std::basic_string<char, std::char_traits<char>, TestAllocator<char> >’:
/tmp/cirrus-ci-build/test/filesystem_test.cpp:588:116:   required from here
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3142:63: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3142 |       typedef typename _Alloc::template rebind<_CharT>::other _CharT_alloc_type;
      |                                                               ^~~~~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3160:68: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3160 |       typedef __gnu_cxx::__normal_iterator<pointer, basic_string>  iterator;
      |                                                                    ^~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3162:61: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3162 |                                                             const_iterator;
      |                                                             ^~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3163:53: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3163 |       typedef std::reverse_iterator<const_iterator> const_reverse_iterator;
      |                                                     ^~~~~~~~~~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3164:52: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3164 |       typedef std::reverse_iterator<iterator>      reverse_iterator;

I was exploring whether this library would be a good replacement for devtoolset on centos7 since the base system libstdc++ is fairly old there, and some the std filesystem library often requires compiled components (the devtoolset workflow is easiest to deploy if the produced artifacts have not runtime dependencies on devtoolset libraries).

I have tested this by extending the Cirrus CI setup with centos7 and centos8 platforms here. The build and test steps pass for centos8, but centos7 fails with above error.

Ideally it should be possible to use this library to statically compile in a libstdc++-neutral std::filesystem-like functionality (either by using header-only components or by a static link).

I understand that there is no official support for centos platforms, so please close this if it is out of scope.

Describe the bug
Building in Visual Studio with the wide character flag: GHC_WIN_WSTRING_STRING_TYPE
With clang 11, and c++17, I have these warnings/errors:

1>In file included from D:\filesystem\test\filesystem_test.cpp:62: 1>D:\filesystem\include\ghc/filesystem.hpp(4509,50): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(4510,42): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(4511,46): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(1649,13): error : no matching function for call to 'toUtf8' 1>D:\filesystem\include\ghc/filesystem.hpp(2628,29): message : in instantiation of function template specialization 'ghc::filesystem::path::path<std::basic_string_view<wchar_t, std::char_traits<wchar_t> >, ghc::filesystem::path>' requested here 1>D:\filesystem\include\ghc/filesystem.hpp(1536,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1542,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1571,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1581,20): message : candidate template ignored: could not match 'const charT *' against 'std::basic_string_view<wchar_t, std::char_traits<wchar_t> >'

Please find my Visual Studio project file for filesystem_test attached. (just trim the .txt extension)

Note! clang compiles with only the first three warnings with c++11 (without the actual basic_string_view error)

To create the Visual Studio solution and projects, I ran:
cmake -G "Visual Studio 15 2017 Win64"

I am working with latest on master: commit 9a047b9

To Reproduce
Compile as above

Expected behavior
Compiles, no errors. Warnings are negotiable. :)

Additional context

Thank you for creating this library. I was working with the Windows filesystem, which I found did not support path appending as expected. And I want to have standard cross-platform support.

filesystem_test.vcxproj.txt

Is your feature request related to a problem? Please describe.

This is a header-only library therefore it is usually built with the warning level of the project which includes it. With some additional warnings enabled the filesystem.hpp does not compile cleanly.

Describe the solution you'd like

Build the project with -Wconversion -Wsign-conversion -Wpedantic and fix the reported warnings.

Describe the bug
I use the class filesystem::directory_entry and I am checking some attributes using the methods is_regular_file(), is_directory(), and exists(std::error_code &). For the later, I also check the content of the error_code object. In case, I reuse the same std::error_code object and one of the previous path is not valid, then the content of the reused std::error_code object is always false (i.e. the value() is not equal to 0) for the other paths even if the method exists returns true. Based on C++ reference, the implementation proposed by Visual Studio, or the one included in XCode 11 with macOS 10.15, the error_code object should be cleared. I also checked the document N4687 but it is not mentioned what to do in this case (see 30.10.12.3).

To Reproduce

    std::error_code ec;
    auto d1 = std::filesystem::directory_entry();
    std::cout << d1.exists(ec) << std::endl; // should be false
    std::cout << ec.value() << std::end; // should not be equal to 0
    auto d2 = std::filesystem::directory_entry(std::filesystem::current_path());
    std::cout << d2.exists(ec) << std::endl; // should be true
    std::cout << ec.value() << std::end; // expected to be equal to 0 but this is not!

Expected behavior
The std::error_code object should be cleared if the directory entry exists

Additional context
From my comprehension of the source code, the problem is in the method directory_entry::status(std::error_code &) (see line 4699).

In case, the status' type is not none, then the std::error_code object should be cleared. I proposed the following implementation to fix the issue.

GHC_INLINE file_status directory_entry::status(std::error_code& ec) const noexcept
{
    if (_status.type() != file_type::none) {
        ec.clear();
        return _status;
    }
    return filesystem::status(path(), ec);
}

Describe the bug

/home/builder/build/test/bench/filesystem.hpp: In instantiation of 'ghc::filesystem::path::path(const Source&, ghc::filesystem::path::format) [with Source = std::basic_string_view<char>; <template-parameter-1-2> = ghc::filesystem::path]':
/home/builder/build/test/bench/filesystem.hpp:2409:35:   required from here
/home/builder/build/test/bench/filesystem.hpp:1455:27: error: no matching function for call to 'toUtf8(const std::basic_string_view<char>&)'
     : _path(detail::toUtf8(source))
             ~~~~~~~~~~~~~~^~~~~~~~
/home/builder/build/test/bench/filesystem.hpp:1342:20: note: candidate: template<class charT, class traits, class Alloc, typename std::enable_if<(sizeof (charT) == 1), int>::type size> std::__cxx11::string ghc::filesystem::detail::toUtf8(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&)
 inline std::string toUtf8(const std::basic_string<charT, traits, Alloc>& unicodeString)
                    ^~~~~~
/home/builder/build/test/bench/filesystem.hpp:1342:20: note:   template argument deduction/substitution failed:
/home/builder/build/test/bench/filesystem.hpp:1455:27: note:   'const std::basic_string_view<char>' is not derived from 'const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>'
     : _path(detail::toUtf8(source))
             ~~~~~~~~~~~~~~^~~~~~~~

To Reproduce
Compile with G++ 7 on Debian.

Expected behavior
Compiles.

Additional context
The GCC7 seems to be missing string_view or is not included.

Hi!
First of all, thank you for this library!

Describe the bug
I'm trying to use this library as a fallback when the compiler does not support the final standard version of <filesystem> library.
However, I've found that in some cases the result returned by path::lexically_normal() of ghc is different from the one returned by the C++17 standard version.
In particular, the normal version of path("../") should be "..", while ghc retains the trailing '/' (eventually converted to '\\' on Windows).
On Windows, the same applies also to path("..\\"), which should be normalized to ".." and instead still retains the trailing '\\'.

To Reproduce
To reproduce the behavior, I've created a small program to compare the different results between ghc::filesystem and std::filesystem:

#include <iostream>
#include <filesystem>
#include "ghc/filesystem.hpp"

using std::cout;
using std::endl;

int main() {
    cout << "[std] path(\"../\").lexically_normal() == ";
    cout << std::filesystem::path("../").lexically_normal() << endl;
    
    cout << "[ghc] path(\"../\").lexically_normal() == ";
    cout << ghc::filesystem::path("../").lexically_normal() << endl;

#ifdef _WIN32
    cout << "[std] path(\"..\\\\\").lexically_normal() == ";
    cout << std::filesystem::path("..\\").lexically_normal() << endl;
    
    cout << "[ghc] path(\"..\\\\\").lexically_normal() == ";
    cout << ghc::filesystem::path("..\\").lexically_normal() << endl;
#endif
    return 0;
}

I've tested the program using various compilers and operating systems.

• On Windows 10 using

  • MSVC 2017
  • MinGW-w64 9.2.0
  • clang 9.0
  
  

  the program output is:

  [std] path("../").lexically_normal() == ".."
  [ghc] path("../").lexically_normal() == "..\\"
  [std] path("..\\").lexically_normal() == ".."
  [ghc] path("..\\").lexically_normal() == "..\\"

• On Ubuntu 18.04 LTS using

  • g++ 8.3.0
  • clang 6.0
  
  

  the program output is:

  [std] path("../").lexically_normal() == ".."
  [ghc] path("../").lexically_normal() == "../"

Expected behavior
In all these cases path::lexically_normal() of ghc should return a path("..") in order to conform to the standard.

External References

• std::filesystem::path from cppreference.com

  A path can be normalized by following this algorithm:
  ...
  7. If the last filename is dot-dot, remove any trailing directory-separator.

• Relative Paths from Open Standards

  A path in normal form has no redundant current directory (dot) elements, no redundant parent directory (dot-dot) elements, and no redundant directory-separators.

Is your feature request related to a problem? Please describe.

When ghc::filesystem is used in multiple places of a project, there is a compiletime overhead due to the header-only, everything-inline nature of the library. In such situations it might be nice to be able to concentrate the implementation into a single place and include only the nessesery elements in all the other places.

This also comes with the advantage, that the needed system includes (e.g. windows.h) will only be needed at the implementation place, so thes do not pollute the global namespace in every place filesystem is used.

Describe the solution you'd like

Two small wrapping headers should be added, they set some defines and include the normal filesystem.hpp header. One called fs_fwd.hpp will ensure everything needed to use ghc::filesystem is visible, the other, called fs_impl.hpp will lead to the expansion of all implementation code and use the system headers needed to implement the functionality.
The implementation header must be included in a cpp before the visibility of a fs_fwd.hpp to take precedence in the code extraction.

First implementation will be done on a feature branch.

This feature is related to some wishes from #3.

In line 3027 of filesystem.hpp, if the copy()-method resolves to a copy_file with two file paths given, the options parameter is not passed on.
copy_file(from, to, ec);
This causes issues e.g. if the target file already exists and overwrite_existing or update_existing was passed as an option.

Is your feature request related to a problem? Please describe.
my project is compiled with -fno-exception.
and throw is forbidden with -fno-exception

Describe the solution you'd like
maybe we can make a macro protect

Expected behavior

fs::path("a/b").lexically_relative("a") == "b";
fs::path("a/b").lexically_relative("a/") == "b";

Actual behavior

fs::path("a/b").lexically_relative("a/") == "b";
fs::path("a/b").lexically_relative("a/") == "../b";

(tested on AppleClang 11.0.0 on macOS 10.15.3)

Hi it's me again. Another issue i forgot to mention that is breaking the build is

class resursive_directory_iterator
{
...
using difference_type = ptrdiff_t;

should be 

using difference_type = std::ptrdiff_t;

Describe the bug
When using a debug build (-DCMAKE_BUILD_TYPE=Debug) and a recursive directory iterator. If the directory to iterate is fairly large, a use after free or heap overflow occurs

To Reproduce

1. Debug Build - Enable address sanitizer
2. Use a recursive directory iterator on a directory that has at least 12 levels of sub
   directories

Expected behavior
After some execution you should see a crash located at filesystem.hpp:4570:

==32507==ERROR: AddressSanitizer: unknown-crash on address 0x62d0011f6328 at pc 0x555e71d4f2d5 bp 0x7ffd5f191200 sp 0x7ffd5f1911f0
READ of size 280 at 0x62d0011f6328 thread T0
.
.
.
0x62d0011f6430 is located 0 bytes to the right of 32816-byte region [0x62d0011ee400,0x62d0011f6430)
allocated by thread T0 here:
    #0 0x7f25c1f1db50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f25c064c9b5 in opendir (/lib/x86_64-linux-gnu/libc.so.6+0xdf9b5)

SUMMARY: AddressSanitizer: unknown-crash {PATH}/filesystem.hpp:4570 in i_readdir_r
Shadow bytes around the buggy address:
  0x0c5a80236c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c5a80236c60: 00 00 00 00 00[00]00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c80: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32507==ABORTING

Additional context
For some reason it is listed as an unknown-crash but it has all the symptoms of a heap overflow or a use-after-free error.

Hi.
Believe I've found a bug:
Error C2065 'DWORD': undeclared identifier ghc\filesystem.hpp(1089)

Noticed this now (on Windows, vs2019) when upgrading from an older release of your repo.

So I get this when using the forward declarations. Apparently it now relies on having DWORD already defined which wasn't the case before. My application does not include windows.h, hence there's no DWORD.

I see a few options:

• Change the declaration of make_system_error() to not use DWORD.
• Include windows.h always, not only for implementation.
• typedef unsigned long DWORD if not already defined.

Not sure what works best here but the latter two are probably best to avoid if possible.

The copy_file function uses shared_ptr as a RAII wrapper around close through a custom deleter.

It seems like this code is using shared_ptr as a mechanism for RAII. shared_ptr isn't necessarily free: it will likely allocate storage for the lamdas and their captures because it has to type erase the custom deleter internally. I find it confusing to see shared_ptr used this way. This is not really its intended use, so the intent of the code wasn't really clear on my first read.

I don't see why RAII is advantageous here: it would make sense if any of the functions called in copy_file could throw, because the destructor would ensure close is called. However, the function is noexcept. If anything throws in here, the application will be terminated anyway.

This is just something I noticed while I was digging around the source. I could submit a PR addressing these comments if you agree with them. Otherwise, I'd be really interested to hear why the code is written like this.

Cheers!

In MSVC there's special macro called _MSVC_LANG that works fine without /Zc:cplusplus flag
So you can build your own macro, for example:
https://github.com/oneapi-src/oneTBB/blob/8632c278467f99941caff8a666fa33e57650c53f/include/tbb/detail/_config.h#L39-L47

Our unit tests all harness boost::filesystem::path, which has some differences from std::filesystem::path and ghc::filesystem::path.
Are these strictly defined in the standard?

ex1/

p1 = "c:/bbb"
p2 = "";
p3 = p1 / p2;   == "c:/bbb/"    (with ghc)
p3 = p1 / p2;   == "c:/bbb"     (with boost)

ex2/

p1 = "c:/bbb"
p2 = "/ccc";
p3 = p1 / p2;   == "c:/ccc"       (with ghc)
p3 = p1 / p2;   == "c:/bbb/ccc"   (with boost)

If I want the same behaviour, can you recommend how?

I considered making path::append virtual, to:
ex1/ skip appending if the rhs is empty
ex2/ strip the leading or trailing separator on entry

Or add a ghc::filesystem::path member to my path class, instead of inheriting. But this requires forwarders for every path function.

fs::remove_all() previously only returned the number of regular files removed, not counting in directories, but as directory is a file_type, they should count too.

First of all, thanks for this extremely useful project!
I would like to have it run in the browser by compiling it to web-assembly using emscripten. Emscripten itself does not currently provide support for std::filesystem and it seems to me this might be in scope for your library.

I was able to have it compile using emscripten by simply adding

#elif defined(__EMSCRIPTEN__)
#define GHC_OS_WEB

to your platform determination part in filesystem.hpp.

It was immediately usable, but the following simple example gives a crash, so I suppose there may be some more effort necessary to get things running correctly:

for(auto& p: fs::recursive_directory_iterator("."))
        std::cout << p.path() << '\n';

fails after two prints

./proc
./proc/self
Operation not permitted: './proc/self'

Note that the "filesystem" in this case is the virtual browser filesystem -- I have not yet researched the nuances that might make the browser platform different from others.

Describe the bug
I'm not sure if this meets the C++ standard, but both msvc and mingw implementations do this.

To Reproduce

auto path = fs::path("test/");
fs::create_directory(path);
auto itor = fs::directory_iterator(path);
while (itor != fs::directory_iterator()) {
    ++itor;
}
fs::remove_all(path);
fs::create_directory(path); // throw an exception

When itor is equal to end(), it releases the resources held by itor. fs::create_directory will not throw an exception.

Expected behavior
The second fs::create_directory works fine.

Additional context