scille / winfspy Goto Github PK

Another issue with common file formats.
Adobe Reader DC cannot open any PDF docuents stored in the provided memfs.
To reproduce, just copy any PDF document on the memfs.
Either by double clicking in Explorer on the document or via File Open Dialog, Adobe Reader reports the file can't be opened, Access denied.

To avoid this behaviour the following workaround helps: Disable "Run in App Container" in Adobe Acrobats Settings.
(Settings - Security (extended) - middle checkbox at the top )

Even this workaraound will help, it is not an option for many Users in secured environments or if the do not have this option due security policies in enterprises.

I started investigations in this topic to find out whats the difference of this setting on FS level, but I still have no more information yet.

The rules on Windows are that the . and .. directory entries must always be present in full (i.e. Pattern==NULL) listings, except for the root directory where they should not be there at all. For example, if you have a directory with the files foo and bar, you should report the following directory entries:

• If the directory is not the root directory:

  • .
  • ..
  • foo
  • bar

• If the directory is the root directory. This is optional and your file system will work correctly if you always return the dot and dot-dot entries. But it is what the Microsoft file systems do.

  • foo
  • bar

Thanks @billziss-gh for reporting this issue and providing the relevant information!

Context

I have written a custom HDFS (hadoop) filesystem using fs_spec and used winfspy to mount the filesystem in Windows. I have adapted the memfs-exmple from this repository accordingly. But I had to realize that Windows sends a lot of get_security_by_name open ans close calls to the file system, which makes the mounted filesystem really slow. As suggested and discussed here, mounting a remote filesystem as a network drive (via VolumePrefix) should improve the mounting performance.

To my understanding in winfspy the VolumePrefix can be set via the **volume_params ->prefix argument of the FileSystem-constructor

prefix gets mapped to FSP_FSCTL_VOLUME_PARAMS->Prefix[] in:

Problem

Mounting my filesystem with setting prefix="\\server\hdfs" or prefix="/server/hdfs" like:

fs = FileSystem(
        str(mountpoint),
        operations,
        ...
        prefix="\\server\\hdfs"
)

does not have any effect and it is mounted as normal drive under the mountpoint.

Question

Is this a bug? Or am I'm doing something wrong? How can I use winfspy to mount my filesystem as a network drive?

I'm trying to get a sid from pywin32 passed into FspPosixMapSidToUid() but I can't figure out how to convert it from a PySID to the cffi PSID generated in this library.

I can get the raw bytes of the PySID using memoryview().tobytes() ot .tolist() but I can't seem to use them as an initializer for PSID because they're not the same size (28 bytes versus 12 bytes respectively)

The root problem I'm trying to chase is detailed more in tannercollin/standardnotes-fs#15.

Any thoughts?

Hi Emmanuel,
Is saw some of your recent code changes and there are still a bug in calculating filename sizes:

1. The filename size is calculated without a NULL byte. You can see this by
   executing a os.listdir with python, all filenames have (incorrectly) that extra null byte.
2. encoding a name to utf16 just for getting its 2 byte unicode size is a little bit overkill.
   len(file_name.encode("utf16")) is much slower then 2*len(file_name) and you do that pretty often
   reading a large directory.

Best Regards
Michael

This is how the winfsp DLL is loaded at the moment:

Note that WINFSP_LIBRARY_PATH is used three times:

• %WINFSP_LIBRARY_PATH%/inc: used when building the _bindings module
• %WINFSP_LIBRARY_PATH%/lib: used when building the _bindings module
• %WINFSP_LIBRARY_PATH%/bin: used to load the dll

This makes it hard to load a debug DLL without patching the source code, since the path to to the debug DLL typically looks like this:

C:\\Users\\User\\Projects\\winfsp\\build\\VStudio\\build\\Debug

I guess there are several ways to go about it. The most obvious way would be to add a %WINFSP_DEBUG_PATH% that overrides %WINFSP_LIBRARY_PATH%/bin when it is defined. In this case, adding a warning (e.g "The debug library is being loaded") might make sense.

Thanks to @billziss-gh for the suggestion!

The VolumeLabelLength in VolumeInfo is incorrect because it needs to report the length of the VolumeLabel in bytes (7 * 2 = 14) and not characters. Windows and WinFsp uses WCHAR’s which are 2 bytes long.

Note: the VolumeLabel does not contain a L'\0' terminating character.

To be fixed here:

Thanks @billziss-gh for reporting this issue and providing the relevant information!

Observe output from VS Debugger immediate window:

DirInfo
0x00000156d7fdb690 {Size=110 FileInfo={FileAttributes=0 ReparseTag=0 AllocationSize=0 ...} NextOffset=...}
    Size: 110
    FileInfo: {FileAttributes=0 ReparseTag=0 AllocationSize=0 ...}
    NextOffset: 0
    Padding: 0x00000156d7fdb6e0 ""
    FileNameBuf: 0x00000156d7fdb6f8 L".."

First the size should be 108 and not 110. This is because FileNameBuf should not contain a L'\0' terminating character. Compute DirInfo->size = sizeof(FSP_FSCTL_DIR_INFO) + FileNameLengthInBytes.

Also, the directory entry is not reported as a directory (observe that FileAttributes=0).

To be fix there:

Thanks @billziss-gh for reporting this issue and providing the relevant information!

Following an investigation on Parsec, we have a recurring error in the test pipeline.

tests/core/mountpoint/test_winfstest.py::test_winfstest[10_GetSetSecurity.t] XFAIL
tests\core\mountpoint\test_winfstest.py:59

>       assert errno == getattr(winerror, expected), f"Expected {expected}, got {message}"
E       AssertionError: Expected ERROR_PATH_NOT_FOUND, got The directory name is invalid.
E       assert 267 == 3
E         -267
E         +3

expect(
    "CreateFile %s\\bar GENERIC_WRITE 0 0 CREATE_NEW FILE_ATTRIBUTE_NORMAL 0" % name,
    "ERROR_PATH_NOT_FOUND",
)

** Running command:
-> CreateFile %s\1c8ab2fd-6429-44d0-a4a1-bdf60a98cd04\bar GENERIC_WRITE 0 0 CREATE_NEW FILE_ATTRIBUTE_NORMAL 0
-> Expecting: ERROR_PATH_NOT_FOUND
-> Got: errno=267, message='The directory name is invalid.'

This might be a breaking change in WinFSP 1.7 (B2) :
winfsp/winfsp@9066338

From what we can understand, there's no more error thrown "ERROR_PATH_NOT_FOUND" (STATUS_OBJECT_NAME_NOT_FOUND = ENOENT), but the Win32 throw NotADirectoryError (WinError 267 = 'The directory name is invalid.')

Some pipelines for parsec-cloud tests passed because the "XFAIL" in winfstest may not trigger a bad test.

https://ci.appveyor.com/project/touilleMan/winfspy/builds/28795168/job/cl7s80fug8gyxfh8 :'(

This will give us very high confidence that winfspy works correctly in all circumstances.

See the testing strategy section of the winfsp docs:

Winfsp-tests: This test suite provides comprehensive testing of WinFsp’s capabilities under various scenarios. This includes general Win32 (and NTDLL) file API testing, but also includes WinFsp specific tests, such as incorrectly functioning user mode file systems. The non-WinFsp specific tests are verified against NTFS.

This test suite is developed together with WinFsp. It is written in C/C++ and provides a form of gray box testing.

Link to winfsp-tests.

Thanks to @billziss-gh for the suggestion!

This should be labelled as question.
When dealing with winfsp in fuse mode, open in operations will be called as
def open(self, path, flags):
in winfspy open looks like
def open(self, file_name, create_options, granted_access):

so far so good.

Now let's assume I have (a kind of) passthrough filesystem.
using fuse i can open the real file which is somewhere in the ntfs filesystem this way:

FSP_FUSE_UF_HIDDEN = 0x00008000
    FSP_FUSE_UF_READONLY = 0x00001000
    FSP_FUSE_UF_SYSTEM = 0x00000080
    FSP_FUSE_UF_ARCHIVE = 0x00000800

    CREATE_NEW = 1
    CREATE_ALWAYS = 2
    OPEN_EXISTING = 3
    OPEN_ALWAYS = 4
    TRUNCATE_EXISTING = 5
    FILE_SHARE_READ = 0x00000001
    FILE_SHARE_WRITE = 0x00000002
    FILE_SHARE_DELETE = 0x00000004
    FILE_SHARE_VALID_FLAGS = 0x00000007
    FILE_ATTRIBUTE_READONLY = 0x00000001
    FILE_ATTRIBUTE_NORMAL = 0x00000080
    FILE_ATTRIBUTE_TEMPORARY = 0x00000100
    FILE_FLAG_DELETE_ON_CLOSE = 0x04000000
    FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000
    FILE_FLAG_RANDOM_ACCESS = 0x10000000
    GENERIC_READ = 0x80000000
    GENERIC_WRITE = 0x40000000
    DELETE = 0x00010000
    NULL = 0

    _ACCESS_MASK = os.O_RDONLY | os.O_WRONLY | os.O_RDWR
    _ACCESS_MAP = {os.O_RDONLY: GENERIC_READ,
                   os.O_WRONLY: GENERIC_WRITE,
                   os.O_RDWR: GENERIC_READ | GENERIC_WRITE}

    _CREATE_MASK = os.O_CREAT | os.O_EXCL | os.O_TRUNC
    _CREATE_MAP = {0: OPEN_EXISTING,
                   os.O_EXCL: OPEN_EXISTING,
                   os.O_CREAT: OPEN_ALWAYS,
                   os.O_CREAT | os.O_EXCL: CREATE_NEW,
                   os.O_CREAT | os.O_TRUNC | os.O_EXCL: CREATE_NEW,
                   os.O_TRUNC: TRUNCATE_EXISTING,
                   os.O_TRUNC | os.O_EXCL: TRUNCATE_EXISTING,
                   os.O_CREAT | os.O_TRUNC: CREATE_ALWAYS}


    def os_open(file, flags, mode=0o777, *, share_flags=FILE_SHARE_VALID_FLAGS):
        '''
        works better then os.open() in windows
        '''
        if not isinstance(flags, int) and mode >= 0:
            raise ValueError('bad flags: %r' % flags)

        if not isinstance(mode, int) and mode >= 0:
            raise ValueError('bad mode: %r' % mode)

        if share_flags & ~FILE_SHARE_VALID_FLAGS:
            raise ValueError('bad share_flags: %r' % share_flags)

        access_flags = _ACCESS_MAP[flags & _ACCESS_MASK]
        create_flags = _CREATE_MAP[flags & _CREATE_MASK]
        attrib_flags = FILE_ATTRIBUTE_NORMAL

        if flags & os.O_CREAT and mode & ~0o444 == 0:
            attrib_flags = FILE_ATTRIBUTE_READONLY

        if flags & os.O_TEMPORARY:
            share_flags |= FILE_SHARE_DELETE
            attrib_flags |= FILE_FLAG_DELETE_ON_CLOSE
            access_flags |= DELETE

        if flags & os.O_SHORT_LIVED:
            attrib_flags |= FILE_ATTRIBUTE_TEMPORARY

        if flags & os.O_SEQUENTIAL:
            attrib_flags |= FILE_FLAG_SEQUENTIAL_SCAN

        if flags & os.O_RANDOM:
            attrib_flags |= FILE_FLAG_RANDOM_ACCESS

        h = _winapi.CreateFile(file, access_flags, share_flags, NULL,
                               create_flags, attrib_flags, NULL)

        return msvcrt.open_osfhandle(h, flags | os.O_NOINHERIT)

Means I get the access_flags and create_flags out of flags passed by the fuse operation open.

When create_options & CREATE_FILE_CREATE_OPTIONS.FILE_NON_DIRECTORY_FILE are not 0, I know to create a file by using
0x40000000 for access_flags and 0x4 for create_flags.

But I can''t find any valid mapping from CREATE_FILE_CREATE_OPTIONS to find the proper parameters for the winapi CreateFile method.

How can this be adressed ?
Or even asked the other way around, how can the create_options and granted_access flags from winfspy open operations be converted to flags which can be used with standard os.open() ?

The reason that the file in the filter is not found is because winfspy reports it as L"\0" and the FSD compares it to L"". Windows does not use NULL-terminated strings internally, so the FSD finds that these 2 strings are not the same.

In fact, removing the +2 from this line does fix the problem:

Thanks @billziss-gh for reporting this issue and providing the relevant information!

When activating support for ADS there's a bug in operations.py

Line 772 should be changed from

cooked_file_context = ffi.from_handle(file_context, buffer, length, p_bytes_transferred)

to

cooked_file_context = ffi.from_handle(file_context)

however when overwriting operations, my personal implementation looks like

    @operation
    def get_stream_info(self, file_context, buffer, length: int, p_bytes_transferred):
        pass

I assume this should return some info about the streams which are associated, but at the moment it seems, operations.py does not handle this info at all

So by checking ADS in get_security_by_name for example like this

    @operation
    def get_security_by_name(self, file_name):
        file_name = PureWindowsPath(file_name).as_posix()
        stream = None
        if ":" in file_name: #Check for alternate streams
            file_name, stream = file_name.split(":")
        # do whatever with stream now  

as well as in open and create we already can deal with ADS if just line 772 in operations.py is fixed, even if the implementation is not complete yet. But it's better then crashing like it does without this change ;)

When saving a ms office 365 file (Word, Excel) the provided memfs crashes immediatly.

So far I could target the error in in
winstuff.py
def evolve

The return value of lib.FspSetSecurityDescriptor is not checked there.
Changing the method to:

def evolve(self, security_information, modification_descriptor):
    psd = ffi.new("SECURITY_DESCRIPTOR**")
    if NTSTATUS.STATUS_SUCCESS == lib.FspSetSecurityDescriptor(
        self.handle, security_information, modification_descriptor, psd):
        handle = psd[0]
        size = lib.GetSecurityDescriptorLength(handle)
        return type(self)(handle, size)
    else:
        raise RuntimeError(
            f"Cannot create new security descriptor"
            f"{cook_ntstatus(lib.GetLastError())}"
        )

solves the problem, but I'm not quiet sure if this is enough so far.