How to tell if a tombstone exists for a file path? about projfs-managed-api

[FeralSquid]

Ok, after playing around with this, I think I see what is happening, though I still have a question towards the bottom and it would be helpful to know if I missed something...

It looks like Utils.TryGetOnDiskFileState() will invoke GetPlaceholderInfo() to test if the file exists in the backing store (if needed), and it then uses that (in combination with what it knows about the local file, if anything) to return false in all situations where an application should treat the file as not existing, which include:

• No local file, no backing store file
• No local file, backing store has file but there is a local tombstone (user deleted "virtual" file locally)

... else it will return true.

Does that sound correct?

Also, assuming that is correct, my remaining question is ... under what circumstance would Utils.TryGetOnDiskFileState() return OnDiskFileState.Tombstone?

Thanks

[cgallred]

Utils.TryGetOnDiskFileState() will only return OnDiskFileState.Tombstone when the provider is not running. That routine is really not meant to be used by a running provider. It is a utility routine meant for use by tools. A provider that needs to know whether a file has been deleted needs to register for NotificationType.FileHandleClosedFileDeleted and use that to keep track of what's been deleted.

What Utils.TryGetOnDiskFileState() does is try to open the file at the given path and query the contents of its reparse point, if any. If the file doesn't exist on disk then you'll see GetPlaceholderInfo() called (this is why the documentation for Utils.TryGetOnDiskFileState() says that a running provider needs to be careful calling it because "it may cause callbacks to be invoked in the provider").

In the case of a tombstone you have a file on disk with a special "tombstone" reparse point. Utils.TryGetOnDiskFileState() will open that tombstone file, see that it is a tombstone, and report OnDiskFileState.Tombstone.

However there is a caveat. The job of a tombstone is to make it look like there's nothing there, but there are certain circumstances where we need to be able to reveal the tombstone's existence. For instance, if you want do rmdir /s on your virtualization root we can't very well keep hiding the tombstones or you'd never be able to delete them. So when the ProjFS file system filter driver sees an attempt to open a tombstone it checks to see whether the provider that owns the tombstone is running. If so, it makes it look like there's nothing there. Otherwise it allows the tombstone to be opened (and thus deleted). The filter driver can't tell the difference between an open issued by Utils.TryGetOnDiskFileState() and any other random open, which is why Utils.TryGetOnDiskFileState() won't report OnDiskFileState.Tombstone while the provider that owns the tombstone is running.

[FeralSquid]

Thanks for the clarification and additional info!

It is unfortunate that there isn't a way to query the VirtualizationInstance for its understanding of the local file state (only - so no callback invocation), since it is the source of truth for that, while I am already the source of truth for the backing store (so I don't need to necessarily know what my own GetPlaceholder callback says, for example).

If I were to go the notification route where I have to handle and persist the notion of all files the user has deleted, I fear it is fragile, as my process can be killed before I get a change to handle and persist all notifications. Further, users can delete things while my process isn't running ... which suggests I'd need to scan the NTFS journal to catch up/find missed notifications or something.

Not insurmountable, just annoying, especially since that local file state I want must be readily available to you just on the other side of that api fence ;)

Thanks again for your prompt and detailed response!

[cgallred]

I just remembered that there might be a different direction you could approach this. If you use FindFirstFileEx with the FIND_FIRST_EX_ON_DISK_ENTRIES_ONLY flag (and then of course FindNextFileW) you'll get an enumeration of only what is on disk. That is, it will not return any virtual files and the results will contain any tombstones that might be present.

• The WIN32_FIND_DATAW structures that you get back report the reparse tag, if any, in the dwReserved0 member if "the dwFileAttributes member includes the FILE_ATTRIBUTE_REPARSE_POINT attribute".
• A tombstone will have the reparse point tag value of IO_REPARSE_TAG_PROJFS_TOMBSTONE (0xA0000022).

The issue of how to handle modifications to the virtualization root while the provider isn't running still exists, but even if Utils.TryGetOnDiskFileState() could tell you about tombstones when the provider is running it wouldn't help. A tombstone is only created if your provider is running when the file is deleted. That's because to create a tombstone ProjFS has to know whether or not the file being deleted exists in the provider's backing store. If it does not, then there's no reason for ProjFS to create a tombstone. If the provider is not running when the delete happens, then ProjFS can't ask the provider whether the file exists in the backing store. So the file is simply deleted. When the provider comes back up it is re-projected as if it had never been hydrated in the first place.

[FeralSquid]

Ah, that sounds interesting, I'll give that a try.

Also good info/point about deletions that happen when provider isn't running.

Thanks!

[FeralSquid]

I finally got around to playing with this today.

I think this will be useful/helpful!

One thing I wasn't expecting, though, is that this does still invoke GetPlaceholderInfo() callbacks on the sub-directories between the virtualization root and the parent directory of the file/directory passed to FindFirstFileEx() ... my ideal would be for it to just return ERROR_PATH_NOT_FOUND or error 369 (provider unavailable) if the subdirectory doesn't exist locally and has no placeholder, but this still will greatly reduce the number of placeholders requested/cached in my use case.

Is that expected?

Actually, given that behavior, I suppose I could query each subdirectory myself, stopping as soon as one is not found, to prevent any placeholders from being created...

Thanks again!

[cgallred]

This is expected. The FIND_FIRST_EX_ON_DISK_ENTRIES_ONLY flag only applies to the actual enumeration, not to the NtOpenFile call that FindFirstFileEx issues to get a handle to the directory that is going to be enumerated.

That is an interesting suggestion though, and I can see how that behavior would be more consistent. I'll put it on my backlog for future consideration.