gemtalk / filesystemgs

Port of FileSystem implementation to GemStone/S from Pharo

FileSystem for GemStone/S

Port of Pharo FileSystem implementation derived from pharo-project/pharo.

Installation

1. Set Environment variables
   1. ROWAN_PROJECTS_HOME
   2. TOPAZ_SRC_DIRECTORY - topaz src files will be generated into this location
   3. GEMSTONE
   4. ARCHBASE - should correspond to the repo version used to build $GEMSTONE
2. Clone repositories in $ROWAN_PROJECTS_HOME
   1. FileSystemGs
      • URL: https://github.com/GemTalk/FileSystemGs
      • Branch: development
   2. Rowan
      • URL: https://github.com/GemTalk/Rowan
      • Branch: kurt.IntegrateWithFileSystemGs
3. Start NetLDI
4. Start Stone with a Rowan extent
5. Generate topaz source files by evaluating $ROWAN_PROJECTS_HOME/FileSystemGs/scripts/read_and_write_file_system.gs in topaz.
6. Restart Stone using a non-Rowan extent.
7. Install Rowan and FileSystemGs by evaluating $ROWAN_PROJECTS_HOME/FileSystemGs/scripts/installRowan.tpz in topaz.
8. Load test cases by evaulating (Rowan projectNamed: 'Rowan') loadProjectSet: Rowan platformConditionalAttributes, #('tests' 'v2' 'v2Only' 'testsV2' 'stubs' 'tonel') in Topaz. Ensure you commit after this expression if you want the tests to stick around.

filesystemgs's Issues

DirectoryEntryTest Failures

The DirectoryEntryTest cases all appear to fail in a similar way. They all appear to fail with an #_errorNotIndexable.

FileReferenceTest - Insufficient testing of selectors unsupported by the File/Directory references

Many of the FileReferenceTest ensure that selectors like #children work correctly for Directories. Correct error conditions are not tested for when #children is sent to FileReference representing Files. File-only selectors like #writeStream do not test Directory error conditions as either.

The tests should be updated to test such error conditions.

Proposed FileMode class structure to replace #write and #read psuedo-modes

Added the proof of concept of FileMode with subclasses to be passed instead of #write and #read.
The classes which have been introduced are:

			FileAppendMode
				FileAppendAndReadMode
					FileAppendAndReadBinaryMode
				FileAppendBinaryMode
			FileReadMode
				FileReadBinaryMode
			FileReadWriteMode
				FileReadWriteMode
					FileReadWriteBinaryMode
					FileReadWriteTruncatedMode
						FileReadWriteTruncatedBinaryMode
			FileWriteMode
				FileWriteBinaryMode

These classes return the appropriate mode string, (ex: ‘w+’) and know if the file is writable.
Example of change - the text in blue shows the change.

Existing format for opening a stream over a file:

		(filesystem open: aFileReference path writable: #write) ```

Proposed format for opening a stream over a file:

	(filesystem open: aFileReference path writable: self store readWriteTruncatedMode) ```

The #readWriteTruncatedMode method would return an instance of FileReadWriteTruncatedMode

If a form of this proposal were to be adopted, then the appropriate methods should be renamed to have mode.
As an example, #open:writeable: would become #open:mode:

LibC Tests are insufficient

The TestCases testing LibC wrapper classes do not sufficiently cover the functions we are using. The FileSystem tests exercise LibC and guided the Linux implementation. Porting to a new platform would be easier using lower-level tests against the LibC interface directly.

The TestCase classes should cover, at minimum, the functions and structs we use.

UUID and UUIDGenerator class triage

The UUID and UUIDGEnerator classes are used by Smalltalk and for the port of Rowan to SmalltalkCI (hpi-swa/smalltalkCI#387), I have started work on porting those two classes the base ... not a log of progress yet, but these two projects can both benefit from that work.

Note that for GemStone we will be exclusively using the FastUUIDGenerator class.

FileReference - remove matching selectors

The following selectors are unused by Rowan. They seems like they could belong in an extension package in the future but not in the basic FileSystem implementation.

There are a some senders in Pharo but Rowan does not send them. Regardless, the pattern matching differs between Pharo and Gemstone. GemStone appears to do a prefix test. Pharo support wildcards in the pattern.

• childrenMatching:
• allChildrenMatching:
• directoriesMatching:
• allDirectoriesMatching:
• filesMatching:
• lastFileFor:extension:
• nextNameFor:extension
• childGeneratorBlock:matching:

DiskFileSystemTest failure

Several bits of functionality are missing which prevent the DiskFileSystemTest from passing.

• stat open file descriptor
• create/delete file
• set position in a file
• create/delete directory
• rename file

FileReferenceTest Failures

The stat function and associated data structures are not yet available in GemStone. An FFI call should be generated and the associated structures should be created.

List of class references in FileSystemGs that need to be resolved

This task/issue should be used to triage the disposition of each of the classes with separate tasks/issues for each of the classes that require significant work and or disucssion.

For each class we will decide to resolve the reference

• by eliminating code that references class
• replacing references with a GemStone equivalent
• porting the class from Pharo

When a decision is made as to how the reference is resolved, let's create an issue/task for that class with the planned resolution ... the issue will serve as the focal point for further discussion ... when the issue is resolved, we can check off the class

Master list of classes (from #1):

• CodeImporter
• File
• FileDoesNotExistException (task #3)
• Message
• NotFound
• OSPlatform
• PackageManifest (#7)
• SessionManager
• Smalltalk
• Stdio
• UIManager
• UUID (#8)
• UUIDGenerator (#8)
• WeakRegistry
• ZipArchive
• ZnBufferedReadStream
• ZnBufferedReadWriteStream
• ZnCharacterReadStream
• ZnCharacterWriteStream
• ZnMimeType

Port `FileDoesNotExistException` from Pharo

Pretty clear that this exception class should be created initially based upon the Pharo 7.0 implementation.

POSIX Version Target

Linux supports extensions to POSIX not available on other platforms. For instance, O_PATH is an option that can be provided to open. This extension does not exist on macOS. The same is true in reverse. macOS supports O_SHLOCK which isn't present on Linux.

Initial versions of FileSystemGs targeted Linux and included various Linux extensions like O_PATH. In adding support for macOS, many of these differences were found and isolated into Linux specific classes.

macOS is certified against Unix 3 (POSIX.1-2001). Linux does not have a specific certification. It is up to each distro to provide as much (or little) POSIX compliance as it would like. (This is my understanding anyways.) In practice, we only use features in glibc which should be present on any system we will run on.

It seems like a better option, for the time being, to only support what is specified in POSIX (perhaps targeting POSIX.1-2001). Lower-level tests would target the behavior we use from this version of POSIX. Each test would test semantics rather than platforms-specific, low-level details such as the value of O_CREAT.

Separate low-level classes would persist to encode the details of O_CREAT and the function name to call.

Once we have a supported base, we could expand into platform-specific features as necessary.

See: #48

Set current working directory

FileReference should have a selector that sets the current working directory. The selector should be called #setAsWorkingDirectory.

In the event that FileReference does to reference a Directory, DirectoryExpected should be raised.

FileSystemVisitor removal

There are several FileSystem visitors that are used for deletion, copying, etc... These really aren't necessary. It looks as though each algorithm would be straight forward to implement iteratively.

Struct field access

FileSystem is being implemented via the FFI. Field sizes and offsets are being calculated and hardcoded into Smalltalk code so we are able to access fields in a struct. In the case of the stat struct, the man page explicitly states the following.

 Note: the order of fields in the stat structure varies somewhat  across
       architectures.   In  addition,  the  definition above does not show the
       padding bytes that may be present between some fields on various archi‐
       tectures.  Consult the glibc and kernel source code if you need to know
       the details.

This suggests that FileSystem is going to need a more generic way off accessing the fields in the struct or we will need to use different struct classes w/ appropriate offsets for different architectures.

macOS stat function

On Linux, stat is a macro which ultimately calls __xstat. On macOS, stat is a function. Several years ago, they added support for 64-bit inodes w/ the stat64 function. This is now deprecated and callers should use normal stat function.

Invoking stat in C calls stat$INODE64. The original stat function is still available and can be called via dlopen/dlsym. This function does not produce the same results as stat$INODE64. Presumably, it is the old version of the function which operated on 32-bit inodes.

Remove LibcLibrary classes

Per #46, the remaining LibcLibrary classes should be removed. The CCallout instances will live in the LibcInterface subclasses.

Update/Create build script to include creation of dummy classes

From @nrgiii ... Here is the code to initialize the database with the phony classes:

run
|blk eblk list |
blk := [:name|
  Object subclass: name instVarNames: #() classVars: #() classInstVars: #() poolDictionaries: #() inDictionary: Globals options: #()].
eblk := [:name|
  Exception subclass: name instVarNames: #() classVars: #() classInstVars: #() poolDictionaries: #() inDictionary: Globals options: #()].

"Install exception subclasses"
eblk value: 'FileDoesNotExistException' .
eblk value: 'NotFound' .


list := {
'PackageManifest' .
'SessionManager' .
'Smalltalk' .
'File' .
'CodeImporter' .
'ZnMimeType' .
'ZnBufferedReadStream' .
'ZnCharacterReadStream' .
'UIManager' .
'UUID' .
'OSPlatform' .
'ZnBufferedReadWriteStream' .
'ZnCharacterWriteStream' .
'Message' .
'WeakRegistry' .
'ZipArchive' .
'UUIDGenerator' .
'Stdio' } .

"Install Object subclasses"
list do:[:e| blk value: e] .
true
%

And here are the things needed to be done:

• create topaz script that loads Rowan
• create dummy classes
• loads FileSystemGs bypassing initialization of the FileSystemGs classes

FileReferenceTest>>#testWriteStream*

FileReferenceTest has a large set of test cases which are currently failing. Several tests, for instance #testAllChildren, fail because they cannot create files. The current implementation opens and closes a write stream to create the basic file.

Fixing the writeStream related tests should make progress on the other failing tests cases easier.

Calling LibC functions on macOS via FFI

On Linux, we bind to libc by binding to libc.so.6. On macOS, it seems the equivalent in older versions of macOS is called libsystem_c.dylib and is stored in /usr/lib/system/. This doesn't appear to exist on Big Sur.

How does macOS expect an FFI to bind to libc and call libc functions? Has this changed w/ Big Sur? Will the implementation have to take this into account?

support for posix permissions for files and directories deferred

No direct support for permissions in GsFile class >> #stat:isLstat: probably need additional primitive support ... defer handling of symbolic links until a later date ...

Script/doit for running FileSystemGs tests ...

I've written a topaz script for Rowan to run tests ... should do the same for FileSystemGs ...

support for symbolic links for files and directories deferred

The implmentation of GsFile class >> #stat:isLstat: is not too clear in relation to handling symbolic links for files and directories ... defer handling of symbolic links until a later date ...

Write/Read Stream Object Structure

UTF-8 stream decoding unsupported causing test failures

The following tests fail because ZnUTF8Encoder does not support reading the next code point from a stream nor does it support moving backward on a stream.

FileReferenceTest>>#testUpToAll
ZnCharacterEncoderTests>>#testUTF8Back

potential issues related to #exists and file permissions

For details take a look at this travis build... specifically stack was a result of attempting to read the file named st_launcher_home.ston in this directory:

$ ls -altr $HOME/.st_launcher
total 16
drwxr-x--- 25 travis travis 4096 Jul 17 19:19 ..
-rw-r--r--  1 root   root    425 Jul 17 19:19 st_launcher_default.env
-rw-r--r--  1 root   root    361 Jul 17 19:19 st_launcher_home.ston
drwxr-xr-x  2 root   root   4096 Jul 17 19:19 .

While this travis build based on the same st_launcher commit runs without error with these permissions:

$ ls -altr $HOME/.st_launcher
total 16
-rw-rw-r--  1 travis travis  425 Jul 17 19:19 st_launcher_default.env
-rw-rw-r--  1 travis travis  361 Jul 17 19:19 st_launcher_home.ston
drwxr-x--- 25 travis travis 4096 Jul 17 19:19 ..
drwxr-xr-x  2 root   root   4096 Jul 17 19:19 .

Fix failing Path test cases

• #testFullName
• #testPathString

arm64 support

arm64 support is required for macOS and Linux.

Known differences:

• macOS
  • arm64 uses stat instead of stat$INODE64
• Linux
  • O_NOFOLLOW and O_DIRECTORY have different values
  • struct stat is smaller on aarch64
  • struct stat offsets differ

FileReference - #isReadable and #isWritable assume ownership

Permissions check selectors fail to take into account that a user might not own a file. For instance, a file like /dev/sda is readable to root and its group but is not readable by a normal user.

FileReference should reflect the permissions of the currently active user rather than the owner of the file.

FileLocator class>>resolver should be persistent ...

right now it's stuck in sesson temps which means that it isn't easy to extend the list of resolvers ... would like to add some Rowan resolvers, but right now I have to modify SystemResolver which is not cool ... probably need to have user-specific resolvers stashed in UserGlobals, system-wide resolvers, and session specific resolvers ....

Test Triage: MemoryFileSystemTest and MemoryHandleTest up first

Currently there are 462 tests failing out of 478 total.

Looking over the tests, I am reminded that the memory-based filesystem is probably a good place to start. There are (likely) no primitives involved and these tests should flush out the the Pharo-specific methods (MNUs for #ifEmpty and #split: in the first two tests I tried), so these two tests should probably be tackled first:

• MemoryFileSystemTest
• MemoryHandleTest
  When these two tests are running clean, we are likely to be left with significantly few test failures and we can revisit the remaining list for the next round of triage.

Here's the current list failures/errors:

FileSystemGs tests for GemStone '3.2.15'
478 run, 16 passed, 10 failed, 452 errors
  errors
	BreadthFirstGuideTest>>#testAll
	CollectVisitorTest>>#testBreadthFirst
	CollectVisitorTest>>#testPostorder
	CollectVisitorTest>>#testPreorder
	CopyVisitorTest>>#testAll
	DeleteVisitorTest>>#testBeta
	DirectoryEntryTest>>#testCreationTimeIsADateAndTimeInstance
	DirectoryEntryTest>>#testIsDirectory
	DirectoryEntryTest>>#testIsFile
	DirectoryEntryTest>>#testIsNotDirectory
	DirectoryEntryTest>>#testIsNotFile
	DirectoryEntryTest>>#testModificationTimeIsADateAndTimeInstance
	DirectoryEntryTest>>#testReference
	DirectoryEntryTest>>#testSize
	DiskFileSystemTest>>#testBinaryReadStream
	DiskFileSystemTest>>#testBinaryReadStream
	DiskFileSystemTest>>#testBinaryReadStreamDo
	DiskFileSystemTest>>#testBinaryReadStreamDo
	DiskFileSystemTest>>#testBinaryReadStreamDoIfAbsent
	DiskFileSystemTest>>#testBinaryReadStreamDoIfAbsent
	DiskFileSystemTest>>#testBinaryReadStreamIfAbsent
	DiskFileSystemTest>>#testBinaryReadStreamIfAbsent
	DiskFileSystemTest>>#testChildrenAt
	DiskFileSystemTest>>#testChildrenAt
	DiskFileSystemTest>>#testChildrenSorting
	DiskFileSystemTest>>#testChildrenSorting
	DiskFileSystemTest>>#testChildrenSortingRoot
	DiskFileSystemTest>>#testChildrenSortingRoot
	DiskFileSystemTest>>#testCopy
	DiskFileSystemTest>>#testCopy
	DiskFileSystemTest>>#testCopyAndDelete
	DiskFileSystemTest>>#testCopyAndDelete
	DiskFileSystemTest>>#testCopyDestExists
	DiskFileSystemTest>>#testCopyDestExists
	DiskFileSystemTest>>#testCopySourceDoesntExist
	DiskFileSystemTest>>#testCopySourceDoesntExist
	DiskFileSystemTest>>#testCopyWithCorrectBasename
	DiskFileSystemTest>>#testCopyWithCorrectBasename
	DiskFileSystemTest>>#testCreateDirectory
	DiskFileSystemTest>>#testCreateDirectory
	DiskFileSystemTest>>#testCreateDirectoryExists
	DiskFileSystemTest>>#testCreateDirectoryExists
	DiskFileSystemTest>>#testCreateDirectoryNoParent
	DiskFileSystemTest>>#testCreateDirectoryNoParent
	DiskFileSystemTest>>#testCreateDirectoryNotCreateParent
	DiskFileSystemTest>>#testCreateDirectoryNotCreateParent
	DiskFileSystemTest>>#testCreateFile
	DiskFileSystemTest>>#testCreateFile
	DiskFileSystemTest>>#testCreateFileNotCreateParent
	DiskFileSystemTest>>#testCreateFileNotCreateParent
	DiskFileSystemTest>>#testDefaultWorkingDirectory
	DiskFileSystemTest>>#testDefaultWorkingDirectory
	DiskFileSystemTest>>#testDelete
	DiskFileSystemTest>>#testDelete
	DiskFileSystemTest>>#testDelimiter
	DiskFileSystemTest>>#testDelimiter
	DiskFileSystemTest>>#testDirectory
	DiskFileSystemTest>>#testDirectory
	DiskFileSystemTest>>#testEnsureDirectory
	DiskFileSystemTest>>#testEnsureDirectory
	DiskFileSystemTest>>#testEnsureDirectoryCreatesParent
	DiskFileSystemTest>>#testEnsureDirectoryCreatesParent
	DiskFileSystemTest>>#testEnsureDirectoryExists
	DiskFileSystemTest>>#testEnsureDirectoryExists
	DiskFileSystemTest>>#testEntriesAt
	DiskFileSystemTest>>#testEntriesAt
	DiskFileSystemTest>>#testEntryAt
	DiskFileSystemTest>>#testEntryAt
	DiskFileSystemTest>>#testEqual
	DiskFileSystemTest>>#testEqual
	DiskFileSystemTest>>#testFile
	DiskFileSystemTest>>#testFile
	DiskFileSystemTest>>#testFileNames
	DiskFileSystemTest>>#testFileNames
	DiskFileSystemTest>>#testIsDirectory
	DiskFileSystemTest>>#testIsDirectory
	DiskFileSystemTest>>#testIsDiskFileSystem
	DiskFileSystemTest>>#testIsDiskFileSystem
	DiskFileSystemTest>>#testMoveMemoryToDisk
	DiskFileSystemTest>>#testMoveMemoryToDisk
	DiskFileSystemTest>>#testMoveTo
	DiskFileSystemTest>>#testMoveTo
	DiskFileSystemTest>>#testMoveToFailingExistingDestination
	DiskFileSystemTest>>#testMoveToFailingExistingDestination
	DiskFileSystemTest>>#testMoveToFailingMissingDestination
	DiskFileSystemTest>>#testMoveToFailingMissingDestination
	DiskFileSystemTest>>#testMoveToFailingMissingSource
	DiskFileSystemTest>>#testMoveToFailingMissingSource
	DiskFileSystemTest>>#testNonExistentEntryAt
	DiskFileSystemTest>>#testNonExistentEntryAt
	DiskFileSystemTest>>#testNonExistentFileSize
	DiskFileSystemTest>>#testNonExistentFileSize
	DiskFileSystemTest>>#testReadingAfterWriting
	DiskFileSystemTest>>#testReadingAfterWriting
	DiskFileSystemTest>>#testReadStream
	DiskFileSystemTest>>#testReadStream
	DiskFileSystemTest>>#testReadStreamDo
	DiskFileSystemTest>>#testReadStreamDo
	DiskFileSystemTest>>#testReadStreamDoIfAbsent
	DiskFileSystemTest>>#testReadStreamDoIfAbsent
	DiskFileSystemTest>>#testReadStreamIfAbsent
	DiskFileSystemTest>>#testReadStreamIfAbsent
	DiskFileSystemTest>>#testReferenceTo
	DiskFileSystemTest>>#testReferenceTo
	DiskFileSystemTest>>#testRoot
	DiskFileSystemTest>>#testRoot
	DiskFileSystemTest>>#testRootExists
	DiskFileSystemTest>>#testRootExists
	DiskFileSystemTest>>#testRootIsDirectory
	DiskFileSystemTest>>#testRootIsDirectory
	DiskFileSystemTest>>#testRootIsNotAFile
	DiskFileSystemTest>>#testRootIsNotAFile
	DiskFileSystemTest>>#testWorking
	DiskFileSystemTest>>#testWorking
	DiskFileSystemTest>>#testWriteStream
	DiskFileSystemTest>>#testWriteStream
	DiskFileSystemTest>>#testWriteStreamDo
	DiskFileSystemTest>>#testWriteStreamDo
	DiskFileSystemTest>>#testWriteStreamDoIfPresent
	DiskFileSystemTest>>#testWriteStreamDoIfPresent
	DiskFileSystemTest>>#testWriteStreamIfPresent
	DiskFileSystemTest>>#testWriteStreamIfPresent
	FileHandleTest>>#testAt
	FileHandleTest>>#testAt
	FileHandleTest>>#testAtPut
	FileHandleTest>>#testAtPut
	FileHandleTest>>#testAtPutBinaryAscii
	FileHandleTest>>#testAtPutBinaryAscii
	FileHandleTest>>#testAtWriteBinaryAscii
	FileHandleTest>>#testAtWriteBinaryAscii
	FileHandleTest>>#testClose
	FileHandleTest>>#testClose
	FileHandleTest>>#testCreatedOpen
	FileHandleTest>>#testCreatedOpen
	FileHandleTest>>#testEnsureClosed
	FileHandleTest>>#testEnsureClosed
	FileHandleTest>>#testIO
	FileHandleTest>>#testIO
	FileHandleTest>>#testReadBufferTooLarge
	FileHandleTest>>#testReadBufferTooLarge
	FileHandleTest>>#testReadOnly
	FileHandleTest>>#testReadOnly
	FileHandleTest>>#testReference
	FileHandleTest>>#testReference
	FileHandleTest>>#testSizeAfterGrow
	FileHandleTest>>#testSizeAfterGrow
	FileHandleTest>>#testSizeNoGrow
	FileHandleTest>>#testSizeNoGrow
	FileHandleTest>>#testTruncate
	FileHandleTest>>#testTruncate
	FileHandleTest>>#testWriteStream
	FileHandleTest>>#testWriteStream
	FileLocatorTest>>#testAsAbsolute
	FileLocatorTest>>#testBasename
	FileLocatorTest>>#testCommaAddsExtension
	FileLocatorTest>>#testCommaAddsExtensionAgain
	FileLocatorTest>>#testContainsLocator
	FileLocatorTest>>#testContainsPath
	FileLocatorTest>>#testContainsReference
	FileLocatorTest>>#testCPath
	FileLocatorTest>>#testEqual
	FileLocatorTest>>#testExtension
	FileLocatorTest>>#testFileSystem
	FileLocatorTest>>#testImageDirectory
	FileLocatorTest>>#testIsAbsolute
	FileLocatorTest>>#testIsNotRoot
	FileLocatorTest>>#testIsRelative
	FileLocatorTest>>#testIsRoot
	FileLocatorTest>>#testLocalDirectory
	FileLocatorTest>>#testMoveTo
	FileLocatorTest>>#testMoveTo
	FileLocatorTest>>#testOriginBasename
	FileLocatorTest>>#testParent
	FileLocatorTest>>#testResolveAbsoluteReference
	FileLocatorTest>>#testResolveCompoundString
	FileLocatorTest>>#testResolvePath
	FileLocatorTest>>#testResolveRelativeReference
	FileLocatorTest>>#testResolveString
	FileLocatorTest>>#testSlash
	FileLocatorTest>>#testWithExtensionAddsExtension
	FileLocatorTest>>#testWithExtensionReplacesExtension
	FileReferenceAttributeTests>>#testCreationTime
	FileReferenceAttributeTests>>#testCreationTime
	FileReferenceTest>>#testAllChildren
	FileReferenceTest>>#testAllChildren
	FileReferenceTest>>#testAllDirectories
	FileReferenceTest>>#testAllDirectories
	FileReferenceTest>>#testAllEntries
	FileReferenceTest>>#testAllEntries
	FileReferenceTest>>#testAsAbsoluteConverted
	FileReferenceTest>>#testAsAbsoluteIdentity
	FileReferenceTest>>#testAsReference
	FileReferenceTest>>#testBaseAndExtension
	FileReferenceTest>>#testBasename
	FileReferenceTest>>#testBasenameWithoutExtension
	FileReferenceTest>>#testCanonicalization
	FileReferenceTest>>#testChildDirectories
	FileReferenceTest>>#testChildDirectories
	FileReferenceTest>>#testChildFiles
	FileReferenceTest>>#testChildFiles
	FileReferenceTest>>#testChildOfPath
	FileReferenceTest>>#testChildOfReference
	FileReferenceTest>>#testChildren
	FileReferenceTest>>#testChildren
	FileReferenceTest>>#testCommaAddsExtension
	FileReferenceTest>>#testCommaAddsExtensionAgain
	FileReferenceTest>>#testContainsLocator
	FileReferenceTest>>#testContainsPath
	FileReferenceTest>>#testContainsReference
	FileReferenceTest>>#testContents
	FileReferenceTest>>#testDeleteAll
	FileReferenceTest>>#testDeleteAll
	FileReferenceTest>>#testDeleteAllChildren
	FileReferenceTest>>#testDeleteAllChildren
	FileReferenceTest>>#testDeleteIfAbsent
	FileReferenceTest>>#testDoesContainReferenceFileSystem
	FileReferenceTest>>#testDoesNotContainReferenceWhenUsingDifferentInstancesOfMemoryFileSystem
	FileReferenceTest>>#testDoesntContainLocator
	FileReferenceTest>>#testDoesntContainPath
	FileReferenceTest>>#testDoesntContainReferencePath
	FileReferenceTest>>#testEnsureDelete
	FileReferenceTest>>#testEnsureDeleteAll
	FileReferenceTest>>#testEnsureDeleteNonEmptyDirectory
	FileReferenceTest>>#testEntries
	FileReferenceTest>>#testEntries
	FileReferenceTest>>#testEqual
	FileReferenceTest>>#testEqualityRelativeVsAbsolute
	FileReferenceTest>>#testExists
	FileReferenceTest>>#testGlob
	FileReferenceTest>>#testGlob
	FileReferenceTest>>#testGrandchildOfReference
	FileReferenceTest>>#testHasChildren
	FileReferenceTest>>#testHasChildren
	FileReferenceTest>>#testHasDirectories
	FileReferenceTest>>#testHasDirectories
	FileReferenceTest>>#testHasFiles
	FileReferenceTest>>#testHasFiles
	FileReferenceTest>>#testIndicator
	FileReferenceTest>>#testIsAbsolute
	FileReferenceTest>>#testIsNotAbsolute
	FileReferenceTest>>#testIsNotRelative
	FileReferenceTest>>#testIsNotRoot
	FileReferenceTest>>#testIsRelative
	FileReferenceTest>>#testMakeRelative
	FileReferenceTest>>#testParent
	FileReferenceTest>>#testParentResolutionWithAbsoluteReference
	FileReferenceTest>>#testParentResolutionWithPath
	FileReferenceTest>>#testParentResolutionWithReference
	FileReferenceTest>>#testParentResolutionWithRemoteReference
	FileReferenceTest>>#testParentUpTo
	FileReferenceTest>>#testParentUpTo
	FileReferenceTest>>#testPathRelativeTo
	FileReferenceTest>>#testReadStream
	FileReferenceTest>>#testReadStreamDo
	FileReferenceTest>>#testReadStreamDoifAbsent
	FileReferenceTest>>#testReadStreamDoifAbsentNot
	FileReferenceTest>>#testReadStreamDoNotFound
	FileReferenceTest>>#testReadStreamIfAbsent
	FileReferenceTest>>#testReadStreamNotFound
	FileReferenceTest>>#testRelativeToPath
	FileReferenceTest>>#testRelativeToReference
	FileReferenceTest>>#testRename
	FileReferenceTest>>#testRename
	FileReferenceTest>>#testRenameTargetExists
	FileReferenceTest>>#testRenameTargetExists
	FileReferenceTest>>#testResolve
	FileReferenceTest>>#testSiblingOfReference
	FileReferenceTest>>#testSimpleResolution
	FileReferenceTest>>#testSlash
	FileReferenceTest>>#testTempFilePrefixSuffix
	FileReferenceTest>>#testUnequalContent
	FileReferenceTest>>#testUnequalSize
	FileReferenceTest>>#testUpToAll
	FileReferenceTest>>#testWithExtentionAddsExtension
	FileReferenceTest>>#testWithExtentionReplacesExtension
	FileReferenceTest>>#testWithoutExtension
	FileReferenceTest>>#testWorkingDirectoryParent
	FileReferenceTest>>#testWriteStream
	FileReferenceTest>>#testWriteStreamDo
	FileReferenceTest>>#testWriteStreamDoExists
	FileReferenceTest>>#testWriteStreamDoifPresent
	FileReferenceTest>>#testWriteStreamDoifPresentNot
	FileReferenceTest>>#testWriteStreamExists
	FileReferenceTest>>#testWriteStreamifPresent
	FileReferenceTest>>#testWriteStreamifPresentExists
	InteractiveResolverTest>>#testCached
	InteractiveResolverTest>>#testNew
	MemoryFileSystemTest>>#testBinaryReadStream
	MemoryFileSystemTest>>#testBinaryReadStreamDo
	MemoryFileSystemTest>>#testBinaryReadStreamDoIfAbsent
	MemoryFileSystemTest>>#testBinaryReadStreamIfAbsent
	MemoryFileSystemTest>>#testChildrenAt
	MemoryFileSystemTest>>#testChildrenSorting
	MemoryFileSystemTest>>#testChildrenSortingRoot
	MemoryFileSystemTest>>#testCopy
	MemoryFileSystemTest>>#testCopyAndDelete
	MemoryFileSystemTest>>#testCopyDestExists
	MemoryFileSystemTest>>#testCopySourceDoesntExist
	MemoryFileSystemTest>>#testCopyWithCorrectBasename
	MemoryFileSystemTest>>#testCreateDirectory
	MemoryFileSystemTest>>#testCreateDirectoryExists
	MemoryFileSystemTest>>#testCreateDirectoryNoParent
	MemoryFileSystemTest>>#testCreateDirectoryNotCreateParent
	MemoryFileSystemTest>>#testCreateFile
	MemoryFileSystemTest>>#testCreateFileNotCreateParent
	MemoryFileSystemTest>>#testDelete
	MemoryFileSystemTest>>#testDelimiter
	MemoryFileSystemTest>>#testDirectory
	MemoryFileSystemTest>>#testEnsureDirectory
	MemoryFileSystemTest>>#testEnsureDirectoryCreatesParent
	MemoryFileSystemTest>>#testEnsureDirectoryExists
	MemoryFileSystemTest>>#testEntriesAt
	MemoryFileSystemTest>>#testEntryAt
	MemoryFileSystemTest>>#testFile
	MemoryFileSystemTest>>#testFileNames
	MemoryFileSystemTest>>#testModifiedTimeWhenFileCreated
	MemoryFileSystemTest>>#testModifiedTimeWhenFileModifiedByWriteStream
	MemoryFileSystemTest>>#testModifiedTimeWhenFileModifiedWithBinaryWriteStream
	MemoryFileSystemTest>>#testModifiedTimeWhenFileWrittenTo
	MemoryFileSystemTest>>#testModifiedTimeWhenHandleTruncated
	MemoryFileSystemTest>>#testMoveTo
	MemoryFileSystemTest>>#testMoveToFailingExistingDestination
	MemoryFileSystemTest>>#testMoveToFailingMissingDestination
	MemoryFileSystemTest>>#testMoveToFailingMissingSource
	MemoryFileSystemTest>>#testNonExistentEntryAt
	MemoryFileSystemTest>>#testNonExistentFileSize
	MemoryFileSystemTest>>#testReadingAfterWriting
	MemoryFileSystemTest>>#testReadStream
	MemoryFileSystemTest>>#testReadStreamDo
	MemoryFileSystemTest>>#testReadStreamDoIfAbsent
	MemoryFileSystemTest>>#testReadStreamIfAbsent
	MemoryFileSystemTest>>#testReferenceTo
	MemoryFileSystemTest>>#testRootIsNotAFile
	MemoryFileSystemTest>>#testWriteStream
	MemoryFileSystemTest>>#testWriteStreamDo
	MemoryFileSystemTest>>#testWriteStreamDoIfPresent
	MemoryFileSystemTest>>#testWriteStreamIfPresent
	MemoryHandleTest>>#testAt
	MemoryHandleTest>>#testAt
	MemoryHandleTest>>#testAtPut
	MemoryHandleTest>>#testAtPut
	MemoryHandleTest>>#testAtPutBinaryAscii
	MemoryHandleTest>>#testAtPutBinaryAscii
	MemoryHandleTest>>#testAtWriteBinaryAscii
	MemoryHandleTest>>#testAtWriteBinaryAscii
	MemoryHandleTest>>#testClose
	MemoryHandleTest>>#testClose
	MemoryHandleTest>>#testCreatedOpen
	MemoryHandleTest>>#testCreatedOpen
	MemoryHandleTest>>#testEnsureClosed
	MemoryHandleTest>>#testEnsureClosed
	MemoryHandleTest>>#testIO
	MemoryHandleTest>>#testIO
	MemoryHandleTest>>#testReadBufferTooLarge
	MemoryHandleTest>>#testReadBufferTooLarge
	MemoryHandleTest>>#testReadOnly
	MemoryHandleTest>>#testReadOnly
	MemoryHandleTest>>#testReference
	MemoryHandleTest>>#testReference
	MemoryHandleTest>>#testSizeAfterGrow
	MemoryHandleTest>>#testSizeAfterGrow
	MemoryHandleTest>>#testSizeNoGrow
	MemoryHandleTest>>#testSizeNoGrow
	MemoryHandleTest>>#testTruncate
	MemoryHandleTest>>#testTruncate
	MemoryHandleTest>>#testWriteStream
	MemoryHandleTest>>#testWriteStream
	PathTest>>#testAbsolutePath
	PathTest>>#testAbsolutePrintString
	PathTest>>#testAbsoluteWithParents
	PathTest>>#testAsReference
	PathTest>>#testBasename
	PathTest>>#testBasenameNoParent
	PathTest>>#testBasenameWithoutExtension
	PathTest>>#testCanonicalization
	PathTest>>#testCommaAddsExtension
	PathTest>>#testCommaAddsExtensionAgain
	PathTest>>#testContains
	PathTest>>#testContainsLocator
	PathTest>>#testEqual
	PathTest>>#testExtendingPath
	PathTest>>#testExtensions
	PathTest>>#testFullName
	PathTest>>#testGrandchildOfPath
	PathTest>>#testIsAbsolute
	PathTest>>#testIsAbsoluteWindowsPathReturnsFalseWhenNoWindowsAbsolutePathProvided
	PathTest>>#testIsAbsoluteWindowsPathReturnsTrueWhenWindowsAbsolutePathProvided
	PathTest>>#testIsChildOfPath
	PathTest>>#testIsChildOfReference
	PathTest>>#testIsNotAbsolute
	PathTest>>#testIsNotRelative
	PathTest>>#testIsNotRoot
	PathTest>>#testIsRelative
	PathTest>>#testMakeRelative
	PathTest>>#testMakeRelativeFrom2RelativePaths
	PathTest>>#testParent
	PathTest>>#testParentParent
	PathTest>>#testParentResolution
	PathTest>>#testParentUpTo
	PathTest>>#testParse
	PathTest>>#testParseBogus
	PathTest>>#testParseTrailingSlash
	PathTest>>#testParseWindowsPathWithUnixDelimiters
	PathTest>>#testPathString
	PathTest>>#testPrintPathOn
	PathTest>>#testPrintPathOnDelimiter
	PathTest>>#testPrintRelativeWithParent
	PathTest>>#testPrintWithDelimiter
	PathTest>>#testRedundantSeparators
	PathTest>>#testRelativeFromString
	PathTest>>#testRelativeFromStringNormalization
	PathTest>>#testRelativeFromStringNormalizationParent
	PathTest>>#testRelativeFromStringParent
	PathTest>>#testRelativePrintString
	PathTest>>#testRelativeTo
	PathTest>>#testRelativeToBranch
	PathTest>>#testRelativeWithParents
	PathTest>>#testResolveAbsolute
	PathTest>>#testResolvePath
	PathTest>>#testResolveRelative
	PathTest>>#testResolveString
	PathTest>>#testSiblingOfPath
	PathTest>>#testSimpleResolution
	PathTest>>#testSlash
	PathTest>>#testUnequalContent
	PathTest>>#testUnequalSize
	PathTest>>#testUnixAbsolutePathName
	PathTest>>#testWindowsAbsolutePathName
	PathTest>>#testWithExtentionAddsExtension
	PathTest>>#testWithExtentionReplacesExtension
	PathTest>>#testWorkingDirectoryParent
	PlatformResolverTest>>#testCache
	PlatformResolverTest>>#testHome
	PostorderGuideTest>>#testAll
	PreorderGuideTest>>#testAll
	SelectVisitorTest>>#testBreadthFirst
	SelectVisitorTest>>#testBreadthFirstSelect
	SelectVisitorTest>>#testPostorder
	SelectVisitorTest>>#testPostorderSelect
	SelectVisitorTest>>#testPreorder
	SelectVisitorTest>>#testPreorderSelect
	SystemResolverTest>>#testChanges
	SystemResolverTest>>#testImage
	SystemResolverTest>>#testImageDirectory
	SystemResolverTest>>#testLocalDirectory
	SystemResolverTest>>#testVmBinary
	SystemResolverTest>>#testVmDirectory
	WindowsStoreTest>>#testAbsoluteFullName
	WindowsStoreTest>>#testAbsolutePath
	WindowsStoreTest>>#testPrintString
	WindowsStoreTest>>#testRelativeFullName
	WindowsStoreTest>>#testRelativePath
  failures
	FileReferenceAttributeTests>>#testExists
	FileReferenceAttributeTests>>#testFileSize
	FileReferenceAttributeTests>>#testIsBlock
	FileReferenceAttributeTests>>#testIsCharacter
	FileReferenceAttributeTests>>#testIsDirectory
	FileReferenceAttributeTests>>#testIsFile
	FileReferenceAttributeTests>>#testIsReadable
	FileReferenceAttributeTests>>#testIsSymlink
	FileReferenceAttributeTests>>#testIsWriteable
	FileReferenceAttributeTests>>#testModificationTime

FFI calls should use #callWith:errno:

The old style of obtaining errno leaves open the possibility that Smalltalk could obtain an errno value for another FFI call.

CCallout errno: 0.
callout callWith: { arg1 .. argN }.
errno := CCallout errno.

In the above example, errno is obtained by sending a Smalltalk message. When the #errno send is performed, interrupts could cause the errno value to be invalidated. For instance, if a GC is triggered and Ephemeron finalization occurs after #callWith: completes but before #errno returns a value, another FFI call could occur. A similar issue can occur between the send of #errno: and #callWith:.

errnoHolder := { 0 }.
callout
  callWith: { arg1 .. argN }
  errno: errnoHolder.
errno := errnoHolder at: 1.

This code example is equivalent to the one above. The second example guarantees errno will be set to value provided in errnoHolder (if it isn't nil) and that the value in the holder will be the errno associated with the FFI call. This is true even if Ephemeron finalization occurs or other interrupts invalidate the #errno value stored in the GsProcess.

LibC Wrapper Library

Martin and I had a discussion regarding issue #46. I walked him through the process of discovering the actual stat function I need to invoke on macOS. The process is labor intensive and there doesn't appear to be a straight forward way of obtaining actual function names for posix functions. In the case of stat, I needed to disassemble a program in LLDB in order to obtain the name stat$INODE64.

We came to the belief that wrapping LibC w/ a GemStone specific library shim library and the binding to that would be more useful. This would also make Smalltalk code less platform specific.

The library can also expose the size of structs, as well as the offsets for each field. This would make porting to new platforms and architectures a matter of compiling the wrapper library and running.

This would is likely to start once the existing implementation is ported to macOS, arm64, and UTF-8 decoding is implemented. This will allow the integration of FileSystem and Rowan sooner.

Merge FileSytem work from GemTalk/Rowan FileSystem to GemTalk/FileSystemGs

This post seems to be the "best way" to move files from Rowan to Filesystem while preserving commit histories (I think) ... @regkrock has done a lot of work on the port and I want to preserve the history if possible ..

FileAttributes

Right now, there are various calls into FileSystem and FileSystemStore to access basic attributes of a file. There is also a second object called FileSystemDirectoryEntry which provides access to all of these attributes of a file independently. Examples include creation time and whether it is a file or directory.

The various calls in FileSystem/FileSystemStore should be removed. FileSystemDirectoryEntry should be deprecated and replaced by a FileAttributes class which provides access to the file's attributes.

The existing test cases are insufficient. In several places, they ensure the response is not nil rather than actually validating the response. In some cases, stubbing out methods w/ ^0 was enough to make the tests pass. This should be rectified in the tests for FileAttributes.

MemoryFileSystemTest failures

#testModifiedTime*
#testNonExistentFileSize

PackageManifest is a Pharo meta-data class and should be ignored

The class comment for PackageManifest tells the tail:

"
I store meta-data for a package. I'm the common superclass of all package Manifest.
"

We can simply remove the classes ...

ZnEncodedStream expect to encode/decode UTF8 at the character level

The current implementation is stylized to pass tests ... I don't think it is practical to incorporate the ICU streaming API for 3.2.17 and the stylized implementation will be sufficient for reading and writing Tonel and FileTree packages ... but for 3.4/2.5 and beyond, we ought to bite the bullet and implement the ICU streaming API

FileReference tests use memory based file system

FileReference tests should be run against each FileSystem to ensure high-level behavior is properly tested against each file system. The tests should be factored to use a sandbox directory. Test files and directories should be created in the sandbox.

The sandbox for disk-based file systems should be within an appropriate temporary directory.

UnixFileDescriptor Finalization

UnixFileDescriptor can leak system file descriptors unless we incorporate a finalization mechanism which closes the descriptor before garbage collection.

Martin created ExternalObjectRegistry to handle this case. FileSystem does not use it yet.

The CLibrary associated w/ libc.so.6 should not hardcode a full library path

The path /lib64/libc.so.6 is not valid on all flavors of linux. Instead, the library should be looked up in the library path by name alone.

MacOS support

The scope of this is not yet know.

Areas of known problems:

• #31 Struct field offsets will likely be different
• Functions like stat are implemented as macros on Linux so the implementation calls statx instead

Migrate to UnixFileOpeningOptions

To get the tests working, file opening used a simple writable flag. Instead, we should migrate to using UnixFileOpeningOptions which allow the code to more clearly specify options like 'should we create the file if it does not exist' rather than assuming yes in low-level code.

Add support for basic writing to a file descriptor

Basic read from UnixFileDescriptor support

Add #read: function to UnixFileDescriptor which will read up to the provided number of bytes. The bytes read should be return in a ByteArray.

Renamed DirectoryExpected and FileExpected

The word expected doesn't quite capture the intention of the exception. Instead, DirectoryRequired and FileRequired should be used.

MacOSPlatform vs MacOSXPlatform

These classes represent Classic Mac OS vs Mac OS X. Classic Mac OS is no longer relevant. Mac OS X has been renamed to macOS. macOS is not different from Mac OS X in a way that suggests the distinction should be maintained.

Removed MacOSXPlatform and simplify to MacOSPlatform. It should return true to #isMacOS and #isMacOSX. #family should report #MacOS.

FileReferenceTest - Excessive direct use of FileSystem

FileReferenceTest calls FileSystem methods directly. In most cases, this seems wrong since it avoids using FileReference. In several instances, this results in FileReference outside of the test sandbox directory.

These references should be cleaned up. The FileSystem instance should only be used directly when essential.

FileReferenceTest - Missing tests for various selectors

• #binaryReadStream
• #binaryWriteStream
• #copyTo:
• #delete
• #fullName
• #fullPath
• #hash
• #modificationTime
• #size
• Files w/o permissions

FileReferenceAttributeTests are insufficient

Testing of file attributes is inconsistently done in various test cases. FileSystemTests fails to actually validate data. FileReferenceAttributeTests fails to properly test that modification time changes when writing.

Attributes of a file should be tested at the FileReference level. It would probably be better to ignore it at the level of FileSystem since FileReference's test will test it indirectly.

FileSystemGs class initialization fails during load..

All three of these classes fail when #initialize is sent during load (only classes with initialize methods):

• MemoryStore
• FileLocator
• DiskStore

Currently each of the classes fails with the same error: a SessionManager class does not understand #'default'.

ERROR 2010 , a MessageNotUnderstood occurred (error 2010), a SessionManager class does not understand  #'default' (MessageNotUnderstood)
topaz 1> where
==> 1 MessageNotUnderstood >> defaultAction         @2 line 3
2 MessageNotUnderstood (AbstractException) >> _signalWith: @5 line 25
3 MessageNotUnderstood (AbstractException) >> signal @2 line 47
4 SessionManager class (Object) >> doesNotUnderstand: @9 line 10
5 SessionManager class (Object) >> _doesNotUnderstand:args:envId:reason: @7 line 12
6 DiskStore class >> initialize                 @2 line 2
7 RwGsMethodAdditionSymbolDictPatch (RwGsMethodPatch) >> runInitializer @5 line 6
8 [] in  ExecBlock1 (RwGsPatchSet_254) >> runInitializers @8 line 17
9 OrderedCollection (Collection) >> do:         @5 line 10
10 RwGsPatchSet_254 >> runInitializers           @17 line 14
11 RwGsPatchSet_254 >> apply                     @30 line 33
12 RwGsImage class >> applyModification_254:visitorClass:instanceMigrator: @10 line 9
13 RwGsImage class >> applyModification_254:instanceMigrator: @8 line 12
14 RwPrjLoadTool >> _loadProjectSetDefinition_254:instanceMigrator: @17 line 18
15 [] in  ExecBlock0 (RwPrjLoadTool) >> _doProjectSetLoad_254:instanceMigrator:originalProjectSet:processedClassNames: @2 line 4
16 ExecBlock0 (ExecBlock) >> on:do:              @3 line 42
17 RwPrjLoadTool >> _doProjectSetLoad_254:instanceMigrator:originalProjectSet:processedClassNames: @2 line 6
18 RwPrjLoadTool >> loadProjectSetDefinition:instanceMigrator: @3 line 7
19 RwPrjLoadTool >> loadProjectSetDefinition:    @4 line 4
20 RwPrjLoadTool >> loadProjectFromSpecUrl:      @10 line 6
21 [] in  Executed Code                          @4 line 6
22 ExecBlock0 (ExecBlock) >> on:do:              @3 line 42
23 Executed Code                                 @4 line 7
24 GsNMethod class >> _gsReturnToC               @1 line 1

The class SessionManager is on the list of unimplemented classes, so we won't be able to resolve the initialization issues, until we've at leat got SessionManager defined.