MSDOS fs buffer hang & more about elks

[georgp24]

I tried to reproduce this and cannot. ELKS shows the right file sizes on my FAT16 image and I did not get your buffer error.

Is the wina20.38_ a compressed file or what type of file? Why is there an underscore at the end of the file extension?

[Mellvik]

The trailing underscore was the ‘standard’ M$ distribution format for MSDOS on floppies. The files are compressed but the file system does (obviously) not know anything about that, so a dir listing on DOS lists the physical sizes. I can send you a listing of the entire directory, notice that some of the dates are wrong too. The buffer problem is repeatable, possibly not occuring at exactly the same point in a chain of commands …

[georgp24]

I can see on your photo that the file sizes are wrong. If your FAT16 hard disk image is smaller than 10 MB you may attach it to your post so I can test it.

[Mellvik]

I have no way to get the disk image off the machine… There is no network on the old klunker (at this time, I ordered an old NE2k card the other day, so maybe in a few weeks). Here’s the list - the sizes and dates are frequently different from what I get from DOS and none of the files are actually directories, so there are several issues here. I’m attaching a photo of part of the dos listing for reference. I can get you a full dos listing too if it’s useful. cat /mnt/floppy/xxx tmp: -rwxr-xr-x 1 root root 6261 May 31 1994 ansi.sy_ -rwxr-xr-x 1 root root 7467 May 31 1994 append.ex_ -rwxr-xr-x 1 root root 11208 May 31 1994 attrib.exe -rwxr-xr-x 1 root root 38 May 31 1994 autoexec.bat -r--r--r-- 1 root root 3523999830 Apr 09 2015 av.gr_ -rwxr-xr-x 1 root root 25265287 Jan 25 2084 bk.gr_ -rwxr-xr-x 1 root root 4143072905 Aug 04 2001 bkav.gr_ -rwxr-xr-x 1 root root 2000435527 Jun 28 1971 bkud.gr_ drwxrwxrwx 3 root root 2048 Aug 06 1980 bkudav.gr_ -rwxr-xr-x 1 root root 73755 May 31 1994 busetup.exe -rwxr-xr-x 1 root root 12241 May 31 1994 chkdsk.exe -rwxr-xr-x 1 root root 16770 May 31 1994 chkstate.sy_ -rwxr-xr-x 1 root root 1754 May 31 1994 choice.com -rwxr-xr-x 1 root root 54645 May 31 1994 command.com -rwxr-xr-x 1 root root 13 May 31 1994 config.sys -rwxr-xr-x 1 root root 26936 May 31 1994 country.sys -rwxr-xr-x 1 root root 4337 May 31 1994 country.tx_ -rwxr-xr-x 1 root root 29544 May 31 1994 dblspace.bi# -rwxr-xr-x 1 root root 5087 May 31 1994 dblwin.hl_ -rwxr-xr-x 1 root root 15718 Dec 12 2015 debug.exe -rwxr-xr-x 1 root root 79065 May 31 1994 defrag.exe -rwxr-xr-x 1 root root 4284 May 31 1994 defrag.hl_ -rwxr-xr-x 1 root root 11336 May 31 1994 deloldos.ex_ -rwxr-xr-x 1 root root 7249 May 31 1994 deltree.ex_ -rwxr-xr-x 1 root root 6762 May 31 1994 diskcomp.co_ -rwxr-xr-x 1 root root 8306 May 31 1994 diskcopy.co_ -rwxr-xr-x 1 root root 9797 May 31 1994 display.sy_ -rwxr-xr-x 1 root root 6277 May 31 1994 dmdrvr.bi_ -rwxr-xr-x 1 root root 2576 May 31 1994 doshelp.hl_ -rwxr-xr-x 1 root root 4213 May 31 1994 doskey.co_ -rwxr-xr-x 1 root root 14607 May 31 1994 dossetup.ini -rwxr-xr-x 1 root root 4032 May 31 1994 driver.sy_ -rwxr-xr-x 1 root root 66294 May 31 1994 drvspace.bin -rwxr-xr-x 1 root root 181840 May 31 1994 drvspace.exe -rwxr-xr-x 1 root root 28521 May 31 1994 drvspace.hl_ -rwxr-xr-x 1 root root 1272 May 31 1994 drvspace.in_ -rwxr-xr-x 1 root root 8518 May 31 1994 drvspace.sy_ -rwxr-xr-x 1 root root 15709 Dec 12 2015 edit.hl_ -rwxr-xr-x 1 root root 18340 May 31 1994 ega.cp_ -rwxr-xr-x 1 root root 19049 May 31 1994 ega2.cp_ -rwxr-xr-x 1 root root 15558 May 31 1994 ega3.cp_ -rwxr-xr-x 1 root root 50058 May 31 1994 emm386.ex_ -rwxr-xr-x 1 root root 7539 May 31 1994 fasthelp.ex_ -rwxr-xr-x 1 root root 7829 May 31 1994 fastopen.ex_ drwxrwxrwx 3 root root 4096 Oct 31 2027 fc.ex_ -rwxr-xr-x 1 root root 29336 May 31 1994 fdisk.exe dr--r--r-- 3 root root 122880 Jan 20 2076 find.ex_ -rwxr-xr-x 1 root root 22974 May 31 1994 format.com -r--r--r-- 1 root root 1458342741 Jul 31 2049 graphics.co_ -r--r--r-- 1 root root 2337691628 Oct 30 2077 graphics.pr_ dr--r--r-- 3 root root 49152 Aug 29 1971 help.com -r--r--r-- 1 root root 855918219 Aug 01 2049 help.hl_ -rwxr-xr-x 1 root root 105286170 Apr 12 2019 himem.sy_ dr--r--r-- 3 root root 4096 Jan 03 2027 interlnk.ex_ -rwxr-xr-x 1 root root 1342719629 Feb 06 1981 intersvr.ex_ -rwxr-xr-x 1 root root 40774 May 31 1994 io.sys -rwxr-xr-x 1 root root 15750 May 31 1994 keyb.com -rwxr-xr-x 1 root root 34598 May 31 1994 keyboard.sys -rwxr-xr-x 1 root root 11759 May 31 1994 keybrd2.sy_ drwxrwxrwx 3 root root 6144 Jan 07 2001 label.ex_ -r--r--r-- 1 root root 604390538 Mar 22 1982 loadfix.co_ -rwxr-xr-x 1 root root 19512 May 31 1994 mem.ex_ -rwxr-xr-x 1 root root 76228468 Apr 08 2046 memmaker.exe dr--r--r-- 3 root root 2048 Aug 07 2009 memmaker.hl_ -rwxr-xr-x 1 root root 2337691398 Apr 11 2098 memmaker.in_ -r--r--r-- 1 root root 3921685253 Mar 30 1984 mode.co_ drwxrwxrwx 3 root root 4096 Jan 01 2038 monoumb.38_ dr--r--r-- 3 root root 14336 Jan 29 2000 more.com dr--r--r-- 3 root root 18432 Feb 05 1981 mouse.co_ drwxrwxrwx 3 root root 34816 Aug 12 2023 move.ex_ drwxrwxrwx 3 root root 8192 Jan 10 1984 msav.exe -rwxr-xr-x 1 root root 2298758147 Apr 02 2044 msav.hl_ -r--r--r-- 1 root root 4134993912 May 31 2020 msavhelp.ov_ -r--r--r-- 1 root root 2583578251 Dec 01 2048 msavirus.ls_ -rwxr-xr-x 1 root root 1996445271 Jan 12 2045 msbackdb.ovl -rwxr-xr-x 1 root root 3897951350 Aug 07 2049 msbackdr.ovl dr--r--r-- 3 root root 16384 Jan 14 2049 msbackfb.ovl -r--r--r-- 1 root root 2314618509 Feb 05 2050 msbackfr.ovl dr--r--r-- 3 root root 2048 Feb 16 2096 msbackup.ex_ -r--r--r-- 1 root root 2337758464 Jul 13 1987 msbackup.hl_ dr--r--r-- 3 root root 6144 Jan 09 2096 msbackup.ovl -rwxr-xr-x 1 root root 509078667 Jan 05 2026 msbconfg.hl_ -r--r--r-- 1 root root 310146 Jan 14 1996 msbconfg.ovl -rwxr-xr-x 1 root root 25361 May 31 1994 mscdex.exe -r--r--r-- 1 root root 201755778 Sep 02 2048 msd.exe -rwxr-xr-x 1 root root 38138 May 31 1994 msdos.sys drwxrwxrwx 3 root root 49152 Apr 11 2007 mstools.dl_ -r--r--r-- 1 root root 2935160831 Jan 31 2005 mwav.ex_ drwxrwxrwx 3 root root 8192 Oct 22 2049 mwav.hl_ dr--r--r-- 3 root root 43008 Aug 06 2023 mwavabsi.dl_ drwxrwxrwx 3 root root 34816 Aug 09 2049 mwavdlg.dl_ -r--r--r-- 1 root root 538976288 Nov 13 2030 mwavdosl.dl_ -rwxr-xr-x 1 root root 1635021600 Mar 20 2032 mwavdrvl.dl_ dr--r--r-- 3 root root 2048 Jan 13 1985 mwavmgr.dl_ -rwxr-xr-x 1 root root 542332236 Oct 13 2018 mwavscan.dl_ -rwxr-xr-x 1 root root 1565928820 Aug 31 2025 mwavsos.dl_ -r--r--r-- 1 root root 1679848815 Dec 31 1995 mwavtsr.ex_ dr--r--r-- 3 root root 2048 Mar 09 2038 mwbackf.dl_ -rwxr-xr-x 1 root root 2334556700 Aug 22 1995 mwbackr.dl_ -rwxr-xr-x 1 root root 538976288 Dec 31 1995 mwbackup.ex_ dr--r--r-- 3 root root 43008 Mar 14 1996 mwbackup.hl_ drwxrwxrwx 3 root root 2048 Dec 31 1995 mwgrafic.dl_ -r--r--r-- 1 root root 1937011297 Mar 20 1996 mwundel.ex_ -r--r--r-- 1 root root 1937055858 Jan 13 1985 mwundel.hl_ -rwxr-xr-x 1 root root 19951 May 31 1994 networks.txt -rwxr-xr-x 1 root root 7036 May 31 1994 nlsfunc.exe -rwxr-xr-x 1 root root 6358 May 31 1994 os2.txt -rwxr-xr-x 1 root root 6761 May 31 1994 packing.lst dr--r--r-- 3 root root 30720 Aug 11 2072 power.ex_ -rwxr-xr-x 1 root root 544825888 Dec 31 2036 print.ex_ -rwxr-xr-x 1 root root 194309 May 31 1994 qbasic.exe -rwxr-xr-x 1 root root 784303150 Jul 24 1991 qbasic.hl_ dr--r--r-- 3 root root 6144 Feb 08 1980 ramdrive.sy_ -rwxr-xr-x 1 root root 86838 May 31 1994 readme.txt dr--r--r-- 3 root root 2048 Sep 30 2082 replace.ex_ -rwxr-xr-x 1 root root 522308901 Jan 10 2045 restore.ex_ -rwxr-xr-x 1 root root 124262 May 31 1994 scandisk.exe -rwxr-xr-x 1 root root 6920 May 31 1994 scandisk.ini -rwxr-xr-x 1 root root 111299 May 31 1994 setup.exe -rwxr-xr-x 1 root root 1830 May 31 1994 setup.msg dr--r--r-- 3 root root 6144 Mar 30 1982 setver.ex_ drwxrwxrwx 3 root root 8192 Oct 22 2049 share.ex_ -rwxr-xr-x 1 root root 1027178524 Jan 24 2097 sizer.ex_ -rwxr-xr-x 1 root root 3681878076 Apr 03 2079 smartdrv.ex_ -r--r--r-- 1 root root 2337666117 Jan 02 2005 smartmon.ex_ -r--r--r-- 1 root root 4043508851 Aug 04 2076 smartmon.hl_ -r--r--r-- 1 root root 2784225745 Nov 19 2097 sort.ex_ dr--r--r-- 3 root root 10240 Jul 08 2054 spatch.bat -rwxr-xr-x 1 root root 1577632899 Feb 16 2096 sstor.sy_ -r--r--r-- 1 root root 2345337817 Jun 01 2101 subst.ex_ -rwxr-xr-x 1 root root 9432 May 31 1994 sys.com drwxrwxrwx 3 root root 4096 Oct 31 2027 tree.co_ -rwxr-xr-x 1 root root 1735357008 Jan 13 1985 ud.gr_ -rwxr-xr-x 1 root root 1868767333 Mar 19 1996 udav.gr_ dr--r--r-- 3 root root 2048 Nov 15 2031 undelete.exe -rwxr-xr-x 1 root root 175637068 Jan 06 2038 unformat.com -rwxr-xr-x 1 root root 87995 May 31 1994 uninstal.exe -rwxr-xr-x 1 root root 2332034502 Mar 30 1982 vfintd.38_ -rwxr-xr-x 1 root root 1646272544 Dec 31 1995 vsafe.co_ -rwxr-xr-x 1 root root 1988843095 Jan 12 2045 wina20.38_ dr--r--r-- 3 root root 2048 Mar 07 1996 wntools.gr_ -rwxr-xr-x 1 root root 17188086 Apr 02 1996 xbios.ov_ -rwxr-xr-x 1 root root 11093 May 31 1994 xcopy.ex_

[georgp24]

I found images of the old DOS 6 installation disks on the net. So I copied these files on the FAT16 image I generated with the script in the elkscmd/test/fat directory. (After modifying to FAT16) The files show up correctly then. So the problem is difficult to reproduce.

However, I also found that ELKS hangs after you deleted a file on the FAT16 disk.

[Mellvik]

Hmm,, are you getting the delete character error I reported a few days ago? Console bios driver? I may need to move the dev environ off of my raspberry pi to check, but it’s really convenient to have minix FS readily supported (no intel linux at hand, just macs).

[georgp24]

I did not test these issues. Since I came up with the FAT file system I did look into the issues related to that.

Maybe you can run Qemu on the RPI, Qemu is usually used by the ELKS developers.

[Mellvik]

Your idea about getting the fat-image is the best. I'll figure out a way to get the image off the klunker tomorrow. I have Venix running on one partition, dd-compress-split and floppy may work.

[Mellvik]

Here is the DOS image. The 'offending' files are in the TMP directory. This is just the partition, not the entire disk image.
dos286.img.gz

[georgp24]

Helge,

I could not mount you disk image in Qemu with ELKS. When I looked at it with fdisk on Linux, the partition table seemed to be corrupted.

This is what sudo fdisk -l dos286.img lists:

'Befehl (m für Hilfe): p
Festplatte dos286.img: 15,2 MiB, 15925248 Bytes, 31104 Sektoren
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x6f727265

Device Boot Start End Sectors Size Id Type
dos286.img1 ? 1684955424 3386954047 1701998624 811,6G 6c unbekannt
dos286.img2 ? 1998616933 2542722764 544105832 259,5G 6e unbekannt
dos286.img3 ? 538988361 1077964648 538976288 257G 79 unbekannt
dos286.img4 ? 1394614304 1394635640 21337 10,4M 53 OnTrack DM6 Aux3'

This does not look good and I can understand that ELKS has problems with that.

Georg

[jbruchon]

That is because there is no partition table; this is a filesystem image as you'd see on a floppy, not an MBR-partitioned hard disk image. The boot sector is a MS-DOS 6.22 boot sector. This is not an unusual configuration for small flash drives nor for floppy drives, so it should be handled. All block devices should have some sort of partition table detection and partition sub-device handling.

[georgp24]

I made a script to write the partition image provided by Helge to a hard disk which can be mounted with ELKS:

#!/bin/bash

echo "ELKS FAT Hard Drive Image Builder"
echo

set +x
#Hard drive image file name
HD_IMAGE=fat16_hdd

if [ $UID != 0 ]
then echo -e "\nThis script must be run with root permissions\n" >&2
exit 1
fi

#makes flat hard disk image of 32 MB - sfdisk needs 32.2 MB image for that
echo -e "\nGenerating 32 MB disk image:"
dd if=/dev/zero of=$HD_IMAGE bs=1024 count=32200

#put partition table on this disk - id=0x80, =bootable first partition
echo -en "\nMaking partition table:"
echo ',,0E,' | sfdisk --quiet -D $HD_IMAGE 2>/dev/null

#write dos286.img partition image to sector 0 in partition 1 of the image
echo "Writing partition image"
dd if="./dos286.img" of="$HD_IMAGE" seek=63 bs=512 count=34000 conv=notrunc

#make user owner of HD image (if using sudo)
if env | grep -q SUDO_USER
then
USERNAME=$(env | grep SUDO_USER | cut -d= -f 2)
chown $USERNAME:users "$HD_IMAGE"
fi
#allow all to read, owner to write
chmod 644 "$HD_IMAGE"

set +x
test -e "$HD_IMAGE" &&
echo -e "\nRun $HD_IMAGE with qemu.sh\n"

This way I can reproduce the error reported by Helge. However, I currently do not know how to fix this.

Georg

[Mellvik]

Sorry for not mentioning right away that this was a partition, not a disk image. That said, the partition mounted easily on all Linux systems I have access to. Now that you have turned it into a disk image, we can focus on the problem at hand, which looks like a data skew bug. Good work.

[lithoxs]

The problem is deeper than a simple bug.
First, msdosfs does not have the notion of inodes, much less inode numbers. All information is stored at the msdos directory entry.

In ELKS, the VFS insists on having an inode and its inode number. Then, for msdosfs, there has to be a mapping between every directory entry and a number (to be used as an inode number). This mapping is simply a combination the sector number of the directory entry and the offset of the directory entry (see file fs/msdos/misc.c, line 247).
Now, this number can be greater than 16 bits. In Linux this is no problem, but in ELKS only the 16 least significant bits are taken, resulting in the inode number pointing to somewhere else on the disk for inode information.

A solution can be to use 32 bit inode numbers (see file include/linuxmt/types.h, line #28), but this needs changes at many places.

[jbruchon]

Assuming we're not currently worried about supporting FAT32, the starting cluster number could be used as the inode number. Hard linking is not permitted on FAT filesystems, so the starting cluster number is always unique between files and is always 16 bits wide or less.

[lithoxs]

Might work. The idea of the inode number is to easily locate the inode information (directory information for msdos) on disk. It can be added the sector and offset of the directory entry in the section specific for msdos in the inode structure.
The change needs a clear understanding of the functions in fs/msdos directory

[Mellvik]

I took a closer look at the data (on the partition), and sure enough - the directory blocks giving offending data are further out on the volume than the ones giving correct results. And there is never a mix, correct and wrong data coming from the same directory block. Adding a new dir (using DOS) and copying data to it, gives all wrong metadata. Isn't it weird though that the file names come out correct all the time? Since the file names and the metadata are stored together, shouldn't we be able to get the correct metadata along with the name?

[lithoxs]

When the VFS looks for a file, searches the file name in directories. If all directories traversed have 16 bits inodes, there will not be problem to find it. You are right when you say that all metadata is readily available at this moment, but instead function msdos_get_entry() creates a 32 bit inode number and truncates it to 16 bits, without reading any metadata.
The function which reads the metadata is msdos_read_inode(). It locates the directory entry using the inode number. First locates the sector (see file fs/msdos/inode.c, line 243) and then the directory entry within the sector (line 246). Size and time are read in lines 271 and 274.
But if the original inode number was larger than 16 bits, the inode number got truncated to 16 bits, so the original sector number was lost and the sector to read will be something else. This explains why the names are correct but metadata is garbage.
I tracked this as the cause of the directory listing problem, but there may be additional problems.
At the very least this is a bug.

[mfld-fr]

Even if it does not solve completely the problem, why not improving the current code by not using the sector number, but the FAT cluster number as suggested by Jody, as the first part of the pseudo-inode, and not the sector offset, but the directory entry index, as the second part of that pseudo-inode ? This puts limitations below FAT16 capabilities, but it could work for a small FS.

A complete solution would be to implement a full map between the pseudo-inode and the pair (cluster, directory entry). It could also support FAT32, but would have a higher impact on existing code and memory consumption.

The choice of the solution is driven by the actual need : still targeting legacy HW where FAT16 was above capabilities (so no big cluster number and no many directory entries, that both can fit in 16 bits), or targeting emulators and recent HW where we can reach easily the FAT16 limits ?

@georgp24 : before adding features to ELKS by copying/pasting existing code, it would be nice to think a bit about the requirements... and understand deeper what is done in that code against that requirements before merging it...

[jbruchon]

Generate a synthetic 16-bit inode number starting at 1 for every entry in any directory that's read e.g. with readdir() and store in a table the 16-bit cluster number for that directory on disk and a 16-bit directory index. For large numbers of files it would run out of memory quickly (64K = 10922 of these entries) but it would work for smaller volumes.

[mfld-fr]

To see if the first solution could work for @Mellvik, I decoded the BPB he provided in his dump:

• formatter: MSDOS5.0
• 512 bytes / sector
• 4 sectors / cluster (= 2048 bytes / cluster)
• 1 reserved sector for boot
• 2 FATs
• 512 entries in root directory
• disk type: HD
• 34 sectors by FAT (= 8702 clusters max)
• 17 sectors / track
• 5 heads
• hidden sectors: 49215
• total sectors: 34000 (= 16 MB)
• disk identifier: 80h (HD)
• signature: 29h (OK)
• disk serial number: 213011EEh
• disk name: DOS622

It is not possible to pack the cluster number (up to 8702) and the directory index (up to 512) into a 16 bits inode number. The FS is too large for the first solution to work for that issue.

[georgp24]

I am surprised to read mentioned here that FAT32 is not supported. It IS supported and that has been documented in the fat_fs.txt file. In the elkscmd/test/fat directory there is a s script to generate a FAT32 hard disk image which I used for testing.

Probably FAT32 support only works for a few files on the disk as well. However, using Qemu you can also access flash disks formatted with FAT32 from ELKS. I did access a 480 MB partition on the flash disk with a few files on it.

[georgp24]

As Juan mentioned I tried to changed to 32 bit inode numbers but did not get it to work. So I left it as it is for a start. I think it is important to be able to copy files from and to a FAT disk to ELKS and that can be done with the existing code.

[jbruchon]

Though not necessarily the best solution, the 16-bit variant of jodyhash (I have built and tested that variant with ELKS) might work for building a 16-bit hash table of directory entries for "touched" inodes that is purged on umount. That way a 16-bit inode number could be used even with FAT32.

[lithoxs]

Writing code for a table to relate a 16 bit inode number with the corresponding sector and offset will require more effort and code size than simply increasing the inode number to 32 bits. Besides, the romfs also needs 32 bit inode numbers.
Regarding support for msdos:
FAT12 is fully supported.
FAT16 and FAT32 are supported as long as:
1. All directory entries are within the first 2 Mbytes from the start of partition.
2. Maximum partition size is 64 Mbytes.

[mfld-fr]

@Mellvik : do you need any more action on that issue, or could we close it now ?

[Mellvik]

I'll test this with the latest sources next weekend.

[mfld-fr]

Latest fixes merged in #128 and #129.

[Mellvik]

Observations from todays tests - most of these should be repeatable in Qemu. Read/write/ls performance is still bad, compared to DOS and a 1985 version of Mtools with Venix. Date (actually year) is not always copied correctly: My date is jan 1st 1970 (because ELKS does not pick up the date from the BIOS), while the date on the copied files is jan 1st 2098. An MSDOS filesystem cannot be umounted after mods have been made to the file system. (Resource busy) Copy many files internally on the MSDOS fs some times hang the system There is an occasional file delete problem (or a name cache problem): A file can be deleted (rm) and still show in ls. Rm many (‘*’) from a directory (‘many’ being about 30) cause the system to print a page of garbage on the screen, then softhang. See picture for this one. I will be posting a short video to visualize the speed of ls, file copy. Conclusion - it seems early to close this issue - or maybe I would open several new ones?

[mfld-fr]

Thank you @Mellvik for this latest round of testing ! It confirms my opinion about the FAT support in ELKS : it was introduced last year in #111 in somewhat a rush, with a poor quality level in both design & implementation.

Despite this matter of fact, I think it is worth to improve it, because today FAT is the de facto standard to load a boot image (UEFI...). So I will come back to this issue after the current round (i.e. building the kernel with a full GNU tool chain), and I keep it open until then.

[Mellvik]

I agree on both accounts - reliable and efficient FAT-support is important. Just let me know what I can do to help with there testing. BTW - here is some more from last nights test: Time stamp error: The case is different than previously reported. ELKS does set time stamps correctly, but lists DOS time stamps 1-1-98 as 2098 instead of 1998. The rm * problem sometmes, but not always corrupts the file system. Further - in my testes ‘many’ files is more like 90-120 than 30. The previously reported file delete problem appears to be cache related. When rm * crashes the system, many files have been successfully unlink()’ed, but the changes haven´t made it to the disk. The rm * problem turns out to be VFS (‘VFS: No free nodes’) - see shared picture. Strange. The fact that fdisk reports weird partition numbers should probably be posted as a bug.

[Mellvik]

A few physical machine tests (compaq 286) (I did a git fetch last night, assuming it's reasonably current):

• Unrelated to the FAT issues, and probably recorded somewhere else: ls after login shows no files, pwd shows '/', ls -a shows . .. .profile, cd /; ls shows the expected directories. So pwd is reporting wrong right after login.

-- FAT: The metadata issues are gone, ls -l reports correct data, wrong year (as reported elsewhere).
-- ls in a big FAT directory (130 files) is somewhat slow initially (like before), then noticeably slower as it proceeds. ls -l does not show this effect.
-- any operation that writes to the FAT filesystem, hangs the system - mkdir, cp, ...

More testing later this week.

H

[georgp24]

I also think it is not user friendly to end up in an empty directory. I suggest to put a file in there so the user gets an orientation and does not feel lost.

[ghaerr]

A few physical machine tests (compaq 286) (I did a git fetch last night, assuming it's reasonably current):

Thank you @Mellvik for your testing!

• Unrelated to the FAT issues, and probably recorded somewhere else: ls after login shows no files, pwd shows '/', ls -a shows . .. .profile, cd /; ls shows the expected directories. So pwd is reporting wrong right after login.

This isn't a bug: you're sitting in /root/.profile, where there are no files. Only ls -a shows .profile, as expected. Things are normal after you cd /.

-- FAT: The metadata issues are gone, ls -l reports correct data, wrong year (as reported elsewhere).

My testing shows proper 2020 dates on `ls -l'.

I would like to know more information about the type of FAT filesystem mounted, can you report what the FAT: line says immediately after mount?

-- ls in a big FAT directory (130 files) is somewhat slow initially (like before), then noticeably slower as it proceeds. ls -l does not show this effect.
-- any operation that writes to the FAT filesystem, hangs the system - mkdir, cp, ...

Any chance of sending over an image of this FAT filesystem you are using to test?
I have not tested speed, so there's likely improvement to be made there, but am wondering
about the FAT filesystem type for these other hangs (none of which I've seen after my fixes).

@mfld-fr: Can we move this out of #123 and into a new FAT filesystem testing issue?

[mfld-fr]

@ghaerr : I prefer to keep this old thread as one piece, because there are still some problems today that were discussed earlier in that thread, to keep all the information in one place.

[Mellvik]

BTW - this is Minix boot on 1.44 floppy.

10. feb. 2020 kl. 17:04 skrev Gregory Haerr ***@***.***>: A few physical machine tests (compaq 286) (I did a git fetch last night, assuming it's reasonably current): Thank you @Mellvik <https://github.com/Mellvik> for your testing! Unrelated to the FAT issues, and probably recorded somewhere else: ls after login shows no files, pwd shows '/', ls -a shows . .. .profile, cd /; ls shows the expected directories. So pwd is reporting wrong right after login. This isn't a bug: you're sitting in /root/.profile, where there are no files. Only ls -a shows .profile, as expected. Things are normal after you cd /.

The bug is that pwd reports the wrong directory.

-- FAT: The metadata issues are gone, ls -l reports correct data, wrong year (as reported elsewhere). My testing shows proper 2020 dates on `ls -l'.

These are files from 1998, reported date is plus a hundred years [2098], I believe someone else reported that some days ago.

I would like to know more information about the type of FAT filesystem mounted, can you report what the FAT: line says immediately after mount?

[MSDOS-FS 0.3 FAT16]

-- ls in a big FAT directory (130 files) is somewhat slow initially (like before), then noticeably slower as it proceeds. ls -l does not show this effect. -- any operation that writes to the FAT filesystem, hangs the system - mkdir, cp, ... Any chance of sending over an image of this FAT filesystem you are using to test? I have not tested speed, so there's likely improvement to be made there, but am wondering about the FAT filesystem type for these other hangs (none of which I've seen after my fixes).

I have a hard time getting the partition off the system, Most of the disk is Venix, I haven't found a dos tool to turn a partition into a file so I can ship it off the system. Maybe ELKS can do that now, btw. ...

@mfld-fr <https://github.com/mfld-fr>: Can we move this out of #123 <#123> and into a new FAT filesystem testing issue?

Mellvik

[ghaerr]

The bug is that pwd reports the wrong directory.

I see, which directory is it reporting?

[MSDOS-FS 0.3 FAT16]

That mount message is before I fixed the hang problem and create file crash problem.

The current kernel message should read 'FAT: 1440k fat16', so it looks like you're running at least one ELKS source rev behind.

@mfld-fr: I figured we might run into issues without knowing the exact ELKS build being run. That would be a nice addition, but for the time being, I'll submit a PR that 1) shows detailed mounted FAT superblock information, and 2) a temporary FATFS version number incremented on each patch to FAT for ease of understanding user test results.

[mfld-fr]

@ghaerr : more simple: just get the current commit SHA and append it to the ELKS version 😄

[ghaerr]

-- FAT: The metadata issues are gone, ls -l reports correct data, wrong year (as reported elsewhere).

@Mellvik: I found and corrected a longstanding date bug - MSDOS dates start with 1980. Since ELKS boots without the date being properly set, it defaults to Jan 1 1970, which resulted in a negative year value. Long story short, this was converted and displayed as a date with a year like 2096, instead of 1970. Fixed on mkdir. If you see more date problems, run date and report that too, thanks!

[Mellvik]

New test using a fresh download of the entire tree and the default config:
mount -t msdos /dev/bda4 /mnt
mount failed: No such device

Tried with the same config as used yesterday, when the mount worked: No difference.
cat /dev/bda4 > /dev/null works fine

[ghaerr]

New test using a fresh download of the entire tree and the default config:
mount -t msdos /dev/bda4 /mnt
mount failed: No such device

@mfld-fr: I have only tested the FAT filesystem with /dev/fd0 and /dev/fd1 mounts. I have not looked into what the FAT BPB might need to be for hard disks, and need input for that. We may need code for handling a MBR boot partition.

@Mellvik: Can you test using /dev/fd1 as the mount point point (or test floppies rather than hard drives)? Please send a screenshot for any floppy mount or filesystem failures, and thanks for your testing!

[Mellvik]

Sorry, I don't have any dual floppy systems operational @ this time. That said you gave me an incentive to wrap up the (dual floppy) project on my workbench since before xmas. That may take a few days though. Mellvik

[Mellvik]

New test from updated sources as of an hour ago: - mount of bda4 now works and initial testing of file/directory creation/deletion works fine (se attached picture). - no changes in the performance of file listings (I didn't expect that, but running DOS on the machine and listing directories this afternoon, reminded me how far off the speed is). - probing of floppies (yes, I guess I should put this in a different issue #) is sort of broken, possibly a compaq special (see attached picture): Probing of /dev/fd1 starts since the BIOS reports it (wrongly) as present. A known feature of Compaq BIOSes, please consider checking the CDH values in the probe routine. If they are all 0, consider the probe to have failed (non existent drive). - There is no change in the reporting of dates in ls (see pic). BTW it's not only the year that is wrong in these dates, the entire date seems to be totally off. Mellvik

[Mellvik]

More test results related to FAT file system and dates: [from commit dc3baae]

• The date (year) problem is still there, and my claim that it also pertains to the rest of the date, is incorrect. This is indeed a year-only issue.
• I believe the speed (directory listing) issue is worth looking into at some point, maybe open it as its own issue (non-urgent).
• There are other date issues - which do not belong here.
  
  

[ghaerr]

• The date (year) problem is still there, and my claim that it also pertains to the rest of the date, is incorrect. This is indeed a year-only issue.

Thanks for the screenshots. However, they each show different filenames. Can you capture an example of the same file, under ELKS and MSDOS, which will shows the year handled incorrectly? That will allow me to understand what the problem is. I'm glad to hear its only the year, and not the time, although I'm unable to get ELKS ls to show a file time, only the date.

• I believe the speed (directory listing) issue is worth looking into at some point, maybe open it as its own issue (non-urgent).

Yes, there are still number of improvements that could/should be made to the FAT filesystem. It was enough work to just get it to operate bug-free, which is the aim of this release.

• There are other date issues - which do not belong here.

Other than the date not being set at boot, and date not able to set a new date? Let us know, thanks!

[Mellvik]

Sorry @ghaerr; I sent the wrong picture. Here is the correct pair. And I'm including a 3rd picture — demonstrating (at the bottom)(a well known problem???) how ELKS bombs when running out of memory. After the error message, the system is dead. This is the other issue I was referring to. BTW - I'll take a look at date.c. —Mellvik

[ghaerr]

Sorry @ghaerr; I sent the wrong picture. Here is the correct pair.

No pictures were attached!

And I'm including a 3rd picture — demonstrating (at the bottom)(a well known problem???) how ELKS bombs when running out of memory. After the error message, the system is dead.

I'm aware that there isn't much memory to run programs, for instance bc won't run at all anymore.

The kernel source is ancient and needs to be gone over to pair down overly-large data and code segment uses, such as miscellaneous printk's, panic messages, and other things I've noticed.

However, issue #241 (configurable process heap + stack size) would possibly help greatly here, for possibly /bin/init and various important system utilities. I was thinking of diving into that, but it won't be for this next release.

[Mellvik]

[I guess the pics were pruned because of the size, these are smaller] —Mellvik

[Mellvik]

[ghaerr]

@Mellvik: I see the problem now - the year is off by exactly 100! That is going to be an easy fix and I think I know why. I'll look into this after skiing today and submit a PR. A picture is worth a thousand words! :)

[Mellvik]

@ghaerr,
the problem in date.c (not busyelks/cmd/date.c but elkscmd/sh_utils/date.c - why are there two?) is that strtok() has changed. The second parameter is now expected to be a string, not a char. so if you change the line
p = strtok(datestring, "-");
to doublequotes like above, date works.

[Mellvik]

Also, just in case you missed it, @ghaerr, it may be useful to take a look at the memory error on the bottom of the last picture.
sys_brk failed; len 27972 > endseg 19792 Not enough memory. Use chmem to allocate more : Out of memory

[ghaerr]

Also, just in case you missed it, @ghaerr, it may be useful to take a look at the memory error on the bottom of the last picture.
sys_brk failed; len 27972 > endseg 19792 Not enough memory. Use chmem to allocate more : Out of memory

No I didn't miss that, thanks: I mentioned that working on #241 might help fix our ELKS application memory problems.

I'll fix the off-by-100 date problem as well as the date.c strtok() bug. Nice catch - maybe you better join the programming team!? :)

[Mellvik]

great - and thanks but no thanks, @ghaerr. I'll stay on the sidelines for now and enjoy the ride :-) Being an almost total stranger to Git, I have a lot of catch-up to do before I can possibly contribute directly. —Mellvik

[ghaerr]

@Mellvik: I've fixed the date -s problem, but I've looked into the displayed year off-by-100 problem in your screenshot and can't get it to duplicate. See attached pictures, they show the correct dates in years from 1998-2014 in ELKS and a DOS system! I'm not sure why your system is showing differently yet.

[Mellvik]

@ghaerr, I will do some more testing on this later today. Maybe you can try files with a jan 1st date.

…

--Mellvik

[ghaerr]

@Mellvik : Also set your system date using the fixed date -s to Jan 1 1998, then do a "cat > file" and "ls -l file" to see what it shows. You can also try varying the date set to try to frame this very strange problem!

[Mellvik]

@ghaerr, indeed strange. I set the date to 1997, 1998, 1999 - varying dates, and they all come out correctly with ls -l. And here’s the real weirdo: The machine starts with date zero (1970…) I create a (FAT)file and the date is 1-1-1980 !!!!! Looks like the bug may be somewhere else than expected? I need to look at the actual raw data in the directory — but before that I’d like to add whatever code is necessary to get ELKS to pull and set the date&time from the hardware. —Mellvik

[ghaerr]

@ghaerr, indeed strange. I set the date to 1997, 1998, 1999 - varying dates, and they all come out correctly with ls -l.

Yes, that's what I figured would happen, after looking at the source.

And here’s the real weirdo: The machine starts with date zero (1970…) I create a (FAT)file and the date is 1-1-1980 !!!!! Looks like the bug may be somewhere else than expected?

No - this is normal: FAT date 0 = 1980 while UNIX date 0 = 1970. This has not to do with the bug.

I need to look at the actual raw data in the directory — but before that I’d like to add whatever code is necessary to get ELKS to pull and set the date&time from the hardware.

Absolutely, the very best method would be to somehow get a copy of the disk or at least its directory contents. Do you have any other systems that can read MSDOS and ELKS-created floppies? That would help.

[Mellvik]

New FAT mount/access test (ELKS as of this afternoon):
mount -t msdos /dev/fd1 /mnt
/dev/fd1 is a 1.2M 5.25 FAT12 floppy
This system (compaq portable 386/20) has a bad screen - barely readable, so I can't send images, so here's a summary:
Just after mount, a
ls -l /mnt
which goes like this
/mnt:
/mnt/<garbled chars>: Invalid argument
/mnt/<garb>ibm.3.3: Invalid argument
/mnt/<garbled>: Invalid argument
/mnt/ failure.$co: Invalid argument (The space is literal)
/mnt/$.<garbage>: No such file or directory
/mnt/2.m<garb>: Invalid argument
Then a normal listing of files, equivalent to what I get in DOS for the same disk, except that the hidden files are included and there is one more of the 'Invalid argument' entries further down. My guess is that the 'Invalid argument' entries are deleted files that for some reason show up.

I'll make a copy of the floppy image and test on VirtualBox tomorrow.

--Mellvik

[ghaerr]

New FAT mount/access test (ELKS as of this afternoon):
mount -t msdos /dev/fd1 /mnt
/dev/fd1 is a 1.2M 5.25 FAT12 floppy

Please send over or take a picture of the two "FAT: " kernel messages displayed directly after the mount. This will give an indication of exactly what the kernel thinks the FAT disk parameters are.

[Mellvik]

@ghaerr,
Correction: The image is 360k, not 1.2M.

The screen is still barely readable, but with some effort - here's what I get:
FAT: me=fd,csz=2,#f=2,floc=1,fsz=2, rloc=5,#d=112,dloc=12,#s=720,ts=0
FAT: 360k, fat12 format

I guess the ts= is the only 'abnormal' here ...??

[ghaerr]

The screen is still barely readable, but with some effort - here's what I get:
FAT: me=fd,csz=2,#f=2,floc=1,fsz=2, rloc=5,#d=112,dloc=12,#s=720,ts=0
FAT: 360k, fat12 format

I guess the ts= is the only 'abnormal' here ...??

No, ts=0 is normal for FAT12 disks. The two big differences with your disk that have not been tested are that the cluster size is 2 (1k blocks), and number of root dir entries is 112, half of normal. There's obviously a big bug in FAT's handling of one of these two items.

Is there any way you could send a disk image to me for testing?

[Mellvik]

FWIW - and while waiting for a transfer of the image, I ran ELKS (from this morning) on VirtualBox, mounted a (random) 360 FAT12 image, and the ts= is a big number. Plus -
touch /mnt/abc.xyz
hangs ELKS - hard hang.

[Mellvik]

This does not happen when the images is 1.2M. I guess we're closing in.

Link to the 360K image (the one from the message above, not the one from yesterday):
https://www.dropbox.com/s/f80kb418w877wbm/atdg207.img?dl=0

[ghaerr]

FWIW - and while waiting for a transfer of the image, I ran ELKS (from this morning) on VirtualBox, mounted a (random) 360 FAT12 image, and the ts= is a big number. Plus -
touch /mnt/abc.xyz
hangs ELKS - hard hang.

This disk also has 1k clusters and 112 root dirs. Note, that it might be dangerous to your disks to run any application that tries to write to the FAT filesystem, when ls shows incorrect kernel FAT reading... this could permanently damage the FAT integrity of the floppy, depending on whether and where anything was written.

If a disk image can be sent over, I'll look into it. You might also try running minfo or mshowfat on the disks, to see what a standard tool thinks of them (especially after writing and hang).

[ghaerr]

This does not happen when the images is 1.2M. I guess we're closing in.

Your 1.2M image uses a 512 byte cluster size, the 360k version uses 1k cluster size...

Link to the 360K image (the one from the message above, not the one from yesterday):

Got it.

Well @Mellvik, with all your historical equipment, you've really pulled up an oldie here: it's a properly-formatted IBM 2.0 disk! 1k cluster size, 112 directory entries. The FAT BPB is short, and doesn't yet include the ts= (total size field). Confirmed with minfo.

I'll use this image to debug the FAT filesystem. I suspect it has something to do with either 1k cluster size or 112 root dir entries. Thank you!

[ghaerr]

@Mellvik: The 360k image you sent me works fine, can't get any failures, including using touch. This disk is verified as having 1k cluster size and 112 root dir entries... Are you sure you sent me the proper failing disk image?

[Mellvik]

What can I say? I have the same file in two different places and indeed they have a different checksum.
Let's try again:
https://www.dropbox.com/s/qswhzej42yw0teo/atdg207.img?dl=0
This is definitely the one I mounted and reported on ...

[ghaerr]

What can I say? I have the same file in two different places and indeed they have a different checksum.
Let's try again:
https://www.dropbox.com/s/qswhzej42yw0teo/atdg207.img?dl=0
This is definitely the one I mounted and reported on ...

Ok - this one displays all the directory contents properly, but fails on touch. I'll look into it.
Are you fairly sure this is a working MSDOS disk that hasn't been written to by ELKS in the past, including touch? ELKS could have corrupted a directory or the FAT table, for instance, on any failure/crash while writing.

[Mellvik]

'Fairly sure' is about right. Let's say 85% ... It is indeed suspicious that I have two different images with the same name. On the other hand I have never had a reason to mount this image on elks before. And the one from yesterday with all the Invalid agrument messages, is an entirely different image, a DOS 3.31 boot disk. I'll check in the morning. BTW - FWIW - I have tested the failing image with DOS, and it works fine.

…

--Mellvik

[ghaerr]

@Mellvik, I've got thetouch bug reproducing, please send over the DOS 3.31 disk image with the invalid argument messages, if you can.

[Mellvik]

OK; here i is - symptoms verified reproducible in emulator:
https://www.dropbox.com/s/3lupqz5mnneyae0/fd1.img?dl=0

Comments:

1. This may not be the most important issue given all that is pending right now, so feel free to put it on the back burner. On the other hand I guess it's efficient to get it resolved while fresh - and it would indeed be nice to get it out of the way.
2. I had a crash on the system demonstrating the FAT date problem, will get back to that.
3. The floppy probing works but is extremely slow (and noisy) compared to what any other OS is doing. It may be a good idea to open an improvement issue on the one (5+ seconds, 5 full seeks back and forth, ...)

[ghaerr]

OK; here i is - symptoms verified reproducible in emulator:
https://www.dropbox.com/s/3lupqz5mnneyae0/fd1.img?dl=0

Thanks. I'll take a look at it.

I've found the problem with your previous disk image :)

[Mellvik]

Update:
Here's a floppy image with the directory that triggers the date problem. Turns out it triggers the other problems too - in VirtualBox, presumably in other emu's. Some consolation maybe, Raspbian has problems with this image too, and reports the date like this:

-rwxr-xr-x 1 root root 12065 Nov 25 1961 FC.EX_
-rwxr-xr-x 1 root root 29336 Nov 25 1961 FDISK.EXE
-rwxr-xr-x 1 root root 4879 Nov 25 1961 FIND.EX_
-rwxr-xr-x 1 root root 22974 Nov 25 1961 FORMAT.COM
-rwxr-xr-x 1 root root 11376 Nov 25 1961 GRAPHICS.CO_

It even has some of the other problems/symptoms:

helge@raspberrypi:~ $ ls -l /mnt/x
ls: cannot access '/mnt/x/?'$'\016''?'$'\a''.<$u': Input/output error
ls: cannot access '/mnt/x/ failure.$co': Input/output error
total 7959922
d????????? ? ? ? ? ? failure.$co
-rwxr-xr-x 1 root root 772195871 Jan 11 1980 $.???
drwxr-xr-x 13 root root 2048 Jul 23 1999 2.m??
d????????? ? ? ? ? ? ????.<$u
-rwxr-xr-x 1 root root 3469790928 Dec 25 1941 ?&?.??&
-rwxr-xr-x 1 root root 0 Jan 18 1980 ?(?IBM.3.3
-rwxr-xr-x 1 root root 6261 Nov 25 1961 ANSI.SY_

https://www.dropbox.com/s/39wrfterzow1eum/testimg144.img?dl=0

[Mellvik]

I've looked closer into the FAT date representation problem - discussed a number of times on this thread. Here's the problem summary & verdict:

• Problem: ELKS reports the date of some files 100 years off - 2098 instead of 1998.
• If we set the date (in ELKS) to 1.1.1998 and create a file in the same directory, the date is correct in DOS and ELKS
• In the first case, the date entry in the directory is 21ec (118 years after 1980), in the latter it is 2124 (18 years after 1980).
  So, per the book, ELKS is right on both accounts and DOS is using +100 as some kind of flag (these files are coming off of a DOS installation floppy).

@ghaerr, I'll leave the rest to you.

[ghaerr]

@Mellvik, regarding the above 100 year off date problem: The standard mdir tool also shows the dates on your 3rd disk as being in 2098 as well:

Gregs-MBP2:elks-gh greg$ mdir -i fd3-fat-bad.img 
 Volume in drive : has no label
Directory for ::/

ANSI     SY_      6261 2098-01-01   0:04 
APPEND   EX_      7467 2098-01-01   0:04 
ATTRIB   EXE     11208 2098-01-01   0:04 
AUTOEXEC BAT        38 2098-01-01   0:04 

So, per the book, ELKS is right on both accounts and DOS is using +100 as some kind of flag (these files are coming off of a DOS installation floppy).

Thus, I am not going to try any further date changes to FAT at this point. Should you find other date bugs in FAT, lets open a new issue, so this one can be closed, provided that my latest PR fixes the 3 disk images you uploaded.

[Mellvik]

@ghaerr, Sounds reasonable. Like I mentioned before, even mainline Linux (at least debian, and raspbian which is my ELKS cross devel/test environ) has this (minor) problem. I'd rather see (yes, I know I'm repeating myself) focus on the speed of FAT directory listings - eventually. Maybe as a side-effect of the malloc cleanup activities? Sligtly related: Now that 'msdos' has been (generally) replaced by 'fat' in the kernel (maybe eventually 'VFAT'), how about updating userland similarly: mount -t fat ...

…

--Mellvik

22. feb. 2020 kl. 21:56 skrev Gregory Haerr ***@***.***>:  @Mellvik, regarding the above 100 year off date problem: The standard mdir tool also shows the dates on your 3rd disk as being in 2098 as well: Gregs-MBP2:elks-gh greg$ mdir -i fd3-fat-bad.img Volume in drive : has no label Directory for ::/ ANSI SY_ 6261 2098-01-01 0:04 APPEND EX_ 7467 2098-01-01 0:04 ATTRIB EXE 11208 2098-01-01 0:04 AUTOEXEC BAT 38 2098-01-01 0:04 So, per the book, ELKS is right on both accounts and DOS is using +100 as some kind of flag (these files are coming off of a DOS installation floppy). Thus, I am not going to try any further date changes to FAT at this point. Should you find other date bugs in FAT, lets open a new issue, so this one can be closed, provided that my latest PR fixes the 3 disk images you uploaded. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Mellvik]

Testing build 398ef39 - no new FAT related issues.
I agree the this issue may be closed.

One observation FWIW: Filesystem type 6, FAT16B, mounts and works fine. The debug message at mount reports FAT: 0k. If that's suspicious, now it has been reported.

[ghaerr]

One observation FWIW: Filesystem type 6, FAT16B, mounts and works fine. The debug message at mount reports FAT: 0k.

@Mellvik: Yes, looks like the size reporting is only working for FAT12 images. Do you have a FAT16 image you could upload for my testing on this?