cnvogelg / amitools

Various tools for using AmigaOS programs on other platforms

amitools - various AmigaOS tools for other platforms

• written by Christian Vogelgsang [email protected]
• under the GNU Public License V2

Introduction

amitools is a collection of Python 3 tools that I've written to work with Amiga OS binaries and files on macOS and all other *nix-like platforms supporting Python. Windows might work as well, but is heavily untested. However, patches are welcome.

I focus with my tools on classic Amiga setups, i.e. a 680x0 based system with Amiga OS 1.x - 3.x running on it. However, this is an open project, so you can provide other Amiga support, too.

The tools are mostly developer-oriented, so a background in Amiga programming will be very helpful.

Prerequisites

• Python >= 3.7
• pip3

Optional Packages

• lhafile - FS Edition: required to use .lha file scanner
• machine68k: required to run vamos

Installation

Stable/Release Version

If you only need the tools without vamos then you can install the pure Python version:

pip3 install amitools

If you want to run vamos then you need the CPU emulator from the machine68k package and you can install this dependency with:

pip3 install 'amitools[vamos]'

Note:

• on Linux/macOS may use sudo to install for all users
• the version may be a bit outdated. If you need recent changes use the current version.

Current Version from GitHub

If you wan to run vamos then first install the CPU emulator machine68k:

pip3 install -U git+https://github.com/cnvogelg/machine68k.git

Then install amitools directly from the git repository:

pip3 install -U git+https://github.com/cnvogelg/amitools.git

Note:

• This will install the latest version found in the github repository.
• You find the latest features but it may also be unstable from time to time.
• Repeat this command to update to the latest version.

Developers

• Follow this route if you want to hack around with the amitools codebase
• Clone the Git repo: amitools@git
• Ensure you have Cython and machine68k installed:

pip3 install cython machine68k

• Enter the directory of the cloned repo and install via pip:

pip3 install -U -e .

This install amitools in your current Python environment but takes the source files still from this repository. So you can change the code there and directly test the tools.

Contents

The new Documentation of amitools is hosted on readthedocs

Tools

• vamos V)irtual AM)iga OS

  vamos allows you to run command line (CLI) Amiga programs on your host Mac or PC. vamos is an API level Amiga OS Emulator that replaces exec and dos calls with its own implementation and maps all file access to your local file system.

  Note: vamos requires the package machine68k installed first!

• xdftool

  Create and modify ADF or HDF disk image files.

• xdfscan

  Scan directory trees for ADF or HDF disk image files and verify the contents.

• rdbtool

  Create or modify disk images with Rigid Disk Block (RDB)

• romtool

  A tool to inspect, dissect, and build Amiga Kickstart ROM images to be used with emulators, run with soft kickers or burned into flash ROMs.

• hunktool

  The hunktool uses amitools' hunk library to load a hunk-based amiga binary. Currently, its main purpose is to display the contents of the files in various formats.

  You can load hunk-based binaries, libraries, and object files. Even overlayed binary files are supported.

• typetool

  This little tool is a companion for vamos. It allows you to dump and get further information on the API C structure of AmigaOS used in vamos.

• fdtool

  This tool reads the fd (function description) files Commodore supplied for all of their libraries and dumps their contents in different formats including a code structure used in vamos.

  You can query functions and find their jump table offset.

Python Libraries

• Hunk library amitools.binfmt.hunk

  This library allows to read Amiga OS loadSeg()able binaries and represent them in a python structure. You could query all items found there, retrieve the code, data, and bss segments and even relocate them to target addresses

• ELF library amitools.binfmt.elf

  This library allows to read a subset of the ELF format mainly used in AROS m68k.

• .fd File Parser amitools.fd

  Parse function descriptions shipped by Commodore to describe the Amiga APIs

• OFS and FFS File System Tools amitools.fs

  Create or modify Amiga's OFS and FFS file system structures

• File Scanners amitools.scan

  I've written some scanners that walk through file trees and retrieve the file data for further processing. I support file trees on the file system, in lha archives or in adf/hdf disk images

amitools's Issues

Should hunktool.py learn how to extract version information?

I've found it very useful to be able to recursively scan archives/directories/binaries with hunktool, but my purpose was to gather lots of version information.

To do this I hacked together a quick fork of hunktool.py which uses Python's re module to search for the $VER: cookie. It's not complete because it ignores part of the version string, but it was sufficient for my needs.

Is there any interest in having this polished up and made part of hunktool? (or a separate tool specifically for printing version strings).

You can see my hackfork at: https://github.com/cmsj/amitools/blob/e94cc3dacd8cc1a987101d34be23425a117210ba/amitools/tools/version_hunktool.py

'NoneType' object has no attribute 'addr_base'

I tested with espeak the new math-libraries.
The test succeeded until/including
"16.3.2018 added fake stub generation fc5f0e8"

Starting with 16.3.2018 lib_mgr only returns lib addr 6c8ca13
I get the following error:

# lib_mgr only returns lib addr 6c8ca13ea3d2026bf271e8e6e6a2eea9a4a5c61f
developer@cn-vm-afritsch:~/espeak_src/espeak-source/src$ vamos -m 8192  -- espeak-m68000  --path=amiga -w hello.wav "Hello Alexander"
Traceback (most recent call last):
  File "<string>", line 3, in call_stub
  File "/home/developer/amitools/amitools/vamos/lib/ExecLibrary.py", line 120, in OpenLibrary
    addr = self.lib_mgr.open_lib(name, ver)
  File "/home/developer/amitools/amitools/vamos/LibManager.py", line 165, in open_lib
    return lib.addr_base
AttributeError: 'NoneType' object has no attribute 'addr_base'
developer@cn-vm-afritsch:~/espeak_src/espeak-source/src$ 

Error reopening file for writing

PPAMI.zip

This program opens file for reading to decompress it, then, it reopens the same file for writing. But, when opening for writing happens, it clears original file, and this program cannot pack it because of zero size.

File: test1.zip

Example 1 (works, because it creates test1.new instead of decrypted test1.rnc):
vamos -v ppami.exe u d .new work:test1.rnc
Example 2 (doesn't work):
vamos -v ppami.exe u d work:test1.rnc

This program works in WinUAE.

rdbtool fsadd uses always DOS1 as dostype

rdbtool -f /tmp/test.rdb create size=20Mi +  init rdb_cyls=6 + add size=50% dostype=PFS3 name=CH0 bootable=true automount=true  + fsadd /path/to/pfs3_aio-handler dostype=PFS3

rdbtool /tmp/test.rdb info still shows FileSystem #0 DOS1 version=18.5 size=69388

I have already fixed this (and no, I will not create another pull request 😄 ) but the partition is still not visible on the workbench. (However it is recognized in the early startup menu.)

So the question is: Is it even possible to create a valid hard disk image with custom fs? And if so, how do I do it?

Installation process misses data files

I get following error after normal installation:

$ python setup.py install --user
$ vamos -C 68020 hello-ks13                                                                                       (master|…)
18:13:36.765     libmgr:  ERROR:  [create_lib] can't create auto lib without FD file: exec.library
18:13:36.766     libmgr:  ERROR:  [create_lib] can't create auto lib without FD file: dos.library
Traceback (most recent call last):
  File "/Users/cahir/Library/Python/2.7/bin/vamos", line 11, in <module>
    load_entry_point('amitools==0.1.0', 'console_scripts', 'vamos')()
  File "/Users/cahir/Library/Python/2.7/lib/python/site-packages/amitools-0.1.0-py2.7-macosx-10.9-x86_64.egg/amitools/tools/vamos.py", line 112, in main
    shell=args.shell, cwd=cwd)
  File "/Users/cahir/Library/Python/2.7/lib/python/site-packages/amitools-0.1.0-py2.7-macosx-10.9-x86_64.egg/amitools/vamos/Process.py", line 14, in __init__
    input_fh = self.ctx.dos_lib.file_mgr.get_input()
AttributeError: 'NoneType' object has no attribute 'file_mgr'

I fixed the problem by adding MANIFEST.in file containing:

recursive-include amitools/data *

Is it possible to identify library func call?

Fo example, I have -$132(a6) such call. And I know value of a6. Is it specified anywhere in Amitools code which func will be called, and how to identify destination function?

xdftool pack No Free Blocks

I want to repack one adf-file with adding my own file to existing adf (workbench's one).
But when I'm trying to add a file, it tells me this:

$ xdftool.exe newimage.adf pack Workbench3.1
'pack Workbench3.1' FSError: No Free Blocks(7):node=ADFSDir:'Utilities'(@768),file_name=PPAMI.EXE,want 87

what it means and how to fix that?

Some loadable modules not working in romtool

Firstly, I love the idea of using my Linux box to build ROMs instead of relying on Remus under emulation!

OS ROMs build for me successfully, however it seems not all loadable modules are recognised.

For example, KeymapGB from Remus works OK, but MoveVBR from BlizKick does not.

MoveVBR is added and the used ROM size increases. I can see the version text in the ROM but it is not recognised by the Amiga. Other Blizkick modules fail too.

I built a ROM with only exec and MoveVBR:
romtool scan romtool.rom
@000000ae +00004192 NT_LIBRARY +105 exec.library exec 45.20 (6.1.2002)
@00004192 +00004244 NT_UNKNOWN -55 alert.hook alert.hook

Same ROM from Remus:
romtool scan remus.rom
@000000ae +00004192 NT_LIBRARY +105 exec.library exec 45.20 (6.1.2002)
@00004192 +00004244 NT_UNKNOWN -55 alert.hook alert.hook
@00004246 +000042e0 NT_UNKNOWN +104 MoveVBR MoveVBR 1.2 (11.9.96)

Here is the output from romtool when building:
2017-03-10 00:12:32,624 INFO root Welcom to romtool
2017-03-10 00:12:32,624 INFO root building 512 KiB Kick ROM @00f80000
2017-03-10 00:12:32,624 INFO root @00000000: adding module 'Test/exec_45.20(A4000)'
2017-03-10 00:12:32,625 INFO root @00004244: adding module 'Test/MoveVBR'
2017-03-10 00:12:32,625 INFO root @00004308: padding 507104 bytes with ff
2017-03-10 00:12:32,714 INFO root saving ROM to 'test.rom'

Any ideas?

Can't compile musashi

$ make
mkdir BUILD
gcc -g -O2 --std=c99 -c -o BUILD/m68kmake.o m68kmake.c
gcc -g -O2 --std=c99 -o m68kmake BUILD/m68kmake.o
./m68kmake

    Musashi v3.3 68000, 68010, 68EC020, 68020 emulator
    Copyright 1998-2000 Karl Stenerud ([email protected])

In m68k_in.c, near or on line 2650:
    Unknown M68KMAKE directive
make: *** [m68kops.h] Błąd 1

hunktool -v key error

File "hunktool.py", line 55, in handle_file
    if args.dump:
NameError: global name 'args' is not defined

basm: duplicate relocs in hunk

hi, basm from http://aminet.net/dev/asm/BarflyDisk2_00.lha cannot be load due 'duplicate relocs in hunk'.
what does this mean?
I tried to understand HunkLoadSegFile.py but failed.
Is there maybe a problem with HUNK_RELOC32SHORT?

xdftool `write` command misses last file on RDB disks

If I use the xdftool write command to write a directory full of files to a partition on an RDB disk, the last file in the directory is missed off. This does not happen on non-RDB disks.

This script demonstrates the problem:

#!/usr/bin/env bash -e

AMITOOLS="${HOME}/Dropbox/Personal/Amiga/amitools"
RDBTOOL="${AMITOOLS}/rdbtool"
XDFTOOL="${AMITOOLS}/xdftool"

HDF="test.hdf"
GEOMETRY="chs=100,16,63"

mkdir -p testdata
touch testdata/file1
touch testdata/file2
touch testdata/file3
touch testdata/file4
touch testdata/file5

echo === Disk without RDB ===
[ -f "${HDF}" ] && rm "${HDF}"
${XDFTOOL} ${HDF} create ${GEOMETRY} + format testdata ffs intl
${XDFTOOL} ${HDF} open ${GEOMETRY} + write testdata
${XDFTOOL} ${HDF} open ${GEOMETRY} + list

echo === Disk with RDB ===
[ -f "${HDF}" ] && rm "${HDF}"
${RDBTOOL} ${HDF} create ${GEOMETRY} + init
${RDBTOOL} ${HDF} open ${GEOMETRY} + add
${XDFTOOL} ${HDF} open ${GEOMETRY} part=0 + format testdata ffs intl
${XDFTOOL} ${HDF} open ${GEOMETRY} part=0 + write testdata
${XDFTOOL} ${HDF} open ${GEOMETRY} part=0 + list

I see this output:

=== Disk without RDB ===
testdata                                         VOLUME  --------  11.02.2016 16:57:20 t00  DOS3:ffs+intl
  testdata                                          DIR  ----rwed  11.02.2016 16:57:20 t00
    file1                                             0  ----rwed  11.02.2016 16:57:20 t00
    file2                                             0  ----rwed  11.02.2016 16:57:20 t00
    file3                                             0  ----rwed  11.02.2016 16:57:20 t00
    file4                                             0  ----rwed  11.02.2016 16:57:20 t00
    file5                                             0  ----rwed  11.02.2016 16:57:20 t00
sum:             7  3.5Ki          3584
data:            0    0Bi             0   0.00%
fs:              7  3.5Ki          3584  100.00%
=== Disk with RDB ===
creating: 'DH0' (1, 99) DOS3
testdata                                         VOLUME  --------  11.02.2016 16:57:21 t00  DOS3:ffs+intl
  testdata                                          DIR  ----rwed  11.02.2016 16:57:21 t00
    file1                                             0  ----rwed  11.02.2016 16:57:21 t00
    file2                                             0  ----rwed  11.02.2016 16:57:21 t00
    file3                                             0  ----rwed  11.02.2016 16:57:21 t00
    file4                                             0  ----rwed  11.02.2016 16:57:21 t00
sum:             6  3.0Ki          3072
data:            0    0Bi             0   0.00%
fs:              6  3.0Ki          3072  100.00%

Note that in the second case, where rdbtool is used to create the disk image and part=0 is given to xdftool to select the RDB partition, then file5 is not inserted into the disk image.

(This is a great set of tools, btw 😃 )

Crash when Amiga library function returns multiple results

I'm trying to run binaries compiled with gcc -noixemul using vamos. There're several calls missing from libraries:

22:37:32.011 lib:WARNING: [ exec.library] ? CALL: 558 InitSemaphore( sigSem[a0]=000047cc ) from PC=002890 -> d0=0 (default)
22:37:32.011 lib:WARNING: [ dos.library] ? CALL: 576 GetProgramName( buf[d1]=00004808, len[d2]=00000100 ) from PC=002120 -> d0=0 (default)
22:37:32.011 lib:WARNING: [ exec.library] ? CALL: 564 ObtainSemaphore( sigSem[a0]=000047cc ) from PC=00279a -> d0=0 (default)
22:37:32.012 lib:WARNING: [ exec.library] ? CALL: 186 Allocate( freeList[a0]=00004908, byteSize[d0]=00000010 ) from PC=002834 -> d0=0 (default)
22:37:32.012 lib:WARNING: [ exec.library] ? CALL: 570 ReleaseSemaphore( sigSem[a0]=000047cc ) from PC=00284e -> d0=0 (default)
22:37:32.012 lib:WARNING: [ utility.library] ? CALL: 150 SDivMod32( dividend[d0]=00000014, divisor[d1]=0000000a ) from PC=001d3a -> d0=0 (default)

For a starter I wanted to implement SDivMod32 in UtilityLibrary.py, but I run into a problem:

**** Unexpected exception in library stub: an integer is required File "", line 5, in call_stub
File "musashi/pycpu.pyx", line 63, in musashi.emu.CPU.w_reg (emu.c:1380)
self.w_reg_internal(reg,val)

It seems to me that code crashes because SDivMod32 needs to return two values. Could you verify that?

mathieeesingbas.library

I try to use http://sun.hasenbraten.de/vbcc/ but mathieeesingbas.library is missing. Are there plans to add it?
The library is normally in ROM therefore I don't have a file based version of it.

musashi fails to compile on gcc 4.9.0 (compiler doesnt specificy C99 mode)

g++ --version
g++ (Debian 4.9.0-7) 4.9.0
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

amitools/musashi$ make
gcc -g -O2 -o m68kmake BUILD/m68kmake.o
./m68kmake

            Musashi v3.3 68000, 68010, 68EC020, 68020 emulator
            Copyright 1998-2000 Karl Stenerud ([email protected])

Generated 1962 opcode handlers from 513 primitives
gcc -g -O2 -c -o BUILD/traps.o traps.c
traps.c: In function ‘trap_init’:
traps.c:50:3: error: ‘for’ loop initial declarations are only allowed in C99 or C11 mode
for(int i=0;i<(NUM_TRAPS-1);i++) {
^
traps.c:50:3: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code

Windows version

Funny, I was thinking of building a similar tool for the exact purpose (running SAS/C compiler). Luckily I've found your solution. I might be able to contribute a pull request for changes required to build a Windows version (mainly musashi).

setup.sh depends on 'make' being GNU Make

vamos seems to work very well on FreeBSD 11.1.

However, when installing in FreeBSD according to README.md, the make call in setup.py does not generate the files required for the musashi build. In FreeBSD make is the BSD Make. GNU Make is typically installed as gmake by the package system. Changing make into gmake in setup.py results in a successful build, usable for running the m68k GCC test suite.

It could maybe be possible to auto-detect existence of gmake and fall back to make if not found. Or pick up the name from an environment variable.

Following is the error message. The file m68kops.h is indeed missing with BSD Make, but is created with GNU Make.

$ python2.7 setup.py build
[...]
cc -fno-strict-aliasing -O2 -pipe -fstack-protector -fno-strict-aliasing -DNDEBUG -fPIC -Imusashi -Igen -I/usr/local/include/python2.7 -c musashi/m68kcpu.c -o build/temp.freebsd-11.1-RELEASE-amd64-2.7/musashi/m68kcpu.o
musashi/m68kcpu.c:41:10: fatal error: 'm68kops.h' file not found
#include "m68kops.h"
         ^~~~~~~~~~~
1 error generated.
error: command 'cc' failed with exit status 1

C++ port

It would be great to have ability to wrap or to embed vamos library features into the external code, like a C++.
I know it requires so much time to port vamos into C++, but, until amount of code lines is not so big, could you start to port it? Your knowledges (as of an admin of this project) of internal variable types is much better than somebody other, so, only you port it the best way.

Is it possible in near future?

hunktool throws an exception if python-lha is not installed

Sure it does not need python-lha to work correctly, nonetheless it tries to import it... IMO tools that do not require python-lha should not crash nor warn user if the library is missing.

Unexpected exception in library stub: 'NoneType' object has no attribute 'w_s' File "<string>", line 3, in call_stub

When calling sc with two arguments like for ex.

vamos sc main.c ave.c

I get the following exception

**** Unexpected exception in library stub: 'NoneType' object has no attribute 'w_s' File "", line 3, in call_stub
File "/home/luca/vamos-python/lib/python2.7/site-packages/amitools-0.1.0-py2.7-linux-x86_64.egg/amitools/vamos/lib/ExecLibrary.py", line 113, in OpenLibrary
lib = self.lib_mgr.open_lib(name, ver, ctx)
File "/home/luca/vamos-python/lib/python2.7/site-packages/amitools-0.1.0-py2.7-linux-x86_64.egg/amitools/vamos/LibManager.py", line 97, in open_lib
lib.inc_usage()
File "/home/luca/vamos-python/lib/python2.7/site-packages/amitools-0.1.0-py2.7-linux-x86_64.egg/amitools/vamos/AmigaLibrary.py", line 362, in inc_usage
self.lib_access.w_s("lib_OpenCnt", self.ref_cnt)

and many more.

I have fixed it changing the method _unregister_open_lib in LibManager.py

def _unregister_open_lib(self, lib, libbase):
# we never really flush libraries, unless the library
# builds its own libbase on each invocation.
# unregister lib
#del self.open_libs_addr[lib.addr_base]
# own base per open?
if libbase != None and libbase != lib.addr_base:
del self.open_libs_addr[libbase]
del self.open_libs_name[lib.name] <======== I changed this! Before it was commented out

The change was uncommenting the line below.
But maybe there was a good reason for commenting it out ?

I used the following files:

---- main.c ----------------------------------------------------------

#include <stdlib.h>

extern void ave( void );

int main()
{
ave();
return 0;
}

---- ave.c -----------------------------------------------------------

#include <stdio.h>

void ave( void )
{
printf( "Ave, Vamos!\n" );
}

---- .vamosrc --------------------------------------------------------

.vamosrc - vamos config file for SAS C compiler

[volumes]
wb310=/amiga/Emulation/shared/dir/System
sc=/vamos-amiga/sc/sasc
net=~/vamos-amiga/src/AmiTCP-SDK-4.3

[assigns]

cxxinclude=sc:cxxinclude
include=sc:include
lib=sc:lib
includeasm=sc:include

netinclude=net:netinclude
netlib=net:netlib

c=wb310:c
t=root:tmp

[path]
path=sc:c,wb310:c

"auto" mode: either use amiga library if available and fall back

to vamos internal lib if not amiga library was found

[icon.library]
mode=auto

[vamos]
quiet=True
ram_size=2048

hunktool -x key error

sc.zip

print_hex(main['data'],indent=8)
KeyError: 'data'

Subj.

Issues with SAS/C

Hi,

Having a bit of trouble with vamos, I followed your blog post to setup SAS/C. It seems it cannot read/write in current directory.

[volumes]
src=~/Dropbox/emu/amiga/src
work=~/Dropbox/emu/amiga/hdd/Work
system=~/Dropbox/emu/amiga/hdd/DH0

[assigns]
c=system:c
sc=work:program/sc
include=sc:include
lib=sc:lib
c=system:c,sc:c
t=root:tmp

[path]
path=sc:c,system:c

# disable loading library in any case
[68040.library]
mode=off

# "auto" mode: either use amiga library if available and fall back
# to vamos internal lib if not amiga library was found
[icon.library]
mode=auto

[vamos]
quiet=True
ram_size=2048

When trying smake: (there is a smakefile in current directory)

~/Dropbox/emu/amiga/src$ vamos -c vamosrc smake main
SAS/C_SMAKE 6.58 (27.12.96)
Copyright (c) 1988-1995 SAS Institute, Inc.
smakefile: cannot open

Related issue if I try to compile a simple helloworld program with ´sc´:

SAS/C Amiga Compiler 6.58
Copyright (c) 1988-1995 SAS Institute Inc.
CXWRN: Can't create intermediate file
QName: main.q
ERROR: 205

LockManager bug in generate_key()

Doesn't work on Windows:
https://github.com/cnvogelg/amitools/blob/master/amitools/vamos/lib/dos/LockManager.py#L21

avoid: can't create auto lib without FD file

Is there a way to avoid the message:
14:10:17.266 libmgr: ERROR: [create_lib] can't create auto lib without FD file: cachefile.library

I have in my .vamosrc:
[cachefile.library]
mode=amiga

and also the lib in libs:. But the message still appears.

Cannot load vamos on Windows (ImportError: DLL load failed)

d:\git\amitools>python vamos
Traceback (most recent call last):
File "vamos", line 14, in
from musashi import emu
ImportError: DLL load failed: ═х эрщфхэ єърчрээ√щ ьюфєы№.

Compilation with Msys2 was without any error. And there is emu.pyd file in musashi library.

hunktool: has invalid padding

RESULT_INVALID_HUNK_FILE HUNK_INDEX
amiga.zip

OverflowError: value does not fit into word

Starting with

26.3.2018 "correctly handle signed types in amiga struct 4eb3d20"

I see a new, different error:

developer@cn-vm-afritsch:~/espeak_src/espeak-source/src$ vamos -m 8192  -- espeak-m68000  --path=amiga -w hello.wav "Hello Alexander"
Traceback (most recent call last):
  File "<string>", line 3, in call_stub
  File "/home/developer/amitools/amitools/vamos/lib/ExecLibrary.py", line 490, in InitSemaphore
    self.semaphore_mgr.InitSemaphore(addr)
  File "/home/developer/amitools/amitools/vamos/lib/lexec/SemaphoreManager.py", line 19, in InitSemaphore
    semaphore.w_s("ss_QueueCount",0xffff)
  File "/home/developer/amitools/amitools/vamos/mem/access.py", line 12, in w_s
    self.mem, self.struct_addr, name, val, do_conv)
  File "/home/developer/amitools/amitools/vamos/mem/astruct.py", line 197, in write_field
    self._write_base(mem, base_type, addr, val, do_conv)
  File "/home/developer/amitools/amitools/vamos/mem/astruct.py", line 183, in _write_base
    mem.writes(width, addr, val)
  File "musashi/pymem.pyx", line 209, in musashi.emu.Memory.writes (musashi/emu.c:7391)
    self.w16s(addr, value)
  File "musashi/pymem.pyx", line 169, in musashi.emu.Memory.w16s (musashi/emu.c:6327)
    raise OverflowError("value does not fit into word")
OverflowError: value does not fit into word

vamos runs one command too many times

When I'm trying to run this tool: http://aminet.net/package/util/pack/RNC_ProPack
vamos executes "PPAMI.EXE p d system:README" command too many times. But it should run it only once.
Few month before it worked good.

Should romtool be able to read/create AmigaOS ROM Update files?

OS 3.9 (and maybe 3.5?) use a bundle of ROM modules to load resident with SetPatch, AmigaOS ROM Update. It seems like the format of the file has been figured out (see http://eab.abime.net/showthread.php?t=79968 )

I think it would be useful to be able to build a ROM Update file, and to be able to extract them. I'm prepared to put some time into attempting to teach amitools to do this, but before I start, I'd like to know if it would be accepted into the main tree.

Thanks!

is_interactive call in DosLibrary.py/FileHandle.py has mismatched parmeters.

The class FileHandle from FileHandle.py defines is_interactive() as...

def is_interactive(self, fh):
fd = fh.obj.fileno()
.....

However this is called by DosLibrary.py as ...

amitools/vamos/lib/DosLibrary.py:445: res = fh.is_interactive()
amitools/vamos/lib/dos/FileHandle.py:78: def is_interactive(self, fh):

Im pretty sure the fh parameter isnt needed. on FileHandle.py, you could use self in that method instead.

Missing library calls for simple binaries linked against libnix

I'm trying to run gcc testsuite for amigaos-cross-toolchain using vamos. But I get plenty of following errors:

Testing gcc.c-torture/execute/20000113-1.c,  -Os
doing compile
pid is 84596 -84596
pid is -1
output is  status 0
spawning command  vamos -C 68020 -s 4096 -m 8192 /Users/cahir/workspace/amigaos-cross-toolchain/.build-m68k/build/gcc-2.95.3/gcc/testsuite/20000113-1.x5
pid is -1
Shell closed.
Output is 22:30:43.839       path:WARNING:  ami_command_to_sys_path: ami_path='libs:utility.library' not found!
22:30:43.842        lib:WARNING:  [     dos.library]  ? CALL:  576 GetProgramName( buf[d1]=00403698, len[d2]=00000100 ) from PC=001ff8 -> d0=0 (default)
22:30:43.842        lib:WARNING:  [    exec.library]  ? CALL:  564 ObtainSemaphore( sigSem[a0]=0040365c ) from PC=00292e -> d0=0 (default)
22:30:43.842        lib:WARNING:  [    exec.library]  ? CALL:  186 Allocate( freeList[a0]=00403798, byteSize[d0]=00000010 ) from PC=0029c8 -> d0=0 (default)
22:30:43.842        lib:WARNING:  [    exec.library]  ? CALL:  570 ReleaseSemaphore( sigSem[a0]=0040365c ) from PC=0029e2 -> d0=0 (default)

FAIL: gcc.c-torture/execute/20000113-1.c execution,  -Os

The test is pretty straightforward and doesn't use any fancy features of libnix:

struct x {
  unsigned x1:1;
  unsigned x2:2;
  unsigned x3:3;
};

foobar (int x, int y, int z)
{
  struct x a = {x, y, z};
  struct x b = {x, y, z};
  struct x *c = &b;

  c->x3 += (a.x2 - a.x1) * c->x2;
  if (a.x1 != 1 || c->x3 != 5)
    abort ();
  exit (0);
}

main()
{
  foobar (1, 2, 3);
}

I guess vamos is missing implementation of GetProgramName, Allocate and Deallocate. Would you be able to quickly add those?

ExNext() impementation missing

There is Examine() function realization, but there is no ExNext() function (and maybe some more: ExAll, ExAllEnd) to examine directories properly.

objdump doesn't disassemble code

I'm using "-o" and "-A" and "m68k-elf-objdump" binary. But I don't see any disassembled code in output.

Also, vda68k disassemble output is not very good: strings are interpreted as code, no branches marking, etc.

storing 64-bit inodes in fl_Key

The current system takes the host system's inodes, and stores them in the fl_Key field of the FileLock structure. This works on systems with 32-bit inodes, but fails with a OverflowError: long int too large to convert error with 64-bit inodes, like Cygwin on 64-bit Windows. Is it safe to truncate this number to 32-bit? AFAIK the value of fl_Key is specific to the file system handler, and programs shouldn't really use it.

Feature request: physical device resize

When moving an RDB image to a different physical media/block device I wanted to patch the RDB to extend to the bigger size. Patching the RDB manually didn't work due to checksum problems - looking into this myself should be easy (didn't yet get to it), but this is functionality I'd love to see in amitools, too. Thanks for the great work!

Segment class - start and addr fields

There are two fields in Segment class: Start and Addr.

self.addr = addr
self.start = addr + 8

What every address means? And which one points to segment bytes?

CR: Pool support missing (Patch included)

I noticed that the Exec is missing CreatePool/DeletePool/AllocPooled support so i created a patch.

https://www.dropbox.com/s/0s9ltmzhs9iu2jz/basicpoolsupport.patch?dl=0

Its a little rough (it doesnt support freeing small blocks yet etc but its a start that lets more apps work).

_read_long fails on Ubuntu 12.04 LTS

I tried running vamos on Ubuntu 12.04 using Python 2.7.3, and got the following error:

File "vamos", line 104, in
proc = Process(vamos, args.bin, args.args, stack_size=cfg.stack_size*1024)
File "/home/bszili/amitools/amitools/vamos/Process.py", line 14, in init
self.ok = self.load_binary(bin_file)
File "/home/bszili/amitools/amitools/vamos/Process.py", line 57, in load_binary
self.bin_seg_list = self.ctx.seg_loader.load_seg(ami_bin_file)
File "/home/bszili/amitools/amitools/vamos/SegmentLoader.py", line 73, in load_seg
seg_list = self._load_seg(ami_bin_file,sys_bin_file)
File "/home/bszili/amitools/amitools/vamos/SegmentLoader.py", line 142, in _load_seg
datas = relocator.relocate(addrs)
File "/home/bszili/amitools/amitools/binfmt/Relocate.py", line 63, in relocate
self._reloc(segment.id, data, r, to_addr, to_id)
File "/home/bszili/amitools/amitools/binfmt/Relocate.py", line 70, in _reloc
delta = self._read_long(data, offset) + reloc.addend
File "/home/bszili/amitools/amitools/binfmt/Relocate.py", line 79, in _read_long
return struct.unpack(">i",d)[0]
struct.error: unpack requires a string argument of length 4

Is there any workaround for this?

ReadArgs /A/M working incorrectly

Arguments are parsed wrong if template like 'arg1/A,arg2/A,argn/A/M' is used.
The first parameters are assigned to argn, the last to arg2 and the second last to arg1.
Correct behavior would be that the first two args are assigned to arg1 and arg2 and the remain to argn.
This was working correctly in older releases of vamos. Maybe broken since c722b20.

Vamos parses binary args as its own

This binary: http://aminet.net/package/util/pack/RNC_ProPack

Command line is such one: ppami4 p d -m1 -k 0x1234 -p 10240 -l -o -v .rnc system:PACK_TEST.TXT
But I'm getting this as output:

usage: vamos.py [-h] [-c CONFIG_FILE] [-S] [-l LOGGING] [-v] [-q] [-b]
                [-L LOG_FILE] [-I] [-t] [-T] [-r] [-C CPU] [-y MAX_CYCLES]
                [-B CYCLES_PER_BLOCK] [-m RAM_SIZE] [-s STACK_SIZE]
                [-H HW_ACCESS] [-x] [-D DATA_DIR] [-O LIB_OPTIONS] [-a ASSIGN]
                [-V VOLUME] [-A AUTO_ASSIGN] [-p PATH] [-d CWD] [-P]
                bin [args [args ...]]
vamos.py: error: argument -l/--logging: expected one argument

xfdtool generates adf files that are too large for HxC floppy emulator

I'm doing development on the Amiga and using the tool to generate adf files. These work fine in UAE, but HxC will not allow me to convert to their format to use w/ the HxC.

More info here:

http://www.torlus.com/floppy/forum/viewtopic.php?f=5&t=3133&p=15674#p15674

Hunk parsing issue when using VBCC

I compiled up a hello world application with vbcc (I'll upload a simple binary to dropbox)...

https://www.dropbox.com/s/24647vv97g3utd8/hello?dl=0

12:37:13.857 proc: ERROR: failed loading binary: Error loading '/home/stephen/compileZone/amitools/hello': Unsupported hunk type: 0000

I investigated this a bit and It seems to fail to parse a 1015 hunk (one it converts to 1020). It underreads the segment by 2 bytes so the f.read(4) on the next segment is not aligned to the start of the segment.

Hunkinfo parses this successfully as.... I wonder if its something to do with LoadSeg?

$ ./hunktool -d info hello
hello (0.0032s) TYPE_LOADSEG
header (segments: first=0, last=5, table size=6)
#000 CODE size 00000c54 alloc size 00000c54 file header @0000002c data @00000034
reloc drel32 #6
To Segment #0: 22 entries
To Segment #1: 36 entries
To Segment #2: 8 entries
To Segment #3: 2 entries
To Segment #4: 2 entries
To Segment #5: 43 entries
#1 DATA size 00000008 alloc size 0000003c file header @00000d8c data @00000d94
#2 DATA size 00000010 alloc size 00000010 file header @00000da0 data @00000da8
#3 DATA size 0000000c alloc size 0000000c file header @00000dbc data @00000dc4
reloc drel32 #1
To Segment #0: 1 entries
#4 DATA size 00000010 alloc size 00000010 file header @00000de0 data @00000de8
reloc drel32 #1
To Segment #0: 2 entries
#5 BSS size 00000018 alloc size 00000018 file header @00000e0c
RESULT_OK : 1

rdbtool: "add" seems to be ignoring `heads` count

I am creating a new disk image using rdbtool giving a specific drive geometry. I then add a new partition that is supposed to take up 100% of the disk.

But when I query back the RDB, the new partition is showing the expected low/high cylinder values but the total block count is wrong:

# rdbtool ~/tmp/amiga2gb_new.img create chs=7544,16,32 + init + add size=100% + info
creating: 'DH0' (1, 7543) DOS3
PhysicalDisk:               0     7543     3862528  1.8Gi  heads=16 sectors=32
LogicalDisk:                1     7543     3862016  1.8Gi  rdb_blks=[0:511,#512] used=[hi=1,#2] cyl_blks=512
Partition: #0 'DH0'         1     7543      241376  117Mi    6.25%  DOS3/0x444f5303 max_transfer=0xffffff mask=0x7ffffffe num_buffer=30

It appears that the total block count is not taking the "heads" geometry into account. This disk has 16 heads; 6.25% is one-sixteenth of the disk. A different geometry using 8 heads resulted in the partition filling 12.5% of the disk - one-eighth of the disk.

xdftool fails with FSError: Invalid Root Block

I'm trying to unpack couple of adfs but xdftool fails on all of them. Here's the example output:

xdftool -v perihelion_1of5.adf info
auto open command: ['info']
auto open exit_code: 0
opening volume: perihelion_1of5.adf
'info' FSError: Invalid Root Block(2):block=RootBlock:@880
closing volume: perihelion_1of5.adf
closing image: perihelion_1of5.adf

It fails on all five files with similar message, only difference being RootBlock, for example:

xdftool -v perihelion_2of5.adf info
auto open command: ['info']
auto open exit_code: 0
opening volume: perihelion_2of5.adf
'info' FSError: Invalid Boot Block(1):block=BootBlock:@0
closing volume: perihelion_2of5.adf
closing image: perihelion_2of5.adf

The adf files in question can be found here.

As a side not, adfs work as they should with FS-UAE.

FDFormat and mathieeedoubbas_lib.fd

Out of curiosity I tried to run PhxAss under vamos... and it fails :-(

After short debugging session I figured out that read_fd doesn't handle a case when an argument to function is passed through two registers. That's the case with mathieeedoubbas as it operates on 64-bit floating point numbers. Following line fails to parse:

IEEEDPFix(parm)(d0/d1)

xdftool repack creating errors on disk, read command not able to delve into directories correctly

Creating a new disk with xdftool and verifying it with xdfscan works:

~ xdftool test.adf format "test" ffs + boot install + boot show
BootBlock(0):
 dos_type:  0x444f5301 DOS1 (valid: True) is_ffs=True is_intl=False is_dircache=False
 root_blk:  880 (got 880)
 chksum:    0xe33d0e72 (got) 0xe33d0e72 (calc) -> bootable: True
 valid:     True
 boot_code: 84 bytes
~ xdfscan test.adf
           boot  ok   test.adf  

Repacking a known-good Workbench 3.1 ADF onto the disk results in errors:

~ shasum wb.adf
486ea9520e60051c20ec01329d5a0fe3bb10b4fc  wb.adf
~ xdftool -f test.adf repack wb.adf
~ xdfscan -v test.adf
      E002 boot NOK   test.adf  
@001657:ERROR:File ext block #0 of 'asl.library' (@1656) has invalid parent: got 1572 != expect 1656
@000357:ERROR:File ext block #0 of 'amigaguide.datatype' (@356) has invalid parent: got 355 != expect 356

xdftool can not successfully delve into some directories:

~ xdftool wb.adf read libs/asl.library asl + read classes/datatypes/amigaguide.datatype amg
'read classes/datatypes/amigaguide.datatype amg' FSError: Invalid File Name(9):node=,file_name=classes/datatypes/amigaguide.datatype
~ xdftool test.adf read libs/asl.library asl2 + read classes/datatypes/amigaguide.datatype amg2
'read classes/datatypes/amigaguide.datatype amg2' FSError: Invalid File Name(9):node=,file_name=classes/datatypes/amigaguide.datatype
~ cmp asl asl2

Comparing the datatype files after extracting them using an emulator shows they are also binary identical though. I couldn't tell if the errors were in xdftool alone or also in xdfscan, and weren't able to pin them down. Sorry I can't give more useful info.

xdftool not aware of Latin 1, can't handle all valid AmigaDOS filenames

Using xdftool to work on disks with filenames making use of certain Latin 1 characters will fail:

xdftool new_WB3.1_40.42_install.adf repack WB3.1_40.42_install.adf

'repack WB3.1_40.42_install.adf' FSError: Invalid File Name(9):node=ADFSDir:'Install'(@882),file_name='Portugu?s.info'

Running on Windows error

Error happening at line (when running on Windows with system Python, not by MSYS2):

from musashi import emu

Fatal Python error: PyThreadState_Get: no current thread

Error opening a previously created RDB file

I'm running on OS X 10.11.3:

$ ./rdbtool test.hdf create size=100Mi
$ ./rdbtool test.hdf open + info
Traceback (most recent call last):
  File "./rdbtool", line 665, in <module>
    code = queue.run()
  File "./rdbtool", line 76, in run
    exit_code = CommandQueue.run(self)
  File "/Users/cmsj/hacking/scratch/amitools/amitools/util/CommandQueue.py", line 50, in run
    exit_code = self.run_next(cmd_line, cmd)
  File "./rdbtool", line 160, in run_next
    exit_code = cmd.run(self.blkdev, self.rdisk)
  File "./rdbtool", line 55, in run
    return self.handle_rdisk(rdisk)
  File "./rdbtool", line 236, in handle_rdisk
    lines = rdisk.get_info()
  File "/Users/cmsj/hacking/scratch/amitools/amitools/fs/rdb/RDisk.py", line 84, in get_info
    pd = self.rdb.phy_drv
AttributeError: RDBlock instance has no attribute 'phy_drv'
$ 

Amitool installation actually requires setuptools_scm

While trying to install amitools following the instruction of the README.md I found out that several packages need to be installed for the installation to work successfully when using the "develop --user" install option:
setuptools_scm
pytest-runner

These are not mentioned in the README.md and should probably be added.

Cheers.