Convert llvm-mos ELF objects to ca65 objects in llvm-objcopy about llvm-mos

[asiekierka]

gcc-6502 uses cc65's assembler and linker, so it uses the same object file formats as cc65.

[johnwbyrd]

May no longer be necessary, given https://github.com/rweather/o65utils.

[mysterymath]

EDIT: Whoops, my initial comment on this had confused it for the issue in the other direction. Sorry!

Yeah, that actually seems like a pretty plausible route, since ca65 has a o65->xo65 converter. We'd still want this capability in-tree, and we'd want to make sure o65 is expressive enough for the ELF relocations we do/intend to emit.

[rweather]

FYI, here is the mapping I used from ELF relocations to o65 relocations:

• R_MOS_IMM8 -> unsupported
• R_MOS_ADDR8 -> LOW
• R_MOS_ADDR16 -> WORD
• R_MOS_ADDR16_LO -> LOW
• R_MOS_ADDR16_HI -> HIGH
• R_MOS_PCREL8 -> unsupported
• R_MOS_ADDR24 -> SEGADR
• R_MOS_ADDR24_BANK -> SEG
• R_MOS_ADDR24_SEGMENT -> WORD
• R_MOS_ADDR24_SEGMENT_LO -> LOW
• R_MOS_ADDR24_SEGMENT_HI -> HIGH
• R_MOS_FK_DATA_4 -> unsupported
• R_MOS_FK_DATA_8 -> unsupported
• R_MOS_ADDR_ASCIZ -> unsupported
• R_MOS_IMM16 -> unsupported

My tool only converts executables to o65 at present, not object files. I haven't seen an ELF executable with the unsupported types yet. My assumption is that they have already been fixed up at ELF link time, particularly the IMM8, IMM16, and PCREL8 types. Let me know if I am wrong.

The xo65 object file format is a completely different format from what I can tell, and would probably be better when converting object files from llvm-mos to ca65. The documentation for xo65 is spotty: https://github.com/Rapdorian/xo65

[asiekierka]

Let me know if I am wrong.

You're right, for the most part. There's nothing technically stopping an ELF from using the unsupported types, but:

• R_MOS_IMM8 can be mapped to R_MOS_ADDR8 and R_MOS_IMM16 can be mapped to R_MOS_ADDR16; typically, relocating a symbol via an immediate is equivalent to relocating a symbol via an address. The difference is merely that of semantics/providing additional information to the linker.
• R_MOS_FK_DATA_4 and R_MOS_FK_DATA_8 should not be used in code, but in non-code segments (think DWARF debug data, so supporting it is unnecessary.
• R_MOS_ADDR_ASCIZ was added to generate non-code information (the C64 BASIC header), so not supporting it here should be likewise acceptable.
• R_MOS_PCREL8 is very unlikely to be emitted in even partially-relocated code due to its short range.

[rweather]

IMM8 and IMM16 have been mapped as suggested.

[johnwbyrd]

We'd still want this capability in-tree, and we'd want to make sure o65 is expressive enough for the ELF relocations we do/intend to emit.

Why do we want this capability in-tree? I'm all for it existing in other repos, but I don't yet see why it would need to be married to the project.

[mysterymath]

Being able to use llvm-mos to generate code for an existing ca65 assembly link is a pretty landmark feature if it works well; I'd hope that if that could be made to work at all, it would work out-of-the-box.

[rweather]

Would it be easier to convert o65/xo65 object files into ELF .o's and use llvm-mos as the linker?

[mysterymath]

Actually no, the xo65 object format is more difficult to support in our linker than our ELF files are in theirs. xo65 doesn't exactly have relocation types, instead there's an encoded expression tree containing an arbitrary number of symbols and the math to perform on them.

[johnwbyrd]

I think so. Ulrich specifically refused requests to document the xo65 format, and attempts to reverse engineer it have only partially been successful to date.

From our perspective, it would be more supportable in the long run to invite more file formats INTO our toolchain, rather than trying to export more file formats OUT. I think at least prototyping an xo65toelf tool makes sense; we could test it on a bunch of real -world binaries and see how well it does. (We think that SOME xo65 binaries cannot be rendered in ELF; I'm curious how many actually break that compatibility.)

I considered implementing ca65 support in the assembler, and you'll find some stubs in the code to permit this to happen at some point in the future. But this is a big task. The ca65 macro set is crazy huge.

Again, I am all for building converters to and from various 6502 specific formats; aside from an ease of use argument, I just don't understand why those tools need to occur in this tree.

[asiekierka]

The idea of xo65toelf is, in my opinion, not the right approach.

We had similar discussions in the Game Boy homebrew scene, whose ASM-centric toolchain also uses expression trees in its object format; after bringing this up to said toolchain's maintainer and analyzing the complexity of each solution, we also decided that the best way would be to have the more "powerful" linker support ELFs (or a converter to be made) rather than the less "powerful" linker support complex object files.

However, in that community, the lack of a stability promise for the object format in question was another reason which does not apply here.

[mysterymath]

(We think that SOME xo65 binaries cannot be rendered in ELF; I'm curious how many actually break that compatibility.)

I think they can be reasonably represented in ELF with a suitable extension to our relocation types. Without this, even the prototypical example that I'd like to work, famitone, wouldn't. It was extremely difficult to port famitone2 over to our assembler format, and I'd like to have a solution for using the huge library of NES assembly fragments in the context of a LLVM link.

Again, I am all for building converters to and from various 6502 specific formats; aside from an ease of use argument, I just don't understand why those tools need to occur in this tree.

I know of a few reasons to have them in tree: ease of use, ability to take advantage of superior LLVM ELF handling libraries (with our relocations), and a quite good integration testing setup via llvm-lit. Is there a more compelling argument to keep them out?

[johnwbyrd]

Two arguments. First, the upstreaming argument. There's no way that upstream will accept a bunch of 6502 formats in lld. If you go ahead, be aware that that code will likely need to be removed someday. Second, the draw-the-line argument. The xo65 format isn't the only game in town; there's nesasm and beebasm and o65 and a bunch of other important but specialized per-platform assemblers, in common use today, that probably deserve attention as target formats as well. I don't want to support 5 new output formats in lld. There's your reasons why not, and the question remains: why not experiment with some independent specialized conversion tools, and integrate them into the llvm-mos batteries-included release later? You get the same end result, with none of the downsides.

[mysterymath]

Two arguments. First, the upstreaming argument. There's no way that upstream will accept a bunch of 6502 formats in lld. If you go ahead, be aware that that code will likely need to be removed someday. Second, the draw-the-line argument. The xo65 format isn't the only game in town; there's nesasm and beebasm and o65 and a bunch of other important but specialized per-platform assemblers, in common use today, that probably deserve attention as target formats as well. I don't want to support 5 new output formats in lld. There's your reasons why not, and the question remains: why not experiment with some independent specialized conversion tools, and integrate them into the llvm-mos batteries-included release later? You get the same end result, with none of the downsides.

If it would go anywhere in LLVM, it would be llvm-objcopy; LLD isn't patterned after binutils' arbitrary "binary format descriptor" system, while llvm-objcopy is. AFAICT the original intent was to allow translation between various object file formats, it's just that currently only a handful are implemented.

As to other formats, are there any other 6502 assemblers that produce object files in unique formats that can be linked together by a separate linker? That's all of the scope here; if it's not relocatable, there's nothing we can do with it. To my knowledge xo65, o65, and llvm-mos ELF are the only relocatable object formats for the 6502.

[johnwbyrd]

Seems like a valid question, I'll look into it.

[johnwbyrd]

I'm finding out about some of these projects for the first time, so my understanding may be incomplete. But I've focused solely on assemblers that support the 6502 and seem to have an explicit linking and relocation step.

The wla-dx assembler, https://github.com/vhelin/wla-dx , is an assembler with 504 stars on Github, and what seems like a fairly loyal user base. It has an object file format and a library file format, with some non-trivial support for banking. Docs are at https://wla-dx.readthedocs.io/en/latest/ ; see especially section 12.

The ASxxxx assembler package, https://shop-pdp.net/ashtml/asmlnk.htm, is a cross assembler and linker package which supports the 6502. (Search for 6500 in the docs.) The assembler has the concept of direct page built in. The linker is a relocating linker.

There is Macross, an assembler developed by LucasArts, that has support for linking and relocations. This one may only be of historical interest; it seems that a few news sites have picked it up, but I don't see any active users of it.
https://github.com/Museum-of-Art-and-Digital-Entertainment/macross/blob/master/doc/linkerReleaseNotes

There is gcc-6502 at https://github.com/itszor/gcc-6502-bits . Descending from gcc, it of course has an explicit link step, but I can't find anything out about its relocations or even whether it uses ELF.

The 64tass assembler package, https://tass64.sourceforge.net/ . Generates Atari XEX, Apple II, and CBM output formats among others. See the --nonlinear output file option specifically. Not sure if this qualifies or not.

And since we now support 65816 assembly: there is the Apple IIgs Object Module Format (OMF) versions 1 and 2. See section 1-4 of https://www.goldstarsoftware.com/applesite/Documentation/AppleIIgsProgrammersWorkshopC.PDF . See also the Merlin32 assembler, a modern assembler that emits OMF files.

I'm surprised that Kick Assembler doesn't seem to have an intermediate file format.

I'll keep looking.

[mysterymath]

There's a huge practical documentation gap between our linker and theirs, given cc65's age. There's an ecosystem network effect that cc65 benefits from: it continues to be the primary toolchain recommended to beginners on the platforms I'm aware of, and more of those beginners are likely to improve the cc65 ecosystem of documentation than ours, widening the gap.

My original hope for this is that we would be allow users to slot llvm-mos into their linker ecosystem; that would allow us also to benefit somewhat from their network, and to be an easier recommendation. But, considering the technical aspects of this a bit more; you'd need to set up a llvm-mos linker script to get it to emit sections corresponding to cc65 segments, and that isn't at all trivial to do. We'd need to set up a SDK target for that to be easy; but you'd then lose any of the headers for the target.

As John mentioned, this difficulty would need to be addressed separately for each other ecosystem we'd want to buy into. cc65 is the biggest I know of, but there may be others that are particularly used on other systems. And we wouldn't be able to buy into linker-less ecosystems at all without writing some kind of horrid asm translator, so I think the idea overall just doesn't scale.

So, I'm closing this one out. It doesn't sufficiently help make llvm-mos easier to get started with, and if you don't want an easy time getting started, you might as well use the stuff in the SDK. I would really like to find a way to avoid this project becoming Plan 9, i.e., better, but not enough better, than cc65, but this isn't the way.