miss-inputs / romniscience Goto Github PK

http://www.smspower.org/Development/ROMHeader

Probably works for SG-1000 as well

So, with the inclusion of CD stuff, we've noticed a problem.

It is in fact entirely possible that a given hash may in fact refer to two different things. Currently, ROMniscience just makes the assumption that the first hash it finds is correct.

Mainly seen with audio tracks, but it's possible that there are hash collisions anywhere else.

For example, 3B-C6-D1-19-C8-83-35-04-31-6E-B4-37-82-DC-1B-2B refers to not only an audio track on Pac-Man World: 20th Anniversary (USA), but also Pac-Man World: 20th Anniversary (France), and also something called Euro Demo 54.

The question is, what do we really do about that? It'll probably have a nasty performance hit to keep going through every datfile after a match is found to find more matches. As far as displaying it goes I guess the first match would show up in "Datfile game name / Datfile ROM name" / etc and have more columns "Datfile ROM name 2" and whatnot.

Well, right now it's because I skip over that code that would otherwise insert an image cell, but if I didn't do that, it would proceed to not display the image and screws up the cells to the right of where the image would be.

This is probably an upstream Mono bug but I'll leave it here for now until I get around to looking more into it.

Here's where it might get interesting.

For unknown file types, while we have let the user choose any handler to try and read it, maybe they don't know what it is.

Anyway, we'd have to add a thing to the Handler base class (providing a useless virtual method to make it optional for classes to override), and I guess it just takes a Stream or ROMFile and returns a bool, true if this could be handled by this handler (from looking at magic numbers, etc) or false if not.

Or perhaps it should return an enum with 4 possible values of it is a valid file, it might be a valid file, it probably isn't a valid file, and it definitely isn't? Or maybe that's too complicated.

I'm beginning to realise I really just gotta do this. It'll solve all the things. It'll probably solve #20. It'll make it usable on high-DPI screens.

Also by having multiple frontends by way of multiple build targets, we can probably have a CLI frontend too, which will solve #6 and #10. This should help with #9 as well.

Since the only reason we're hanging on to WinForms is portability, this also gives us the option to use WPF for Windows. That doesn't have to happen right away, as technically WinForms does work under normal Windows, but it becomes an option once we do the thing.

I'm not sure how this all works on Mac OS X, and I may have to enlist the help of someone else to ftest stuff, since I don't exactly have the hardware to run modern Mac OS X. Not sure if there's other platforms Mono runs on either. Until I can find out though, I'd better keep WinForms around.

Also I've just wanted to try GTK#.

Include a command line switch to input the information from the view individual file functionality to the command line instead of a dialog box. I'd want to make it a command line switch for that so that I don't break dragging and dropping a file onto the executable to show a GUI.

I guess one might also want to extract the images via command line too.

If it's like FILE "blah.iso" BINARY, it'll add that ISO twice, once normally and twice as referred to from the cue sheet.

It's not harmful, I guess, but might want to add a check to ROMScanner.processNormalFile to not add files that are already processed somehow.

You could argue that it's because they are improper themselves, but anyway, according to file(1) where most cartridges would have the Nintendo logo, they instead have 0xC8604FE201708FE2; according to GBATEK that's just the ARM7 entry point in PassMe programs so it would be better to look for "DSBooter" at 0xa0 and "PASS" at 0xac; do that at the start of the DS handler because everything else is basically invalid

To be precise if there's stuff that contains \n, like DS banner titles, that will mess up the export when the platform newline character is just \n and not \r\n, because I don't quote the fields. I should quote the fields.

Bleh....

Alright me, I know you don't want to do this. But in the interests of UX, we must do this.

Other than switching to Qt or reimplementing an actually good layout manager in WinForms somehow or something, it's gonna have to be done.

So, here's my proposal.

Right now, we have what attempts to be a normal form layout for now. Name of the thing, file path text box, and browse button all in one line, squished in between one another.

This would work for anything that isn't WinForms. However, since you insist on using WinForms for cross-platforminess (which is kind of ironic the more I think about it), it won't work.

Here's what we'll do instead:

------ (yeah, one of those good ol' horizontal bars, or maybe something that just visually separates each thing)
Name of handler
File path text box + button
[x] Skip header when calculating hashes
[x] Some other option to do with this particular handler (for example, you could skip datfile identification for specific handlers if they take too long because they're so big and/or you're comfortable with your dumps being valid)
[Anything we want really! We can add all sorts of functionality to the settings dialog this way]
----

Much easier to implement and also scale correctly and whatnot, plus makes it easier to add new things.

It'll take more vertical space, yes, but we're using a lot of that anyway. Should be the most readable thing that we can have in this situation, I think.

Because I wanted to clean up the TODO comments and use those more for minor cleanups and whatnot, rather than major feature additions which I feel make more sense as issues.

This is a bit low priority I guess, since I'm not even sure what uses the old text-based format instead of XML, but they're floating around out there.

A ROM should show how big the cartridge was for the system including banana for scale

http://wiki.nesdev.com/w/index.php/Family_Computer_Disk_System#.FDS_format

Sounds good do that

This will be tricky because I only really have homebrew stuff at the moment to experiment with, but anyway, since the documentation exists:

https://www.3dbrew.org/wiki/NCSD (this one is for .3ds dumps of normal carts)
https://3dbrew.org/wiki/3DSX_Format (.3dsx homebrew)

I wouldn't worry about the other formats like SMDH personally.

This is yet another issue I'm raising for myself where I had an idea about how to improve the code to be more flexible and extensible and this is a use case for such an idea.

So, if Wii homebrew games are going to be a thing... you're probably not going to get much out of just the .dol files. But they're all designed for the Homebrew Channel, so you have a meta.xml and an icon.png, which contain the things.

But how do we get at them, when the ROMFile passed to the handler only represents boot.dol? Well, because I like thinking too far ahead and planning the implementation before I even start on the concept, here's what I'm thinking:

Put a method in ROMFile: Stream getSiblingFile(string filename)
Returns a file with that name in the same folder as this file. Maybe validate against absolute paths or anything involving . or .. so we don't end up with any stupid security flaws (I'm not sure what the impact would be, given it'd be only called internally, but I just don't like the idea of blindly grabbing any relative path without making sure we're using things how we expect to be using them). Also, we're trying to get rid of InputStream and use StreamExtensions instead, so that cleanup might have to be done first.

But also if we're in an archive, get a file with the same name that's on the same level inside this archive. Remember that TODO comment where I wanted to refactor ROMFile so that it'd have a different class which is derived from some other base class for archived files and whatnot? That's probably a good idea to do first.

I'm not sure who archives their Wii homebrew folders since neither Homebrew Channel or Dolphin work that way, but it'd come in handy for other situations where we need another file that goes next to a given ROM.

There's some things in here that's picked up by file(1) but not ROMniscience:

ARM9 offset being less than 0x4000 implies homebrew, since that's not actually supposed to happen
Secure area ID (first 8 bytes) being zeroed out implies multiboot
Secure area ID being 0xE7FFDEFFE7FFDEFF means decrypted (as the BIOS overwrites this once verified)
0x1000 being zeroed out apparently indicates the ROM is encrypted, however this should be noted: https://twitter.com/Myriachan/status/964580936561000448 (but perhaps we can do something with that anyway)

While I'm here and reading GBATEK, ARM9 offset >= 0x8000 implies no secure area, and maybe I could add that as an info item

For systems where we're calculating the checksum for validation anyway, include an option to recalculate the checksum and put that back in the stored checksum in the ROM, so it's now valid if it wasn't before, etc.

I dunno, stuff. That should be enough info for me talking to myself via Github issues.

A continuation of #13 but I'm setting it aside for after 0.5.

This will probably require extending chooseChoices so I can add an additional button that gets placed on the generated form, and that button brings you to the list of all handlers to force.

Alternatively, a checkbox that toggles between showing handlers that handle the given file extension, or all handlers.

Either way I'm not getting out of modifying chooseChoices.

Sorry, me! You don't get out of touching that settings dialog forever. Not while I care about UX.

When a handler is disabled, the other items in that part of the dialog regarding that handler should also be disabled, e.g. the text box and calculate hashes checkbox; and of course should re-enable when you click the enabled button.

This is a weird one, because we can tell that the hashes are being calculated correctly, for example, Aquaman: Battle for Atlantis (USA) comes up with 55-2F-5D-D3-8C-E5-FE-18-9F-69-0B-E6-53-47-30-48 as expected. By formatting that properly (i.e. removing the dashes) and searching for it manually in the Gamecube datfile from Redump, it can be seen that this is indeed the correct checksum; however ROMniscience does not display it as being identified.

meow

hi

Will this project compile in visual studio 2019 and what nuget packages do I need?

Thx

At the very least do this when you have an extension that isn't recognized by any other handlers, since right now it just does nothing in that scenario. If we can figure out an intuitive and user friendly way to do that when we already know what handler to use but the user wants to force another one for some reason, then yeah, do that as well I guess.

So, we have generic CD handling. That's good. Yay!

Now that we've started adding weird computers and whatnot to the thing, we also have.. floppy disks. And tapes. And I mean like, MAME handles those generically, or something like that. Kinda makes sense, rather than implementing the same thing for every thingajiggy that uses a floppy disk or tape.

Well, I'm not feeling particularly good-at-wording, but you're me, so you'll understand what I'm on about. I hope.

Oh yeah, some formats are still specific to one system, e.g. .2mg still only makes sense for Apple IIgs, so keep that in mind I guess.

Perhaps we could also do .wav files for tapes and whatnot? Can System.Media.SoundPlayer be useful here?

On Windows at least. Maybe that's just how WinForms is, I dunno, but it bugs me and I feel like I should put something in there to cut it off so it at least displays something, or something

Hmm... maybe it could be an option.

My line of thinking is that the header sorta has to be there for the ROM to be useful (headerless ROMs are only much use if whatever's using them knows about the exact ROM already identified by checksum), so they sorta reasonably have to be there. No-Intro thinks they shouldn't be there though, just because they're a header.

Perhaps I could do this for Atari 7800 too. Maaaaybe SNES?

You're gonna forget it even exists otherwise.

I'm still not sure if it should be off by default or not. Hmm...

The secret real reason I'm giving myself this issue is so I can force myself to revamp the settings system, because we'll need to do that. The issue itself is self-explanatory.

But I'm thinking we'll need to do something like this:

In Handler.cs, have a virtual property customSettings that returns IDictionary<string, SettingType> where the key is the name and value is an enum that represents the type of the setting, or perhaps if I can just use classes I'll do that.

So for DS it'll return {"display invalid banner versions", bool} or something like that.

Then for SettingsDialog.cs in FolderEditor (let's change the name of that class btw) we'll need to make new form elements appear underneath the folder text box according to the handler's customSettings property, which you pass in to the FolderEditor constructor. Append those to settingsToSave accordingly.

I guess as far as actually storing the values goes, you'd have things like DS_display invalid banner versions as the key. That just means you won't be able to use "enabled" as a custom setting without things getting screwy, but why would you?

Currently, ROMniscience only allows one folder for one handler. This is just because I store my ROMs like that (one system and one folder), and I figure most people do that too. But what if they don't? Especially when you have systems that you might want to group together, like Wii and WiiWare, which I don't put in the same folder but some people might, or maybe Megadrive and Mega CD once I get the latter working, or... whatever. I don't judge. And one of my software philosophies is that I don't judge the user on what weird shit they want to do.

Anyway, this shouldn't require too much thought on the backend, just as long as the settings file is backwards compatible (who knows, maybe someone who isn't me uses this program and I just don't know that yet). Configuring it in the GUI in a user-friendly way is the tricky part. As a last resort I guess we just have folders separated by | or ;, but that's ugly and makes the behaviour of the "choose folder" button non-obvious and confusing.

Annoyingly, Dolphin doesn't really have documentation on this, so all you'd have to go by is the source: https://github.com/dolphin-emu/dolphin/blob/master/Source/Core/DiscIO/CompressedBlob.cpp
https://github.com/dolphin-emu/dolphin/blob/master/Source/Core/DiscIO/CompressedBlob.h

Alright, time to actually give myself issues for the things that I say I'm going to get around to doing one day.

For this one:

• Convert .smd to .md (or whatever we want to call the Megadrive ROM extension for plain dumps)
• Convert .v64 and .n64 to .z64
• Remove copier headers from SNES ROMs
• Anything else I forgot at this time, but those three are the most "important" I guess

Use the detected format instead of the filename extension, because extensions are often wrong with this sort of thing.

It should create a copy of the file, leave the original untouched. If it involves the original filename being wrong, like say we have to remove copier headers from something.sfc when something.sfc is actually a Super Wild Card file, normally we would output something.sfc but in order to avoid messing with anything output something.sfc.sfc or something (2).sfc instead (undecided, but the point is, avoid conflicts)

This would just go in a menu called "Manage" or something, and it would operate on multiple selected ROMs, and if any of them have errors it would collect all those errors for one error message box.
For the individual file view, have a button down the bottom perhaps.

Should also be accessible from the command line, speaking of, there's my next issue for myself...

Have "Open" in file menu to just open stuff that way, but also "Open selected" to view the currently selected file (because the view individual file thing contains different information compared to the table)

I hope I can still understand what I just wrote when I come back to this one in the future because I can't right now

We don't need to view every single file in a cue sheet. Definitely not the audio tracks. Just refer to the relevant track, i.e. 1 in most cases, 2 for PC Engine CD, etc..

This will definitely make the UX design a lot easier...

Shrek Superslam (USA) and Wind Waker (PAL) load the icon fine, but throw an ArgumentOutOfRangeException trying to decode the banner text.

Some others aren't able to load the icon at all, and throw an IndexOutOfRangeException trying to do so:

Metal Gear Solid: The Twin Snakes (USA) (both discs)
Super Smash Bros Melee (PAL), also 20XX 4.05
Pokemon Colosseum (PAL)
Sonic Heroes (USA)
Mega Man X Collection (USA)
Star Fox Adventures (USA) rev 1
F-Zero GX (USA)
Beyond Good & Evil (USA)
Pokemon XD (PAL)
Baten Kaitos (USA) disc 2 (disc 1 is fine, which is very odd)
Eternal Darkness: Sanity's Requiem (USA)
Pokemon Colosseum Bonus Disc (Japan)
Pokemon Channel (Australia)

I can't drag stuff onto the .exe even though Mono exists, so that kinda bugs me, and I guess that's just Thunar's fault, but still.

It would also mean I could have ROMniscience available as an "open with" option and other integrationy things, I guess. Probably?

If it's a shell script I could also put it in /usr/local/bin.

It's just more convenient at the moment.

I'm tired.

Likewise with the other ROM management ideas, this would be accessible from a menu and operate on several ROMs at once and be accessible from the command line, etc.

Anyway, this should take the datfile match if any, and rename the ROM to what it says, except with these options:

• Don't change "The" to ", The"; I'll have to change it back
• Don't rename anything starting with [BIOS] to the wrong extension, just leave that extension alone
• Remove cruft like (SGB Enhanced) or (GB Compatible)
  It's already known how I feel about those, but maybe some people like them (I'm sure the No-Intro people do). Hence option.

Naturally avoid overwriting any existing file with the new name, but maybe it could be useful to at least inform the user after a batch rename if that ended up happening.

Add a generic way to handle disc images, abstracting the difference between .iso (2048 byte sectors) with .bin/.cue (with two files of course, and also 2352 bytes per sector because of 16 control bytes at the front which I don't think I need to read, and 288 error correction bytes at the end that I also don't think I need to read; although for datfile purposes I'll just read the whole thing first, it would only matter for any headers inside).

A good start would be Mega CD, since I already have the Megadrive handler and it'd be the same format, just with the need to take sector alignment into account with .bin/.cue as it ends up starting at 0x110 instead of 0x100.

Possibly add other formats like .chd if I can?

For DVD-based formats, there's no weird difference in sectors; as with GameCube/Wii discs. Presumably Dreamcast as well, though those just confuse me.

Well, this would be accessible from the menu in the main window, but also the view individual file dialog of course.

This could be pretty useful, being able to see a full directory structure and also extract files from it (it'd pop up in a separate window with "Extract all" and "Extract selected" options). Also, to finish implementing the PSP handler so it can read UMD dumps, we'll need to implement ISO9660 more or less completely anyway.

Gamecube's filesystem would be a decent place to start since I've half-implemented it already to find the banner anyway, I can't remember if Wii uses the same thing or if it uses a slightly different filesystem implementation? And then DS, and ISO9660 (since PSP as mentioned uses that, and many CD based systems like Mega CD (IIRC) as well). Anyway yeah, stuff and things.

First of all, get rid of Tuple anywhere it's used, it's ugly and I hate it.

Secondly, Handler is fine and all, but instead of just adding to ROMInfo it should do something like add a BlahROM() which is a subclass of ROM(), or something like that. I'll figure it out as I go along, but the way it is now with this ROMInfo.info dictionary is definitely ugly.