matortheeternal / merge-plugins

Standalone application for merging bethesda plugin files.

Description

Merge Plugins Standalone (or just Merge Plugins), is a tool for combining mods from Bethesda games which have .ESP/.ESM files. It uses the xEdit codebase as an API, and can be used with Skyrim, Oblivion, Fallout New Vegas, and Fallout 3. Merge Plugins Standalone is the successor of my Merge Plugins xEdit Script. Feel free to check out the screenshot gallery to get an idea of the look and feel of Merge Plugins.

You can also check out GamerPoet's video tutorial.

Compiling

This project can be compiled with RAD Studio Delphi XE, or newer. You can download compiled builds from the releases section here on GitHub.

Usage

The exectutable will automatically detect load orders for Skyrim, Oblivion, Fallout New Vegas, and Fallout 3. This exectuable should be run through Mod Organizer if you want it to use mods installed with Mod Organizer. There are multiple integration features for working with Mod Organizer, so it is the recommended load order tool for use with this utility.

Setup

1. Unpack Merge Plugins somewhere. E.g. C:\Program Files\Merge Plugins
2. Start Merge Plugins.
3. If you don't see a profile for your game, right-click in the box and choose "New profile", then specify a profile name, game mode, and game path. You can browse for your game path by clicking on the browse icon.
4. If your profile is valid, its background color will turn a shade of light green. You can then select the profile by clicking the game icon (or any other part of the panel).
5. Once you've selected a profile, click "OK".

Configuration

When you first start Merge Plugins you should do some initial configuration. To start, click on the gear icon near the top of the window, this will open the Options Form. Once there, you'll want to:

(1) configure integrations

Go to the Integrations Tab. If you're using Mod Organizer, check the "I'm using Mod Organizer" checkbox. Then click Detect Integrations. If any integrations aren't found, you'll want to install them or browse to find them on your system. Make sure you verify the paths found by Merge Plugins, especially for Mod Organizer if you have multiple installations.

(2) set your asset destination directory

Go to the Merging Tab and click the browse icon by the text box. The folder you choose here will be where merges are placed. If you're using Mod Organizer this should be your Mod Organizer mods folder.

(3) set merging options

Go to the Merging Tab and check the boxes corresponding to the options you want to use. You'll want to use "Extract BSAs" or "Build Merged BSA" if you aren't using Mod Organizer. You can leave everything else as the defaults.

(4) register a username

Go to the General Tab and enter a username in the username text box. Click "Check" to verify the username is available. Once you have a username that's available, click "Register". This username is put on reports submitted to the backend. A username is required if you want to submit reports on plugins you've merged.

Contact

If you're looking for support or want to contribute, join the Modding Tools discord server.

You can view project progress and user stories on the trello board.

Credits

• xEdit: used for interfacing with Bethesda plugin files. xEdit was designed by zilav, hlp, Sharlikran and ElminsterAU
• superobject: used for handling json.
• Abbrevia: used for handling zip archives.
• ZEOSDBO: used for interfacing with MySQL database.

License

Merge Plugins is licensed under the MPL. Please see license.txt for more details.

merge-plugins's Issues

Need to update status panel language after language changed

mpEditForm filename missing .esp should be invalid

Filename should be invalid if it doesn't end with .esp.
When a user enters the edFilename TEdit, .esp should not be highlighted so they can just enter a filename.

Cancelled merge status

Shouldn't let users build a merge if they cancel it after renumbering has started. It seems that reloading the plugins isn't working very well - the resulting plugins have masters issues. It'd be best to not reload plugins until this can be figured out (if it can be). Limitations of the xEdit API kind of suck. 👎

Open Options Form when new profile created

Guide the user to configure their options when they create a new profile.

Type-searching for all listviews

Description

Like windows explorer type to jump between files in a directory.

Implementation details

• If delay between two key press events is greater than a certain value (1 - 2 seconds), clear buffer and add that key to the beginning of the buffer
• If space is pressed as first character of buffer, process it as a "check this checkbox" input
• If we've just cleared the buffer, clear user selection and start searching at beginning of the list
• Only buffer [a-z][A-Z][0-9], space, and some filename-supported symbols
• Each time KeyPress event is detected, jump to next item in list (from selected item) that matches buffer
• If no item found matching buffer between currently selected item and end of list, leave selection on current item and make a beep
• Should be implemented as a decoupled method that takes a ListView and a BufferString

Switching from MO checkbox to NMM checkbox wonks out GUI

Sometimes when changing the selection in the plugins list view the details don't update

Seems fairly random. Really odd. I don't think this existed before so I feel like it might be related to the new Application.OnMessage handler I added for handling the CHM.

Fix TESV_Papyrus_Flags browse filter

Filter is too specific, file doesn't appear for some users. Remove filter entirely?
See the step forum thread.

Not saving profiles

The latest 593 wont load a previous profile on start, even after making a new profile on restart it shows an empty select profile screen.

Background loader hint is coming up when form isn't in foreground

Need to not bring it up if the main form isn't in the foreground

Build button glyph glitching out when switching from disabled to enabled

Copied from Asana.

No idea how to resolve this while maintaining my desired functionality asides making a completely custom control to replace TSpeedButton. Man, the way windows handles two glyphs is terrible. 👎

Disable integration-dependent options when integrations not set up

• Build Merged BSA
• Handle Script Fragments

Exception when trying to detect integrations

Doesn't handle Fallout MCMs

They appear to work via GECK scripts. The GECK scripts often reference their parent file by a hardcoded filename.

As of right now, there is no way to compile GECK scripts outside of the GECK itself, but I can modify the source and guide users to recompile all scripts in the Merged Plugin after merging.

Record merging fails with "Duplicate FormID" on CELL records

After a number of other successful merges I've encountered errors attempting to merge two plugins of a particular Fallout New Vegas mod:
  NVInteriors Core: http://www.nexusmods.com/newvegas/download/1000014216
and
  NVInteriors Combo AWOP: http://www.nexusmods.com/newvegas/download/110976

Log contents here: http://pastebin.com/FTsNQ0iw

I had to snip part of the log to fit in the 512kb limit but the errors are all of the same type, "Duplicate FormID", as shown here:

Exception copying [REFR:0C000B47] (places StreetLitterDark02 [STAT:000401D2] in GRUP Cell Temporary Children of NVInteriorsSouthSewers "South Sewers" [CELL:0C0F7FE6]): Duplicate FormID [020F7FE6] in file [0E] MERGE - NVInteriors Core w-ComboEdition_AWOP.esm

The merge completes with 3347 such errors, and the resulting plugin has 3425 less records than expected (assuming sum of records from both mods).

The only other things in the load order are the core game files and DLCs. I also tried without the DLCs present, with the same result.

I don't know if the FormID starting with "02" is a clue. The "02" mod prefix doesn't correspond to a mod that is involved in the merge but perhaps the formid renumbering is a normal artifact of the merging process..

The only difference I have noticed between these mods and others that have successfully merged is that NVInteriors Combo AWOP has 455 ONAM records, but I don't know if this is relevant.

Please let me know if I can provide any more information.

-profile target line parameter

Grab from smash.

Avoid annoying disk prompt when detecting integrations

Fix exception when trying to update dictionary

Tracker.OnLogEvent not cleaned up from the splash form.

Script fragment remapping to different ordinal but same local formID failing

Source scripts
QF_BQ01_00095125.psc
QF_BQ02_000BD77F.psc
QF_BQ03_000BD78C.psc
QF_BQ04_000CBA9A.psc

Produced scripts
QF_BQ01_03.psc
QF_BQ02_03.psc
QF_BQ03_03.psc
QF_BQ04_03.psc

Expected scripts
QF_BQ01_03095125.psc
QF_BQ02_030BD77F.psc
QF_BQ03_030BD78C.psc
QF_BQ04_030CBA9A.psc

Report
Report submitted by Linthar on the STEP forum thread for merging Investment Price Config.esp and Bounty Gold.esp.

Related issue
Processing of loose scripts reports scripts being renumbered that were renumbered as part of BSA extraction. Logging/skipping logic needs to be fixed.

Access Violation in Merges Popup Menu > Plugins > Check plugins for errors

Need to investigate. Reported by Dreadflopp on the STEP forum thread.

Refactor mpMerge

As per the style from Smash's unit msSmash.

Change TDetailsListEditor to a custom TStringGrid

Will use custom data and draw methods.

Disable ok button if manager integration is invalid

If the user has specified they're using NMM or MO but haven't entered valid paths for the manager and the mods directories on the OptionsForm, disable the OK button.

Overwriting profile with incorrect values on exit

See InsanePlumber's post.

I'm fairly certain this is due to profile loading using the global settings object.

Script fragments won't be altered when master count changes

Copied from Asana.

I haven't verified the game won't find a script fragment if it isn't at a proper local file ordinal (number of masters in the file), but I assume that it probably doesn't work. So, as of right now, renumber conflicting may not lead to functioning mods. This will only happen if a plugin being merged has Script fragments and another mod in the merge has a master that the plugin with script fragments doesn't have.

Copy general assets using game data directory in some cases

With loose/unmanaged ESPs, possibly.

Reported by Equimanthorn on Nexus Mods.

See this log:

robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Vendor Sale Delay - GONE" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm VendorSaleDelay-Gone.seq VendorSaleDelay-Gone.ini /xd VendorSaleDelay-Gone.esp translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Improved Container Names" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm "Improved Container Names.seq" "Improved Container Names.ini" /xd "Improved Container Names.esp" translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\NARC - No Animals Report Crimes" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm NoAnimalsReportCrimes-DG+DB.seq NoAnimalsReportCrimes-DG+DB.ini /xd NoAnimalsReportCrimes-DG+DB.esp translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Farmhouse Chimneys" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm HearthfireChimneys.seq HearthfireChimneys.ini /xd HearthfireChimneys.esp translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Ragdoll Paralysis" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm Paralyze.seq Paralyze.ini /xd Paralyze.esp translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\ClamsDropPearls" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm ClamsDropPearls.seq ClamsDropPearls.ini /xd ClamsDropPearls.esp translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\Data" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm "Beards and Brows.seq" "Beards and Brows.ini" /xd "Beards and Brows.esp" translations "TES5Edit Backups"
robocopy "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\83Willows 101BugsHD Butterfly Dragonfly Luna-Moth Torchbug" "G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks" /e /xf meta.ini *.esp *.esm 83Willows_101BUGS_V4_LowRes.seq 83Willows_101BUGS_V4_LowRes.ini /xd 83Willows_101BUGS_V4_LowRes.esp translations "TES5Edit Backups"

Building merge: Tweaks

Merge is using plugin: Tweaks.esp
Merge data path: G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks\
Adding masters...
Done adding masters

Renumbering Conflicting FormIDs
  Renumbering FormIDs in Moss Rocks Merged.esp
  Renumbering FormIDs in 83Willows_101BUGS_V4_LowRes.esp
  Renumbering FormIDs in Beards and Brows.esp
  Renumbering FormIDs in ClamsDropPearls.esp
  Renumbering FormIDs in Paralyze.esp
  Renumbering FormIDs in HearthfireChimneys.esp
  Renumbering FormIDs in NoAnimalsReportCrimes-DG+DB.esp
  Renumbering FormIDs in Improved Container Names.esp
  Renumbering FormIDs in VendorSaleDelay-Gone.esp

Copying records
  Copying records from VendorSaleDelay-Gone.esp
  Copying records from Improved Container Names.esp
  Copying records from NoAnimalsReportCrimes-DG+DB.esp
  Copying records from HearthfireChimneys.esp
  Copying records from Paralyze.esp
  Copying records from ClamsDropPearls.esp
  Copying records from Beards and Brows.esp
  Copying records from 83Willows_101BUGS_V4_LowRes.esp
  Copying records from Moss Rocks Merged.esp

Handling assets
  Handling assets for VendorSaleDelay-Gone.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Vendor Sale Delay - GONE
  Handling assets for Improved Container Names.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Improved Container Names
  Handling assets for NoAnimalsReportCrimes-DG+DB.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\NARC - No Animals Report Crimes
  Handling assets for HearthfireChimneys.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Farmhouse Chimneys
  Handling assets for Paralyze.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Ragdoll Paralysis
  Handling assets for ClamsDropPearls.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\ClamsDropPearls
  Handling assets for Beards and Brows.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\Data
  Handling assets for 83Willows_101BUGS_V4_LowRes.esp
    Copying general assets from G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\83Willows 101BugsHD Butterfly Dragonfly Luna-Moth Torchbug
  Handling assets for Moss Rocks Merged.esp

Removing unncessary masters
  Removing master HearthfireChimneys.esp
  Removing master ClamsDropPearls.esp
  Removing master Beards and Brows.esp
  Removing master 83Willows_101BUGS_V4_LowRes.esp
  Removing master Moss Rocks Merged.esp

Discarding changes to source plugins
  Reloading Moss Rocks Merged.esp from disk
  Reloading 83Willows_101BUGS_V4_LowRes.esp from disk
  Reloading Beards and Brows.esp from disk
  Reloading ClamsDropPearls.esp from disk
  Reloading Paralyze.esp from disk
  Reloading HearthfireChimneys.esp from disk
  Reloading NoAnimalsReportCrimes-DG+DB.esp from disk
  Reloading Improved Container Names.esp from disk
  Reloading VendorSaleDelay-Gone.esp from disk

Saving: G:\Steam\steamapps\common\Skyrim\ModOrganizer\mods\Tweaks\Tweaks.esp
Done merging Tweaks (440.631s)


All done!

-offline target line parameter

Prevents application from connecting to the internet.

Not saving merges if background loader not complete

Possibly due to RenameSavedPlugins being called when the background loader is not yet complete, causing some kind of exception between wbFileForceClosed and the background loader to cause the save thread to terminate early, thus missing the opportunity to save merges.

Fix by:

• Saving merges before renaming saved plugins
• Using exception handling in RenameSavedPlugins per-plugin in the loop
• Only calling RenameSavedPlugins if the background loader is done (user can't fix errors until the background loader is done anyways)

Handle plugins that have merged plugins as masters during load

Plugins that rely on a merged plugin as a master currently cause loading to fail with an initialization exception.

Possible solutions

• Allow loading of merged plugins (not preferred)
• Don't throw an initialization exception when a plugin fails to load
• Don't throw an initialization exception if a plugin fails to load because it had a merged plugin as a master
• Don't throw an initialization exception for any plugin that fails to load because it was dependent (directly or indirectly) on a merged plugin as master (so like the previous item, but allowing for chain dependencies to fail to load as well)
• Find a way to pre-process plugin file headers, so users can cleanly specify what plugins they want to load from a plugin selection screen (with respect to masters). (which would also involve allowing loading of merged plugins as an option)

Why I don't want to load merged plugins
Building a merge into an existing file is a bad idea, and something I've been trying to avoid for several reasons.

• To safely build a merge into an existing file I have to strip all data (records, masters, etc.) out of that file. Doing this makes it basically equivalent to merging into a new file.
• Existing merged plugins could be at arbitrary load order positions, which can interfere with the process of merging
• Users might try to add merged plugins (that they are managing) to other merges, which causes issues with merge priority and build order, and generally greatly complexifies everything around merging and the application. It's also completely unnecessary (just make a single merge of those files...)
• Merged plugins take up load order slots.
• Renaming a merged plugin becomes less reasonable if you're loading it into the program (how will the program delete the original file if it is using it itself?)
• Would require save-buffering as is done with fixing errors, which is annoying

Not throwing initialization exceptions
This is a better option, but not without its flaws. One issue with this approach is it seems to hide the fact that things may not have happened according to the user's expectations during initialization/plugin loading. The user might notice this, or they might not. We don't want to rely on that. We could only stop throwing exceptions for this if we clearly and directly inform the user that one or more plugins failed to load. (this seems like a good idea)

Pre-processing file headers
This is an interesting idea, and something that will probably be valuable for other projects in the near future. I'm not sure how it would be done, but it's worth looking into.

Better plugin loading

Don't allow a user to proceed from the plugin selection form if their plugin selection is invalid.

Unhandled access violation when building a merge

See scythiss's post

Hide balloon hints when user brings up a child form

Merge Form doesn't resize components properly when resizing from top/left

Not sure why this is happening. Might be related to intercepting certain messages.

Create plugin selection form

Display form prior to the splash form from which the user can select what plugins they want to load.

Empty script fragment filenames not being caught

Somehow empty script fragment filenames are getting through the script fragment code, and then (of course) failing to copy. We should handle empty script fragment filenames.

Spaces after merge name or merge filename cause failure

Need to trim both of them before saving them to a merge from the mpEditForm.

Trailing backslashes on paths in settings window

If a path is entered without a trailing backslash, the program should add it automatically.

See this post on AFKMods from Uhuru N'Uru

(was also reported by other users prior to this report)

UTF8 String Support

Update xEdit API to support UTF8 (via UTF8 patch?)

See lotress's post on Nexus Mods.

GUI element widths need to be increased for language support

Overly verbose "Failed to copy" messages when copying script fragments

See scythiss's post.

Should be able to resolve by keeping track of which script fragments are handled (so we don't try to handle a script fragment twice, from BSA and loose).

Report on merge incorrect report path

"Unable to save report user\reports\submitted<report_name>. Unable to create directory."

Bug reported by xgatt. See the post on Nexus Mods.

Support translation of list view column captions

Documentation framework

Requirements

• Searchable
• Can jump to links
• Can display images
• Easily maintainable

Approach

So the approach to this is really uncertain.

• We could make our own document browser... which would be absolutely insane.
• We could try to integrate a CHM file, which Delphi seems to already have some stuff built for.
• We could just go the route of a web page which could be locally stored on the user's machine and then opened either inside Delphi (through some kind of pre-existing HTML document display component) or in the user's internet browser.
• We could see if there's a way to open PDFs linking to certain sections/pages

Assessment

I think the best choice is a web page, because I want to make the documentation available on the web anyways, and that also allows us to easily try the CHM file methodology. The real question is whether or not there's an easy way to convert my already-existing ODT documentation to a web page without too much trouble, or some way of automatically digesting and processing the documentation from one format to the other.

Non-ascii characters in path causes building merges to freeze

Possibly related to #25

Access violation on startup

Due to changelog code being executed when changelog is not yet loaded. Line 94 of mpChangeLogForm.pas.

Path or filename truncation issue

Reported by dreadflopp on the STEP forum thread.

Invalid username, when username registered

On the General settings page, the username will be reported as invalid in the status label after the user has been registered.

Finish up documentation

Need to complete/fix documentation for the following sections:

1. Modified Main Application Window (Check for Errors button addition)
2. Help button now functional in QuickBar
3. Modified merges tab (plugins not loaded status grey)
4. Modified Options Window (and options window screenshots)
5. Links to general reference sections where appropriate/applicable
6. Nexus Mod Manager integration tutorial
7. Updating tutorial
8. Submitting Reports tutorial
9. Debugging tutorial
10. Arguments
11. More information on renumbering/copying modes
12. Plugin selection form
13. Changelog form
14. Reorder images to come after headerS so section links don't skip them
15. Limitations general reference section
16. Update the CHM documentation with all changes in ODT/PDF documentation.

Verify file access permissions

Button on the merging tab of the options form which verifies Merge Plugins has full read/write access on the user's filesystem.

GetFilesList malformed if statement applying file filters to directories in addition to directory filters

Issue on lines 574-577 of mteHelpers.pas, when a file is a directory and passes through the directory filter, it has to pass the file filter as well, else it will be skipped. This is incorrect functionality.

See this reddit comment from tempest420. Thanks tempest!