pgaskin / kobopatch

An improved patching system for Kobo eReaders.

kobopatch

An improved patching system for Kobo eReaders. See https://www.mobileread.com/forums/showthread.php?t=297338. Download patches for v4.9.11311+ here.

Features

• Zlib replacement.
• Add additional files.
• Add additional symlinks.
• Translation file support.
• Simplified BLX instruction replacement.
• Multi-version configuration file.
• Extensible patch file.
• Built-in generation of Kobo update files.
• Additional instructions.
• Single executable.
• Automated testing of patches.
• Comprehensive log file and error messages.
• Modular and embeddable.
• Structured patch file format.
• Backwards-compatible with old patch format.

Usage

Usage: kobopatch [OPTIONS] [CONFIG_FILE]

Options:
  -f, --firmware string   firmware file to be used (can also use a testdata tarball from kobopatch-patches)
  -h, --help              show this help text
  -t, --run-tests         test all patches (instead of running kobopatch)

If CONFIG_FILE is not specified, kobopatch will use ./kobopatch.yaml.

cssextract extracts zlib-compressed from a binary file
Usage: cssextract BINARY_FILE

symdump dumps symbol addresses from an ARMv6+ 32-bit ELF executable
Usage: symdump BINARY_FILE

Usage: kobopatch-apply [OPTIONS]

Options:
  -h, --help                  show this help text
  -i, --input string          the file to patch (required)
  -o, --output string         the file to write the patched output to (will be overwritten if exists) (required)
  -p, --patch-file string     the file containing the patches (required)
  -f, --patch-format string   the patch format (one of: kobopatch,patch32lsb) (default "kobopatch")
  -v, --verbose               show verbose output from patchlib

kobopatch's Issues

Improve CSS patching heuristics

The CSS minifier and detector have a lot of room for improvement in accuracy, efficiency, and speed.

For the detector:

• It is slower than it needs to be.
• It has false-positives for things like SVG documents (fixing this will change the output of cssextract, but it's only intended to be used for extracting CSS in the first place).

For the minifier:

• It is quite slow.
• It has quite a few cases where it will replace things incorrectly (i.e. : inside a string) (this bug hasn't ever been hit, but it might as well be fixed).
• It doesn't do as much as it could. For example, it could:
  • Remove comments.
  • Replace whitespace in more places.
  • Collapse unnecessary repeated whitespace.
  • Spaces around relativity selectors.
  • Etc...
  • Note: Improving this may break some existing patches which depend on replacing previously minified CSS. I'll fix those patches manually if necessary, and do what I can to preserve backwards compatibility where possible (note that this will be the first truly backwards-incompatible change I've ever made in kobopatch).

kobopatch.yaml overrides with user-provided strings

Hi there! Thanks for making the kobopatch system, it's very useful. When converting my existing set of enabled patches to the override yaml file, I noticed I'm not able (or I just don't know the correct way) to include customizable user strings for patches that use them, so I can just have one master override file rather than edit the individual patch file to include my strings.

Example kobopatch.yaml with the monospace font patch:

overrides:
  src/librmsdk.so.1.0.0.yaml:
    Default ePub monospace font (Courier):
      - Enabled: yes
      - FindReplaceString: { Find: 'CourierStd.otf', Replace: 'normal/Hack' }
      - FindReplaceString: { Find: 'CourierStd-Bold.otf', Replace: 'bold/Hack' }
      - FindReplaceString: { Find: 'CourierStd-Oblique.otf', Replace: 'italic/Hack' }
      - FindReplaceString: { Find: 'CourierStd-BoldOblique.otf', Replace: 'bolditalic/Hack' }

When running to the tool I get the following error:

kobopatch v0.10.4
https://github.com/geek1011/kobopatch

Loading configuration from kobopatch.yaml
Error: could not load config file: error reading kobopatch.yaml: yaml: unmarshal errors:
  line 42: cannot unmarshal !!seq into bool

If this isn't supported behavior, please consider this a feature request!

[Feature Request] Support add folders from kobopatch.yaml

At the moment, kobopatch.yaml support files to transfer files to Kobo devices. I'd like to have folders support to transfer folders as well. That would be helpful in case I want to keep apps such as Plato and KoReader updated in sync with Kobo devices.

Example:

folders:

	src/plato: mnt/onboard/.add/plato
	src/koreader: mnt/onboard/.add/koreader

Add note about using spaces for indentation to parsing error

Zlib support

• Figure out issues with headers (see below)
• Test zlib extraction and replacement
• Implement zlib in patchlib
• Decide on zlib replacement instructions
• Test with existing patches
• Figure out cgo cross-compiling (probably something like what I did with seriesmeta), biggest issue will be for macOS (will need to compile osxcross)

BLX validation when testing patches

Pretty much all *BLX instructions need to point to a valid symbol. Checking this should reduce accidental segfaults due to typos.

This would require resolving the relocations for the PLT.

Improve error messages

Part of #22. Here's the list of improvements which are being made to error messages:

• Patches will be applied in a consistent order to make errors easier to fix.
• All instructions will add their name to error messages.
• All error messages will include line numbers.
• Error checking will be done in stages and will give an error as soon as possible during parsing.
• Multiple patches in a PatchGroup will mention the other patch.

Refactor kobopatch parsing

I am planning to rewrite the kobopatch parsing with go-yaml v3 using the new yaml.Node to allow more control over the parsing and give more useful error messages. This will also allow the code to be organized more cleanly instead of being a huge switch statement. This will be completely backwards-compatible except for the error message format, which will give line numbers as well as instruction numbers.

suggestion: rename kobopatch.yaml to kobopatch-example.yaml in releases?

Having the option to use a global kobopatch.yaml (with overrides for the usually applied patches) over several releases is - at least for me - one of the biggest advantages of kobopatch over the old patch32lsb approach to patching. As long as a patch is still available for a later firmware version, patching is just a mouse-click away without having to go through several patch files and manually changing 'no' to 'yes'.

Would it make sense to rename the kobopatch.yaml file in the release zip files to something like kobopatch-template/example/default.yaml to:

1. make it clear that the customized kobopatch.yaml can be re-applied for patching a newer firmware version
2. prevent the customized kobopatch.yaml from being accidentally overwritten by updating kobopatcher

Tool to convert kobopatch zlib patches to patch32lsb ones

Make tool to automatically check whether all patches apply successfully

Show message box for errors on Windows

MR:post-4101498

I will call the win32 MessageBoxW function via syscall.NewLazyDLL to use the native dialogs and keep the binary size small.

Support more than one PatchGroup per patch

Fix nickel patches

There is an inconsistency between some nickel patches. I'll look into it a bit more later. The other patches are fine though.

Tool to apply patch file to binary

Will have the following arguments:

• -i, --input: input file to patch
• -p, --patch-file: patch file
• -o, --output: output file
• -f, --patch-format: patch format
• -d, --diff: show hex diff (maybe)

cc @geoffreynolds

Add possibility to include other "package" in KoboRoot.tgz

Hello,

It may be good to have possibility to merge other KoboRoot.tgz packages in the output one.
for example :

• add a packages folder on the same level than src
• put all packages tgz in this folder (for exemple kfmon.tgz, nightmode.tgz, ...)
• list in kobopatch.yaml packages names
• kobopatch will merge packages in order of the list.

kobopatch.yaml example:

overrides:
src/nickel.yaml:
merges:
packages/kfmon.tgz
packages/nightmode.tgz

Refactor unit tests

The unit tests work, but are a mess. I'm planning to make them more modular and readable by splitting them up more and using testing.T.Run for subtests. I'm also probably going to add regression testing based on the current patches in kobopatch-patches.

Negative offsets

The keyboard patches do not work. This is because the offsets need to support negative numbers, and they are supposed to be an int32.

Convert patch32lsb to kobopatch format

This could probably be done by taking the generated struct and converting it to a kobopatch one, then marshaling it as yaml.

Custom header/footer captions font-family not applied to Custom header/footer page number text

Selecting an alternate font to Avenir in "Custom header/footer captions" (nickel.yaml) is not applied to "Custom header/footer page number text" (libnickel.so.1.0.0.yaml). Unfortunately, there is no option to define the font-family for the latter.

Unless this is intentional, a header and footer with 2 differing fonts displays.

Allow copying files to more than one location in KoboRoot.tgz

An additional file should be able to be copied to more than one location in KoboRoot.tgz. For backwards compatibility, files should accept either a string or a list.

Will probably need to use something from go-yaml/yaml#100.

Support including whole directories

The files feature is nice, but it would be even better if we could include everything below a directory, too. Something like

  /path/to/foobar/: mnt/onboard/fonts/

would copy everything from the directory foobar/ into the fonts directory.

Thanks for you consideration.

Log checksums for patched files

This will make it easier to test major changes.

Support loading extra yaml config files

Hi Patrick,
thanks for the many updates and the added files section, I am adding a few issues of feature requests that might be useful.

The first is about loading additional yaml files. ATM we need to copy the overrides section, and the files section for each release. It would be nice if one could say

kobopatch -c ../overrides.yaml -c ../files.yaml

to add two yaml files. They can contain whatever yaml code there is, and it is merged into the main configuration overriding previously set values if already set.

If this is too complicated, loading automatically without command line args only one additional config file like ../local.yaml would be an option, too. It would be in .. to be outside the current directory and can be reused for new versions.

Thanks for consideration

Switch to go modules

Off-by-one symbol offsets

The symbol offsets are all +1 due to not accounting for "For the purposes of relocation the value used shall be the address of the instruction (st_value &~1)." (https://static.docs.arm.com/ihi0044/g/aaelf32.pdf).

Fixing this will be a breaking change, so I'll be deprecating the *Symbol instructions and replacing them with *Sym ones (which will have the bug fixed).

Command to convert the Enabled instruction to kobopatch.yaml overrides

Error when using ReplaceString with unicode character

Instead of "_" (\xF5) I would like to use "»" (\xBB) to replace "\x20" in the libnickel patch "Allow searches on Extra dictionaries". This works with patch32lsb or if use a hexeditor (where I can even go up to \xFF). Unfortunately, kobopatch seems to allow only ASCII characters, the last printable character that works is "~" (\x7E) and the very last is the control character \x7F. Starting from \x80, kobopatch shows the following error in log.txt:

applying patchAllow searches on Extra dictionaries looping over instructions skipping non-instruction Enabled(), PatchGroup() or Description() FindBaseAddressString("\x00Extra: ") | hex:0045787472613a20 ReplaceString(7, " ", "\u0080") could not apply patch: ReplaceString: length mismatch in byte replacement Fatal: Could not apply patch file src/libnickel.so.1.0.0.yaml: ReplaceString: length mismatch in byte replacement

Built-in firmware download

Add more symbol resolution options

I'm planning on adding better symbol resolution stuff based on my work from today: https://gist.github.com/geek1011/167a87c3a0360366084a2e8e0cf95b83.

This will be done along with #31, and will deprecate #21. I will also be rewriting the BLX encoding to be more efficient and also support the B winstruction

The following new instructions/options will be added/changed:

• ReplaceBytes / ReplaceBytesAtSymbol
  • FindInstBLX (arg is one of Offset/SymPLT/SymPLTTail/Sym)
  • FindInstB (arg is one of Offset/SymPLT/SymPLTTail/Sym)
  • ReplaceInstBLX (arg is one of Offset/SymPLT/SymPLTTail/Sym)
  • ReplaceInstB (arg is one of Offset/SymPLT/SymPLTTail/Sym)
• FindBaseAddressSym (arg is one of Offset/SymPLT/SymPLTTail/Sym)
  • Can also specify an additional offset.

Note: The args will be able to be specified as either an offset directly, or an object with one of the following string properties: SymPLT/SymPLTTail/Sym.

The following instructions/options will be deprecated:

• ReplaceBytes / ReplaceBytesAtSymbol
  • FindBLX
• FindBaseAddressSymbol (as part of #31)
• ReplaceBLX (in favour of the FindInstBLX and ReplaceInstBLX options of ReplaceBytes)

Advantages:

• Most of the current version-dependent patches will likely become multi-version (or almost multi-version).
• No more segfaults from address typos.
• Significantly easier patch updates.
• Simpler code.

Disadvantages:

• Another ~500 lines of code.
• Adding an ARM disassembly lib for parsing the PLT (rsc.io/arm).
  • Slightly larger kobopatch binary size.
• Needing to emulate a subset of ARM instructions during patching (mov, ldr) for parsing the PLT.
  • Note that I've already tested the code against most versions of libnickel, including old ones.
  • Note that incorrect emulation will be automatically detected, as the bytes won't match for the FindInstBLX and FindInstB instructions.

Add support for old patch format to kobopatch

Just port the code from the patch32lsb folder

ReplaceBLX instruction

Implement a ReplaceBLX instruction which replaces BLX instructions in a binary.

• BLX instruction assembler
• RepaceBLX instruction