Coder Social home page Coder Social logo

vn-tools / arc_unpacker Goto Github PK

View Code? Open in Web Editor NEW
559.0 45.0 82.0 39.4 MB

CLI tool for extracting images and sounds from visual novels.

License: GNU General Public License v3.0

C++ 95.47% HTML 0.24% CMake 0.28% JavaScript 3.48% Common Workflow Language 0.03% Shell 0.06% Python 0.44%

arc_unpacker's Introduction

arc_unpacker - the visual novel extractor

arc_unpacker is a command line tool for extracting images, sounds, music and miscellaneous resources from visual novels (and some other Japanese games).

Usage

Drag and drop the archive or file onto arc_unpacker. It will guess the format and unpack it for you.

Caveats:

  1. The file format might be detected by two or more decoders at once. In such cases you need to tell the program which one to use by supplying --dec=... option.

  2. The file might need more parameters to be correctly unpacked. In such cases you need to supply them manually. For example, XP3 archives need --plugin that tells what kind of decryption to use. To tell it to use Fate/Stay Night decryption, supply --dec=krkr/xp3 --plugin=fsn.

To learn what parameter your game needs, you can either use --help to see all available parameters, or refer to your game details in the game list that lists all required parameters for every supported game.

To learn how to pass parameters to arc_unpacker, refer to this question.

Q&A

  • I drag the game files onto arc_unpacker and it immediately closes.

    Try running it from the command prompt to see the output and possible error message. Usually this behavior is a sign that the game requires additional parameters which are described in the game list, or it's not supported.

  • What is command prompt?

    On Windows, it's cmd.exe. On Linux, I think you already know the answer to this question.

  • How do I pass additional options / parameters?

    (for Windows users)

    1. Open up cmd.exe and navigate to the directory where you downloaded arc_unpacker.exe by typing a command that looks like this:

       cd "C:\downloads"
      

      Alternatively, navigate to the arc_unpacker directory in Explorer, hold Shift and click with right mouse button anywhere, and select "Open command window here".

    2. To run arc_unpacker with your game without extra parameters, type a command that looks like this:

       arc_unpacker "C:\games\your game\file you want to unpack.dat"
      

      or

       arc_unpacker "C:\games\your game\directory you want to unpack"
      

      Alternatively, type arc_unpacker (without hitting Enter) and drop a file onto command prompt. It should enter the path like above, saving you the effort of typing it manually.

    3. To pass additional parameters such as --dec etc., run arc_unpacker like this:

       arc_unpacker "C:\games\Touhou 08\th08.dat" --dec=team-shanghai-alice/pbgz
      

      To get list of possible parameters, see arc_unpacker --help. If your game needs any extra parameters, they're also outlined inside the game list.

  • Why command line? Why no windows / GUI?

    It'd take a lot of effort to make GUI for arc_unpacker:

    • cross platform - I'd need to use Qt or Gtk, and this adds to the project complexity (consider how we support five different compilers).
    • design - it's difficult to make a GUI that caters to most of use cases (recursive unpacking, selecting decoder, passing game key etc.)
    • effort - implementing it easily scales to dozens, if not hundreds of man hours. I'd rather focus on supporting more games.

    That being said, it's not entirely impossible for this to happen in the future. Once C++17 comes out, I plan to ditch some of the non-conforming compilers, which should ease things up a bit.

  • What do I do with .wavloop files?

    These files are audio loops. You can play them like normal .wav, however, most of the players will ignore looping information that is contained in such files. If you're looking for players that do support looped .wavs, I recommend either vgmstream, foo_input_wave_loop or wavosaur. The extension .wavloop was chosen so that it stands out from normal .wavs and for compatibility with foo_input_wave_loop.

  • I get std::bad_alloc, what gives?

    One option is that an archive contains a very large file, which causes arc_unpacker to run out of RAM while it tries to decode that file. To circumvent this (short of buying more memory), you can try running arc_unpacker with -t=1 --no-recurse which should reduce its memory footprint. Since there only were a few such archives spotted, no special mechanism was developed to work around this issue, although this might change in the future. Other options include corrupt game files or a specific kind of bug in arc_unpacker's decoders, but both are unlikely. If you are unable to unpack the files, do not hesitate to report the issue to the issue tracker.

  • How can I help with development?

    Thanks for asking this! There are a number of ways you can help:

    • By documenting games that are already supported but are not present on the games list.
    • By porting existing decoders from other projects. Example projects can be found in THANKS.md.
    • By reverse engineering games that are not yet supported, and coding new decoders for them.

    For details, check out CONTRIBUTING.md.

  • Packing / encoding support?

    Not going to happen. Compiling files for games, especially from fan translation standpoint, almost always needs rolling your own script compiler, modifying the game .exe, fixing weird quirks, etc. - many things that wouldn't make sense for arc_unpacker to support. At the same time, the user base that might find encoders useful would be extremely small, and writing encoders is often much more difficult than writing decoders, so it's not very practical.

Contact

arc_unpacker's People

Contributors

brliron avatar drdaxxy avatar extend2to3meters avatar futile avatar javantea avatar nhkpanda avatar nightfeather avatar porygon2 avatar rr- avatar xiongnemo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

arc_unpacker's Issues

Create a separate page with game list outside of README.md

I'd like the table to include information such as:

  • CLI invocation
  • Engine
  • List of games affected
  • Features
    • Unpacking
      • Script decompilation
      • Graphic extraction
    • Packing
      • Script compilation
      • Graphic repacking

Github doesn't render big tables too well, so I'd like to create a Github Page for this project.

Game request: [BISHOP]Sansha Mendan

https://vndb.org/v6357

Hello, my ultimate goal with this game is to be able to extract and inject the UI and script, however I am very new to this. I already understand that you don't do injection requests, but if you could get the script decoded and extracted that would be more than enough. If you can't help any tips or guides that have helped you in the past would be nice as well.

Game request: [Nitroplus][160630]Sonicomi: Communication with Sonico

Hi. Can you please add support for the English version of Sonicomi by JAST USA? This version uses NPK2 files (note that this is different from the original Japanese release, which used NPA instead), but the game isn't supported yet. If I try unpacking the files using the Tokyo Necro plugin, I get a "Premature end of file" error message.

VNDB: https://vndb.org/v4234
I don't think the Japanese title is necessary, since this request applies to the localized version exclusively, but here it is just in case: ソニコミ

Thank you!

Switch to boost filesystem

Current path management feels very inferior.

  • Research the cost of integrating boost::filesystem to the project.
  • If the costs are negligible, integrate.

Game request: [Type-Moon]Fate/Hollow Ataraxia

My discoveries so far:

  • FHA is protected with CXDEC algorithm.
  • CXDEC uses following:
    • LSFR-based randomization
    • complex key derivation that uses ADLR chunk.
    • the decryption process is mostly linear and simple - just xor input in a few predictable ranges with values derived from the key.
  • Each game using CXDEC protection uses very little customization. These are as follows:
    • Substitution table - used to derive the key. It lies unencrypted in cxdec.tpm file.
    • Configurable order of code blocks used in key derivation algorithm.
    • 32-bit private key.

In the case of FHA the 32-bit private key seems to be 008F0313.

The main difficulty lies in reversing the key derivation algorithm.

Dies Irae std::bad_alloc error

There is an error when decoding the picture.lib file.
For Dies Irae Amantes amentes:
error decoding "bg" std::bad_alloc
error decoding "chara" std::bad_alloc
error decoding "event" std::bad_alloc
"mask" decodes correctly

For Dies irae Acta est Fabula
error decoding "bg" std::bad_alloc
"chara" decodes correctly
error decoding "bg" std::bad_alloc
"mask" decodes correctly

Vndb: https://vndb.org/v548
Both are Regular Editions with Install versions. (No patches applied)
Both were tested on Windows 8.1 Pro 64bit

Research unit test frameworks

Right now the state of the unit tests is pretty sad: each one takes years to compile and creates a new executable. There should be a nicer way to make them work.

Game request: [Almond Collective][110507]Bloomers Lesson

Game info

Developer: アーモンドコレクティブ
Release date: 2013/05/07
Original title: ゴリマッチョとブルマレッスン
English title: Bloomers Lesson
DLsite work number: RJ076908

Details

Would like to extract game resources (videos, etc.) from the "Data.pac" file.

Game request: [Heiantei][160324]Aniyome Kyouka-san to Sono Haha Chikako-san ~Bijin Tsuma to Bijukubo to Issho~

VNDB link: https://vndb.org/v18803
Developer: Heiantei
Release date: 2016/03/24
Original title: 兄嫁京香さんとその母千佳子さん~美人妻と美熟母といっしょ~
English title (if available):

Hello. I'd like to request to support this game. Unfortunately the closest I've gotten to is ripping the files encrypted, but the elements cannot be seen.

Files are: dPi.dat (I think these are the CG/char sprites), includes a dPih.dat file related to it.
dSo.dat/dSoh.dat, which are music files.

There's another game I'm interested in requesting as well, but I also would request the game Lbzttt is requesting as well.

Build on Visual C++

Hi,

First thanks for your work. I have been looking for a tool like this for long.

I try to build it on Visual C++. At least it successfully builds and it seems working on Touhou 13. test not tried yet.

I have made two modifications.

  1. In formats/archives.cc function void Archive::unpack(File &file, FileSaver &file_saver), the lambda expression captures file_saver_proxy which is not initialized. VC++ (I'm using Visual Studio 2013 v120) does not support that. By creating a p_file_saver_proxy and capture that by reference, and after setting p_file_saver_proxy, pointing p_file_saver_proxy to p_file_saver_proxy, it can be successfully built. (I'm not sure whether this modification is legal, according to the logic of your program.)
  2. In formats/kirikiri/tlg_converter.cc, std::unique_ptr<File> decode_tlg_0(File &file), change ::decode to decode. (In VC++, functions in inline namespace can not be refered to as this.)

Another issue is that usually the aux folder can not be created on Windows. Would you mind giving it another name?

Though it is working, I don't think it's easy to have fully support for VC++ (or some other build tools supporting VC++ like CMake). But if you like, I can have a try.

Add correct extension

Currently all conversions happen on data-level. Even if the script correctly decodes everything, the extension remains the same.

There needs to be a way of providing correct file extension.

Rought scenario I imagined

  1. Introduce File structure. It has data and name.
    No lazy loading, just fully loaded data and file name.
  2. Archive unpacker produces Files.
  3. OutputFiles is given Files. It consumes and discards them ASAP like it does now.
  4. InputFiles produces Files.
  5. Archive packer is given Files. It consumes and discards them ASAP like it does now.
  6. Converters are given Files and produce Files.
  7. Image.raw_to_boxed is given raw data and original File and produces new File.
  8. Same goes to Sound.raw_to_boxed.
  9. Image and Sound fill the file name with their own extensions.

The only problem may arise with changing Image.raw_to_boxed. There's a chance I abuse it and assume it produces raw PNG data, which is a bold assumption. That, however, can be easily dealt with with minimal effort.

Make target folder optional

The target folder path should be figured out automatically if it isn't supplied manually.

Combined with #6 this should allow users to drag and drop folders onto executable without hassle.

Merge binaries

file_decoder and arc_unpacker should be merged into one library. What I have in mind:

arc_unpacker file

Saves file~ in its directory if it's recognized.

arc_unpacker archive

Saves archive~ in its directory if it's recognized.

arc_unpacker file archive

Saves file~ in file's directory and archive~ in archive's directory.

arc_unpacker file archive --out DIR

Saves both file~ and archive~ in DIR.

arc_unpacker --fmt=xp3 file archive

Tries to treat file and archive as xp3, if it fails - prints error (= doesn't try to guess the format by itself).

 arc_unpacker file.xp3 archive.nsa --plugin=noop

Tries to unpack file.xp3 with noop plugin, and archive.nsa with noop plugin. XP3 is unpacked successfully, while NSA is not, because there is no such plugin as noop for NSA archives.

Game request: [HARUKAZE]Nora to Oujo to Noraneko Heart

I've been wanting to extract the music for ノラと皇女と野良猫ハート and was trying arc_unpacker on it.

Unfortunately it failed with the following error:

[task 0] arc3~.dat: initial recognition...
[task 0] arc3~.dat: guessing decoder among 1 decoders...
[task 0] arc3~.dat: recognized as whale/dat.
[task 0] arc3~.dat: archive contains 6 files.
[task 6] arc3~.dat: decoding "unk_5.dat"...
[task 6] arc3~.dat: Unknown hash: b0a8290f4eb61ead. File cannot be unpacked.
[task 6] arc3~.dat: decoding ommitted.
[task 5] arc3~.dat: decoding "unk_4.dat"...
[task 5] arc3~.dat: Unknown hash: bab502bf52eaa1fd. File cannot be unpacked.
[task 5] arc3~.dat: decoding ommitted.
[task 4] arc3~.dat: decoding "unk_3.dat"...
[task 4] arc3~.dat: Unknown hash: 2802801e254bbc31. File cannot be unpacked.
[task 4] arc3~.dat: decoding ommitted.
[task 3] arc3~.dat: decoding "unk_2.dat"...
[task 3] arc3~.dat: Unknown hash: ce66d85b0acb8dd6. File cannot be unpacked.
[task 3] arc3~.dat: decoding ommitted.
[task 2] arc3~.dat: decoding "unk_1.dat"...
[task 2] arc3~.dat: Unknown hash: dad04601b5141e79. File cannot be unpacked.
[task 2] arc3~.dat: decoding ommitted.
[task 1] arc3~.dat: decoding "unk_0.dat"...
[task 1] arc3~.dat: Unknown hash: 90cb6c1971148e65. File cannot be unpacked.
[task 1] arc3~.dat: decoding ommitted.
Executed 7 tasks in 0.10s (6 problems, 0 saved files)

Any idea what it'd take to get this game to extract properly? Thanks!

Possible problem with GXT file alpha?

In Steins;Gate 0, the font files (FONT.GXT and FONT2.GXT), when converted to PNG, have completely opaque black backgrounds, which looks incorrect ingame:

The correct appearance would be the following:

Game request: [Almond Collective][131103]Ojiki to mei to

Game info

Developer: アーモンドコレクティブ
Release date: 2013/11/03
Original title: 叔父貴と姪と
DLsite work number: RJ124336

Details

Would like to extract game resources (videos, etc.) from the "Data.pac" file.

Assistance for a VN.

I left you a message in the email that is indicated here. Please check it for more details.

Game request: multiple Purple Software games

Please support archive .cpz
extract archives cg :(

archive .cpz
images .pb3

CMVS

Extract Games
-Hapymaher
-Hapymaher -Fragmentation Dream
-Chrono Clock

Info Game:

https://vndb.org/p132
http://www.purplesoftware.jp/main.php

I need help please
there is no extract tool cg

Thank you
I'll be waiting... :)

let me know if you need the game files

Thank you

Please

more information tools:
http://blog.sina.com.cn/s/blog_5e29812f0100r8ku.html

Please Support.....Thank you ;)

cmvs32.exeを
00051350 75 18 → EB 59

Game Request: [Akabei Soft2][100624]Kourin no Machi, Lavender no Shoujo

VNDB: https://vndb.org/v3095

It's an older game with an awesome soundtrack that has never been released AFAIK, so I'm trying to extract it instead.

I gave this game a try with --plugin=noop but the resulting files were unreadable. It's a kirikiri/xp3 game, so I'm guessing it will probably require a custom plugin, but unfortunately I have no idea how to proceed further because that's the extent of my technical knowledge.

Let me know if you need me to upload the data files or tpm file somewhere or if there's anything else I can do to help!

Thanks!

Introduce utils

For now, all the "junk" code is stored inside string/ directory. The responsibility of string/ directory has been very sketchy so far and what I mean by that can be seen from what it stores:

There is a growing list of other things that might be shared across the classes like the stuff in string/, that doesn't belong to string/ no matter how you look at it. A good example would be the recently checked in Mersenne Twister used by QLiE or some of the basic RGB manipulations in Touhou's ANM decoder.

My proposition is to get rid of string/ in favor of util/ and throw all the junk in there.

Add fuzz tests

The project would benefit from these as it would smoke out possible seg faults.

Consider technology shift

  1. Some areas suffer form very poor performance.
  2. This has prompted me to start creating native extensions in C instead of writing it in plain Ruby.
  3. The obvious advantage lies in the gained performance.
  4. There are, however, numerous disadvantages of such approach:
    1. The codebase becomes impure, it mixes Ruby with C.
    2. It imposes great requirements on the end user - he must have gcc and make around. I assume it is a great PITA to get this right on Windows (used by the target audience) without using Cygwin.
    3. There is RubyInline, but it causes multiple problems on every machine I try to install it on (surprisingly, biggest problems kept happening on Debian). It doesn't resolve the issues mentioned previous points either.
    4. Even if I port the most sensitive parts to C, the remaining code is still quite slow.

The biggest issue with this is that when I started this project, I've chosen Ruby because I wanted it to be as clean and nice as possible... to developers. Now I feel like that as a byproduct, this tool can be used only by developers.

My suggestions are following:

  • Ignore this issue. Which is what I'm going to do for now.
  • Switch to other language.
    1. Go. I'm a bit reluctant to rely on anything from Google, but from what I heard about Go, this seems like a good choice. The syntax is weird.
    2. Rust. For now, it's too immature - core modules might change any moment. The syntax is crazy. I'm afraid of compile times - no incremental building.
    3. D. I know nothing about D, except that it falls into similar niche.
    4. C/C++. Makefiles are nightmare, unit testing that needs to use macros is a nightmare, includes that expand to 70k SLOC is a nightmare. Everything reeks of 80s and C++14 doesn't fix that.

How prevent merging image (purple software)

to change 'standing CG' (clothes -> naked)
I need un-merged files
but the program keep merging images
what should I do

p.s. don't worry about 'repack' the 'arc_conv' can do it .......maybe XD

Premature end of file for Tokyo Necro's `cg.npk`

This problem doesn't occur on Linux, however, it's present in nightly Windows builds.
Probably it has to do with the fact that the file in question is over 5GB since other files work fine.

The first thing to check would be to verify sizeof(size_t) which is used all over the place for file offsets. If it's 32-bit only, we're screwed and need to change size_t to something else everywhere. Further more if Windows implementation of fseek uses 32-bit integers, we're screwed even more and probably need to use fseeko or _fseek_whatever.

Game request: [Regista][160427]Root Double -Before Crime * After Days-

VNDB link: https://vndb.org/v5000
Developer: Regista


Something I have learned from the game:
The game store its data in the .cpk files. The .cpk files can be unpacked using quickbms with cpk.bms
http://aluigi.altervista.org/bms/cpk.bms
However, unpacking it will result in 3 types of file

Game request: [Leaf][080425]ToHeart2 AnotherDays

Game info

VNDB link: https://vndb.org/v90
Developer: Leaf
Release date: 2008
Original title: ToHeart2 AnotherDays

Details

Game request
I'm trying to unpack the voice files in this game. While I can unpack the .a files it uses, the files I get are in .g format which nothing seems to recognise.

Improve makefile

Once we have to deal with only one binary executable thanks to #13, a generic Makefile should be much easier to create. The absolute requirement is to make make figure dependencies out by itself, in an optimal way - otherwise there's nothing to gain.

Improve format autodetection

Right now the program just loops through all possible format readers and tries to unpack with each one of them. Exception happened? Try another reader.

This causes ugly anomalies. For example:

  • I see XP3, and I know for sure it's XP3 thanks to XP3\r\nWhatever magic number inside. But I haven't supplied --plugin, which causes me to throw an exception. The file is mistakenly delivered to next reader.
  • I see ANM archive, and it's nice. It correctly unpacks 0 files. But in reality it was a NWA sound file all along, I just didn't happen to know this because ANM was earlier on my check list.

This should be fixed. I could add new exception, say, RecognitionError and catch only that, but this won't make any good to detecting false positives (example from second point).

What should be done instead:

  • Add to every Transformer (#14) a new method, is_recognized
  • Before calling unpack, loop through all Transformers
  • See how many of them recognize the file
    • If it's just 1 - great! Call unpack on it, and then call it a day.
      • The unpacker now throws an exception urging you to provide --plugin parameter. Everything goes well, because the file is no longer passed to the next transformer.
    • If it's more than 1 - this is good, too. I can tell it to the user and advise him to select the --fmt manually.
    • If it's 0, tough luck. File is not recognized.

Cons of this solution are limited and, in my opinion, negligible:

  • no account for situation when is_recognized needs --plugin to be able to recognize the file perfectly. In these cases we can try looking at file extensions, their names, etc.
  • I need to implement is_recognized for every support format. This is going to be tough!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.