Coder Social home page Coder Social logo

itchio / capsule Goto Github PK

View Code? Open in Web Editor NEW
176.0 14.0 14.0 2.65 MB

๐ŸŽฌ Cross-platform hotkey short video capture for games

License: GNU General Public License v2.0

CMake 2.04% C++ 61.68% C 30.02% Objective-C 0.10% Objective-C++ 1.19% JavaScript 0.63% Rust 4.26% Makefile 0.07%
video-game itchio sharing

capsule's Introduction

capsule

capsule augments any game so that pressing a key records a video+audio clip. Those clips can be easily shared later, see this tweet for example.

It's being developed mainly for integration with the itch.io app, but is designed as a separate pair of projects:

libcapsule is responsible for detecting which video & audio APIs a game uses, and intercepting calls to them so that it may capture video and audio frames.

capsulerun is responsible for launching the game, injecting libcapsule, establishing a connection to it, receiving video and audio frames, encoding, and muxing them.

See Build instructions if you know enough to give it a shot :)

Design goals

  • Easy to use - no configuration, hit a key = record (think FRAPS)
  • Fast - should slow down games as little as possible
  • Quality recordings - high FPS, preserve colorspace when possible
  • Desktop audio - all captured gameplay should come with audio
  • Easy sharing - output should be immediately useful (H264/AAC)

Non-goals

Things that are out of scope (ie. that capsule will not officially support):

  • Compositing (more than one video source, etc.)
  • Live-streaming
  • Any platforms that aren't Linux, macOS, or Windows (especially mobile, BSDs, etc.)
  • Long videos - haven't settled on a maximum yet but "a few minutes"
  • Hooking into existing processes - capsulerun must be used to launch game

Status

Core

  • System
    • capsulerun launches child, relays stdout/stderr, injects 32-bit & 64-bit processes properly
    • capsulerun handles child quitting unexpectedly, relays exit code
    • capsulerun & libcapsule communicate via a pair of fifos/named pipes, send frames via shared memory
    • capsulerun has 'headless' mode where it doesn't launch a child, acts as just an encoder (HTML5 games, etc.)
  • Video
    • encoder outputs variable fps h264 video, aac audio, in an mp4 container

Linux

  • Control
    • X11 recording hotkey support (hardcoded to F9)
  • Video
    • OpenGL capture
    • Vulkan capture
  • Audio
    • ALSA: Intercepts ALSA API calls, only F32LE supported so far
    • PulseAudio: Captures monitor of default output device

macOS

  • Control
    • Cocoa recording hotkey support (hardcoded to F9)
  • Video
    • OpenGL capture
    • Vulkan capture
  • Audio
    • CoreAudio: Intercepts API calls, only F32LE supported so far

Windows

  • Control
    • Win32 recording hotkey support (hardcoded to F9)
  • Video
    • Direct3D 9 capture
    • Direct3D 10 capture
    • Direct3D 11 capture (with GPU color conversion & dummy overlay)
    • Direct3D 12 capture
    • Vulkan capture
  • Audio
    • Wasapi: Intercepts API calls, only F32LE supported so far
    • Wasapi fallback: captures default loopback device

Build instructions

This project is being ported to Go & Rust. These instructions are OUTDATED.

capsule is a (set of) CMake projects, which handles things like:

  • Download some dependencies
  • Generating Makefiles, MSVC project files, etc.
  • Doing 32/64-bit tests
  • Bundling required libraries with the final executable

Like typical CMake projects, capsule should be built in a subfolder, never directly in the source tree, as you'll see in platform-specific instructions

Building on Linux

Notes: some distributions ship libav, most distributions ship outdated versions of ffmpeg, save yourself some trouble and build x264 and ffmpeg 3.2+ yourself into a prefix. You can then pass it to cmake with -DCAPSULE_DEPS_PREFIX=/path/to/prefix

64-bit and 32-bit versions are built separately. Example of 64-bit build:

cd /path/to/capsule
mkdir build64
cd build64
cmake ..
make install -j4

capsulerun usage:

/path/to/capsule/build64/dist/capsulerun -- path/to/some/game.x86_64

If nothing's happening, the game isn't using OpenGL, you've got the wrong capsule architecture (64-bit for a 32-bit game for example), or there's something capsule doesn't support yet.

NB: Launcher scripts may give you a bit of trouble at the moment, as capsulerun will establish a connection with the shell and not the game, which will fail. This will be addressed later.

Building on macOS

Pretty much the same as Linux, except you don't need to compile x264 & ffmpeg yourself, a stripped-down home-cooked universal build will be downloaded by cmake.

The resulting capsulerun & libcapsule are universal, which means there's no separate 32-bit/64-bit build processes and they should work with every app, no matter what their architecture.

Example build:

cd /path/to/capsule
mkdir build
cd build
cmake ..
make install -j4

capsulerun supports launching .app bundles and invididual executables. Example usage:

/path/to/capsule/build/dist/capsulerun -- path/to/some/Game.app

Building on Windows

Important notes:

  • Microsoft Visual C++ (MSVC) is needed, 2015 is what we're using internally.
  • CMake shipped with msys2 won't cut it, you need a Windows release of CMake (which includes the Visual Studio project generators): https://cmake.org/

You probably won't get anywhere with an MSVC older than 2015.

From the Visual C++ 2015 Native Build Tools Command Prompt, run:

cd C:\path\to\capsule
mkdir build32
cd build32
cmake -G "Visual Studio 14 2015" ..

To compile, run this from the build32 folder:

msbuild INSTALL.vcxproj

(You can use /maxcpucount:n to speed up the build process, since capsule is broken down in a few projects - where n is the maximum number of CPUs you want to use)

To build a 64-bit version, add Win64 to the generator name (the -G parameter).

C:\path\to\capsule\build32\dist\capsulerun.exe -- some_game.exe

NB: Some games with launchers (for example UE4 games) may give you a bit of trouble at the moment, as capsulerun will establish a connection with the launcher and not the game, which will fail. This will be addressed later.

License

capsule is released under the GPL v2 license, see the LICENSE file.

It uses the following libraries:

Some of its code is derived from the following projects:

Special thanks to:

capsule's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

capsule's Issues

Dedupe PixFmt enums

Right now they're duplicated in messages.fbs (wire format) and capsule/constants.h.

That's dumb, and I haven't made the same mistake for audio - the wire format should have the canonical constant, since it's now harmless to include it

(there used to be some problems when we used using namespace std and when we didn't have -DNOMINMAX on windows because of windows.h vomiting all over the global namespace).

Investigate electron integration

Here's what I've found so far:

  • BrowserWindow has a getContentSize() method, which is accurate
  • webContents has a startFrameSubscription method, which lets us register a callback to get all frames as BGRA (ignore the dirtyRect part).
  • There doesn't seem to be a suitable shm node module existing, so we'd have to write our own (writing to a fifo works, but is slow).

The main thing for capsule is that capsulerun needs a --headless mode, where it doesn't launch anything but simply prints out the pipe path, and waits for a client to connect, then does the encoding as usual.

The main change for the itch app is that it's the end of simple things ๐Ÿผ having a native module means it'll need to be recompiled at release time, but luckily there's a bunch of tools to facilitate that, and we could even set up automatic builds so there's precompiled binaries, for itch app developers who might not have a native toolchain on their machine.

Add `--control-pipe` option

In a native game setup there's three actors:

  • the launcher + overlay-renderer (itch)
  • the encoder (capsulerun)
  • the game

The current --pipe handles the encoder <-> game communication: it mostly concerns itself with synchronizing the video frame buffer (and capture start/stop).

We need a --control-pipe for launcher <-> encoder, so that the launcher is aware of some metrics (fps, recorded length, etc.) some events (recording start/stop, encoding finished with file path, etc.) and so on. It will also be useful for when we're able to capture mouse/keyboard events from the game, to relay them back to the overlay renderer.

TL:DR we need another pair of fifos/named pipes. Luckily both capsulerun and itch have classes that encapsulate that functionality now, so it should be rather easy. We might however have to change MainLoop to work off an event queue instead of reading directly from a connection, since we'll have several connections to deal with.

HAL support (macOS)

Overland is a wwise game, it's terribly slow in my test VM though so it'd take me a long time to figure out exactly what it's using - but in my last test it didn't seem like the coreaudio hooks were called.

Maybe someone whose mac hardware didn't die on them unexpectedly can help research this?

Flip encoder on its head / don't let audio lag because of video

So currently, the encoder is a single function (boo!) running from params that are a struct, that contains a bunch of callbacks like:

  • receive video format
  • receive video frame
  • receive audio format
  • receive audio frames

There's a bunch of issues with that design:

  • Casts, casts everywhere + public C++ methods that access private fields just so we can pass them as C function pointers in the encoder params
  • Both AudioReceiver (implementations) and VideoReceiver need to assume that just because they sent a frame (in a callback) doesn't mean it has been processed, hence it's not safe to overwrite. So they have logic that checks, on the next call, if there were any frames "pending", and if so, to free them up. This is more work than we need to do.
  • If "ReceiveVideoFrame" hangs (because there are no frames currently available - e.g. the game is loading something or just not rendering frames for any reason), then AudioReceiver doesn't get asked for frames for a long while! This usually causes a buffer overrun, which means audio gets out of sync and everything is screwed forever.

My thinking is:

  • Encoder should be turned into a class
  • It should take references/pointers/smart thingamajis to an AudioReceiver and a VideoReceiver (forget C function pointers, forget has_audio, etc.)
  • It should be responsible for telling AudioReceiver/VideoReceiver when something has been processed
  • VideoReceiver should never block, it should just return immediately when no frames are available
  • encoder should have its own timer (based on the exact same code as receivers, so they're in sync) to know whether it should ask for frames or sleep for a bit, and it should use that to determine whether to ask for audio frames or not, instead of comparing with the current video timestamp

In fact, it would be interesting if video & audio frames were encoded in parallel, and only muxed sequentially. We could do so using a queue or just mutexes to make sure that we're only muxing from one thread.

Research overlay display for OpenGL

I'm not sure what it takes so save/restore OpenGL state or how we're going to have streaming textures that work with a wide range of OpenGL versions, hopefully it's not way more complicated than D3D11.

Give PolyHook a try

Deviare-InProc (Nkt) has nice things like "launch an .exe with injected DLL" or "load DLL remotely in hooked process", but PolyHook does virtual function detour/fp swap/tp swap, and we'll need that for Direct3D.

see

Both Deviare-InProc and PolyHook use a disassembler to make sure the hook code wouldn't overwrite something after the function boundary - OBS doesn't seem to.

OBS also uses separate executables for getting graphics offsets, and injecting a dll into another process

Grandchild handling

This is required for UE4 games, for example, for which the .exe launched by itch isn't the actual game, it's just a launcher, that spawns the actual game.

What's done:

  • We already have a CreateProcess{A,W} hooks, which do the job for UE4 - more hooks might need to be added for other variants. Our hooks use Deviare-InProc's inject methods to make sure it is also injected into the grandchild
  • On Linux/macOS, LD_PRELOAD and DYLD_INSERT_LIBRARIES are inherited by the child, so any grandchilds it spawns have libcapsule loaded up already

What's to do:

  • capsulerun works under the assumption that it has a single connection to the child and that's it. With children, that's no longer the case. The original child might exit early (in which case capsule should probably wait on the grandchild - there's corner cases here, what if the child opens a web browser?), and we should handle connection switching.

Exactly how connection switching is going to work, I'm not sure yet - I introduced capture::SawBackend recently so that maybe the game could send that info via the connection and capsulerun would have an overview of which backends are available where, and pick the best connection to attempt a capture on.

CoreAudio interception

Similar to #40 and #41, already some code in the coreaudio branch, this is significantly easier than the CAPlaythrough+Soundflower approach we had before and it doesn't require installing an extra package (Soundflower) that requires administrator permissions!

Support audio resampling / format conversion

We already have libswresample built-in on all 3 platforms but right now we don't use it, we just de-interleave the samples by hand.

Now that we do audio interception, this might not work out - it's not unreasonable to think some games/OS combos are going to work in a format that isn't f32le: I wouldn't be surprised to see s16le in the wild for example.

Maybe #41 alone will require that, who knows though.

This requires figuring out how libswresample actually works btw, which was still largely a mystery when I first tried.

Fix usage of GPL

Add mentions to file headers, fill out wording in LICENSE, etc.

Wasapi interception

The goal here is to pull off a #40, but for windows audio.

Note that we already have WasapiReceiver, which captures from the loopback audio device. Intercepting would let us only get audio from the game instead of everything that's playing on the computer, though - so that's still a win.

Besides, a second "audio intercept" backend is nice practice for the big fish that awaits me right after this: CoreAudio (on macOS).

The reference for this issue is this msdn sample, I'm going to try and run 90 Second Portraits, intercept CoCreateInstance, and see where it takes us.

ALSA support

Soooooooooooooooo

Some folks don't use pulseaudio on their system, which is great!

ALSA doesn't have out-of-the-box monitor sinks (snd-aloop exists, but it requires configuration) ... which is less great!

I don't feel like reimplementing libasound2.so, nor do I feel like hooking it and potentially destroying audio performance with my naughty naughty code.

Luckily there is a piece of software that already can sit on top of ALSA/OSS and handle monitoring (/loopback) and resampling for us and its name is... pulseaudio!

Contrary to popular belief, pulseaudio isn't that big when you build it without all the nonsense options, look at this beautiful beautiful ldd output:

[amos@testbed bin]$ ldd pulseaudio
        linux-vdso.so.1 (0x00007ffe0f154000)
        libpulsecore-10.0.so => /home/amos/pa-prefix/lib/pulseaudio/libpulsecore-10.0.so (0x00007f90ab9dc000)
        libpulsecommon-10.0.so => /home/amos/pa-prefix/lib/pulseaudio/libpulsecommon-10.0.so (0x00007f90ab766000)
        libpulse.so.0 => /home/amos/pa-prefix/lib/libpulse.so.0 (0x00007f90ab517000)
        libltdl.so.7 => /usr/lib/libltdl.so.7 (0x00007f90ab30d000)
        libgdbm.so.4 => /usr/lib/libgdbm.so.4 (0x00007f90ab101000)
        libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f90aaefd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f90aacdf000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f90aaad7000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f90aa8d3000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f90aa5c0000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f90aa21c000)
        libsndfile.so.1 => /home/amos/pa-prefix/lib/libsndfile.so.1 (0x00007f90a9faf000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f90abc76000)

That's not half bad. To give you an idea of the sizes:

2.3M    /home/amos/pa-prefix/lib/pulseaudio/libpulsecore-10.0.so
1.7M    /home/amos/pa-prefix/lib/pulseaudio/libpulsecommon-10.0.so
1.2M    /home/amos/pa-prefix/lib/libpulse.so.0.20.1
40K     /usr/lib/libltdl.so.7.3.1
48K     /usr/lib/libgdbm.so.4.0.0
20K     /usr/lib/libcap.so.2.25
144K    /usr/lib/libpthread-2.25.so
32K     /usr/lib/librt-2.25.so
16K     /usr/lib/libdl-2.25.so
1.1M    /usr/lib/libm-2.25.so
1.9M    /usr/lib/libc-2.25.so
2.0M    /home/amos/pa-prefix/lib/libsndfile.so.1.0.27
11M     total

some of these we won't ship cause they're builtins (like libm, libc), and some of these are not stripped yet, and .so files usually compress very well, so the download size will be quite small indeed.

That's a custom compile of pulseaudio and libsndfile, of course, since they both tend to want to link against everything.

pulseaudio can be started in the foreground (which means it can be started by capsulerun, if it doesn't detect an already running pulse server), and we can give it a non-standard socket path so other apps don't mistakenly connect to it, like so:

PULSE_RUNTIME_PATH=/tmp/private-pulse $PREFIX/bin/pulseaudio -vvv

And pulse consumers can be run with:

PULSE_SERVER=unix:/tmp/private-pulse/native ./game.x86_64

That's it!

(ps: the caveat is that the pulseaudio server locks exclusive access to the ALSA device so other ALSA apps won't be able to access audio while the game is playing audio.

obviously this will come with a flag to disable it in case it ruins everything.)

Drawing overlay influences capture - macOS 10.12.4

To replicate, check out most recent revision (278d75b), then build, and run a basic LOVE game. When recording the entire image starts fading in and out, and a large red/black/white(?) square flashes in the right bottom part of the screen (not visible in recording). As this appears to be a regression I tried to narrow down when it happened, the best I could come up with is that my guess is that it started between ec2718c and 735da5e. A lot of those builds weren't recording so its hard to narrow it down much more.

Works as expected in cbe5e63

Sorry if this bug report isn't as complete, it is getting made during LD after all.

Relay stdin/stderr/stdout on Windows

There's code for that in https://github.com/itchio/isolate - but it's before we adopted the google c++ styleguide, so I'm going to import it into lab & make sure it's solid, then at a later date we can convert isolate to:

  • follow the c++ styleguide
  • build against MSVC
  • use lab
  • handle unicode arguments properly
  • use argparse for argument parsing

Finish overlay display on D3D11

This is the most advanced codepath for overlay support - it already does streaming texture upload & ninja render in-between game frames, all we need is pretty much to connect it to whichever shared memory area was given to capsulerun.

Maybe for sync it'd be best to do #32 beforehand.

Let's talk design!

Here are the requirements:

  • Should be able to display small overlay in a corner of the screen to show recording controls
  • Should be relatively light: dropping a few FPS, not making a game unplayable (very vague req., will have to see how real-life code performs)
  • Should be usable when target is started before capsule or after capsule - ie. capsule should be able to attach/detach itself from processes (and detect if they're using OpenGL/D3D/Vulkan)
  • Capsule doesn't need to handle transcoding itself, as long as it has a well-defined output format - maybe doing downscaling would be a good feature for capsule to allow running it on low-end systems without destroying the game's fluidity

The closest I could find to my requirements were Taksi, an open-source alternative to Fraps. I imported it to a Github repo, but:

  • It's super old (targets GDI/OpenGL/DX8/DX9)
  • It seems broken atm - my tests with an OpenGL game captured
  • It relies on windows-specific video compression codepaths
  • It has a pretty bad Win32 GUI, but it sounds easy enough to strip out

Another option would be to rely on libobs (from OBS Studio), at first glance:

  • It's a big codebase & already has stuff like transcoding + streaming baked in
  • Uses CMake, doesn't (easily) build with mingw-w64 on Windows, relies on MSVC still (my first attempts at compiling it on Windows were unsuccessful)
  • It's designed with cross-platform support in mind
  • It has no need/desire/support for overlays

TL;DR there seems to be two main strategies:

  • bring taksi up to speed / migrate it to a sane build system or just do a fresh project based on some of the same tricks
  • trim down libobs until it's small enough to fit the reqs + add overlay support to it

libcapsule/libcapsule.so: undefined symbol: XOpenDisplay

I built FFMpeg 3.2.4 and Capsule on Xubuntu 16.04 (x86_64). When trying to run it, I get this:

./capsulerun/capsulerun libcapsule /home/mikko/fractilegames/build/astronautics-0.0.8-linux64/astronautics
thanks for flying capsule on GNU/Linux
pid 14904 given to child /home/mikko/fractilegames/build/astronautics-0.0.8-linux64/astronautics
[capsule] Initializing (pid 14904)...
[capsule] LD_LIBRARY_PATH: (null)
[capsule] LD_PRELOAD: libcapsule/libcapsule.so
[capsule] CAPSULE_PIPE_PATH: /tmp/capsule.rawvideo
/home/mikko/fractilegames/build/astronautics-0.0.8-linux64/astronautics: symbol lookup error: libcapsule/libcapsule.so: undefined symbol: XOpenDisplay

Process just hangs there with no game window opening until I terminate it with Ctrl-C. Should this be working or is it just something not implemented yet?

Support launching .app bundles

The itch app won't use this, probably - because the sandbox needs to create a shim app for us to get proper dock/icon behavior, but it's useful when ran independently.

Support planar video formats

Basically:

  • originally the encoder code assumed interleaved video format
  • so the struct only took a linesize param
  • now: in the wire format (messages.fbs), linesize and offset are arrays
  • ... but the encoder video format still has just linesize and vflip

So we need to adjust the encoder format struct, see definition here and make the encoder code use it properly.

Also, for OpenGL capture, writevideoformat should take care of the stride vflip trick by itself, see these lines.

In general the encoder code needs a big cleanup but this can be done before or after it independently.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.