philippmatthes / neural-greenscreen

Real time background replacement on a mac os driven webcam using the DeepLabV3 neural network for image segmentation and the native CoreMediaIO DAL framework of Mac OS.

Neural Greenscreen

Realtime background removal with neural networks on mac os, providing a virtual camera, which can be accessed by third party applications. ➡️ Read the Medium article

Based on seanchas116/SimpleDALPlugin and johnboiles/coremediaio-dal-minimal-example.

Prerequesites

• A system running mac os with a builtin webcam
• Xcode

Setup

• Check your webcam dimensions and change the code (and the background) if your webcam has a higher resolution than 1280 x 720
• Build neural-greenscreen in Xcode
• Copy neuralGreenscreenMain.plugin into /Library/CoreMediaIO/Plug-Ins/DAL
• Open app that uses your webcam and choose Neural Greenscreen as camera input

You should see the background being replaced.

Troubleshooting

DAL plugins access very low level mac os interfaces. It could be that the plugin is not detected or crashes depending on your system or app. If you find a problem, report it so that we can fix it. Here is a recommended workflow:

• Check your logs (using the mac os console, search for neural)
• Open an issue explaining your problem

License

This project is licensed under MIT License.

neural-greenscreen's Issues

Support for Apple Silicon Macs and third-party webcams

Doesn't seem to work, crashes obs. I tried both code settings for 720p and 1080p. Would be nice if there was a way to choose external "USB Camera" (webcam) if it was plugged in. I'm not sure how to set default video device in mac. Nice work on this regardless if I can get it working.

MacBook Pro (16-inch, 2021)
MacOS Monterey 12.6

default	09:04:49.637339-0400	OBS	Neural Greenscreen: DeviceStartStream(plugin:deviceID:streamID:):
default	09:04:50.180830-0400	OBS	neuralGreenscreenMain/NeuralGreenscreen.swift:154: Fatal error: Failed to render output texture
default	09:04:51.015764-0400	ReportCrash	ASI found [libswiftCore.dylib] (sensitive) 'neuralGreenscreenMain/NeuralGreenscreen.swift:154: Fatal error: Failed to render output texture'

[ Question ] How will you release this as a mac app ?

Will you be able to release your current project with node.js as an mac app ? and also why you haven't gone with coreml since the upcoming apple silicons are going to have ANE in them ? also sorry for using issues page for this.

How to call plugin from C++ or swift application to feed the buffer

trying to get an answer for this over a week now. Any guidance is appreciated.