Advanced Swift library for media conversion and manipulation
Home Page: https://starkdmi.github.io/MediaToolSwift/documentation/mediatoolswift
License: Mozilla Public License 2.0
Is it possible to extract audio data from a video file / container? I'd like to use this to run Whisper transcription on audio tracks to get higher quality captions as text. Thanks
edit:
Would also be great to be able to extract subtitle files too.
Thank you for your work and the MediaToolSwift. It helps me a lot to transcode videos, user import into my app.
As it turns out I have a use-case where the check for a passthrough audio track works different, as a user would expect it.
Sometime, user will import a video that has an audio track already encoded with aac and a bitrate on 32kbit/s. While there is a limitation on the Apple frameworks (does not allow a bitrate lower than 64kbit/s) it seems appropriate to passthrough the source audio stream and skip the compression.
In the method initAudio() in Video.swift there is a check to compare the audio settings.
// Compare source audio settings with output to possibly skip the compression
let defaultSettings = CompressionAudioSettings()
if audioFormatID == codec.formatId,
audioSettings.bitrate == defaultSettings.bitrate,
!((audioFormatID == kAudioFormatMPEG4AAC || audioFormatID == kAudioFormatFLAC)
&& audioSettings.quality != defaultSettings.quality),
audioSettings.sampleRate == defaultSettings.sampleRate
{
variables.hasChanges = false
}The thing is, that the check for audioFormatID == kAudioFormatMPEG4AAC will force MediaToolSwift to re-encode the audio track, even the source is already aac but has a lower bit rate than audioSettings.bitrate.
Also, the check for audioSettings.bitrate == defaultSettings.bitrate will (almost every time) fail as default.bitrate is .auto and in most use cases a user will give a bitrate value.
For myself I use something like that as a work around:
// To avoid re-encoding if source has a lower audio bitrate than selected target audio bitrate, check it.
let sourceAudioBitrate = audioTrack!.estimatedDataRate
let dummySourceAudioSettings = CompressionAudioSettings(codec: audioSettings.codec,
bitrate: .value(Int(sourceAudioBitrate)),
quality: audioSettings.quality,
sampleRate: audioSettings.sampleRate)
if audioFormatID == codec.formatId,
dummySourceAudioSettings.bitrate <= audioSettings.bitrate,
audioSettings.quality == defaultSettings.quality,
audioSettings.sampleRate == defaultSettings.sampleRate
{
variables.hasChanges = false
sourceFormatHint = audioDescription
}Be aware, that sourceFormatHint = audioDescription is needed in my case.
In AudioBitrate.swift I added an Equatable for <=, too.
public static func <= (lhs: CompressionAudioBitrate, rhs: CompressionAudioBitrate) -> Bool {
switch (lhs, rhs) {
case (.auto, .auto):
return false
case (.value(let lhsValue), .value(let rhsValue)):
return lhsValue <= rhsValue
default:
return false
}
}Edit: correct check is dummySourceAudioSettings.bitrate <= audioSettings.bitrate,
I've been using this library, which is great btw, on my app and I noticed that whenever I try to compress an 1-minute, 4k video, it crashes due to running out of memory. I've discovered that it starts compressing, it reaches 60-70% and then it crashes.
Here's my implementation:
class MediaToolVideoCompressionService: VideoCompressor {
private var task: CompressionTask?
func compressVideo(url: URL,
outputURL: @escaping (URL?) -> Void) {
task = nil
Task {
let outputTempURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("tempMovieVideo\(Date().timeIntervalSince1970)").appendingPathExtension("mp4")
task = await VideoTool.convert(
source: url,
destination: outputTempURL,
fileType: .mp4,
videoSettings: .init(
codec: .h264,
bitrate: .value(3 * 1_000_000),
size: .init(width: 1280, height: 720)
),
optimizeForNetworkUse: true,
skipSourceMetadata: false,
customMetadata: [],
copyExtendedFileMetadata: true,
overwrite: false,
deleteSourceFile: false,
callback: { state in
switch state {
case .started:
print("\(Date()) MediaToolVideoCompressionService Started")
case .progress(let progress):
print("Progress MediaToolVideoCompressionService: \(progress.fractionCompleted)")
case .completed(let info):
outputURL(info.url)
print("\(Date()) Done MediaToolVideoCompressionService: \(info.url.path)")
case .failed(let error):
print("\(Date()) Error MediaToolVideoCompressionService: \(error.localizedDescription)")
outputURL(url)
case .cancelled:
outputTempURL.deleteFile()
outputURL(nil)
print("\(Date()) Cancelled MediaToolVideoCompressionService")
}
})
}
}
func cancel() {
task?.cancel()
}
}
Am I doing anything wrong?
Thank you for this wonderful tool to solve video conversion problem.
I was using the tool with older version since few months and now updated to 1.2.0. Changes are done in my code as per new method signatures and properties. As per the latest release notes, .progress() state is removed so using progress property of CompressionTask.
(1) When I try to get the progress status when conversion is going on, it does not give any info about the progress. Here is my code for better understanding.
Task {
let task = await VideoTool.convert(source: source, destination: destination, fileType: .mp4, videoSettings: videoSettings, overwrite: true, callback: { [weak self] state in
guard let self = self else { return }
switch state {
case .started:
debugLog("Started")
case .completed(let mediaInfo):
debugLog("Done: \(mediaInfo.url.absoluteString)")
case .failed(let error):
if let error = error as? MediaToolSwift.CompressionError {
debugLog("Error: \(error.description)")
} else {
debugLog("Error: \(error.localizedDescription)")
}
case .cancelled:
debugLog("Cancelled")
}
})
_ = task.progress.observe(\.fractionCompleted) { progress, _ in
debugLog("Progress: \(progress.fractionCompleted)")
}
}
}
(2) I get below warning at the first line of code where task is initialised.
"Passing argument of non-sendable type '[AVMetadataItem]' outside of main actor-isolated context may introduce data races"
Sorry for raising an issue to ask this question... do you intend to support video composition, overlaying videos?
If I try to scale a video in portrait mode, the resolution of the encoded video is a little bit weird.
I import a video with a resolution of 2160x3840 and set 1280x720 as the target resolution. My expectation is that the encoded video has a resolution of 720x1280. Instead I get 405x720.
In the initVideo() method of Video.swift there's a check for resolution changes.
if let size = videoSettings.size,
videoSize.width > size.width || videoSize.height > size.height {
let rect = AVMakeRect(aspectRatio: videoSize, insideRect: CGRect(origin: CGPoint.zero, size: size))
videoSize = rect.size
}
My workaround: Right before the code above I switch the dimension of videoSettings.size
var videoSettings = videoSettings
if let size = videoSettings.size,
videoSize.width < videoSize.height {
videoSettings.size = CGSize(width: size.height, height: size.width)
}
Result: I import a video with a resolution of 2160x3840 and set 1280x720 as the target resolution. The encoded video has a resolution of 720x1280.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.