fileonq / imaging.heif Goto Github PK
View Code? Open in Web Editor NEWA C#/.NET wrapper around libheif for decoding and processing high efficiency image formats (heif, heic).
License: GNU Lesser General Public License v3.0
A C#/.NET wrapper around libheif for decoding and processing high efficiency image formats (heif, heic).
License: GNU Lesser General Public License v3.0
In #32 and #31 we document the development side of building the automation for release tagging. We now need to create a readme file that documents our release process which includes docs on the following
Add a benchmarking project that validates the performance of the application. This should have standard benchmarks for all documented API workflows.
By: @ahoefling
The libjpef-turbo
dependency is being compiled in debug mode which uses the Visual C Runtime, this makes the entire library dependent on debug binaries that are NOT included in the Visual C++ Redistributable package. We need to ensure all projects in this library are compiled in release mode.
The Library is throwing a DLL not found error. This appears to be due to SetDLLDirectory not working correctly. We should switch over to using AddDLLDirectory instead.
The windows binaries depend on the Visual C++ Redistributable being installed and the hosted GitHub Runner's come pre-installed with a full Visual Studio development environment. That environment has the redistributable installed as well as debug binaries. When running integration tests we may get false positives as the binaries will execute without issues but when used in a downstream project on another machine without visual studio it will fail.
This was first identified as part of #37
We need to implement an integration build action that will run on a clean VM of windows and complete the following steps
When a community member submits a pull request, they will most likely fork this repository and then submit the PR. Right now the benchmarks fail to checkout code correctly from the forked repo. This is due to how the ref
is piped from the first build to the next. It assumes that it is retrieving the ref from the origin repository and doesn't take into account forked repositories.
The benchmark.yml
build checks if it is a new PR or been requested to run via the check-command
job. It will then store the current git ref
and pass that value to the next job benchmark-build
. This job will try and checkout the code and compile, but it assumes that it is checking out the origin repo. If the PR is a forked repository then the checkout command will fail.
Fetching the repository
"C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1
Waiting 14 seconds before trying again
"C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1
Waiting 13 seconds before trying again
"C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
Error: The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1
Update the benchmark.yml
to work with forked repositories
In both .NET 5 and .NET 6 benchmarks the ToSpan
has extra allocations that don't make sense. This API wraps the native pointer from the encoders project and it shouldn't be creating any additional allocations. See benchmark table below.
The benchmarks have been ran on different machines the important value is the memory allocation
Method | Mean | Error | StdDev | Allocated native memory | Native memory leak | Allocated |
---|---|---|---|---|---|---|
Thumbnail_ToSpan (NET 5) | 58.68 ms | 1.150 ms | 1.889 ms | 5,123,597 B | - | 120 B |
PrimaryImage_ToSpan (NET 5) | 2.981 s | 0.0252 s | 0.0236 s | 222,028,216 B | - | 88 B |
Thumbnail_ToSpan (NET 6) | 47.54 ms | 0.425 ms | 0.398 ms | 5,123,853 B | - | 600 B |
PrimaryImage_ToSpan (NET 6) | 2.968 s | 0.0271 s | 0.0241 s | 222,029,080 B | - | 616 B |
Update the NuGet package properties to have all of the correct data for the resulting NuGet package. This includes description, summary, tags, etc.
The DLLs generated are not marked as deterministic when viewing in the NuGet Package Explorer. We need to update the build so they are deterministic.
Update all public APIs in FileOnQ.Imaging.Heif
to have proper xml docs. Verify the docs show up using the NuGet package.
After implementing automated benchmarking it documented there are large memory leaks with the current implementation. The data is available in #20 as well as the table copied below
BenchmarkDotNet=v0.13.0, OS=Windows 10.0.17763.2114 (1809/October2018Update/Redstone5)
Intel Xeon Platinum 8171M CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK=5.0.400
[Host] : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
Job-VLDWLB : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
Runtime=.NET 5.0 InvocationCount=1 LaunchCount=1
UnrollFactor=1
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
---|---|---|---|---|---|---|---|---|---|
Thumbnail_Write | 60.91 ms | 1.208 ms | 2.269 ms | - | - | - | 112 B | 4,937,059 B | 295,469 B |
BenchmarkDotNet=v0.13.0, OS=Windows 10.0.17763.2114 (1809/October2018Update/Redstone5)
Intel Xeon Platinum 8171M CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK=5.0.400
[Host] : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
Job-DXULZI : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
Runtime=.NET 5.0 InvocationCount=1 LaunchCount=1
UnrollFactor=1
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
---|---|---|---|---|---|---|---|---|---|
PrimaryImage_Write | 2.947 s | 0.0303 s | 0.0268 s | - | - | - | 128 B | 210,038,978 B | 18,048,557 B |
We need to ensure all native memory is freed so these result in 0 native memory leaked
When decoding and writing images to jpeg libheif performs a conversion from the native colorspace and chroma to the expected YUV (colorspace) and 420 (chroma). This operation takes about .5s - 1.2s in debug mode. If we detect this prior to attempting a decode and keep it in the same format we may be able to shave time off our image processing. This will require updating our jpeg encoder to handle a variety of colorspace/chromas and have a fallback strategy
Currently the benchmarks project is only targeting .net5. This needs to support 4.8, .net5, and .net6.
There is a very small memory leak with .NET 6 when writing an image to storage. See benchmarking results below
BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19043.1415 (21H1/May2021Update)
AMD Ryzen 9 3950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK=6.0.100
[Host] : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Job-KBXXVN : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Runtime=.NET 6.0 InvocationCount=1 LaunchCount=1
UnrollFactor=1
| Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
|------------------- |---------:|---------:|---------:|------:|------:|------:|----------:|------------------------:|-------------------:|
| Thumbnail_Write | 65.39 ms | 0.928 ms | 0.868 ms | - | - | - | 832 B | 5,982,865 B | 32 B |
| Thumbnail_ToArray | 63.64 ms | 0.366 ms | 0.324 ms | - | - | - | 66,888 B | 5,982,157 B | - |
| Thumbnail_ToSpan | 63.72 ms | 0.403 ms | 0.377 ms | - | - | - | 600 B | 5,981,901 B | - |
| Thumbnail_ToStream | 63.90 ms | 0.388 ms | 0.363 ms | - | - | - | 66,952 B | 5,981,901 B | - |
Benchmarks only support .NET 5 and need to print data for all supported platforms. In #62 we identified a memory leak in .NET Framework that was pre-existing and it is tracked in #65. With proper benchmarking metrics this could have been caught much earlier in the release cycle
Add ability to resize the primary image or thumbnail using the libheif native resize api.
IImage
void Resize(int width, int height); // New API
void Resize(int widthAndHeight); // New API
The code below would generate a 200x200 square thumbnail. This use case is valuable for heic images that don't have an embedded thumbnail.
using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
primary.Resize(200, 200);
primary.Save("Output.jpeg", 90);
}
Compile project for use on Android devices
Update build to sign all FileOnQ generated assemblies from this project.
Both the PR build and the Main build will require code signing of our assemblies.
Add documentation for minimum versions of Visual Studio and dotnet
globals.json
to force dotnet to specific version versionThe build process compiles libjpeg-turbo in debug mode instead of release mode. This generates a file that is roughly 600kb larger than if it was built in release mode. It is a best practice to try and compile upstream libraries in release mode as the code is optimized.
During the initial proof of concept the JPEG encoder that users libjpeg-turbo was failing to save. We were seeing errors when writing to disk. After investigation it was determined that we are going to have to use debug mode for now and try and fix this later
The snippet below is from the original proof of concept and will change in the future
public HeifImage(string file)
{
heifContext = LibHeifContext.heif_context_alloc();
var error = LibHeifContext.heif_context_read_from_file(heifContext, file, IntPtr.Zero);
if (error.Code != LibHeifContext.ErrorCode.Ok)
throw new Exception(Marshal.PtrToStringAnsi(error.Message));
LibHeifContext.ImageHandle* imageHandle;
var imageError = LibHeifContext.heif_context_get_primary_image_handle(heifContext, &imageHandle);
var numberOfThumbnails = LibHeifContext.heif_image_handle_get_number_of_thumbnails(imageHandle);
if (numberOfThumbnails > 0)
{
var itemIds = new uint[numberOfThumbnails];
fixed (uint* ptr = itemIds)
{
LibHeifContext.heif_image_handle_get_list_of_thumbnail_IDs(imageHandle, ptr, numberOfThumbnails);
}
// no idea why this is failing
LibHeifContext.ImageHandle* thumbHandle;
var thumbError = LibHeifContext.heif_image_handle_get_thumbnail(imageHandle, itemIds[0], &thumbHandle);
if (thumbError.Code != LibHeifContext.ErrorCode.Ok)
throw new Exception(Marshal.PtrToStringAnsi(thumbError.Message));
Encode(thumbHandle);
}
else
{
Encode(imageHandle);
}
void Encode(LibHeifContext.ImageHandle* handle)
{
var hasAlpha = LibHeifContext.heif_image_handle_has_alpha_channel(handle) == 1;
var encoder = LibEncoder.encoder_jpeg_init(90);
var options = LibHeifContext.heif_decoding_options_alloc();
LibEncoder.encoder_update_decoding_options(encoder, handle, options);
var bitDepth = LibHeifContext.heif_image_handle_get_luma_bits_per_pixel(handle);
if (bitDepth < 0)
{
LibHeifContext.heif_decoding_options_free(options);
LibHeifContext.heif_image_handle_release(handle);
throw new Exception("Input image has undefined bit-dept");
}
LibHeifContext.Image* outputImage;
var decodeError = LibHeifContext.heif_decode_image(
handle,
&outputImage,
LibEncoder.encoder_colorspace(encoder, hasAlpha),
LibEncoder.encoder_chroma(encoder, hasAlpha, bitDepth),
options);
LibHeifContext.heif_decoding_options_free(options);
if (decodeError.Code != LibHeifContext.ErrorCode.Ok)
{
LibHeifContext.heif_image_handle_release(handle);
throw new Exception(Marshal.PtrToStringAnsi(decodeError.Message));
}
if ((IntPtr)outputImage != IntPtr.Zero)
{
bool saved = LibEncoder.encode(encoder, handle, outputImage, "output.jpeg");
if (!saved)
throw new Exception("Unable to save");
}
LibEncoder.encoder_free(encoder);
}
}
The libjpeg-turbo
code fails on LibEncoder.encode(encoder, handle, outputImage, "output.jpeg")
when the library is built under release mode but works when the library is built under debug mode
The build system requires visual studio 2019 enterprise but should work with pro and community. Let's update the build process to fall back to other versions if enterprise is not installed
When using multiple FileOnQ imaging libraries in the same downstream project there are naming conflicts with build targets for net48. This is because both FileOnQ.Imaging.Heif
and FileOnQ.Imaging.Raw
use the same build <Target>
name of CopyNativeDlls
.
The conflict prevents native assemblies from being copied to the bin directory correctly. This will create a silent error as the build will not fail and the application will have a runtime error.
Make the build target names unique
Compile project for use in Web Assembly (WASM) projects. We should test this with Blazor and Uno Platform WASM
Compile project for use on iOS devices
Add new GitHub Actions to release final NuGets based on git tagging strategy
Tag | NuGet Version |
---|---|
v1.0.0 |
1.0.0 |
The x265 dependency is only used for encoding and at this point FileOnQ.Imaging.Heif
is only interested in decoding. That library is licensed under GPL and is not compatible with LGPL. In the future if we want to add support for encoding, we will need to revisit the license and how we can include it.
Remove x265 from the library and docs
It was discovered that we are seeing a memory leak in FileOnQ.Imaging.Heif when targeting .NET 4.8. Both the thumbnail (234B) and primary image (234B) are impacted. Below is the benchmarks.
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
---|---|---|---|---|---|---|---|---|---|
Thumbnail_Write | 29.09 ms | 0.211 ms | 0.187 ms | - | - | - | 74,504 B | 5,125,345 B | 234 B |
Thumbnail_ToArray | 28.79 ms | 0.171 ms | 0.160 ms | - | - | - | 74,504 B | 5,124,925 B | 234 B |
Thumbnail_ToSpan | 29.12 ms | 0.258 ms | 0.215 ms | - | - | - | - | 5,124,909 B | 234 B |
Thumbnail_ToStream | 29.11 ms | 0.366 ms | 0.343 ms | - | - | - | 140,816 B | 5,124,925 B | 234 B |
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
---|---|---|---|---|---|---|---|---|---|
PrimaryImage_Write | 1.342 s | 0.0043 s | 0.0038 s | - | - | - | 1,951,112 B | 222,030,860 B | 234 B |
PrimaryImage_ToArray | 1.343 s | 0.0031 s | 0.0028 s | - | - | - | 1,951,112 B | 222,030,552 B | 234 B |
PrimaryImage_ToSpan | 1.342 s | 0.0057 s | 0.0053 s | - | - | - | - | 222,029,928 B | 234 B |
PrimaryImage_ToStream | 1.341 s | 0.0043 s | 0.0038 s | - | - | - | 3,894,032 B | 222,030,392 B | 234 B |
Add memory buffer APIs that can return an array or a stream
IImage
byte[] ToArray(int quality = 90)
Stream AsStream(int quality = 90)
ReadOnlySpan<byte> AsReadOnlySpan(int quality = 90)
byte[] buffer;
using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
buffer = primary.ToArray();
}
// do something with buffer
Stream stream;
using (var image = new Heifimage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
stream = primary.AsStream();
}
// do something with stream
// don't forget to dispose of your stream
stream?.Dispose();
This one is a little tricky as Span<T>
gives us access to native memory. If we allow C# to operate on native memory we should ensure it happens within the IDisposable
using block otherwise there will be a memory leak.
ReadOnlySpan<byte> span;
using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
ReadOnlySpan<byte> span = primary.AsReadOnlySpan();
// do something with span
}
Add the entire .github folder to the solution so it is accessible from within Visual Studio. As new files are added we will need to add it to the solution manually
Currently the library only contains a console application for testing. This is great for developers. However to support non-developer testers, we need to have two WinForms apps for testing purposes. One targeting .NET 4.8 and another targeting .NET 5.0.
Add a baseline benchmark to each printed table. This will show the existing benchmark compared to the PR benchmark side-by-side. The data is saved in the repository, so we just need to extract it and merge it with the existing table
Add support for latest build in .NET 6 and update all .NET 5 builds to latest stable patch.
We added the Interop
API which updates the dll search directory for .NET Framework to match behaviors in .NET5+. This means we no longer need to have explicit x86 vs x64 interop classes. They all reference the exact same assembly name. Let's remove the duplicate file but keep the native interop wrapper. See code snippet example below.
internal static IntPtr InitJpegEncoder(int quality)
{
switch (RuntimeInformation.ProcessArchitecture)
{
case Architecture.X64:
case Architecture.X86:
return native.encoder_jpeg_init(quality);
default:
throw new NotSupportedException($"Current platform ({RuntimeInformation.ProcessArchitecture}) is not supported");
}
}
When using the generated NuGet FileOnQ.Imaging.Heif.1.0.0-pr.20
I am unable to run the Save()
code as it fails with a VC++ Runtime Library error. See screenshot below
When I run the build locally it works without issue, it only fails when using the generated artifacts from the build.
The build.yml
follows a similar set of instructions I am doing locally.
FileOnQ.Imaging.Heif
this will also compile all native binariesThe console app will save correctly. If you swap the NuGet out for the generated one it will fail on the save.
I am working on putting together getting started docs to make is easier to get up and running. This hasn't been merged into the main
branch and can be found in the docs
branch.
This project is a complex mixture of .NET Framework, .NET and native C/C++. We should add a developer guide that will explain the different areas of the project which will help contributors understand how to use the solution and make changes.
For example the unit test project automatically builds a nupkg which allows debugging. This developer guide will mention it is a best practice to do a clean and rebuild on the unit test project so you can step into the apis and debug.
Add thumbnail count API to determine how many thumbnail images if any are embedded in the heic image
IHeifImage
// Gets the number of thumbnails embedded in the heif file
// this is typically 0 or 1
int GetThumbnailCount(); // New API
// Gets an array of all available thumbnail ids, if 0
// then the image doesn't have any thumbnails
int[] GetThumbnailIds(); // New API
// Gets the thumbnail at the specified ID
IImage Thumbnail(int id); // New API
The code snippet below will write all thumbnails to disk
using (var image = new HeifImage("MyImage.heic"))
{
int[] thumbnailIds = image.GetThumbnailIds();
for (int i = 0; i , thumbnailIds.Length; i++)
{
using (var thumbnail = image.GetThumbnail(thumbnailIds[i]))
{
thumbnail.Write($"output_{i}.jpeg", 90);
}
}
}
Add native code debugging to simplify debugging between the interop of managed code and native code. This will allow a developer to step into any of the c++ code or 3rd party libraries.
Add automated release notes to be generated as part of the git release tag build from #31. Using github-release-notes we will add all merged pull requests and links to the release notes. At this point in time we aren't as organized with issues so we only care about merged pull requests
Compile project for use on Windows ARM64
During the build process it runs several chocolatey install steps to ensure that all C++ build dependencies are installed. If this step fails it doesn't automatically fail the build.
choco install meson
choco install ninja
choco install nasm
We need to update this script to fail if there is an issue downloading deps
The benchmark.yml
hooks into issue comments which are used in both PRs and Issues, this build is only useful from a PR. We need to update the build to detect if it is running on an issue and stop it from processing. See build log below
https://github.com/FileOnQ/Imaging.Heif/runs/4795761106?check_suite_focus=true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.