Coder Social home page Coder Social logo

unityasyncimageloader's Introduction

Unity Asynchronous Image Loader

ImageConversion.LoadImage and Texture2D.LoadImage are slow when loading large images (greater than 2K) at runtime. They blocks the Unity main thread when loading the image for a duration between a hundred milliseconds and even a few seconds. This is a dealbreaker for those games and applications that want to load those images programmatically at runtime.

This package aims to offload image loading, image decoding and mipmap generation to other threads. It creates smoother gameplay and reduces lag spike on the Unity main thread when loading large images.

This package uses FreeImage. which is the same library used by Unity to process image data.

Unity Version

This package is developed in Unity 2019.1. It may work on Unity 2020 and Unity 2021.

Installation

The package can be installed using the Git URL https://github.com/Looooong/UnityAsyncImageLoader.git by following Installing from a Git URL instructions.

Dependencies

  • Unity Burst
  • Unity Mathematics

Usage

Loader Settings

  /// <summary>Settings used by the image loader.</summary>
  public struct LoaderSettings {
    /// <summary>Create linear texture. Only applicable to methods that create new <c>Texture2D</c>. Defaults to false.</summary>
    public bool linear;
    /// <summary>Texture data won't be readable on the CPU after loading. Defaults to false.</summary>
    public bool markNonReadable;
    /// <summary>Whether or not to generate mipmaps. Defaults to true.</summary>
    public bool generateMipmap;
    /// <summary>Automatically calculate the number of mipmap levels. Defaults to true. Only applicable to methods that create new <c>Texture2D</c>.</summary>
    public bool autoMipmapCount;
    /// <summary>Mipmap count, including the base level. Must be greater than 1. Only applicable to methods that create new <c>Texture2D</c>.</summary>
    public int mipmapCount;
    /// <summary>Used to explicitly specify the image format. Defaults to FIF_UNKNOWN, which the image format will be automatically determined.</summary>
    public FreeImage.Format format;
    /// <summary>Whether or not to log exception caught by this method. Defaults to true.</summary>
    public bool logException;

    public static LoaderSettings Default => new LoaderSettings {
      linear = false,
      markNonReadable = false,
      generateMipmap = true,
      autoMipmapCount = true,
      format = FreeImage.Format.FIF_UNKNOWN,
      logException = true,
    };
  }

Load Image Asynchronously

  var imageData = File.ReadAllBytes();
  var texture = new Texture2D(1, 1);
  var loaderSettings = AsyncImageLoader.LoaderSettings.Default;
  var success = false;

  // =====================================
  // Load image data into existing texture
  // =====================================

  // Use the default LoaderSettings
  success = await AsyncImageLoader.LoadImageAsync(texture, imageData);

  // Similar to ImageConversion.LoadImage
  // Mark texture as unreadable after reading.
  success = await AsyncImageLoader.LoadImageAsync(texture, imageData, true);

  // Use a custom LoaderSettings
  success = await AsyncImageLoader.LoadImageAsync(texture, imageData, loaderSettings);

  // ==================================
  // Create new texture from image data
  // ==================================

  // Use the default LoaderSettings
  texture = await AsyncImageLoader.CreateFromImageAsync(imageData);

  // Use a custom LoaderSettings
  texture = await AsyncImageLoader.CreateFromImageAsync(imageData, loaderSettings);

Load Image Synchronously

The synchronous variants are the same as the asynchronous counterparts but without Async suffix in theirs name. They are useful for debugging and profiling within a frame.

  var imageData = File.ReadAllBytes();
  var texture = new Texture2D(1, 1);
  var loaderSettings = AsyncImageLoader.LoaderSettings.Default;
  var success = false;

  // =====================================
  // Load image data into existing texture
  // =====================================

  // Use the default LoaderSettings
  success = AsyncImageLoader.LoadImage(texture, imageData);

  // Similar to ImageConversion.LoadImage
  // Mark texture as unreadable after reading.
  success = AsyncImageLoader.LoadImage(texture, imageData, true);

  // Use a custom LoaderSettings
  success = AsyncImageLoader.LoadImage(texture, imageData, loaderSettings);

  // ==================================
  // Create new texture from image data
  // ==================================

  // Use the default LoaderSettings
  texture = AsyncImageLoader.CreateFromImage(imageData);

  // Use a custom LoaderSettings
  texture = AsyncImageLoader.CreateFromImage(imageData, loaderSettings);

After Loading

Texture Format

If the image has alpha channel, the format will be RGBA32, otherwise, it will be RGB24.

Mipmap Count

If the LoadImage and LoadImageAsync variants are used with generateMipmap set to true, the mipmap count is set to the maximum possible number for a particular texture. If you want to control the number of mipmap, you can use the CreateFromImage and CreateFromImageAsync instead.

Mipmap Data

The mipmaps are generated using box filtering with 2x2 kernel. The final result won't be the same as Unity's counterpart when using texture import in the editor.

Troubleshooting

There is still lag spike when loading large images

After AsyncImageLoader method finishes executing, the image data are still transfering to the GPU. Therefore, any object, like material or UI, that wants to use the texture afterward will have to wait for the texture to finish uploading and thus block the Unity main thread.

There is no easy way to detect if the texture has finished uploading its data. The workarounds are:

  • Wait for a second or more before using the texture.
  • (Not tested) Use AsyncGPUReadback to request a single pixel from the texture. It will wait for the texture to finish uploading before downloading that single pixel. Then the request callback can be used to notify the Unity main thread about the texture upload completion.

Acknowledgement

This package is inspired by Matias Lavik's unity-async-textureimport.

unityasyncimageloader's People

Contributors

looooong avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

unityasyncimageloader's Issues

CreateFromImageAsync Texture2D does not work async / SupportsTextureFormatNative error

@Looooong I get the following error when calling CreateFromImageAsync from a thread:

UnityException: SupportsTextureFormatNative can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.

Unity 2020.3.1. AFAIK it's impossible to create a new Texture2D on anything but the main thread? Though that seems to render this whole repo useless, which I hope isn't the case cuz it looks GREAT otherwise!

Here's the full stack trace:

UnityException: SupportsTextureFormatNative can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.
Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.
UnityEngine.EditorSystemInfo.SupportsTextureFormat (UnityEngine.TextureFormat format) (at <10564ed154d647e194bef4aef8878649>:0)
UnityEngine.SystemInfoShimBase.SupportsTextureFormat (UnityEngine.TextureFormat format) (at <10564ed154d647e194bef4aef8878649>:0)
UnityEngine.SystemInfo.SupportsTextureFormat (UnityEngine.TextureFormat format) (at <10564ed154d647e194bef4aef8878649>:0)
UnityEngine.Texture.ValidateFormat (UnityEngine.TextureFormat format) (at <10564ed154d647e194bef4aef8878649>:0)
UnityEngine.Texture2D..ctor (System.Int32 width, System.Int32 height, UnityEngine.TextureFormat textureFormat, System.Int32 mipCount, System.Boolean linear, System.IntPtr nativeTex) (at <10564ed154d647e194bef4aef8878649>:0)
UnityEngine.Texture2D..ctor (System.Int32 width, System.Int32 height, UnityEngine.TextureFormat textureFormat, System.Int32 mipCount, System.Boolean linear) (at <10564ed154d647e194bef4aef8878649>:0)
AsyncImageLoader+ImageImporter+<CreateNewTextureAsync>d__17.MoveNext () (at Assets/UnityAsyncImageLoader/Runtime/ImageImporter.cs:100)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.TaskAwaiter`1[TResult].GetResult () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
AsyncImageLoader+<CreateFromImageAsync>d__9.MoveNext () (at Assets/UnityAsyncImageLoader/Runtime/AsyncImageLoader.cs:105)
UnityEngine.Debug:LogException(Exception)
<CreateFromImageAsync>d__9:MoveNext() (at Assets/UnityAsyncImageLoader/Runtime/AsyncImageLoader.cs:108)
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()

and the method I'm calling it from:

private static async Task<Texture2D> ByteArrayToTexture2D(byte[] bytes)
{
    return await AsyncImageLoader.CreateFromImageAsync(bytes);
}

webgl

i am running webgl, and found this error
DllNotFoundException: FreeImage assembly: type: member:(null)

does it work on webgl?

Change interface from byte[] to Memory<byte>

It would be great if the interface didn't take plain byte arrays. The reason being that you can't really allocate those easily from a block based system. Memory on the other hand is wrapper (like Span<>) around parts of memory that can be passed around between functions and stored in fields/properties. It supports pinning just like basic arrays do and byte[] should just be convertible to Memory so users of your library shouldn't be impacted by the API change.

Encode Texture2d Asynchronously

In your library, you implemented a solution to load image asynchronously from byte array.

Could you suggest a way for asynchronous encoding a texture to PNG or other formats?

MacOS

Missing libraries for MacOS. Using the precompiled ones from homebrew doesn't seem to work. Any guidance on where to obtain them and how to modify the scripts to get it to work?

Transparency issue vs using unitys native importer

For some reason after using this image loader into a texture when applied to a sprite has a little transparency across the entire image. I'm not sure if its the texture itself or the way unity utilizes it differently than using the built in editor texture importer. I don't know how to configure it in a way that stops this from happening.

Texture Blinking Issue

I'm using, await AsyncImageLoader.LoadImageAsync(tex, byteArray); and it loads the correct texture and is greatly increasing performance but the main issue is sometimes the texture loads as completely black for 1 frame and it happens multiple times every second causing the texture to blink. I'm not simply loading a static texture but instead loading a series of textures that play a video. Any thoughts on what might be causing this?

video of the issue:
https://user-images.githubusercontent.com/91104156/204570541-7a66809b-432d-443b-95a1-705793665cbe.mp4

LoadImageAsync fails to generate mipmaps

When trying to load an image using AsyncImageLoader.LoadImageAsync with default LoaderSettings, I get the following error:

"InvalidOperationException: The writeable UNKNOWN_OBJECT_TYPE FilterMipmapJob.outputMipmap is the same UNKNOWN_OBJECT_TYPE as FilterMipmapJob.inputMipmap, two containers may not be the same (aliasing)."

It seems to be related to the jobs that are creating the mipmaps.
I can workaround the issue by disabling mipmap generation.

Unity Version: 2021.2.0b4.3123 Personal

OpenEXR loads as full white image instead of its content. Is it supported ATM?

I am a bit confused about how to load OpenEXR Images or if it is supported. Currently I used either LoadImage or CreateFromImage either with automatic loader settings or specific, but the returned data is all white.

Texture size seem correct.

Also, I'm wondering about reading these values as RGBAFloat or RGBAHalf. Is it supported?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.