Can this be used to encode to allow for later playback? about anime4k

[bloc97]

There will be a python/tensorflow version coming soon. You will be able to use ffmpeg-python to encode video if you wish. (Even if it goes against the nature of Anime4K)

[cladoxylopsidus]

Are there any updates on this yet?

[bloc97]

I can't promise a deadline due to the current circumstances, but I think I should at least be able to start working on a Tensorflow/Python port now that v3.0 is released.

Edit: On a different note, you should be able to achieve similar if not even better results using the already available AVISynth filters and the numerous waifu2x encoders out there. But of course, they are pretty slow and not suitable for real-time use.

[arbrog]

Bump. I tried using mpv's record-file option but it looks like it just outputs the original input file resolution not the upscaled post shader player resolution. Does anyone know any tools for rendering videos with shaders? One i found that might work is https://github.com/polyfloyd/shady (is there a meta tag for issues?)

[arbrog]

Awesome! Yeah I know it's against the whole point but if I want to watch on my Ipad, etc (using plex) I don't know of any way to apply the shader live on those platforms.

[arces]

Thank you @bloc97 appreciate it :)

[ccicnce113424]

(The following content was translated from Chinese to English using DeepL)
The HLSL version of this filter can be loaded in AviSynth using the AviSynthShader. Unfortunately, the HLSL version is not the latest version and has stopped updating.
Example

[arces]

Any update @bloc97 . I am sure you are busy and don't want to bug you but thought it would be good to check in a year later and see if there is any update. Thank you!

[bloc97]

If I remember correctly it is already possible to encode video using mpv on linux after applying shaders.
Futhermore, I can release the uncleaned tensorflow code/model, but the line thinning and other non-ML shaders are not implemented in python.

I had planned to release the tensorflow model much earlier but I had no time and my training machine broke. I only recently obtained a personal machine for training and am in the process of updating the code to Tensorflow 2.5 (the old code was for tf 1.15).

[arces]

I have a 3090, 64gb ram, and an i7-9770k happy to train the model if you need.

[bloc97]

@arces I really appreciate your offer! My machine is working, albeit not as powerful as yours... However, even the biggest Anime4K network is very lightweight (under 20k parameters) and trains pretty much on anything. Training time is also very short by using sota training methods such as adaptive momentum, cyclical learning rates and super-convergence. It takes less than 2 hours for successful training for the UL version, and less than 10 minutes for the M version to converge.
The real issue right now is the code is extremely messy and not worthy of being published... The script for TF model -> GLSL conversion is also hardcoded and won't work if the network architecture is modified.

Edit: I'm working on cleaning up the code and allowing it to be run within a standard tf-gpu docker container.

[barneymaydance]

Hi @bloc97 , thanks for your excellent work. I tried it and it is really fast. I want to train the same model by my dataset. Can you please release the model structure of M version? Model structure is enough. Thanks!

[bloc97]

@barneymaydance The M version is tiny. It's simply five 3x3 Conv2D layers (4 features wide) with CReLU activation followed by a Depth to Space upsampling layer. The network predicts residuals instead of the full image, just like in VDSR (CVPR16).
Note that in v3.1 the M variant uses a concatenate-and-reduce dense layer at the end, similar to what is used in DenseNet (CVPR17), but further experiments showed that it is not beneficial for x2 upscaling with a very shallow network and can be removed. The dense layer do help for x2, x3 and x4 upscaling when using a deeper network (6+ layers). I've kept it just for consistency across versions (easier to debug and to train in batch)...

[bloc97]

@barneymaydance Here's the snippet of code that I used

import tensorflow as tf

def SR2Model(input_channels=1, features=4, block_depth=4):

    input_shape = [None, None, input_channels]
    input_lr = tf.keras.layers.Input(shape=input_shape)
    upsampled_lr = tf.keras.layers.UpSampling2D(size=(2, 2), interpolation='bilinear')(input_lr)
    
    x = input_lr
    for i in range(block_depth):
        x = tf.keras.layers.Conv2D(features, (3, 3), padding='same', kernel_initializer='he_normal')(x)
        x = tf.nn.crelu(x)
    
    x = tf.keras.layers.Conv2D(input_channels*4, (3, 3), padding='same', kernel_initializer=tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.001))(x)
    x = tf.nn.depth_to_space(x, 2)
    x = tf.keras.layers.Add()([x, upsampled_lr])

    model = tf.keras.models.Model(input_lr, x)
    model.summary(line_length=150)

    return model

[feanor3]

the python version for encoding will be aviable for windows?