faustomorales / vit-keras Goto Github PK

I am able to save a model after training in .h5 format using save() method without a problem, but facing problems while loading the model.

tensorflow == 2.4.1
(working on GCP)
**Error : **
raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: ClassToken

model or weights can't be saved, getting this error RuntimeError: Unable to create link (name already exists) . I'm guessing two weights got the same name from this Link

import tensorflow as tf
inp = tf.keras.layers.Input(shape=(256,256,3))
base = vit.vit_b16(
    image_size=256,
    pretrained=True,
    include_top=False,
    pretrained_top=False,
)
x = base(inp)
x = tf.keras.layers.Dense(64,,activation='relu')(x)
x = tf.keras.layers.Dense(5,, activation='softmax')(x)
model = tf.keras.Model(inputs=inp, outputs=x)
opt = tf.keras.optimizers.Adam()
loss = tf.keras.losses.CategoricalCrossentropy()
model.compile(optimizer=opt,loss=loss,metrics=['categorical_accuracy'])
model.save('ViT.h5')

Hello,

When trying to run your code:

pip install vit-keras

from vit_keras import vit, utils
image_size = 384
classes = utils.get_imagenet_classes()
model = vit.vit_b16(image_size=image_size,activation='sigmoid',pretrained=True,include_top=True,pretrained_top=True)

I get the following error: NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow_addons/custom_ops/activations/_activation_ops.so: undefined symbol: _ZN10tensorflow14kernel_factory17OpKernelRegistrar12InitInternalEPKNS_9KernelDefEN4absl11string_viewESt10unique_ptrINS0_15OpKernelFactoryESt14default_deleteIS8_EE

This was ran on google colab with Tensorflow 2.4.0, Keras 2.4.3

Thanks for your great contribution!
I notice you offer the VIT npz format models. How could you convert official models from JAX to NPZ?

I want to convert the Deit official models and apply into your repo.
Thanks a lot again.

when i am trying to import the package on kaggle an error shows up saying
" ModuleNotFoundError: No module named 'keras.src.engine' "
please could you tell me how can i solve this ?

please

I want to see models by using load model weights for a known configuration (BASE_URL), as commented.
But this Page is Not Found https://github.com/faustomorales/vit-keras/releases/download/dl.

is there any issue?

So thank You for this implementation of VIT in Tensorflow. But whenever I Try to use this I get the Error
TypeError: Tensors in list passed to 'values' of 'ConcatV2' Op have types [float32, float16] that don't all match.
I Have tried setting all the variables to float32 but that also does not work . Please help me , here is my code
`

`
VIT.zip

Hi,
I have some very small images from an optical system (15x15 px), and realise I can get a huge performance boost by transfer learning from pretrained weights.

Is the minimum size 128x128? Or can smaller sizes be used?

Hello,
First of all, thank you for this nice re-implementation in keras with the pretrained weights available.

It is just to raise an issue when using mixed precision , it seems to be related to some casting.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(op_type_name, name, **keywords)
    437               preferred_dtype=default_dtype,
--> 438               as_ref=input_arg.is_ref)
    439           if input_arg.number_attr and len(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in internal_convert_n_to_tensor(values, dtype, name, as_ref, preferred_dtype, ctx)
   1607             preferred_dtype=preferred_dtype,
-> 1608             ctx=ctx))
   1609   return ret

/opt/conda/lib/python3.7/site-packages/tensorflow/python/profiler/trace.py in wrapped(*args, **kwargs)
    162           return func(*args, **kwargs)
--> 163       return func(*args, **kwargs)
    164 

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
   1508           "Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
-> 1509           (dtype.name, value.dtype.name, value))
   1510     return value

ValueError: Tensor conversion requested dtype float32 for Tensor with dtype bfloat16: <tf.Tensor 'Placeholder:0' shape=(None, 576, 768) dtype=bfloat16>

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-41-d11fa3fc8440> in <module>
     35         K.clear_session()
     36         with strategy.scope():
---> 37             model = compile_new_model_vit(cfg, cfg['name'][num_model], with_sgd=False)
     38 
     39 

<ipython-input-38-6c20ae4b4197> in compile_new_model_vit(cfg, name, with_sgd)
     13 def compile_new_model_vit(cfg, name, with_sgd=False):
     14     with strategy.scope():
---> 15         model = get_model_vit(cfg, name)
     16 
     17         losses = CategoricalCrossentropyCustom(label_smoothing = cfg['smoothing'])

<ipython-input-38-6c20ae4b4197> in get_model_vit(cfg, name)
      2     model_input = tf.keras.Input(shape=(cfg['resize'][0], cfg['resize'][1], 3), name='inputs')
      3     image_size = 384
----> 4     x = vit.vit_b16(image_size=image_size,activation='sigmoid', pretrained=True, include_top=False, pretrained_top=False)(model_input)
      5     #x1 = tf.keras.layers.GlobalAveragePooling2D(name='avg_pool')(x)
      6     #x2 = tf.keras.layers.GlobalMaxPooling2D(name='max_pool')(x)

/opt/conda/lib/python3.7/site-packages/vit_keras/vit.py in vit_b16(image_size, classes, activation, include_top, pretrained, pretrained_top, weights)
    146         activation=activation,
    147         include_top=include_top,
--> 148         representation_size=768 if weights == "imagenet21k" else None,
    149     )
    150     if pretrained:

/opt/conda/lib/python3.7/site-packages/vit_keras/vit.py in build_model(image_size, patch_size, num_layers, hidden_size, num_heads, name, mlp_dim, classes, dropout, activation, include_top, representation_size)
     75     )(x)
     76     y = tf.keras.layers.Reshape((y.shape[1]*y.shape[2], hidden_size))(y)
---> 77     y = layers.ClassToken(name="class_token")(y)
     78     y = layers.AddPositionEmbs(name="Transformer/posembed_input")(y)
     79     for n in range(num_layers):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    950     if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
    951       return self._functional_construction_call(inputs, args, kwargs,
--> 952                                                 input_list)
    953 
    954     # Maintains info about the `Layer.call` stack.

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
   1089         # Check input assumptions set after layer building, e.g. input shape.
   1090         outputs = self._keras_tensor_symbolic_call(
-> 1091             inputs, input_masks, args, kwargs)
   1092 
   1093         if outputs is None:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
    820       return nest.map_structure(keras_tensor.KerasTensor, output_signature)
    821     else:
--> 822       return self._infer_output_signature(inputs, args, kwargs, input_masks)
    823 
    824   def _infer_output_signature(self, inputs, args, kwargs, input_masks):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
    861           # TODO(kaftan): do we maybe_build here, or have we already done it?
    862           self._maybe_build(inputs)
--> 863           outputs = call_fn(inputs, *args, **kwargs)
    864 
    865         self._handle_activity_regularization(inputs, outputs)

/opt/conda/lib/python3.7/site-packages/vit_keras/layers.py in call(self, inputs)
     20         batch_size = tf.shape(inputs)[0]
     21         cls_broadcasted = tf.broadcast_to(self.cls, [batch_size, 1, self.hidden_size])
---> 22         return tf.concat([cls_broadcasted, inputs], 1)
     23 
     24 

/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py in concat(values, axis, name)
   1675           dtype=dtypes.int32).get_shape().assert_has_rank(0)
   1676       return identity(values[0], name=name)
-> 1677   return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
   1678 
   1679 

/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py in concat_v2(values, axis, name)
   1206   _attr_N = len(values)
   1207   _, _, _op, _outputs = _op_def_library._apply_op_helper(
-> 1208         "ConcatV2", values=values, axis=axis, name=name)
   1209   _result = _outputs[:]
   1210   if _execute.must_record_gradient():

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(op_type_name, name, **keywords)
    464                               (prefix, dtype.name))
    465             else:
--> 466               raise TypeError("%s that don't all match." % prefix)
    467           else:
    468             raise TypeError(

TypeError: Tensors in list passed to 'values' of 'ConcatV2' Op have types [float32, bfloat16] that don't all match.

I found the image_size must be int. it means the picture has the same height and width? But how to handle the picture.shape = (620, 32)?

I finetuned the vit_b16 model with new data and saved the new model by "save".
When I load this retrained model, ending up with "TypeError: call () missing 1 required positional argument: 'training' " .
Any suggestions? Thanks!

Exception: URL fetch failure on https://github.com/faustomorales/vit-keras/releases/download/dl/ViT-L_16_imagenet21k+imagenet2012.npz: 404 -- Not Found

Hi, i was trying to understand, what's the meaning of the "FlexErf" node? is it related to the tf.math.erf used by tensorflow to Computes the Gauss error?

Many thanks!!

@faustomorales

Good morning,

I'm using your model with fine-tuning but when I run my program in the line from vit_keras import vit it shows the error message ModuleNotFoundError: No module named 'keras.src.engine'. My tensorflow version is 2.16.1 and Keras 3.1.1. Should I install other versions of these libraries or what is it the problem?

Thank you in advance.

As of TF 2.4.0, loading a pretrained model from the numpy weights results in AssertionError: Transformer/encoderblock_0/MlpBlock_3/Dense_0/kernel:0 not in output weights..

This is because of a breaking change in TF 2.4.0 that makes it so we can no longer rely on the same weights namespacing at the top level of a model. From the release notes.

Code that is overly dependent on the exact names attached to symbolic tensors (e.g. assumes there will be ":0" at the end of the inputs, treats names as unique identifiers instead of using tensor.ref(), etc.) may break.

I probably shouldn't have relied so heavily on the naming the first place, but this will require significant changes to vit_keras.utils.load_weights_numpy.

Do you have any idea if it is possible to generate an attention map per predicted output class?
As an example if there is a dog and a cat in the image I'd like to show the attention given to dog and one given to cat.

Can we use it for grayscale images?

Hi,

Currently, the vit-keras only support square size input ( 256x256), could you change it to support non-square size input image like 256 x 128?

Thank you very much.

There seems to be an error in the visualization file at grid_size = int(np.sqrt(model.layers[5].output_shape[0][-2] - 1)). It is hard coded and seems not suitable for 224 x 224 resolution? Any help is appreciated.

Hello,

I have an image classification problem where each image has 10 channels: (10,255,255). How do I define the model to work with such images?

I use the tensorlfow2.0.0 to run the code.But in layers.py line35, there is an error ValueError: Can't convert Python sequence with mixed types to Tensor. This may be due to input_ Shape is a nonetype, and the initializers shape got an nonetype input.

Hello,

Thank you so much for this great repo. Everything works beautifully so far.

I was wondering if you guys are interested in porting pytorch ViT model from a 2021 NeurlIPS paper "ImageNet-21K Pretraining for the Masses"(https://github.com/Alibaba-MIIL/ImageNet21K), which curated ImageNet-21K and the resulted pre-trained ViT-B-16 model on ImageNet-1K has shown a fantastic result (Top-1 84.4%)?

I would love to help with porting the weights if you guys are interested. Is there any recommendation on what information to look for in order to get the job done?

Thank you,
John

I see the call method in TransformerBlock has 2 input:
def call(self, inputs, training)
But when the script call TransformerBlock in vit.py, it is :

for n in range(num_layers): 
        y, _ = layers.TransformerBlock( 
            num_heads=num_heads, 
            mlp_dim=mlp_dim, 
            dropout=dropout, 
            name=f"Transformer/encoderblock_{n}", 
        )(y) 

Does is work in this way?
May I miss something in this repo?
Thanks a lot.

I was facing an error in loading the model. Resolved by downgrading Tensorflow version to 2.1.0.
Upgrading this repo to latest version of Tensorflow would simply things a little more.
Thanks for the implementation.

I am trying to use vit-keras in Kaggle and when using it with TPU, the below error is getting generated.

The TF version is Tensorflow version 2.2.0

`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in
1 with tpu_strategy.scope(): # creating the model in the TPUStrategy scope means we will train the model on the TPU
----> 2 model = create_model()
3 model.summary()

in create_model()
7 include_top=True,
8 pretrained_top=False,
----> 9 classes=5
10 )
11

/opt/conda/lib/python3.7/site-packages/vit_keras/vit.py in vit_l32(image_size, classes, activation, include_top, pretrained, pretrained_top)
228 activation=activation,
229 include_top=include_top,
--> 230 representation_size=1024 if pretrained and not pretrained_top else None,
231 )
232 if pretrained:

/opt/conda/lib/python3.7/site-packages/vit_keras/vit.py in build_model(image_size, patch_size, num_layers, hidden_size, num_heads, mlp_dim, classes, dropout, activation, include_top, representation_size)
99 y = tf.keras.layers.Reshape((-1, hidden_size))(y)
100 y = layers.ClassToken(name="class_token")(y)
--> 101 y = layers.AddPositionEmbs(name="Transformer/posembed_input")(y)
102 for n in range(num_layers):
103 y, _ = layers.TransformerBlock(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in call(self, *args, **kwargs)
895 # Build layer if applicable (if the build method has been
896 # overridden).
--> 897 self._maybe_build(inputs)
898 cast_inputs = self._maybe_cast_inputs(inputs)
899

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _maybe_build(self, inputs)
2414 # operations.
2415 with tf_utils.maybe_init_scope(self):
-> 2416 self.build(input_shapes) # pylint:disable=not-callable
2417 # We must set also ensure that the layer is marked as built, and the build
2418 # shape is stored since user defined build functions may not be calling

/opt/conda/lib/python3.7/site-packages/vit_keras/layers.py in build(self, input_shape)
32 self.pe = tf.Variable(
33 name="pos_embedding",
---> 34 initial_value=tf.initializers.random_normal(stddev=0.06)(
35 shape=(1, input_shape[1], input_shape[2])
36 ),

AttributeError: module 'tensorflow.keras.initializers' has no attribute 'random_normal'`

I have a question. When we fine tune this model, does it finetune entire network or just the head part? And if possible, how can i obtain specific layers to obtain internal weights?

Thank u so much

Hello!
I don't have "setup.py" in this repository, is that okay?

Thanks to create a nice implementation and package.
I was facing an error in saving model.

My tensorflow environment is below.

pip freeze | grep tensor
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.4.0
tensorflow-addons==0.11.2
tensorflow-datasets==4.0.0
tensorflow-estimator==2.4.0
tensorflow-io==0.17.0
tensorflow-metadata==0.26.0
tensorflow-probability==0.12.1

Saving model as h5 format can be done without any problems.

import tensorflow as tf
import vit_keras.vit as vit
model = vit.vit_b16(
image_size=224,
pretrained=True,
include_top=True,
pretrained_top=False,
)
opt = tf.keras.optimizers.Adam()
loss = tf.keras.losses.CategoricalCrossentropy()
model.compile(optimizer=opt,loss=loss,metrics=['accuracy'])
model.save('ViT.h5')

However, When I try to save model as SavedModel format, I got an error.

import tensorflow as tf
import vit_keras.vit as vit
model = vit.vit_b16(
image_size=224,
pretrained=True,
include_top=True,
pretrained_top=False,
)
opt = tf.keras.optimizers.Adam()
loss = tf.keras.losses.CategoricalCrossentropy()
model.compile(optimizer=opt,loss=loss,metrics=['accuracy'])
tf.saved_model.save(model,"ViT")

WARNING:absl:Found untraced functions such as MultiHeadDotProductAttention_1_layer_call_and_return_conditional_losses, MultiHeadDotProductAttention_1_layer_call_fn, LayerNorm_0_layer_call_and_return_conditional_losses, LayerNorm_0_layer_call_fn, LayerNorm_2_layer_call_and_return_conditional_losses while saving (showing 5 of 480). These functions will not be directly callable after loading.
WARNING:absl:Found untraced functions such as MultiHeadDotProductAttention_1_layer_call_and_return_conditional_losses, MultiHeadDotProductAttention_1_layer_call_fn, LayerNorm_0_layer_call_and_return_conditional_losses, LayerNorm_0_layer_call_fn, LayerNorm_2_layer_call_and_return_conditional_losses while saving (showing 5 of 480). These functions will not be directly callable after loading.
ValueError Traceback (most recent call last)
in
----> 1 tf.saved_model.save(model,"ViT")
2
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py in save(obj, export_dir, signatures, options)
1030 meta_graph_def = saved_model.meta_graphs.add()
1031
-> 1032 _, exported_graph, object_saver, asset_info = _build_meta_graph(
1033 obj, signatures, options, meta_graph_def)
1034 saved_model.saved_model_schema_version = constants.SAVED_MODEL_SCHEMA_VERSION
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py in _build_meta_graph(obj, signatures, options, meta_graph_def)
1196
1197 with save_context.save_context(options):
-> 1198 return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py in _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
1160 function_aliases[fdef.name] = alias
1161
-> 1162 object_graph_proto = _serialize_object_graph(saveable_view,
1163 asset_info.asset_index)
1164 meta_graph_def.object_graph_def.CopyFrom(object_graph_proto)
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py in _serialize_object_graph(saveable_view, asset_file_def_index)
752
753 for obj, obj_proto in zip(saveable_view.nodes, proto.nodes):
--> 754 _write_object_proto(obj, obj_proto, asset_file_def_index,
755 saveable_view.function_name_map)
756 return proto
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py in _write_object_proto(obj, proto, asset_file_def_index, function_name_map)
798 version=versions_pb2.VersionDef(
799 producer=1, min_consumer=1, bad_consumers=[]),
--> 800 metadata=obj._tracking_metadata)
801 # pylint:enable=protected-access
802 proto.user_object.CopyFrom(registered_type_proto)
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py in _tracking_metadata(self)
3077 @Property
3078 def _tracking_metadata(self):
-> 3079 return self._trackable_saved_model_saver.tracking_metadata
3080
3081 def _list_extra_dependencies_for_serialization(self, serialization_cache):
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py in tracking_metadata(self)
53 # TODO(kathywu): check that serialized JSON can be loaded (e.g., if an
54 # object is in the python property)
---> 55 return json_utils.Encoder().encode(self.python_properties)
56
57 def list_extra_dependencies_for_serialization(self, serialization_cache):
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in python_properties(self)
39 def python_properties(self):
40 # TODO(kathywu): Add python property validator
---> 41 return self._python_properties_internal()
42
43 def _python_properties_internal(self):
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py in _python_properties_internal(self)
33
34 def _python_properties_internal(self):
---> 35 metadata = super(ModelSavedModelSaver, self)._python_properties_internal()
36 # Network stateful property is dependent on the child layers.
37 metadata.pop('stateful')
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in _python_properties_internal(self)
57 )
58
---> 59 metadata.update(get_config(self.obj))
60 if self.obj.input_spec is not None:
61 # Layer's input_spec has already been type-checked in the property setter.
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in get_config(obj)
116 # When loading, the program will attempt to revive the object from config,
117 # and if that fails, the object will be revived from the SavedModel.
--> 118 config = generic_utils.serialize_keras_object(obj)['config']
119
120 if config is not None:
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py in serialize_keras_object(instance)
243 name = get_registered_name(instance.class)
244 try:
--> 245 config = instance.get_config()
246 except NotImplementedError as e:
247 if _SKIP_FAILED_SERIALIZATION:
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py in get_config(self)
469 # of self.layers). Note that self._layers is managed by the
470 # tracking infrastructure and should not be used.
--> 471 layer_configs.append(generic_utils.serialize_keras_object(layer))
472 config = {
473 'name': self.name,
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py in serialize_keras_object(instance)
243 name = get_registered_name(instance.class)
244 try:
--> 245 config = instance.get_config()
246 except NotImplementedError as e:
247 if _SKIP_FAILED_SERIALIZATION:
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/layers/core.py in get_config(self)
1227 self.units,
1228 'activation':
-> 1229 activations.serialize(self.activation),
1230 'use_bias':
1231 self.use_bias,
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
199 """Call target, and fall back on dispatchers if there is a TypeError."""
200 try:
--> 201 return target(*args, **kwargs)
202 except (TypeError, ValueError):
203 # Note: convert_to_eager_tensor currently raises a ValueError, not a
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/activations.py in serialize(activation)
499 activation.name in _TF_ACTIVATIONS_V2):
500 return _TF_ACTIVATIONS_V2[activation.name]
--> 501 return serialize_keras_object(activation)
502
503
~/.pyenv/versions/py384/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py in serialize_keras_object(instance)
269 if hasattr(instance, 'name'):
270 return get_registered_name(instance)
--> 271 raise ValueError('Cannot serialize', instance)
272
273
ValueError: ('Cannot serialize', functools.partial(<function gelu at 0x15b6d55e0>, approximate=False))

Also, the same error occurred in ModelCheckpoint callback.

tf.keras.callbacks.ModelCheckpoint("ViT.ckpt", verbose=1, save_best_only=True, save_weights_only=False, save_freq='epoch')

This looks like a really promising resource. Thanks for putting it together.

When I try to use a model (specifically vit_l16) it downloads an NPZ file to my .keras/weights folder. That file is 1,190,422KB. However, that file fails to load:

np.load('ViT-L_16_imagenet21k+imagenet2012.npz', allow_pickle=False)

gives one of two errors:

Traceback (most recent call last):
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 1207, in internal_evaluate_expression_json
    pydevd_vars.evaluate_expression(py_db, frame, expression, is_exec=True)
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_vars.py", line 371, in new_func
    return _run_with_unblock_threads(original_func, py_db, curr_thread, frame, expression, is_exec)
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_vars.py", line 339, in _run_with_unblock_threads
    return _run_with_interrupt_thread(original_func, py_db, curr_thread, frame, expression, is_exec)
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_vars.py", line 310, in _run_with_interrupt_thread
    return original_func(py_db, frame, expression, is_exec)
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_vars.py", line 421, in evaluate_expression
    Exec(_expression_to_evaluate(expression), updated_globals, frame.f_locals)
  File "c:\xxx\.vscode\extensions\ms-python.python-2021.7.1053846006\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<string>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \U', '', ' escape

or

Traceback (most recent call last):
  File "C:\xxxxx\site-packages\numpy\lib\npyio.py", line 230, in __del__
    self.close()
  File "C:\xxxxx\site-packages\numpy\lib\npyio.py", line 221, in close
    if self.zip is not None:
AttributeError: 'NpzFile' object has no attribute 'zip'

Have you seen this before? Is there something obvious I'm doing wrong? The code I am calling is:

model = vit.vit_l16(
        image_size = (input_h, input_w),
        activation = 'sigmoid',
        pretrained = True,
        include_top = False,
        pretrained_top = False,
        classes = n_classes
    )

Could You please Explain me what is the difference Between all the VIT models like VIT b16 vs l16 and VIT l16 vs VIT l32. Any help would he appreciated. Thank you

While trying to load the model I'm getting this Userwarning,

/opt/conda/lib/python3.7/site-packages/tensorflow_addons/activations/gelu.py:81: UserWarning: Default value of `approximate` is changed from `True` to `False`   
"Default value of `approximate` is changed from `True` to `False`

Is this expected or we can just avoid it.. ?

Hi there,

Thank you SO much for this package! I've been trying it out via this Kaggle tutorial:

[https://www.kaggle.com/raufmomin/vision-transformer-vit-fine-tuning?select=test_images]

On my own data, with 36 classes, and I keep getting this error:

Resizing position embeddings from 12 to 7
warnings.warn(

So far, the model also is too big to run on tensorflow-cpu, any pointers?

Hi there,

I saw the implementation using a convolution to generate fixed size hidden vector from a variable size of input image. That's brilliant!
However, I am wondering if the fine-tuning result would be degradated, using a different input image size, say, 224, rather than the official input size, 384, as shown in your example.

Many thanks !