taki0112 / amsgrad-tensorflow

Simple Tensorflow implementation of "On the Convergence of Adam and Beyond" (ICLR 2018)

Junho Kim

👉 Research Scientist at NAVER AI Lab

Recent implementations

• 🔥 tf-torch-template
• Denoising-Diffusion-GANs
• Vision Transformer Cookbook

Recent accepted papers

• 🔥 Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models CVPRW 2023, Oral
• Context-Preserving Two-Stage Video Domain Translation for Portrait Stylization CVPRW 2023
• Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding CVPR 2023

Publications

• ⭐ Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models CVPRW 2023, Oral
• Context-Preserving Two-Stage Video Domain Translation for Portrait Stylization CVPRW 2023
• Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding CVPR 2023
• Rarity Metric : A New Metric to Evaluate the Uncommonness of Synthesized Images ICLR 2023, Spotlight
• Learning Input-agnostic Manipulation Directions in StyleGAN with Text Guidance ICLR 2023
• Generator Knows What Discriminator Should Learn in Unconditional GANs ECCV 2022
• ⭐ Feature Statistics Mixing Regularization for Generative Adversarial Networks CVPR 2022
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks ICLR 2022
• Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing CVPR 2021
• ⭐ U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation ICLR 2020

amsgrad-tensorflow's Issues

TypeError: Input 'y' of 'Mul' Op has type resource that does not match type float32 of argument 'x'.

• Tensorflow 1.12.0
• CUDA 9.0

File "AMSGrad.py", line 93, in _resource_apply_dense
   m_t = state_ops.assign(m, beta1_t * m + m_scaled_g_values, use_locking=self._use_locking)
 File "/home/user/.envs/default/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper
   return func(x, y, name=name)
 File "/home/user/.envs/default/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1131, in _mul_dispatch
   return gen_math_ops.mul(x, y, name=name)
 File "/home/user/.envs/default/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5042, in mul
   "Mul", x=x, y=y, name=name)
 File "/home/user/.envs/default/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 546, in _apply_op_helper
   inferred_from[input_arg.type_attr]))
TypeError: Input 'y' of 'Mul' Op has type resource that does not match type float32 of argument 'x'.

How to use them in the tensorflow fit?

In tensorflow, I use like the following, but I don't know why it doesn't work.
history=NN.fit(x=train_x,y=Y_train,epochs=500,validation_split=0.1,batch_size=128,callbacks=[early_stopping])
The bug report is the following:

Epoch 1/500
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[<ipython-input-68-b504eaeb395a>](https://localhost:8080/#) in <module>()
----> 1 history=NN.fit(x=train_x,y=Y_train,epochs=500,validation_split=0.1,batch_size=128,callbacks=[early_stopping])

1 frames
[/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py](https://localhost:8080/#) in autograph_handler(*args, **kwargs)
   1145           except Exception as e:  # pylint:disable=broad-except
   1146             if hasattr(e, "ag_error_metadata"):
-> 1147               raise e.ag_error_metadata.to_exception(e)
   1148             else:
   1149               raise

NotImplementedError: in user code:

    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step  **
        outputs = model.train_step(data)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 863, in train_step
        self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v1.py", line 792, in minimize
        self.apply_gradients(grads_and_vars)
    File "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v1.py", line 795, in apply_gradients
        self.optimizer.apply_gradients(grads_and_vars, global_step=self.iterations)

    NotImplementedError:

I insert the AMSGrad optimizer in the compile block.

# We apply a fairly simple MLP architecture
def build_network():
  models=[]
  model=Sequential()
  model.add(Dense(64,input_dim=122))
  model.add(Dense(32))
  model.add(Activation('relu'))
  model.add(Dropout(.15))
  model.add(Dense(32))
  model.add(Activation('relu'))
  model.add(Dropout(.15))  
  model.add(Dense(32))
  model.add(Activation('relu'))
  model.add(Dropout(.15))  
  model.add(Dense(5))
  model.add(Activation('softmax'))
  model.compile(loss='categorical_crossentropy',optimizer=AMSGrad(learning_rate=0.015,beta1=0.9, beta2=0.99, epsilon=1e-8),metrics=['accuracy'])
  return model

Anyone know the reason?