bckenstler / clr Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Shouldn't the update rule be:
lr = base_lr + (max_lr - base_lr) * ....
rather than
lr = base_lr + (max_lr - lr) * ...
Otherwise you don't get the linear relationship.
Note that the clr callback updates the learning rate prior to any further learning rate adjustments as called for in a given optimizer.
Hi, @bckenstler , excellent work! I am still confused about what you said about the "order of learning rate augmentation". If the clr callback is added and sets the learning rate after each batch training ends, will a given optimizer (e.g. adam) still adjust the learning rate that the clr just set for updating weights. Thanks!
From readme, "step_size : number of training iterations per half cycle. Authors suggest setting step_size = (2-8) x (training iterations in epoch) . Default 2000."
Does it mean step_size should be "np.ceil(x_train.shape[0]/batch_size/2)" or "2*np.ceil(x_train.shape[0]/batch_size)"?
Hello,
I try to perform clr as described and it works very well with VGG16. But when training other networks like DenseNet I get following error (TypeError: integer argument expected, got float):
TypeError Traceback (most recent call last)
in ()
26 validation_steps = len(val_list) // batch_size + 1,
27 callbacks=[clr],
---> 28 verbose=1)
/opt/conda/lib/python3.6/site-packages/Keras-2.2.4-py3.6.egg/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your ' + object_name + '
call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
/opt/conda/lib/python3.6/site-packages/Keras-2.2.4-py3.6.egg/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1434 use_multiprocessing=use_multiprocessing,
1435 shuffle=shuffle,
-> 1436 initial_epoch=initial_epoch)
1437
1438 @interfaces.legacy_generator_methods_support
/opt/conda/lib/python3.6/site-packages/Keras-2.2.4-py3.6.egg/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
217 batch_logs[l] = o
218
--> 219 callbacks._call_batch_hook('train', 'end', batch_index, batch_logs)
220
221 batch_index += 1
/opt/conda/lib/python3.6/site-packages/Keras-2.2.4-py3.6.egg/keras/callbacks.py in _call_batch_hook(self, mode, hook, batch, logs)
93 'Method (%s) is slow compared '
94 'to the batch update (%f). Check your callbacks.', hook_name,
---> 95 delta_t_median)
96 if hook == 'begin':
97 self._t_enter_batch = time.time()
Has anybody an idea what is the reason for this error? Thank you!
Dima S.
Hi, I've tried to use your class within my training code, but I got the following error: AttributeError: 'CyclicLR' object has no attribute 'on_train_batch_begin'.
My code is the following:
from tensorflow.keras.applications.resnet50 import ResNet50
base_model = ResNet50(weights='imagenet', include_top=False)
base_model.trainable = False
model = tf.keras.Sequential([
base_model,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(units=200, activation='relu'),
tf.keras.layers.Dense(units=5, activation="softmax")
])
model.compile(optimizer=tf.keras.optimizers.Adam(0.001), metrics=["accuracy"],
loss=tf.keras.losses.sparse_categorical_crossentropy)
clr = CyclicLR(base_lr=0.001, max_lr=0.006, step_size=240)
model.fit(train_dataset, epochs=30, steps_per_epoch=60, validation_data=val_dataset,
validation_steps=1, callbacks=[clr])
By the way train_dataset and val_dataset are tf.data.Datasets.
Versions: Python3 and TF 1.10.
Any idea why the issue?
Dear @bckenstler ,
recently I stumbled across an issue with CLR calling from keras 2.1.5:
I ran using
CyclicLR(base_lr=1e-5, max_lr=8e-4, mode='triangular2', step_size=trn_steps//10, scale_mode='iterations')
where trn_steps
is equal to steps_per_epoch
in model.fit_generator
.
Now, my observation is that during the first epoch CLR goes through 10 cycles (as planned), but then lr stays constant throughout the remaining epochs. How do I properly reset the lr cycle? I tried scale_mode='cycle'
as well, but no luck. What am I doing wrong?
Hi,
How can I set the learning rate curve to use Epochs as x axis instead of iterations?
Hi, this callback seems quite interesting. Do you plan to PR to the keras repo?
After using CLR for a bit in models written in Python, I must say CLR makes a huge difference in my work.
Now that R is well served by the keras package, I wonder if you could also write a CLR callback for the R Keras (see its API here)? That would help wonders people who for one reason or another have some models already prepared in R.
Thanks!
It's cool that you've implemented a cyclical learning rate for Keras, but have you considered adding this to Pypi? That way, it's a lot easier for others to incorporate CLR in their own repos
Hi,
Thank you so much for your work.
I want to plot the range of learning rate base_lr= 0.01, max_lr=0.1 for my method
schedulers = torch.optim.lr_scheduler.CyclicLR(optim_backbone, base_lr= 0.01, max_lr=0.1, step_size_up=2000, step_size_down=None, mode='triangular')
Like Figure 2(a) of this paper https://arxiv.org/pdf/1708.07120.pdf
From the paper and your implementation, your examples are only use SGD optimizer. I am wondering if I can use this CLR for Adam or other optimizers. Many thanks.
Hi @bckenstler. This is great. Found your repo and figured the R implementation of keras could also greatly benefit from this. I literally translated your code into R and put it into a new package I plan to develop. Would you consider linking my repo somewhere at the top of your README so people who are looking for the R implementation could find it easily?
Hi @bckenstler ,
Thanks a lot for sharing your implementation. I just read the paper on cyclical learning rate. I'd like to know how you dealt with the following
Thanks to your CLR implementation, I forked and tailored another version for trapezoid schedule which is introduced in this paper:
This is just for your information, you can find it here: https://github.com/daisukelab/TrapezoidalLR
I was thinking I could ask for merge, but I just kept it as another version. I guess the trapezoid schedule might be a temporary solution though I implemented...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.