Welcome to pytorch-optimizer’s documentation!¶

torch-optimizer – collection of optimizers for PyTorch.

Simple example¶

import torch_optimizer as optim

# model = ...
optimizer = optim.DiffGrad(model.parameters(), lr=0.001)
optimizer.step()

Installation process is simple, just:

$ pip install torch_optimizer

AccSGD	https://arxiv.org/abs/1803.05591
AdaBound	https://arxiv.org/abs/1902.09843
AdaMod	https://arxiv.org/abs/1910.12249
Adafactor	https://arxiv.org/abs/1804.04235
AdamP	https://arxiv.org/abs/1804.00325
AggMo	https://arxiv.org/abs/2006.08217
DiffGrad	https://arxiv.org/abs/1909.11015
Lamb	https://arxiv.org/abs/1904.00962
NovoGrad	https://arxiv.org/abs/1905.11286
PID	https://www4.comp.polyu.edu.hk/~cslzhang/paper/CVPR18_PID.pdf
QHAdam	https://arxiv.org/abs/1810.06801
QHM	https://arxiv.org/abs/1810.06801
RAdam	https://arxiv.org/abs/1908.03265
Ranger	https://arxiv.org/abs/1908.00700v2
RangerQH	https://arxiv.org/abs/1908.00700v2
RangerVA	https://arxiv.org/abs/1908.00700v2
SGDP	https://arxiv.org/abs/2006.08217
SGDW	https://arxiv.org/abs/1608.03983
Shampoo	https://arxiv.org/abs/1802.09568
SWATS	https://arxiv.org/abs/1712.07628
Yogi	https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization