Flow-Based Generative Models, Bijective Transforms and Neural Lossless Compression

There was a interesting paper released some time in the past couple of weeks that showed it's possible to do lossless compression via flow-based models. Historically flow based models didn't get as much attention as VAE and definitely not as much as GAN's so I paid very little mind to them. But flow based generative… Continue reading Flow-Based Generative Models, Bijective Transforms and Neural Lossless Compression

Importance of Decoding Algorithms in Neural Text Generation

Text generation and more specifically neural language modeling has recently exploded in popularity. The controversial release of GPT-2 coupled with the impressive generated text has brought language modeling to the forefront of NLP. A paper was recently released exploring the different sampling techniques for generating natural looking text, as well as proposing a new technique.… Continue reading Importance of Decoding Algorithms in Neural Text Generation

Stochastic Weight Averaging and the Ornstein-Uhlenbeck Process

A couple weeks back a blog post was released on the PyTorch blog describing the Stochastic Weight Averaging (SWA) algorithm and it's implementation in pytorch/contrib. The algorithm itself seemed embarrassingly straightforward and relied on averaging snapshots of the the model across a certain learning rate schedule. The authors argued that "SGD tends to converge to… Continue reading Stochastic Weight Averaging and the Ornstein-Uhlenbeck Process