[RB] Replicating GPT-2, the most dangerous NLP model (with Aaron Gokaslan) (Ep. 79)

In this episode, I am with Aaron Gokaslan, computer vision researcher, AI Resident at Facebook AI Research. Aaron is the author of OpenGPT-2, a parallel NLP model to the most discussed version that OpenAI decided not to release because too accurate to be published.
Better Language Models GPT-2

What is GPT-2 and why it is a big deal

In this episode we discuss about image-to-image translation and the dangers of the GPT-2 model. As a matter of fact, there is no better language model than GPT-2 at the moment. With the latest version of GPT-2 it is possible to generate paragraphs of text, given a seed. It seems most of the generated paragraphs make sense. Moreover, it is quite challenging to understand if its author was a human or an algorithm.
Hence, anyone could use the GPT-2 model to generate fake news or facts that are not supported by any piece of evidence.

Better Language Models GPT-2

The future of Artificial Intelligence

In the episode, we switch gears and speak about the future of Artificial Intelligence. Hence, I ask what a researcher like Aaron would see happening in the next 5 years.
Above all, Aaron believes that self-supervision will play a fundamental role in the future of AI. Finally, he provided us with some very interesting resources (added below to the references section) and some demos of the open version of GPT-2 in action. That stuff will definitely blow your mind!

We hope you enjoy the show!

Don’t forget to join the discussion on our Discord server

References

Multimodal image to image translation (not all mentioned in the podcast but recommended by Aaron)
Pix2Pix

CycleGAN

GANimorph: Paper and Code

UNIT:https://arxiv.org/abs/1703.00848
MUNIT:https://github.com/NVlabs/MUNIT
DRIT: https://github.com/HsinYingLee/DRIT

GPT-2 and related

Try OpenAI’s GPT-2
Blogpost
The Original Transformer Paper
Grover: The FakeNews generator and detector: