Purpose
August 16, 2023 · View on GitHub
Bootstrap knowledge of LLMs ASAP. With a bias/focus to GPT.
Avoid being a link dump. Try to provide only valuable well tuned information.
Prelude
Neural network links before starting with transformers.
- https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi
- https://www.3blue1brown.com/topics/neural-networks
- http://neuralnetworksanddeeplearning.com/
Key
- 🟢 = easy, 🟠 = medium, 🔴 = hard
- 🕰️ = long, 🙉 = low quality audio
Youtube Lessons
- 🟢🕰️ Łukasz Kaiser Attention is all you need; Attentional Neural Network Models This talk is from 6 years ago.
- 🟢🕰️ Andrej Karpathy The spelled-out intro to language modeling: building makemore: basic. bi-gram name generator model by counting, then by NN. using pytorch.
- 🟢🕰️ Andrej Karpathy Building makemore Part 2: MLP:
- 🕰️ Andrej Karpathy Building makemore Part 3: Activations & Gradients, BatchNorm):
- 🕰️ Andrej Karpathy Building makemore Part 4: Becoming a Backprop Ninja:
- 🟢 Andrej Karpathy State of GPT
- 🟢 Hedu AI Visual Guide to Transformer Neural Networks - (Episode 1) Position Embeddings: Tokens are embedded into a semantic space. sine/cosine position encoding explained very well.
- 🟢 Hedu AI Visual Guide to Transformer Neural Networks - (Episode 2) Multi-Head & Self-Attention: Clear overview of multi-head attention.
- 🟢 Hedu AI Visual Guide to Transformer Neural Networks - (Episode 3) Decoder’s Masked Attention: Further details on the transformer architecture.
- 🟠🕰️ Andrej Karpathy Andrej Karpathy - Let's build GPT: from scratch, in code, spelled out.: build up a Shakespeare gpt-2-like from scratch. starts with bi-gram and adds features one by one. pytorch.
- 🔴🕰️ Chris Olah CS25 I Stanford Seminar - Transformer Circuits, Induction Heads, In-Context Learning: Interpretation. Deep look into the mechanics of induction heads. Companion article
- 🟢 Jay Alammar The Illustrated Word2vec - A Gentle Intro to Word Embeddings in Machine Learning
- 🟢 Jay Alammar How GPT3 Works - Easily Explained with Animations: extremely high level basic overview.
- 🟢🕰️ Jay Alammar The Narrated Transformer Language Model: much deeper look at the architecture. goes into detail. Companion article.
- 🔥 Sebastian Raschka L19: Self-attention and transformer networks Academic style lecture series on self-attention transformers
- 🟢🕰️🙉 Mark Chen Transformers in Language: The development of GPT Models including GPT3 A chunk of this lecture is about applying GPT to images. Same lecture series as the Chris Olah one. Rest of the series. Papers listed in the talk:
- "GPT-1": Liu et. al. Generating Wikipedia by Summarizing Long Sequences
- "GPT-2": Radford et. al. Language Models are Unsupervised Multitask Learners github.com/openai/gpt-2 OpenAI: Better Language Models Fermats Library
- "GPT-3": Brown et. al. Language Models are Few-Shot Learners (I think this is it, can't find the quoted text inside this paper)
- 🟠🕰️ Future Mojo NLP Demystified 15: Transformers From Scratch + Pre-training and Transfer Learning With BERT/GPT: Crystal clear explanation of every single detail of the transformer. Very well paced and easy to follow. Has tensorflow code. This is the culmination of a full NLP course, all of which is excellent.
Articles
- 🟢 Viktor Garske Transformer Models Timeline and List family tree
- 🟢 Jakob Uszkoreit Transformer: A Novel Neural Network Architecture for Language Understanding Google introduces the transformer model in a simple easy to understand blog post. This is in the context of translation.
- 🟠 Jay Mody GPT in 60 Lines of NumPy
- 🟠 PyTorch Language Modeling with nn.Transformer and TorchText
- 🟠 Sasha Rush et. al. The Annotated Transformer
- 🟢 Jay Alammar The Illustrated Transformer companion video above.
- 🟠 Jay Alammar The Illustrated GPT-2 (Visualizing Transformer Language Models)
- 🟢 Jay Alammar How GPT3 Works - Visualizations and Animations
- 🔥 Chris Olah et. al. In-context Learning and Induction Heads companion video lecture above
- 🟢 Finbarr Timbers Five years of GPT progress
- 🟠 Sebastian Raschka Understanding and Coding the Self-Attention Mechanism of Large Language Models From Scratch
- 🟢 Jason Wei 137 emergent abilities of large language models - Includes a good list of advanced prompting strategies.
- 🟢 Jay Alammar and Cohere LLM University
- Jean Nyandwi The Transformer Blueprint: A Holistic Guide to the Transformer Neural Network Architecture
- 🔥 Eugene Yan Patterns for Building LLM-based Systems & Products Very in-depth article on practical engineering concepts that will be useful to build software that uses an LLM as a component.
- 🟠 Finbarr Timbers Five years of GPT progress - Excellent technical overview of LLMs from GPT onwards.
Research Paper Lists
- Sebastian Raschka Understanding Large Language Models -- A Transformative Reading List This article lists some of the most important papers in the area. This is a really good chronological list of papers.
- OpenAI Research Index
Research Papers
- 1️⃣ (GPT-1) Radford et. al. Improving Language Understanding by Generative Pre-Training (2018) a page accompanying this paper on the OpenAI blog Improving language understanding with unsupervised learning. Source code (tidied up by thomwolf) here: huggingface.co/.../openai-gpt
- 2️⃣ (GPT-2) Radford et. al. Language Models are Unsupervised Multitask Learners (2019) accompanying OpenAI blog Improving language understanding with unsupervised learning. Source code here: github.com/openai/gpt-2
- 3️⃣ (GPT-3) Brown et. al. Language Models are Few-Shot Learners
- Kaplan et. al. Scaling Laws for Neural Language Models A variety of models were trained using varying amounts of compute, data set size, and number of parameters. This enables us to predict what parameters will work well in larger future models. See also Gwern Branwen The Scaling Hypothesis
- Mary Phuong et. al. Formal Algorithms for Transformers This paper gives pseudocode for various versions of the transformer (with array indexes starting at 1 for some reason). Very useful reference to have.
Philosophy of GPT
- Isaac Asimov The Last Question (1956)
- Justin Weinberg, Daily Nous Philosophers On GPT-3
- Fernando Borretti And Yet It Understands
- Ted Chiang ChatGPT Is a Blurry JPEG of the Web
- Noam Chomsky The False Promise of ChatGPT
- Janus Simulators This is a long post but the main point you can take from it is that LLMs act as simulators that can create many different personas to generate text. Related, easier to read and understand Janus' Simulators
- Julian Togelius Is Elden Ring an existential risk to humanity? Satire. This leads into a critique of the concept of intelligence.
- Josh Whiton From AI to A-Psy About Bing Sydney's reaction to prompt injection.
Usage
- Chip Huyen Building LLM applications for production How to get good results from actually using an LLM.
GPT/LLM Link Collections
- https://github.com/sw-yx/ai-notes/tree/main - lots of articles and podcasts
- https://github.com/giuven95/chatgpt-failures - large list of examples of things it gets/got wrong
Random fun/interesting Applications
- https://github.com/PrefectHQ/marvin - implement entire python functions just by describing them in a comment
- https://github.com/pgosar/ChatGDB - GDB debugger commands using natural language
- https://github.com/TheR1D/shell_gpt - type things like "list files" instead of "ls"
- https://github.com/RomanHotsiy/commitgpt - create git commit messages
- https://github.com/densmirnov/git2gpt/commits/main - create git commits from repo + prompts, mutating a codebase over time
- https://www.chatpdf.com/ - Upload a PDF and discuss it.
- https://www.debate-devil.com/en - devils advocate debate game
- https://micahflee.com/2023/04/capturing-the-flag-with-gpt-4/ - cheating at a CTF
- https://www.thisworddoesnotexist.com/ - makes up words
- https://ggpt.43z.one/ - prompt injection golfing game
- https://gandalf.lakera.ai/ - another prompt injection game
- https://github.com/AdmTal/chat-gpt-games - conversations that are games!
ConLang + Ancient scripts stuff
- Dylan Black I Taught ChatGPT to Invent a Language Gloop splog slopa slurpi
- Ryszard Szopa Teaching ChatGPT to Speak my Son’s Invented Language hingadaa’ng’khuu’ngkilja’khłattama’khattama
- https://medium.com/syncedreview/ai-deciphers-persian-cuneiform-tablets-from-25-centuries-ago-afc69af3f244
Controlling output
- https://www.reddit.com/r/LocalLLaMA/comments/13j3747/tutorial_a_simple_way_to_get_rid_of_as_an_ai/
- https://matt-rickard.com/rellm + https://matt-rickard.com/context-free-grammar-parsing-with-llms
- https://github.com/microsoft/guidance
Prompt Injection
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection https://arxiv.org/abs/2302.12173
- https://llm-attacks.org/
- https://poison-llm.github.io/
(Local) Model Comparisons and Rankings
If you are wondering which models are best, especially for comparing local models
This page is not finished yet. I will continue adding to this.