Andrej Karpathy

Quotes are word for word from the source.

• Scaling existing ideas

  “In Computer Vision, the 2012 AlexNet was mostly a scaled up (deeper and wider) version of 1990's ConvNets.”

  Our noteDiscussing how progress in AI often comes from scaling rather than entirely new algorithms.

• Surprising Reality

  “It shouldn't work, but amusingly we live in a universe where it does.”

  Our noteReflecting on the counter-intuitive success of complex neural network training.

• Learning with Constraints

  “I did not tune the hyperparameters too much and ran the experiment on my (slow) Macbook, but after training for 3 nights I ended up with a policy that is slightly better than the AI player.”

  Our noteDescribing the results of training a Pong AI on a standard laptop without heavy optimization.

• Understanding Algorithm Logic

  “In conclusion, once you understand the "trick" by which these algorithms work you can reason through their strengths and weaknesses.”

  Our noteEncouraging a deep understanding of how AI tools actually function to predict their utility.

• Algorithm Constraints

  “The current consensus is that PG works well only in settings where there are a few discrete choices so that one is not hopelessly sampling through huge search spaces.”

  Our noteDiscussing the limitations of policy gradients in large search spaces.

• Practical Implementation Advice

  “And if you insist on trying out Policy Gradients for your problem make sure you pay close attention to the tricks section in papers, start simple first, and use a variation of PG called TRPO”

  Our noteAdvice on how to successfully implement complex reinforcement learning algorithms in practice.

• Model learning goal

  “The goal of the model is to learn the patterns in the data and then generate similar new documents that share the statistical patterns within.”

  Our noteDefining what a generative model is actually trying to achieve with its training data.

• Understanding AI responses

  “When you initialize the document with your prompt, the model's response from its perspective is just a statistical document completion.”

  Our noteExplaining that ChatGPT's responses are essentially statistical completions of a document.

• Model Parameters Definition

  “Parameters The parameters are the knowledge of the model. They are a large collection of floating point numbers (wrapped in Value for autograd) that start out random and are iteratively optimized during training.”

  Our noteExplaining what parameters represent in a neural network architecture.

• Software 3.0 Paradigm

  “Software 3.0 is about programming through prompting. What's in the context window is your lever over the interpreter, and the interpreter is the LLM.”

  Our noteExplaining the shift from writing code to prompting models.

• Optimization potential

  “Today's models are not optimally formulated, and just changing some of the details of the model, loss function, augmentation or the optimizer we can about halve the error.”

  Our noteKarpathy suggests that significant gains are still possible through better engineering of existing models.

• Future of software

  “code is suddenly free, ephemeral, malleable, discardable after single use. Vibe coding will terraform software and alter job descriptions.”

  Our noteReflecting on how AI-generated code changes the cost and utility of software development.

• Vulnerability of models

  “The conclusion seems to be that we can take any arbitrary image and classify it as whatever class we want by adding tiny, imperceptible noise patterns.”

  Our noteExplaining how adversarial noise can fool state-of-the-art neural networks.

• Solving AI flaws

  “In fact, Deep Learning offers tangible hope for a solution, since we can use all the wiggle of composed functions to design more resistant architectures or objectives.”

  Our noteExpressing optimism that deep learning's flexibility can help fix its own vulnerabilities.

• Human performance baseline

  “My classification accuracy ended up at about 94% on 400 images. Why not 100%? Because some images are really unfair!”

  Our noteAndrej explains his results after testing his own ability to classify the CIFAR-10 dataset.

• Data over architecture

  “They were immortal primitive relics from the past, who made up for their faults with the immense amount of data they had experienced.”

  Our noteHe explains how massive amounts of training data can compensate for outdated underlying technology.

• Economical human intervention

  “In these cases, shaper interventions like this one were the most economical ways of rectifying the situation.”

  Our noteUsing human guidance to fix AI errors is often more efficient than automated retraining.

• Human vs Computer learning

  “The strengths, weaknesses and types of data practically available to humans and computers are fundamentally misaligned. Thus, it is unfair to draw direct comparisons and extreme caution should be taken when drawing inspiration from human learning.”

  Our noteHe warns against assuming computers should learn exactly like humans do.

• Industrial cost optimization

  “The cost-driven hyper-optimization of the industry is a deep rabbit hole way beyond the scope of this post.”

  Our noteHe explains how industries sacrifice quality and health for profit margins.