Andrej Karpathy

Quotes are word for word from the source.

• Demystifying technology

  “I want you to note that again there is nothing too scary going on inside.”

  Our noteKarpathy encourages learners by showing that complex algorithms are just composed of simple parts.

• Brute force security

  “Which, due to the nice properties of SHA256, can only be done via brute force search.”

  Our noteDescribing why the security of Bitcoin relies on the computational difficulty of finding specific hash values.

• Bitcoin transaction logic

  “Every Input/Output of any bitcoin transaction must always be fully spent. So if we own 0.001 BTC and want to send half of it somewhere else, we actually have to send one half there, and one half back to us.”

  Our noteExplaining how Bitcoin transactions require spending the entire input and returning change.

• Managing unspent outputs

  “Once we consume this UTXO in its entirety we are free to "chunk up" its value into however many outputs we like, and optionally send some of those chunks back to our own address.”

  Our noteExplaining how users can split their Bitcoin value into multiple new outputs.

• Transaction message encoding

  “So what this message really encodes is just the inputs and the new outputs, their amounts, and their owners (via the locking scripts specifying the public key hash of each owner).”

  Our noteExplaining what data is actually being signed in a Bitcoin transaction.

• Transaction validation process

  “The Bitcoin nodes will decode it, validate it, and include it into the next block they might mine any second now (if the fee is high enough).”

  Our noteDescribing how the network processes a broadcasted transaction based on its validity and fee.

• The 10x Fallacy

  “People have a strange bug built into psychology: a 10x more important or impactful problem intuitively feels 10x harder (or 10x less likely) to achieve. This is a fallacy - in my experience a 10x more important problem is at most 2-3x harder to achieve.”

  Our noteExplaining why people overestimate the difficulty of ambitious goals.

• Active vocabulary choice

  “in your papers you never "study" or "investigate" (there are boring, passive, bad words); instead you "develop" or even better you "propose". And you don't present a "system" or, shudder , a "pipeline"; instead, you develop a "model”

  Our noteKarpathy emphasizes using strong, active verbs to make research sound more impactful.

• Social nature of research

  “Science might have this image of a few brilliant lone wolfs working in isolation, but the truth is that research is predominantly a highly social endeavor”

  Our noteHighlighting that even technical fields rely heavily on community and networking.

• Cheap intelligence

  “But if intelligence really does become too cheap to meter, it will become possible to do a perfect reconstruction and synthesis of everything.”

  Our noteHe reflects on the implications of AI becoming an abundant and inexpensive resource.

• Current state of AI coding

  “Ultimately, vibe coding full web apps today is kind of messy and not a good idea for anything of actual importance. But there are clear hints of greatness and I think the industry just needs a bit of time”

  Our noteAndrej evaluates the current viability of building production apps using only AI prompts.

• Clean code practices

  “Note how we're wrapping all the JS code into a function closure so that we don't pollute the global namespace. This is a very common trick in Javascript that is essentially used to implement classes.”

  Our noteHe explains a technical best practice for organizing code in a library.

• Avoid unverified complexity

  “What we try to prevent very hard is the introduction of a lot of "unverified" complexity at once, which is bound to introduce bugs”

  Our noteHe explains why building from simple to complex is the only way to ensure correctness.

• Generalize special cases

  “I like to write a very specific function to what I'm doing right now, get that to work, and then generalize it later making sure that I get the same result.”

  Our noteA coding strategy to avoid bugs when building complex functionality.

• Performance tradeoff curve

  “Human accuracy is not a point. It lives on a tradeoff curve.”

  Our noteExplaining that human performance varies based on the amount of effort and expertise invested.

• Future intelligence gap

  “Large organizations get to concentrate their vast resources to buy more intelligence. And within the category of "individual" too, the elite may once again split away from the rest of society.”

  Our noteSpeculating on how future AI scaling might favor those with more capital.

• Challenging common wisdom

  “What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train”

  Our noteHe discusses how his experience contradicted the prevailing belief that RNNs were hard to use.

• Flexible data processing

  “The takeaway is that even if your data is not in form of sequences, you can still formulate and train powerful models that learn to process it sequentially.”

  Our noteHe explains that sequential modeling can be applied to non-sequential data types like images.

• Model scale impact

  “We observed that these became less common with larger/better models, but nonetheless, these are the kinds of mistakes that come up.”

  Our noteAndrej noted that increasing model size helps reduce structural errors in generated text.

• Testing limits

  “I wanted to push structured data to its limit, so for the final challenge I decided to use code.”

  Our noteHe described his motivation for training a neural network on the Linux source code.

• Emergence of context

  “Topics and themes that span multiple words (and in general longer-term dependencies) start to emerge only much later.”

  Our noteExplaining that complex understanding in AI models takes longer to develop than basic syntax.

• Innovation in attention

  “The concept of attention is the most interesting recent architectural innovation in neural networks.”

  Our noteIdentifying the most significant architectural shift in neural network research at the time.

• Energy input accounting

  “the energy input to your body is upper bounded by the number of food calories that you eat.”

  Our noteHe simplifies the concept of energy intake for weight management.

• Rapid prototyping power

  “It took us ~100 lines and 10 minutes of Javascript (with a bit of practice) and we tweaked Twitter's look, removed err... undesirable content, made Twitter autorefresh”

  Our noteHe demonstrates how quickly a small amount of code can solve specific user problems.

• Seeking feedback

  “A few patterns stand out for me, and if you notice anything else I'd be happy to hear about in the comments.”

  Our noteHe invites his audience to contribute their observations to his analysis.

• Tractable but difficult

  “I feel like the problems are tractable, they're surmountable, but they're still difficult.”

  Our noteAndrej discusses the difficulty of solving current AI bottlenecks like continual learning.

• Learning by building

  “I do think that there's two types of knowledge, almost. There's the high-level surface knowledge, but when you build something from scratch, you're forced to come to terms with what you don't understand and you don't know that you don't understand it.”

  Our noteAndrej explains why building from scratch is the only way to truly understand a complex system.

• True understanding

  “If I can't build it, I don't understand it. That's a Feynman quote , I believe. I 100% have always believed this very strongly, because there are all these micro things that are just not properly arranged and you don't really have the knowledge.”

  Our noteHe emphasizes that theoretical knowledge is insufficient without the ability to implement it.

• Learning new languages

  “They increase accessibility to languages or paradigms that you might not be as familiar with. I think they're very helpful there as well.”

  Our noteAndrej explains how AI helps him write in Rust despite being new to the language.

• Risk of self-training

  “Any individual sample will look okay, but the distribution of it is quite terrible. It's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse.”

  Our noteAndrej warns about the dangers of models or people training only on their own outputs.