fam/nanochat-omni
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
240 Commits 3 Branches 0 Tags
8203efa9190e9d8da7419de29599e91a9242b968
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrej Karpathy 8203efa919 implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user. 2026-01-16 17:37:51 +00:00
Andrej Karpathy fbf2bbea25 update log with a bunch of attempts 2026-01-16 02:21:17 +00:00
Andrej Karpathy 747ed4491f add negative result on olmo3 pretraining mix 2026-01-16 00:44:01 +00:00
Andrej Karpathy 7d1700c521 add zstd lib 2026-01-16 00:44:01 +00:00
Andrej Karpathy bdcc030ffa oops legacy spurious line now 2026-01-15 23:32:20 +00:00
Andrej Karpathy 22a71aa3d3 fuse adamw into a single torch compiled kernel similar to muon. it's about 1.7X faster, but overall it's so tiny that it's not making a major dent 2026-01-15 23:30:44 +00:00
Andrej Karpathy 255f8b9af6 cleanly separate cpu and gpu sections 2026-01-15 23:30:11 +00:00
Andrej Karpathy 6bb92403d5 changes and optimizations to muon, making it more efficient and simpler/cleaner a bit 2026-01-15 03:20:48 +00:00
Andrej Karpathy 3142ca1a28 minor helpful message 2026-01-15 03:20:21 +00:00
Andrej Karpathy 7312ec9898 fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way 2026-01-13 22:45:27 +00:00
Andrej Karpathy 3b50b77ed3 fix base_loss to report correct loss by switching the dataloader to the new default 2026-01-13 22:09:36 +00:00
Andrej Karpathy f92efce169 add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance 2026-01-13 21:33:54 +00:00
Andrej Karpathy 43c29dd9d5 Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training 
The new DataLoader ensures that every token sequence in train/val batches has a BOS token
at the beginning. Therefore, no token streams start abruptly in the middle of a document,
which could be confusing for the model. Note that this changes the loss scale because there
are fewer confusing tokens in the train/val batches. The main downside is that we now waste
about 35% of tokens due to cropping. This is ok because we have a lot of data. See dev/LOG.md
entry for this change for a lot more information.
 2026-01-13 20:05:47 +00:00
Andrej Karpathy 23985413aa adjust the comment on the regex pattern per recent experimnet see dev/LOG.md 2026-01-13 17:50:39 +00:00
Andrej Karpathy 64b48d0e5c validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs 2026-01-13 17:45:06 +00:00
Andrej Karpathy 238353c998 document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight. 2026-01-13 17:14:29 +00:00
Andrej Karpathy 4610a838a1 record negative result on MTP 2026-01-12 05:23:47 +00:00
Andrej Karpathy 21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway 2026-01-12 03:10:13 +00:00
Andrej Karpathy aa95fb2e03 make miniseries more generic and easier to run and less hard coded 2026-01-12 02:54:35 +00:00
Andrej Karpathy b33e394528 oops actually make SSSL the default window pattern 2026-01-11 21:50:35 +00:00
Andrej Karpathy fbc1484e8c add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb 2026-01-11 21:49:54 +00:00
Andrej Karpathy 2ff7d51252 integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge 2026-01-11 20:33:19 +00:00
Andrej Karpathy 201d705957 recover the ability to load old checkpoints by patching the lambdas if they don't exist in checkpoints 2026-01-11 20:13:12 +00:00
Andrej Karpathy aa530cdad5 Add learnable lambdas that gate the residual connection and a skip connection to the input embeddings, solid bump to val_bpb 2026-01-11 18:47:35 +00:00
Andrej Karpathy 2c4473dd1b Big Muon optimizer changes inspired by latest of modded-nanogpt. Added Polar Express, Adafactor-style variance reduction, cautious weight decay, schedule weight decay linearly to ramp down to zero. Tuned optimum weight decay for multiple model sizes d8, d12, d16, d20 and found a scaling law with optimum wd \propto 1/channels^2, including it as default into code. --weight_decay of base_train is now default on and configured optimally according to all of these experiments. Solid bump to val_bpb observed as a result of these changes. 2026-01-11 16:56:59 +00:00
Andrej Karpathy f5a0ea4d3f take out these gitignore dirs 2026-01-08 18:18:42 +00:00
Andrej Karpathy 4ddc803797 fix adamw slight bug. this chunk was copy pasted originally from modded-nanogpt, which still seems to have the bug 2026-01-08 18:18:42 +00:00
Andrej Karpathy 061f83c152 delete grad_clip. appears to not be necessary at all. not only was it buggy because the clipping happened per gpu before grad synchronization, but it costs ~2% MFU, and it also doesn't even help. I tried deleting it a while ago and back then it did help. So I'm guessing that some hyperparameter tuning obviated the reason for it since then 2026-01-08 02:16:50 +00:00
Andrej Karpathy e8c30c3b19 add notebook used for scaling laws analysis 2026-01-07 22:28:53 +00:00
Andrej Karpathy 3af4dcf6ee also add scaling_laws.sh script if it's a useful reference 2026-01-07 22:25:13 +00:00
Andrej Karpathy 4cc605b940 quick pointer to miniseries post in readme for now 2026-01-07 22:14:21 +00:00
Andrej Karpathy ccf4b7f9bf nudge hyperparameters of the base script with the results of the sweeps and miniseries. vocab size down to 32K. D:N ratio from 20 to 8. add miniseries script 2026-01-07 22:11:59 +00:00
Andrej Karpathy ae0bf52529 tune hyperparameters based on overnight sweeps. warmdown_ratio is the biggest free win, increasing 0.2 -> 0.4, and embedding lr can be larger bumping 0.2 -> 0.3 2026-01-05 18:57:46 +00:00
Andrej Karpathy eec0c79563 also add matplotlib dep so that we can have jupyter notebooks 2026-01-05 18:41:09 +00:00
Andrej Karpathy 54e59c38ad add notebook on deriving the CORE estimates for the GPT-3 miniseries. 2026-01-05 18:40:28 +00:00
Andrej Karpathy 9d4c9b786d many small fixes to base_train: reporting ETA, allowing some additional kwarg flexibility, making sure we don't crash when e.g. depth = 11 - we now calculate the closest num_heads that works 2026-01-05 00:38:09 +00:00
Andrej Karpathy 962b6bfba3 alright add transformers as a dep of the repo because it should be easy to evaluate the CORE score of HF models. Not super happy about it but i tried it and the uv.lock doesn't get bloated as much as i expected 2026-01-04 20:37:28 +00:00
Andrej Karpathy ed2082fbc4 sane secrets management 2026-01-04 19:29:22 +00:00
Andrej Karpathy eb7bbc1b66 delete the configurator in favor of argparse and clean up a lot of kwarg details to make them more consistent across all scripts 2026-01-04 19:14:23 +00:00
Andrej Karpathy 507d54224a fix small bug where this would break if git stage has deleted files 2026-01-04 19:11:43 +00:00
Andrej Karpathy 9c60dfb64c bump nanochat to use the latest stable pytorch that is 2.9.1 . Run e.g. to re-update your local environment if you git pull 2026-01-04 18:36:36 +00:00
Andrej KarpathyandClaude Opus 4.5 be56d29b87 simplify redundant if/elif in bloat metrics 
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-04 01:40:42 +00:00
Andrej KarpathyandClaude Opus 4.5 ee79f29fbd replace files-to-prompt with git ls-files for bloat metrics 
files-to-prompt was including untracked files (knowledge/, dev scripts, etc.) which inflated the bloat metrics. now we use git ls-files to only count tracked source files, which is more accurate and removes an external dependency.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-04 01:38:15 +00:00
Andrej Karpathy da8b7ea4cb also delete the rustbpe test code, this now lives in rustbpe repo that is separate 2026-01-04 01:23:34 +00:00
Andrej Karpathy aa42f40e66 delete the inline rustbpe project. it was ugly to have a project within project and rustbpe is now nicely a separate repo on my github karpathy/rustbpe and it's on pypi etc., so we just add it as a depedency to uv. i think it is appropriate that this is a separate repo because 1) it doesn't have too many knobs, other than the ones that are exposed - the regex pattern and vocab size and 2) all of its complexity is not algorithmic (it's equivalent to minbpe), instead it is efficiency-related, so it is ok to hide relatively speaking 2026-01-03 23:55:28 +00:00
Andrej Karpathy 48abd7d85f simplify, clarify and slightly tune model initialization. should be very slightly better possibly, but certainly a lot clearer 2026-01-01 21:15:09 +00:00
Andrej KarpathyandClaude Opus 4.5 8f979a8bda fix: sample first token independently for each row in multi-sample generation 
Previously, when generating multiple samples (num_samples > 1), the first
token after prefill was sampled once and broadcast to all rows, causing
all samples to start identically. Now the prefill logits are expanded to
num_samples and sampled independently for each row.

Also simplified the generation loop by moving the forward pass to the end
of the loop, eliminating the first_iteration flag and if/else branching.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-28 04:52:13 +00:00
Andrej KarpathyandClaude Opus 4.5 91d76cc690 Replace speedup assertion with warning in batch_encode test 
Performance varies by machine and load, making hard assertions flaky.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-28 04:10:49 +00:00
Andrej 7a8769a40c Merge pull request #383 from barisozmen/master 
3x faster rust encode (`batch_encode`) (12 LoC + 2 tests)
 2025-12-27 20:06:57 -08:00
Andrej 088726aa7d clean up model_tag handling across scripts a bit more. 2025-12-27 20:01:09 -08:00
Andrej Karpathy 2874eda59a update to new os env var to get rid of deprecation warning 2025-12-28 03:32:46 +00:00
Andrej Karpathy e1770a3061 remove spurious cast, gets compiled away anyway but it's confusing people 2025-12-27 23:07:48 +00:00
Andrej Karpathy 49389ecaa8 fix tf32 warning for deprecated api use 2025-12-27 22:03:06 +00:00
Andrej bc51da8bac pad vocab size to 64 for DDP optimizers and efficiency 2025-12-23 09:13:31 -08:00
Andrej d5759400f9 fixing two typos in comments 2025-12-08 20:03:08 -08:00
Andrej e72c3299df fix random.seed() footgun bug for SpellingBee data generation 2025-12-08 19:58:45 -08:00
Andrej 7931e0903a rename checkpoint_dir to checkpoints_dir for consistency. 2025-12-08 18:32:12 -08:00
Andrej 849d95ae1f remove unnecessary check to make the logic in CausalSelfAttention.forward() clearer 2025-12-08 18:30:37 -08:00
Andrej 39cccc527f small bugfix make mid_train script work even with a tiny number of iterations 2025-12-08 18:27:32 -08:00
AndrejandSofie Van Landeghem 8b1cecaa95 Apply suggestion from @svlandeg for nicer looking comparison 
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
 2025-12-08 18:27:06 -08:00
Andrej 58f3e84e01 clean up train/val loader in sft for consistency with mid/base 2025-12-08 18:23:57 -08:00
Andrej 1b2a675c88 Improve KV cache code readability 2025-12-08 18:19:05 -08:00
Andrej d75e6ed711 Fix script comment to reference correct file 2025-12-08 18:16:42 -08:00
Andrej 72a7cf2bc4 Fix distributed Parquet dataloader resume for multi-epoch training 2025-12-08 18:15:02 -08:00
Andrej Karpathy bffdb2ef91 group common code to make things neater in gpt logit computation 2025-12-09 02:01:05 +00:00
Andrej cbf30c842c apply float32 cast before logits softcapping so the tanh is in fp32. torch compile fuses this correctly with no extra memory costs. 2025-12-08 14:17:43 -08:00
Andrej Karpathy 90442de35f fix bug where any rank has to be able to create checkpoint_dir if saving optim 2025-12-08 20:45:19 +00:00
Andrej 2fd0440355 fix: missing val_bpb on resume 2025-12-08 12:35:08 -08:00
Andrej 4a87a0d19f Merge pull request #299 from samjabrahams/rotary_embedding_head_dim_comment_cleanup 
Fix comment: rotary embeddings final dimension size
 2025-11-17 13:29:21 -08:00
Andrej Karpathy bc1fca39f3 mqa -> gqa to reduce confusion 2025-11-15 15:43:37 +00:00
Andrej f66a780f68 Fix torch.dtype mismatching when running engine inline test. 2025-11-14 07:28:29 -08:00
Andrej 4763ce612a Small fixes to typos 2025-11-14 07:25:59 -08:00
Andrej Karpathy 9a71d13688 typo oops 2025-11-13 16:08:30 +00:00
Andrej Karpathy 7b7fd0fe71 thank you Sophie for your help with nanochat 2025-11-13 16:07:54 +00:00
Andrej Karpathy c6abcdfe3a big change: add pretraining resumption logic so that checkpoints can now be approximately resumed and training can continue. this is useful for very long runs when you don't want the anxiety of your run crashing for some reason. alternatively, it's a way to recover training in the event of loss spikes. i mean, this should have been there in v0 but it's ok. the resumption is approximate to control complexity and bloat, but it's possible we want to change that in the future. to use, set --save_every to a step interval to write checkpoints with, and then use --resume_from_step to resume optimization from a given step. only base model training (pretraining) supports this atm, but it's ok because midtraining is comparably quite a bit faster. 2025-11-13 15:34:40 +00:00
Andrej Karpathy 91f09ccd0d minor fix comment in engine 2025-11-13 15:28:18 +00:00
Andrej Karpathy adb5d4a16c uv lock has to change when we removed numpy the other commit 2025-11-13 15:16:27 +00:00
Andrej Karpathy c6b7ab7440 grad clip logging and printing and cosmetics 2025-11-05 21:08:30 +00:00
Andrej 885a4f25e7 Replace fcntl with filelock for Windows compatibility 2025-11-04 16:35:39 -08:00
Andrej 3a2ae631c4 Merge branch 'master' into master 2025-11-04 16:35:02 -08:00
Andrej 12d995f58c Add NPROC_PER_NODE var to speedrun.sh and run1000.sh 2025-11-04 16:26:33 -08:00
Andrej d1558c7873 handle bf16 on MPS by casting to fp32 during load checkpoint 2025-11-04 09:42:50 -08:00
Andrej df25293087 Add explicit UTF-8 encoding on open 2025-11-04 09:38:18 -08:00
Andrej a83646e098 fix(eval): use UTF-8 when reading CORE JSONL and writing CSV 2025-11-03 06:38:33 -08:00
Andrej 8681922328 fix lstrip bug, make it removeprefix, TIL. 2025-11-03 06:37:48 -08:00
Andrej b6da6982f6 fix nanochat logo: the t was placed too far to the right 2025-11-02 08:17:00 -08:00
Andrej c2c4f77e22 oops small bugfix to run1000.sh missing kwarg 2025-11-02 08:14:41 -08:00
Andrej d1ac0b2d07 when loading models on CPU, convert tensors from bfloat16 to float 2025-11-02 07:58:56 -08:00
Andrej Karpathy cf587acb1a move eval bundle download to be lazy and inside the python code so that we can substantially simplify the run bash scripts 2025-11-01 16:04:38 +00:00
Andrej Karpathy 7d2c4a3d95 delete pandas dep in base_eval use csv instead 2025-11-01 15:28:30 +00:00
Andrej ad39db5a23 tiny fix to comment 
Update engine.py with correct error message on assert
 2025-11-01 07:43:57 -07:00
Andrej 630f54ae5a use empty locals and globals in call to eval() in engine tool use 
harden eval: prevent the calc tool from accessing globals and locals
 2025-11-01 07:22:59 -07:00
Andrej Karpathy f15732524a make deepwiki link better 2025-11-01 14:13:29 +00:00
Andrej dfc88334b6 fix tok/sec calculation bug when grad accum steps > 1 
Fix tok/sec metrics for base_train and mid_train when gradient accumulation is not 1
 2025-10-30 08:36:32 -07:00
Andrej eb11bb0e2e remove numpy as dep 
Remove explicit numpy dependency
 2025-10-30 08:28:14 -07:00
Andrej 1ccbaf4416 nit delete redundant catch/raise in execute 
Remove redundant exception handling in chdir
 2025-10-29 08:10:03 -07:00
Andrej 29ff38d94b Merge pull request #35 from bhaskar0210s/master 
fix: return inf instead of crashing when evaluate_bpb has zero total_bytes
 2025-10-29 08:06:24 -07:00
Andrej a1de1f46ad Merge pull request #156 from tlepoint/fix/export-base-dir 
Export the base dir variable in runcpu.sh
 2025-10-28 15:19:08 -07:00
Andrej ee00f523d0 fixing all the typos to make the pull requests stop 
Batch of typo fixes
 2025-10-28 13:36:07 -07:00
Andrej Karpathy fdda5826e3 Merge branch 'haowei01-fix_kv_cache_due_to_resize' 2025-10-28 16:54:30 +00:00