fam/nanochat-omni
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
186 Commits 3 Branches 0 Tags
2f2d7ab80cd07a8b5fab9feebcb185fd3ca37339
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrej KarpathyandClaude Opus 4.5 91d76cc690 Replace speedup assertion with warning in batch_encode test 
Performance varies by machine and load, making hard assertions flaky.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-28 04:10:49 +00:00
Andrej 7a8769a40c Merge pull request #383 from barisozmen/master 
3x faster rust encode (`batch_encode`) (12 LoC + 2 tests)
 2025-12-27 20:06:57 -08:00
Andrej 088726aa7d clean up model_tag handling across scripts a bit more. 2025-12-27 20:01:09 -08:00
Andrej Karpathy 2874eda59a update to new os env var to get rid of deprecation warning 2025-12-28 03:32:46 +00:00
Andrej Karpathy e1770a3061 remove spurious cast, gets compiled away anyway but it's confusing people 2025-12-27 23:07:48 +00:00
Andrej Karpathy 49389ecaa8 fix tf32 warning for deprecated api use 2025-12-27 22:03:06 +00:00
Andrej bc51da8bac pad vocab size to 64 for DDP optimizers and efficiency 2025-12-23 09:13:31 -08:00
Andrej d5759400f9 fixing two typos in comments 2025-12-08 20:03:08 -08:00
Andrej e72c3299df fix random.seed() footgun bug for SpellingBee data generation 2025-12-08 19:58:45 -08:00
Andrej 7931e0903a rename checkpoint_dir to checkpoints_dir for consistency. 2025-12-08 18:32:12 -08:00
Andrej 849d95ae1f remove unnecessary check to make the logic in CausalSelfAttention.forward() clearer 2025-12-08 18:30:37 -08:00
Andrej 39cccc527f small bugfix make mid_train script work even with a tiny number of iterations 2025-12-08 18:27:32 -08:00
AndrejandSofie Van Landeghem 8b1cecaa95 Apply suggestion from @svlandeg for nicer looking comparison 
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
 2025-12-08 18:27:06 -08:00
Andrej 58f3e84e01 clean up train/val loader in sft for consistency with mid/base 2025-12-08 18:23:57 -08:00
Andrej 1b2a675c88 Improve KV cache code readability 2025-12-08 18:19:05 -08:00
Andrej d75e6ed711 Fix script comment to reference correct file 2025-12-08 18:16:42 -08:00
Andrej 72a7cf2bc4 Fix distributed Parquet dataloader resume for multi-epoch training 2025-12-08 18:15:02 -08:00
Andrej Karpathy bffdb2ef91 group common code to make things neater in gpt logit computation 2025-12-09 02:01:05 +00:00
Andrej cbf30c842c apply float32 cast before logits softcapping so the tanh is in fp32. torch compile fuses this correctly with no extra memory costs. 2025-12-08 14:17:43 -08:00
Andrej Karpathy 90442de35f fix bug where any rank has to be able to create checkpoint_dir if saving optim 2025-12-08 20:45:19 +00:00
Andrej 2fd0440355 fix: missing val_bpb on resume 2025-12-08 12:35:08 -08:00
Andrej 4a87a0d19f Merge pull request #299 from samjabrahams/rotary_embedding_head_dim_comment_cleanup 
Fix comment: rotary embeddings final dimension size
 2025-11-17 13:29:21 -08:00
Andrej Karpathy bc1fca39f3 mqa -> gqa to reduce confusion 2025-11-15 15:43:37 +00:00
Andrej f66a780f68 Fix torch.dtype mismatching when running engine inline test. 2025-11-14 07:28:29 -08:00
Andrej 4763ce612a Small fixes to typos 2025-11-14 07:25:59 -08:00
Andrej Karpathy 9a71d13688 typo oops 2025-11-13 16:08:30 +00:00
Andrej Karpathy 7b7fd0fe71 thank you Sophie for your help with nanochat 2025-11-13 16:07:54 +00:00
Andrej Karpathy c6abcdfe3a big change: add pretraining resumption logic so that checkpoints can now be approximately resumed and training can continue. this is useful for very long runs when you don't want the anxiety of your run crashing for some reason. alternatively, it's a way to recover training in the event of loss spikes. i mean, this should have been there in v0 but it's ok. the resumption is approximate to control complexity and bloat, but it's possible we want to change that in the future. to use, set --save_every to a step interval to write checkpoints with, and then use --resume_from_step to resume optimization from a given step. only base model training (pretraining) supports this atm, but it's ok because midtraining is comparably quite a bit faster. 2025-11-13 15:34:40 +00:00
Andrej Karpathy 91f09ccd0d minor fix comment in engine 2025-11-13 15:28:18 +00:00
Andrej Karpathy adb5d4a16c uv lock has to change when we removed numpy the other commit 2025-11-13 15:16:27 +00:00
Andrej Karpathy c6b7ab7440 grad clip logging and printing and cosmetics 2025-11-05 21:08:30 +00:00
Andrej 885a4f25e7 Replace fcntl with filelock for Windows compatibility 2025-11-04 16:35:39 -08:00
Andrej 3a2ae631c4 Merge branch 'master' into master 2025-11-04 16:35:02 -08:00
Andrej 12d995f58c Add NPROC_PER_NODE var to speedrun.sh and run1000.sh 2025-11-04 16:26:33 -08:00
Andrej d1558c7873 handle bf16 on MPS by casting to fp32 during load checkpoint 2025-11-04 09:42:50 -08:00
Andrej df25293087 Add explicit UTF-8 encoding on open 2025-11-04 09:38:18 -08:00
Andrej a83646e098 fix(eval): use UTF-8 when reading CORE JSONL and writing CSV 2025-11-03 06:38:33 -08:00
Andrej 8681922328 fix lstrip bug, make it removeprefix, TIL. 2025-11-03 06:37:48 -08:00
Andrej b6da6982f6 fix nanochat logo: the t was placed too far to the right 2025-11-02 08:17:00 -08:00
Andrej c2c4f77e22 oops small bugfix to run1000.sh missing kwarg 2025-11-02 08:14:41 -08:00
Andrej d1ac0b2d07 when loading models on CPU, convert tensors from bfloat16 to float 2025-11-02 07:58:56 -08:00
Andrej Karpathy cf587acb1a move eval bundle download to be lazy and inside the python code so that we can substantially simplify the run bash scripts 2025-11-01 16:04:38 +00:00
Andrej Karpathy 7d2c4a3d95 delete pandas dep in base_eval use csv instead 2025-11-01 15:28:30 +00:00
Andrej ad39db5a23 tiny fix to comment 
Update engine.py with correct error message on assert
 2025-11-01 07:43:57 -07:00
Andrej 630f54ae5a use empty locals and globals in call to eval() in engine tool use 
harden eval: prevent the calc tool from accessing globals and locals
 2025-11-01 07:22:59 -07:00
Andrej Karpathy f15732524a make deepwiki link better 2025-11-01 14:13:29 +00:00
Andrej dfc88334b6 fix tok/sec calculation bug when grad accum steps > 1 
Fix tok/sec metrics for base_train and mid_train when gradient accumulation is not 1
 2025-10-30 08:36:32 -07:00
Andrej eb11bb0e2e remove numpy as dep 
Remove explicit numpy dependency
 2025-10-30 08:28:14 -07:00
Andrej 1ccbaf4416 nit delete redundant catch/raise in execute 
Remove redundant exception handling in chdir
 2025-10-29 08:10:03 -07:00
Andrej 29ff38d94b Merge pull request #35 from bhaskar0210s/master 
fix: return inf instead of crashing when evaluate_bpb has zero total_bytes
 2025-10-29 08:06:24 -07:00
Andrej a1de1f46ad Merge pull request #156 from tlepoint/fix/export-base-dir 
Export the base dir variable in runcpu.sh
 2025-10-28 15:19:08 -07:00
Andrej ee00f523d0 fixing all the typos to make the pull requests stop 
Batch of typo fixes
 2025-10-28 13:36:07 -07:00
Andrej Karpathy fdda5826e3 Merge branch 'haowei01-fix_kv_cache_due_to_resize' 2025-10-28 16:54:30 +00:00
Andrej Karpathy baf0b3fdda also add a test that failed before the fix and passes now with the fix for kv cache resize 2025-10-28 16:54:17 +00:00
Andrej Karpathy f1db6b4712 delete czar call for help, i'm working through the inbound on that now. add current LLM policy which just asks for disclosure atm 2025-10-28 16:51:41 +00:00
Andrej Karpathy 9415931f85 delete czar call for help, i'm working through the inbound on that now. add current LLM policy which just asks for disclosure atm 2025-10-28 15:17:43 +00:00
Andrej Karpathy c75fe54aa7 readme tweak, link to new discussion and add file structure 2025-10-25 19:39:16 +00:00
Andrej Karpathy 05a051dbe9 fix tokenization bug, there should be no space before first letter. sigh 2025-10-24 15:06:06 +00:00
Andrej Karpathy 8892470f29 add the SpellingBee task so that nanochat can count r in strawberry etc. along the way we had to add a bunch of new functionality, e.g. extend the calculator to support the count function of python. possibly the current TaskMixture uses way too many synthetic examples of SpellingBee because the eval gives us exactly 100% performance on spelling. We can tune this later to reclaim some wall clock time here I think 2025-10-24 14:02:48 +00:00
Andrej Karpathy 81597cd616 move the lr schedule args up in base_train so they are tunable in configurator 2025-10-24 13:27:31 +00:00
Andrej Karpathy cc3636b01c allow the tokenizer visualize_tokenization to also print the exact token id. you can never be paranoid enough 2025-10-24 13:27:05 +00:00
Andrej Karpathy 5eeb2b6ef9 experiment: looking to 'hire' a nanochat repo czar to help the repo, mentioning in readme 2025-10-22 16:55:54 +00:00
Andrej Karpathy 2dda5c4c8d Merge branch 'ulanch-fix/ios-safari-input-overlap' 2025-10-22 16:26:35 +00:00
Andrej Karpathy 80b203ea59 also bump run1000.sh to new uv sync 2025-10-22 16:25:36 +00:00
Andrej Karpathy 51e70f0d3c Merge branch 'lukestanley-fix-cpu-support-with-extras' 2025-10-22 16:11:15 +00:00
Andrej Karpathy 48387cd895 also bump run1000.sh to new uv sync 2025-10-22 16:08:31 +00:00
Andrej 2e938530ce delete spurious torch.empty allocation in adamw 
fix: remove unnecessary tensor allocation in DistAdamW optimizer
 2025-10-21 11:35:17 -07:00
Andrej Karpathy a088b7a6ec use enable_gqa of pytorch sdpa, allows us to delete some code, didnt realize it's available 2025-10-21 18:07:33 +00:00
Andrej Karpathy 94ee507054 quick fix base eval due to fewshot requirement 2025-10-21 17:56:08 +00:00
Andrej 33e8a27f91 Merge karpathy/cpu-mps-dev , adding the ability to run on CPU, on MPS, or on CUDA, with autodetect. Gnarly PR, nonzero chance I broke something. 
add cpu|mps support
 2025-10-21 10:26:04 -07:00
Andrej Karpathy 50bea28ef9 also add readme mention of the cpu mps changes 2025-10-21 17:24:48 +00:00
Andrej Karpathy 5bdc99abfb merge and resolve conflict 2025-10-21 17:19:10 +00:00
Andrej Karpathy dfcb1c16f1 Merge branch 'master' into cpu-mps-dev 2025-10-21 17:15:53 +00:00
Andrej Karpathy bb71c64579 fix silly issue in dataloader, this version is much faster and more portable to mps too 2025-10-21 17:12:50 +00:00
karpathy bb786c5560 i shouldnt have committed the lock file, i missed that. revert to the flagship build which is linux. sorry to pollute the repo history... 2025-10-21 10:07:40 -07:00
Andrej c9ea7a91e2 Add customization instructions to README 
Added a section on customization for nanochat.
 2025-10-21 08:57:10 -07:00
Andrej Karpathy 03cddd9878 actually let's not brick code on git pull. change error to warning 2025-10-21 15:13:25 +00:00
Andrej Karpathy fe5aed940b add personality to nanochat. breaks previous code on git pull and requires download of a new file from s3, but there is a helpful error message so hopefully its ok 2025-10-21 15:04:58 +00:00
karpathy 2e9669e03a upgrading all other files to be able to use cpu/mps as well as cuda. various minor other changes ,e.g. changing max_iterations to num_iterations in sft script for consistency in naming 2025-10-20 10:15:17 -07:00
Andrej a09ac812ed toml changes for cpu only install 2025-10-20 07:53:15 -07:00
Andrej 0f007889dd Add MIT License as a file to the project 2025-10-19 17:22:19 -07:00
Andrej 5a879f4947 export NANOCHAT_BASE_DIR so child processes get it too 2025-10-19 17:07:56 -07:00
Andrej Karpathy c1d2ed1c13 use orig_model in sampling, silly of me to miss this 2025-10-20 00:05:09 +00:00
Andrej Karpathy 2bc521a6de use orig_model in sampling, silly of me to miss this 2025-10-20 00:04:15 +00:00
Andrej Karpathy 9467d83cf2 fix memory leak bug in rust tokenizer ty @mitsuhiko 2025-10-19 23:54:31 +00:00
AndrejandTancrède Lepoint cf2baf9933 fix typo 
Co-authored-by: Tancrède Lepoint <tlepoint@users.noreply.github.com>
 2025-10-17 08:35:41 -07:00
karpathy e4f9b9c64d revert to previous pyproject.toml 2025-10-17 08:08:16 -07:00
Andrej e883b1d597 Merge pull request #99 from burtenshaw/cpu-mps-dev-ben 
Add mps and cpu dependency management
 2025-10-17 07:24:38 -07:00
karpathy ae02650afe update the midtraining script too 2025-10-16 16:33:17 -07:00
karpathy df600b6ed5 many small tweaks. base, eval, core work now i think 2025-10-16 15:46:18 -07:00
Andrej Karpathy d6d86cbf4c update readme with a link to the CPU|MPS branch 2025-10-16 22:03:39 +00:00
Andrej Karpathy ccfe7915ac mention the current d32 chat hosted on nanochat.karpathy.ai, as an example endpoint of the repo 2025-10-16 19:32:44 +00:00
karpathy 786119d593 add autodetect of device and related stuff. getting weird warnings/errors still, so wip 2025-10-16 10:26:19 -07:00
karpathy 279b74312c adjust comment/guidance on device type 2025-10-16 10:06:39 -07:00
karpathy 306bc380ab add support for CPU and for MPS. I had to change a few cosmetic things. I also discovered I think a bit of a bug, where I was casting wte to bfloat16 in the wrong place (the model init) instead of in init_weights 2025-10-16 10:04:43 -07:00
Andrej Karpathy 722da4f543 trying to add basic cpu support, will try mps too 2025-10-16 16:14:38 +00:00
Andrej Karpathy 4346536ab2 also allow regenerating assistant message by clicking it, and make sure to feed good seed to generate 2025-10-16 01:28:37 +00:00
Andrej Karpathy 2846999b8f allow user to click on their message to edit them. conversation after that point is wiped 2025-10-16 01:16:22 +00:00
Andrej Karpathy 92d52ecc92 add slash commands to webui 2025-10-16 01:09:53 +00:00
Andrej Karpathy fae3aca951 add script to train a 000 version of nanochat. currently it's a bit more like 00 and this would run in probably around 33 hours instead of the budget of 41 hours, so we might tune it later. i think it's ok for now 2025-10-15 20:32:22 +00:00