eb7bbc1b66
delete the configurator in favor of argparse and clean up a lot of kwarg details to make them more consistent across all scripts
Andrej Karpathy2026-01-04 19:14:23 +00:00
507d54224a
fix small bug where this would break if git stage has deleted files
Andrej Karpathy2026-01-04 19:11:43 +00:00
9c60dfb64c
bump nanochat to use the latest stable pytorch that is 2.9.1 . Run e.g. to re-update your local environment if you git pull
Andrej Karpathy2026-01-04 18:36:36 +00:00
da8b7ea4cb
also delete the rustbpe test code, this now lives in rustbpe repo that is separate
Andrej Karpathy2026-01-04 01:23:34 +00:00
aa42f40e66
delete the inline rustbpe project. it was ugly to have a project within project and rustbpe is now nicely a separate repo on my github karpathy/rustbpe and it's on pypi etc., so we just add it as a depedency to uv. i think it is appropriate that this is a separate repo because 1) it doesn't have too many knobs, other than the ones that are exposed - the regex pattern and vocab size and 2) all of its complexity is not algorithmic (it's equivalent to minbpe), instead it is efficiency-related, so it is ok to hide relatively speaking
Andrej Karpathy2026-01-03 23:55:28 +00:00
48abd7d85f
simplify, clarify and slightly tune model initialization. should be very slightly better possibly, but certainly a lot clearer
Andrej Karpathy2026-01-01 21:14:26 +00:00
10231dfb40
Fix conversation scroll to bottom on some browsers + remove duplicated padding (#348)
Paweł Krefta2025-12-31 22:03:22 +01:00
389d019a0b
small change to doc string at top of tok_train.py (#402)
helloaidank2025-12-31 20:57:26 +00:00
72a7cf2bc4
Fix distributed Parquet dataloader resume for multi-epoch training
Andrej2025-12-08 18:15:02 -08:00
bffdb2ef91
group common code to make things neater in gpt logit computation
Andrej Karpathy2025-12-09 02:01:05 +00:00
cbf30c842c
apply float32 cast before logits softcapping so the tanh is in fp32. torch compile fuses this correctly with no extra memory costs.
Andrej2025-12-08 14:17:43 -08:00
90442de35f
fix bug where any rank has to be able to create checkpoint_dir if saving optim
Andrej Karpathy2025-12-08 20:45:11 +00:00
2fd0440355
fix: missing val_bpb on resume
Andrej2025-12-08 12:35:08 -08:00
01ea71be39
Fix distributed Parquet dataloader resume for multi-epoch training
sunyujun032025-12-08 00:10:19 -06:00
c6abcdfe3a
big change: add pretraining resumption logic so that checkpoints can now be approximately resumed and training can continue. this is useful for very long runs when you don't want the anxiety of your run crashing for some reason. alternatively, it's a way to recover training in the event of loss spikes. i mean, this should have been there in v0 but it's ok. the resumption is approximate to control complexity and bloat, but it's possible we want to change that in the future. to use, set --save_every to a step interval to write checkpoints with, and then use --resume_from_step to resume optimization from a given step. only base model training (pretraining) supports this atm, but it's ok because midtraining is comparably quite a bit faster.
Andrej Karpathy2025-11-13 15:34:40 +00:00
d54c9cbf8c
CPU Support, as bfloat16 params breaks inference
Manuel Saelices2025-11-01 23:38:50 +01:00
cf587acb1a
move eval bundle download to be lazy and inside the python code so that we can substantially simplify the run bash scripts
Andrej Karpathy2025-11-01 16:04:38 +00:00
Andrej Karpathy