fam/nanochat-omni
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
95 Commits 3 Branches 0 Tags
036a3c5881c7e6430d5565bf8f1224fef54cdc82
Commit Graph
67 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrej dfc88334b6 fix tok/sec calculation bug when grad accum steps > 1 
Fix tok/sec metrics for base_train and mid_train when gradient accumulation is not 1
 2025-10-30 08:36:32 -07:00
Andrej eb11bb0e2e remove numpy as dep 
Remove explicit numpy dependency
 2025-10-30 08:28:14 -07:00
Andrej 1ccbaf4416 nit delete redundant catch/raise in execute 
Remove redundant exception handling in chdir
 2025-10-29 08:10:03 -07:00
Andrej 29ff38d94b Merge pull request #35 from bhaskar0210s/master 
fix: return inf instead of crashing when evaluate_bpb has zero total_bytes
 2025-10-29 08:06:24 -07:00
Andrej a1de1f46ad Merge pull request #156 from tlepoint/fix/export-base-dir 
Export the base dir variable in runcpu.sh
 2025-10-28 15:19:08 -07:00
Andrej ee00f523d0 fixing all the typos to make the pull requests stop 
Batch of typo fixes
 2025-10-28 13:36:07 -07:00
Andrej Karpathy fdda5826e3 Merge branch 'haowei01-fix_kv_cache_due_to_resize' 2025-10-28 16:54:30 +00:00
Andrej Karpathy baf0b3fdda also add a test that failed before the fix and passes now with the fix for kv cache resize 2025-10-28 16:54:17 +00:00
Andrej Karpathy f1db6b4712 delete czar call for help, i'm working through the inbound on that now. add current LLM policy which just asks for disclosure atm 2025-10-28 16:51:41 +00:00
Andrej Karpathy 9415931f85 delete czar call for help, i'm working through the inbound on that now. add current LLM policy which just asks for disclosure atm 2025-10-28 15:17:43 +00:00
Andrej Karpathy c75fe54aa7 readme tweak, link to new discussion and add file structure 2025-10-25 19:39:16 +00:00
Andrej Karpathy 05a051dbe9 fix tokenization bug, there should be no space before first letter. sigh 2025-10-24 15:06:06 +00:00
Andrej Karpathy 8892470f29 add the SpellingBee task so that nanochat can count r in strawberry etc. along the way we had to add a bunch of new functionality, e.g. extend the calculator to support the count function of python. possibly the current TaskMixture uses way too many synthetic examples of SpellingBee because the eval gives us exactly 100% performance on spelling. We can tune this later to reclaim some wall clock time here I think 2025-10-24 14:02:48 +00:00
Andrej Karpathy 81597cd616 move the lr schedule args up in base_train so they are tunable in configurator 2025-10-24 13:27:31 +00:00
Andrej Karpathy cc3636b01c allow the tokenizer visualize_tokenization to also print the exact token id. you can never be paranoid enough 2025-10-24 13:27:05 +00:00
Andrej Karpathy 5eeb2b6ef9 experiment: looking to 'hire' a nanochat repo czar to help the repo, mentioning in readme 2025-10-22 16:55:54 +00:00
Andrej Karpathy 2dda5c4c8d Merge branch 'ulanch-fix/ios-safari-input-overlap' 2025-10-22 16:26:35 +00:00
Andrej Karpathy 80b203ea59 also bump run1000.sh to new uv sync 2025-10-22 16:25:36 +00:00
Andrej Karpathy 51e70f0d3c Merge branch 'lukestanley-fix-cpu-support-with-extras' 2025-10-22 16:11:15 +00:00
Andrej Karpathy 48387cd895 also bump run1000.sh to new uv sync 2025-10-22 16:08:31 +00:00
Andrej 2e938530ce delete spurious torch.empty allocation in adamw 
fix: remove unnecessary tensor allocation in DistAdamW optimizer
 2025-10-21 11:35:17 -07:00
Andrej Karpathy a088b7a6ec use enable_gqa of pytorch sdpa, allows us to delete some code, didnt realize it's available 2025-10-21 18:07:33 +00:00
Andrej Karpathy 94ee507054 quick fix base eval due to fewshot requirement 2025-10-21 17:56:08 +00:00
Andrej 33e8a27f91 Merge karpathy/cpu-mps-dev , adding the ability to run on CPU, on MPS, or on CUDA, with autodetect. Gnarly PR, nonzero chance I broke something. 
add cpu|mps support
 2025-10-21 10:26:04 -07:00
Andrej Karpathy 50bea28ef9 also add readme mention of the cpu mps changes 2025-10-21 17:24:48 +00:00
Andrej Karpathy 5bdc99abfb merge and resolve conflict 2025-10-21 17:19:10 +00:00
Andrej Karpathy dfcb1c16f1 Merge branch 'master' into cpu-mps-dev 2025-10-21 17:15:53 +00:00
Andrej Karpathy bb71c64579 fix silly issue in dataloader, this version is much faster and more portable to mps too 2025-10-21 17:12:50 +00:00
karpathy bb786c5560 i shouldnt have committed the lock file, i missed that. revert to the flagship build which is linux. sorry to pollute the repo history... 2025-10-21 10:07:40 -07:00
Andrej c9ea7a91e2 Add customization instructions to README 
Added a section on customization for nanochat.
 2025-10-21 08:57:10 -07:00
Andrej Karpathy 03cddd9878 actually let's not brick code on git pull. change error to warning 2025-10-21 15:13:25 +00:00
Andrej Karpathy fe5aed940b add personality to nanochat. breaks previous code on git pull and requires download of a new file from s3, but there is a helpful error message so hopefully its ok 2025-10-21 15:04:58 +00:00
karpathy 2e9669e03a upgrading all other files to be able to use cpu/mps as well as cuda. various minor other changes ,e.g. changing max_iterations to num_iterations in sft script for consistency in naming 2025-10-20 10:15:17 -07:00
Andrej a09ac812ed toml changes for cpu only install 2025-10-20 07:53:15 -07:00
Andrej 0f007889dd Add MIT License as a file to the project 2025-10-19 17:22:19 -07:00
Andrej 5a879f4947 export NANOCHAT_BASE_DIR so child processes get it too 2025-10-19 17:07:56 -07:00
Andrej Karpathy c1d2ed1c13 use orig_model in sampling, silly of me to miss this 2025-10-20 00:05:09 +00:00
Andrej Karpathy 2bc521a6de use orig_model in sampling, silly of me to miss this 2025-10-20 00:04:15 +00:00
Andrej Karpathy 9467d83cf2 fix memory leak bug in rust tokenizer ty @mitsuhiko 2025-10-19 23:54:31 +00:00
AndrejandTancrède Lepoint cf2baf9933 fix typo 
Co-authored-by: Tancrède Lepoint <tlepoint@users.noreply.github.com>
 2025-10-17 08:35:41 -07:00
karpathy e4f9b9c64d revert to previous pyproject.toml 2025-10-17 08:08:16 -07:00
Andrej e883b1d597 Merge pull request #99 from burtenshaw/cpu-mps-dev-ben 
Add mps and cpu dependency management
 2025-10-17 07:24:38 -07:00
karpathy ae02650afe update the midtraining script too 2025-10-16 16:33:17 -07:00
karpathy df600b6ed5 many small tweaks. base, eval, core work now i think 2025-10-16 15:46:18 -07:00
Andrej Karpathy d6d86cbf4c update readme with a link to the CPU|MPS branch 2025-10-16 22:03:39 +00:00
Andrej Karpathy ccfe7915ac mention the current d32 chat hosted on nanochat.karpathy.ai, as an example endpoint of the repo 2025-10-16 19:32:44 +00:00
karpathy 786119d593 add autodetect of device and related stuff. getting weird warnings/errors still, so wip 2025-10-16 10:26:19 -07:00
karpathy 279b74312c adjust comment/guidance on device type 2025-10-16 10:06:39 -07:00
karpathy 306bc380ab add support for CPU and for MPS. I had to change a few cosmetic things. I also discovered I think a bit of a bug, where I was casting wte to bfloat16 in the wrong place (the model init) instead of in init_weights 2025-10-16 10:04:43 -07:00
Andrej Karpathy 722da4f543 trying to add basic cpu support, will try mps too 2025-10-16 16:14:38 +00:00
Andrej Karpathy 4346536ab2 also allow regenerating assistant message by clicking it, and make sure to feed good seed to generate 2025-10-16 01:28:37 +00:00
Andrej Karpathy 2846999b8f allow user to click on their message to edit them. conversation after that point is wiped 2025-10-16 01:16:22 +00:00
Andrej Karpathy 92d52ecc92 add slash commands to webui 2025-10-16 01:09:53 +00:00
Andrej Karpathy fae3aca951 add script to train a 000 version of nanochat. currently it's a bit more like 00 and this would run in probably around 33 hours instead of the budget of 41 hours, so we might tune it later. i think it's ok for now 2025-10-15 20:32:22 +00:00
Andrej Karpathy 4c3590c499 fix subtle issue in token decoding in cases where multiple utf8 bytes need to be emitted into a single codepoint. exampels are emoji or foreign languages. basically we have to accumulate token sequences/text and only emit when we get full codepoints 2025-10-15 20:29:54 +00:00
Andrej Karpathy 03fa673b7d add basic logging to chat_web, which i think might be fun 2025-10-15 19:51:06 +00:00
Andrej Karpathy 52bfeea8bd add very basic abuse prevention limits to chat_web so it's ok to host endpoints 2025-10-15 19:42:54 +00:00
Andrej Karpathy 01fb290f53 allow multiple GPUs to do inference in a data parallel way 2025-10-15 19:12:19 +00:00
Andrej Karpathy 190d9515d0 dont evaluate the sampling evals during SFT they are too slow. keep the multiple choice evals. delete unused imports 2025-10-15 16:42:23 +00:00
Andrej Karpathy b8076dd367 fix bug in learning rate multiplier, it was ramping up instead of ramping down. see more in Issue #68. also add --dry_run option useful for experimentation 2025-10-15 16:35:04 +00:00
Andrej 67aaca98f5 export NANOCHAT_BASE_DIR so child processes get it too 
Export the cache directory so that users can use their own cache location
 2025-10-14 16:01:28 -07:00
Andrej dd6ff9a1cc fix bug in fallback case of find_largest_model 
Fix: Handle missing d<number> model tags in find_largest_model
ty
 2025-10-13 14:38:34 -07:00
Andrej 5fd0b13886 Merge pull request #2 from epoyraz/patch-1 
Update README.md
 2025-10-13 10:10:15 -07:00
Andrej 626bd3e260 Add image of the WebUI to readme 2025-10-13 08:03:00 -07:00
karpathy da96b46565 update link to the new discussion 2025-10-13 07:42:09 -07:00
karpathy a53833d04f add nanochat logo png 2025-10-13 06:59:59 -07:00
karpathy 3a5e0bc50b initial commit 2025-10-13 06:49:24 -07:00