Fam Zheng
d923aa1e31
Hopfield + Hebbian hybrid memory system for LLMs. Two nights of experiments (16 iterations), validated on LongMemEval (ICLR 2025). Architecture: - Single-hop: Two-Stage Hopfield (NN top-20 → softmax settle) - Multi-hop: Hebbian W matrix with WTA pattern separation - 64% on LongMemEval (500 questions), retrieval-only, no LLM dependency - 4ms latency @ 20K memories, ~1GB VRAM Key findings: - Hopfield attention solved noise tolerance (20% → 100% vs flat Hebbian) - WTA pattern separation enables 20K+ capacity - Multi-hop associative chains (6 hops, CosSim=1.0) — RAG can't do this - MiniLM-L6 is optimal (discrimination gap > absolute similarity) - Paraphrase cue augmentation: 55% → 100% on synthetic, 36% → 64% on benchmark - SNN encoder viable (CosSim 0.99) but not needed for current architecture
1004 lines
28 KiB
JSON
1004 lines
28 KiB
JSON
[
|
|
{
|
|
"embed_dim": 256,
|
|
"num_neurons": 512,
|
|
"num_steps": 32,
|
|
"param_count": 1050882,
|
|
"final_mse": 0.0009838738478720188,
|
|
"final_cos": 0.9435875415802002,
|
|
"final_firing_rate": 0.051880836486816406,
|
|
"history": {
|
|
"train_mse": [
|
|
0.069805115647614,
|
|
0.03317520907148719,
|
|
0.030660584662109615,
|
|
0.03064859760925174,
|
|
0.029095640778541564,
|
|
0.029248102195560934,
|
|
0.029319513216614725,
|
|
0.02848552754148841,
|
|
0.027690605074167252,
|
|
0.026913467235863207,
|
|
0.02731475606560707,
|
|
0.026665027905255555,
|
|
0.02599358083680272,
|
|
0.026317695248872043,
|
|
0.024538520816713573,
|
|
0.023062017373740674,
|
|
0.021719680260866882,
|
|
0.04732392709702253,
|
|
0.03819723203778267,
|
|
0.021576149947941305,
|
|
0.014994329726323485,
|
|
0.012137911096215249,
|
|
0.01024961080402136,
|
|
0.008987846598029137,
|
|
0.008273491961881518,
|
|
0.007354579144157469,
|
|
0.0066329882014542815,
|
|
0.005774030019529164,
|
|
0.005045031616464257,
|
|
0.004489358142018318,
|
|
0.00404639852931723,
|
|
0.0036757208057679237,
|
|
0.0033611496910452843,
|
|
0.003032777295447886,
|
|
0.002800806146115065,
|
|
0.0025432993890717624,
|
|
0.0023150239139795303,
|
|
0.0021573907462880014,
|
|
0.0020038839371409266,
|
|
0.0018414875026792287,
|
|
0.0016897495719604195,
|
|
0.0015602599596604705,
|
|
0.001453629444586113,
|
|
0.0013390815351158381,
|
|
0.0012668997514992952,
|
|
0.0012032087601255626,
|
|
0.0011454780120402574,
|
|
0.0011037719319574534,
|
|
0.0010552910156548024,
|
|
0.0009910810680594296
|
|
],
|
|
"train_cos": [
|
|
0.0012227444094605744,
|
|
-0.0003334377077408135,
|
|
-0.0003810953057836741,
|
|
-0.002055961755104363,
|
|
-0.0006783111544791609,
|
|
-0.0022491408977657556,
|
|
0.0011839499929919839,
|
|
-0.003466577851213515,
|
|
-0.00026298052398487926,
|
|
-0.0016494732408318669,
|
|
0.0009448173572309315,
|
|
-0.0004931375151500106,
|
|
-0.0010346547293011098,
|
|
0.001151108997873962,
|
|
0.0024685982964001594,
|
|
0.016504984954372047,
|
|
0.13340667318552732,
|
|
0.3529015123844147,
|
|
0.4823670744895935,
|
|
0.5631650984287262,
|
|
0.6201732665300369,
|
|
0.6605822414159774,
|
|
0.6925546258687973,
|
|
0.7168649733066559,
|
|
0.7369866132736206,
|
|
0.7584922999143601,
|
|
0.7759296864271163,
|
|
0.7929383277893066,
|
|
0.807342067360878,
|
|
0.820455914735794,
|
|
0.8307053059339523,
|
|
0.8421838879585266,
|
|
0.851089358329773,
|
|
0.8606858491897583,
|
|
0.8687692016363144,
|
|
0.8779040515422821,
|
|
0.8842260271310807,
|
|
0.8899602562189102,
|
|
0.898174598813057,
|
|
0.9038983464241028,
|
|
0.909727829694748,
|
|
0.9144359439611435,
|
|
0.9191820919513702,
|
|
0.9243983566761017,
|
|
0.9265471071004867,
|
|
0.9307401806116105,
|
|
0.934401735663414,
|
|
0.9366297394037246,
|
|
0.9395659506320954,
|
|
0.9426635056734085
|
|
],
|
|
"epoch_time": [
|
|
0.3276407718658447,
|
|
0.17241811752319336,
|
|
0.14514493942260742,
|
|
0.15177059173583984,
|
|
0.15204524993896484,
|
|
0.14941978454589844,
|
|
0.14756560325622559,
|
|
0.14043617248535156,
|
|
0.13959145545959473,
|
|
0.13918566703796387,
|
|
0.13888287544250488,
|
|
0.1411905288696289,
|
|
0.1664130687713623,
|
|
0.14248085021972656,
|
|
0.13923144340515137,
|
|
0.14377117156982422,
|
|
0.1546025276184082,
|
|
0.15320706367492676,
|
|
0.13819193840026855,
|
|
0.13993167877197266,
|
|
0.14189887046813965,
|
|
0.14249157905578613,
|
|
0.1388845443725586,
|
|
0.13722634315490723,
|
|
0.13905000686645508,
|
|
0.13871026039123535,
|
|
0.13981223106384277,
|
|
0.14063715934753418,
|
|
0.13825201988220215,
|
|
0.14087605476379395,
|
|
0.13816523551940918,
|
|
0.1384143829345703,
|
|
0.1439807415008545,
|
|
0.14021539688110352,
|
|
0.13780498504638672,
|
|
0.1427934169769287,
|
|
0.14551234245300293,
|
|
0.14031195640563965,
|
|
0.1436302661895752,
|
|
0.14116477966308594,
|
|
0.14304280281066895,
|
|
0.14270973205566406,
|
|
0.1403651237487793,
|
|
0.14147257804870605,
|
|
0.14045190811157227,
|
|
0.1416764259338379,
|
|
0.14235329627990723,
|
|
0.14271092414855957,
|
|
0.1398327350616455,
|
|
0.14102721214294434
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"embed_dim": 256,
|
|
"num_neurons": 1024,
|
|
"num_steps": 32,
|
|
"param_count": 1968898,
|
|
"final_mse": 0.0009751874604262412,
|
|
"final_cos": 0.9390977025032043,
|
|
"final_firing_rate": 0.022916078567504883,
|
|
"history": {
|
|
"train_mse": [
|
|
0.06266655307263136,
|
|
0.03312404975295067,
|
|
0.03070711698383093,
|
|
0.03024899298325181,
|
|
0.029628332518041135,
|
|
0.026731413137167693,
|
|
0.028642066195607184,
|
|
0.02664712881669402,
|
|
0.026360327191650868,
|
|
0.026299504935741423,
|
|
0.025126040168106555,
|
|
0.024619465600699185,
|
|
0.02441495517268777,
|
|
0.023426815494894983,
|
|
0.022632006835192443,
|
|
0.019570790976285935,
|
|
0.04966922588646412,
|
|
0.051902918890118596,
|
|
0.025228582415729763,
|
|
0.01610052855685353,
|
|
0.012222643522545696,
|
|
0.010082621034234763,
|
|
0.008429767796769738,
|
|
0.007128522265702486,
|
|
0.00617449355777353,
|
|
0.005383408279158175,
|
|
0.004796477779746055,
|
|
0.004248802922666073,
|
|
0.003693667275365442,
|
|
0.0033036017906852067,
|
|
0.0030596055556088688,
|
|
0.0027902042726054787,
|
|
0.0025377732701599596,
|
|
0.002358842710964382,
|
|
0.0022327777347527443,
|
|
0.002088966954033822,
|
|
0.001983420050237328,
|
|
0.0018329880840610713,
|
|
0.0016787618631497025,
|
|
0.0015517894295044242,
|
|
0.0014650113822426646,
|
|
0.0014209707849659025,
|
|
0.0013487907359376549,
|
|
0.0012802016455680132,
|
|
0.0012260058138053864,
|
|
0.0011834829463623464,
|
|
0.0011421922594308854,
|
|
0.0010926793562248348,
|
|
0.0010493122041225432,
|
|
0.0009922534722136334
|
|
],
|
|
"train_cos": [
|
|
-0.0007222917978651821,
|
|
0.0012521140510216356,
|
|
0.0007950228406116367,
|
|
-0.002621571172494441,
|
|
0.0008132318791467697,
|
|
0.001376264833379537,
|
|
-0.0017023667111061513,
|
|
0.0018350429891142994,
|
|
-3.3404293935745956e-05,
|
|
-0.001872065442148596,
|
|
0.001494578761048615,
|
|
-0.002478212304413319,
|
|
0.005209876946173609,
|
|
0.002390655712224543,
|
|
0.005051728920079768,
|
|
0.07931574312970043,
|
|
0.3074165366590023,
|
|
0.4605686396360397,
|
|
0.5653905600309372,
|
|
0.6322387129068374,
|
|
0.6777178525924683,
|
|
0.7130801320075989,
|
|
0.7397222459316254,
|
|
0.7618081271648407,
|
|
0.7805827468633652,
|
|
0.7970878392457962,
|
|
0.8096365302801132,
|
|
0.8245325773954392,
|
|
0.8369527280330658,
|
|
0.8456784129142761,
|
|
0.8544358879327774,
|
|
0.8640987902879715,
|
|
0.8727038472890853,
|
|
0.8781250268220901,
|
|
0.8830219060182571,
|
|
0.8901462227106094,
|
|
0.8958982437849045,
|
|
0.9022016227245331,
|
|
0.907006460428238,
|
|
0.9107160836458206,
|
|
0.9140749961137772,
|
|
0.9169389426708221,
|
|
0.922037324309349,
|
|
0.9246611595153809,
|
|
0.9269833326339721,
|
|
0.9291444391012191,
|
|
0.9322946518659592,
|
|
0.9344151377677917,
|
|
0.9361900001764297,
|
|
0.939018502831459
|
|
],
|
|
"epoch_time": [
|
|
0.1433243751525879,
|
|
0.14031624794006348,
|
|
0.15794587135314941,
|
|
0.22939205169677734,
|
|
0.15220379829406738,
|
|
0.1658015251159668,
|
|
0.15790677070617676,
|
|
0.15410351753234863,
|
|
0.14745450019836426,
|
|
0.15598845481872559,
|
|
0.18094515800476074,
|
|
0.1554422378540039,
|
|
0.14551401138305664,
|
|
0.15311074256896973,
|
|
0.14874649047851562,
|
|
0.15904831886291504,
|
|
0.14704227447509766,
|
|
0.14524102210998535,
|
|
0.18562626838684082,
|
|
0.15451693534851074,
|
|
0.15463638305664062,
|
|
0.17350339889526367,
|
|
0.17799901962280273,
|
|
0.14171981811523438,
|
|
0.14081430435180664,
|
|
0.14395451545715332,
|
|
0.1444225311279297,
|
|
0.1423017978668213,
|
|
0.1420447826385498,
|
|
0.14295411109924316,
|
|
0.1436612606048584,
|
|
0.14167046546936035,
|
|
0.14470720291137695,
|
|
0.14422297477722168,
|
|
0.13878202438354492,
|
|
0.13988614082336426,
|
|
0.13845181465148926,
|
|
0.13888883590698242,
|
|
0.13750338554382324,
|
|
0.1401503086090088,
|
|
0.16365957260131836,
|
|
0.15123581886291504,
|
|
0.16682744026184082,
|
|
0.19628524780273438,
|
|
0.14569354057312012,
|
|
0.14148402214050293,
|
|
0.15583109855651855,
|
|
0.16861677169799805,
|
|
0.14630436897277832,
|
|
0.15225887298583984
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"embed_dim": 256,
|
|
"num_neurons": 1024,
|
|
"num_steps": 64,
|
|
"param_count": 2493186,
|
|
"final_mse": 0.0006438337150029838,
|
|
"final_cos": 0.951337456703186,
|
|
"final_firing_rate": 0.014148354530334473,
|
|
"history": {
|
|
"train_mse": [
|
|
0.0587492061778903,
|
|
0.031203094776719807,
|
|
0.02755922582000494,
|
|
0.02797670755535364,
|
|
0.026760354451835156,
|
|
0.026476936228573323,
|
|
0.02609567902982235,
|
|
0.023774034529924392,
|
|
0.023597724456340074,
|
|
0.02282807258889079,
|
|
0.0226982275955379,
|
|
0.022114384826272725,
|
|
0.021265622694045305,
|
|
0.020732631254941226,
|
|
0.020519229862838984,
|
|
0.017700643464922904,
|
|
0.03030653465539217,
|
|
0.06265266817063093,
|
|
0.032949542813003066,
|
|
0.017056351387873293,
|
|
0.011676014307886362,
|
|
0.009111061412841081,
|
|
0.007539601088501513,
|
|
0.006191912083886563,
|
|
0.005214437469840049,
|
|
0.004428051761351526,
|
|
0.0037965706549584867,
|
|
0.0033287725527770817,
|
|
0.002944685623515397,
|
|
0.0026589847169816495,
|
|
0.002389865170698613,
|
|
0.0021665571723133324,
|
|
0.002021056128432974,
|
|
0.001846189406933263,
|
|
0.0016920324007514865,
|
|
0.0015431297244504095,
|
|
0.0013931477500591428,
|
|
0.0012811018736101688,
|
|
0.00118755268631503,
|
|
0.0011064097692724318,
|
|
0.00103565962635912,
|
|
0.0009694398962892592,
|
|
0.0009257108526071533,
|
|
0.000885067365015857,
|
|
0.0008306729985633865,
|
|
0.0007921377167804166,
|
|
0.0007456311519490555,
|
|
0.0007143775903386996,
|
|
0.0006877283041831106,
|
|
0.0006588323187315837
|
|
],
|
|
"train_cos": [
|
|
0.00022476123413071037,
|
|
-0.00017563734436407686,
|
|
0.0007876857416704297,
|
|
0.0029204503982327877,
|
|
-0.0016418338927906007,
|
|
-0.0029660439351573585,
|
|
-0.001353371824370697,
|
|
-0.0028749855351634323,
|
|
-0.002180980029515922,
|
|
-0.004068911506328732,
|
|
-2.586673363111913e-05,
|
|
-0.0008906456467229873,
|
|
-0.0008905427646823227,
|
|
0.0025865609699394555,
|
|
0.004204500862397253,
|
|
0.043578516494017096,
|
|
0.28574245870113374,
|
|
0.46765289902687074,
|
|
0.5669307887554169,
|
|
0.6399397909641266,
|
|
0.6930884599685669,
|
|
0.7271538883447647,
|
|
0.7579674512147904,
|
|
0.7834912747144699,
|
|
0.8030357658863068,
|
|
0.8195572316646575,
|
|
0.8335233390331268,
|
|
0.8465323597192764,
|
|
0.857995542883873,
|
|
0.8683356761932373,
|
|
0.8766585230827332,
|
|
0.884472844004631,
|
|
0.8910252630710602,
|
|
0.8978516638278962,
|
|
0.904712438583374,
|
|
0.9096862882375717,
|
|
0.9153447329998017,
|
|
0.9200509130954743,
|
|
0.924096617102623,
|
|
0.9270494729280472,
|
|
0.9307189792394638,
|
|
0.9330470234155654,
|
|
0.9357547521591186,
|
|
0.938254228234291,
|
|
0.9410555541515351,
|
|
0.9429556518793106,
|
|
0.9454453319311142,
|
|
0.9469673454761505,
|
|
0.9485790878534317,
|
|
0.9507022619247436
|
|
],
|
|
"epoch_time": [
|
|
0.2761349678039551,
|
|
0.2765498161315918,
|
|
0.27034854888916016,
|
|
0.2654561996459961,
|
|
0.2851448059082031,
|
|
0.25991320610046387,
|
|
0.25999999046325684,
|
|
0.25831031799316406,
|
|
0.25634336471557617,
|
|
0.2586085796356201,
|
|
0.25832605361938477,
|
|
0.25424814224243164,
|
|
0.25292015075683594,
|
|
0.2530238628387451,
|
|
0.2536354064941406,
|
|
0.2595396041870117,
|
|
0.30223774909973145,
|
|
0.2716667652130127,
|
|
0.28235936164855957,
|
|
0.25617337226867676,
|
|
0.2591276168823242,
|
|
0.2578277587890625,
|
|
0.27134251594543457,
|
|
0.26746392250061035,
|
|
0.26572322845458984,
|
|
0.27844834327697754,
|
|
0.2831892967224121,
|
|
0.2812197208404541,
|
|
0.2728900909423828,
|
|
0.28989267349243164,
|
|
0.30609631538391113,
|
|
0.2729833126068115,
|
|
0.2710697650909424,
|
|
0.2906935214996338,
|
|
0.298297643661499,
|
|
0.27264928817749023,
|
|
0.31177282333374023,
|
|
0.2565786838531494,
|
|
0.27546095848083496,
|
|
0.26677513122558594,
|
|
0.26731371879577637,
|
|
0.26201343536376953,
|
|
0.29913878440856934,
|
|
0.2760806083679199,
|
|
0.27727174758911133,
|
|
0.2798941135406494,
|
|
0.2773592472076416,
|
|
0.2888789176940918,
|
|
0.28528714179992676,
|
|
0.25850486755371094
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"embed_dim": 768,
|
|
"num_neurons": 2048,
|
|
"num_steps": 64,
|
|
"param_count": 15342850,
|
|
"final_mse": 0.0005692970589734614,
|
|
"final_cos": 0.8934202790260315,
|
|
"final_firing_rate": 0.015852510929107666,
|
|
"history": {
|
|
"train_mse": [
|
|
0.09366370979696512,
|
|
0.01991312811151147,
|
|
0.01672689998522401,
|
|
0.016428703907877207,
|
|
0.016496041882783176,
|
|
0.0168199913110584,
|
|
0.016695745754987,
|
|
0.016687909653410316,
|
|
0.01618722341954708,
|
|
0.01627708082087338,
|
|
0.016593078849837184,
|
|
0.015788469603285192,
|
|
0.01721670008264482,
|
|
0.015831851959228517,
|
|
0.017196667240932585,
|
|
0.014922805968672037,
|
|
0.01721709854900837,
|
|
0.013529185857623815,
|
|
0.04155527511611581,
|
|
0.1092149954289198,
|
|
0.10641058832406998,
|
|
0.08261620961129665,
|
|
0.04886667691171169,
|
|
0.02245426448062062,
|
|
0.01265013669617474,
|
|
0.008492178283631802,
|
|
0.006307502626441419,
|
|
0.005064152576960623,
|
|
0.00425044191069901,
|
|
0.0036260800901800395,
|
|
0.003156445873901248,
|
|
0.0027649499941617252,
|
|
0.0024893586058169605,
|
|
0.0022412432124838235,
|
|
0.0020142345281783493,
|
|
0.0018308944418095052,
|
|
0.0016800489975139499,
|
|
0.0015202040725853295,
|
|
0.0013915649149566888,
|
|
0.0012715234246570618,
|
|
0.0011656227638013662,
|
|
0.0010777750052511693,
|
|
0.0009936939517501743,
|
|
0.0009179487911751494,
|
|
0.000850268590147607,
|
|
0.0007846800785046071,
|
|
0.0007293595845112577,
|
|
0.0006784561672247946,
|
|
0.0006321740278508514,
|
|
0.0005917537229834125
|
|
],
|
|
"train_cos": [
|
|
0.0008020128167117946,
|
|
-0.00017133087385445833,
|
|
0.0003534842806402594,
|
|
0.0002611849107779562,
|
|
0.0013856782927177847,
|
|
-0.0003434056998230517,
|
|
-0.0011696210887748748,
|
|
0.0018778305151499809,
|
|
0.00028786000330001117,
|
|
-0.0004938066413160414,
|
|
-0.0008040363260079175,
|
|
0.00010390399256721139,
|
|
-0.0004153235990088433,
|
|
0.0001583939651027322,
|
|
-0.000754376569238957,
|
|
0.0025537379609886558,
|
|
0.0012185127037810163,
|
|
0.024160320637747645,
|
|
0.14571336507797242,
|
|
0.2753903724253178,
|
|
0.3509374648332596,
|
|
0.4031881600618362,
|
|
0.44882077276706694,
|
|
0.5099891409277916,
|
|
0.5695238202810288,
|
|
0.6133517235517502,
|
|
0.6470957666635513,
|
|
0.6744476526975631,
|
|
0.6969929724931717,
|
|
0.7172756731510163,
|
|
0.7340742975473404,
|
|
0.7501888275146484,
|
|
0.7630858838558197,
|
|
0.7764601588249207,
|
|
0.7877005100250244,
|
|
0.7978568434715271,
|
|
0.8077056467533111,
|
|
0.8170691728591919,
|
|
0.82548668384552,
|
|
0.8339649498462677,
|
|
0.8402373313903808,
|
|
0.8472480326890945,
|
|
0.8534927725791931,
|
|
0.8600328654050827,
|
|
0.865775603055954,
|
|
0.8712419509887696,
|
|
0.8763542950153351,
|
|
0.8814421683549881,
|
|
0.8860282808542251,
|
|
0.8900167524814606
|
|
],
|
|
"epoch_time": [
|
|
0.3304593563079834,
|
|
0.2960371971130371,
|
|
0.2895777225494385,
|
|
0.2887406349182129,
|
|
0.30272698402404785,
|
|
0.29352378845214844,
|
|
0.3142101764678955,
|
|
0.31197071075439453,
|
|
0.30093812942504883,
|
|
0.3211641311645508,
|
|
0.30649662017822266,
|
|
0.2945554256439209,
|
|
0.31394052505493164,
|
|
0.3030261993408203,
|
|
0.2888953685760498,
|
|
0.32015419006347656,
|
|
0.29366350173950195,
|
|
0.2821066379547119,
|
|
0.28348708152770996,
|
|
0.29808759689331055,
|
|
0.3085775375366211,
|
|
0.29198551177978516,
|
|
0.2970559597015381,
|
|
0.30441713333129883,
|
|
0.2960829734802246,
|
|
0.32053136825561523,
|
|
0.29787635803222656,
|
|
0.3134956359863281,
|
|
0.2930593490600586,
|
|
0.30736589431762695,
|
|
0.2842867374420166,
|
|
0.2821807861328125,
|
|
0.2860839366912842,
|
|
0.28404903411865234,
|
|
0.2784614562988281,
|
|
0.2799501419067383,
|
|
0.27967238426208496,
|
|
0.28259801864624023,
|
|
0.28206753730773926,
|
|
0.28067946434020996,
|
|
0.2845182418823242,
|
|
0.2860424518585205,
|
|
0.2841989994049072,
|
|
0.28363871574401855,
|
|
0.2832753658294678,
|
|
0.313403844833374,
|
|
0.29127025604248047,
|
|
0.3084988594055176,
|
|
0.2865943908691406,
|
|
0.28415489196777344
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"embed_dim": 768,
|
|
"num_neurons": 4096,
|
|
"num_steps": 64,
|
|
"param_count": 29500674,
|
|
"final_mse": 0.0004611093900166452,
|
|
"final_cos": 0.9164725542068481,
|
|
"final_firing_rate": 0.010140880942344666,
|
|
"history": {
|
|
"train_mse": [
|
|
0.05876353373751044,
|
|
0.016980332927778362,
|
|
0.016626413632184266,
|
|
0.016287324903532862,
|
|
0.016188695328310132,
|
|
0.0163765964563936,
|
|
0.016103057144209742,
|
|
0.0167842373251915,
|
|
0.015967295179143547,
|
|
0.01807353226467967,
|
|
0.01512741087935865,
|
|
0.017879209481179714,
|
|
0.014927968615666032,
|
|
0.01753487759269774,
|
|
0.014885342074558139,
|
|
0.017511264607310294,
|
|
0.014510283712297678,
|
|
0.016356089850887656,
|
|
0.07240878762677312,
|
|
0.1287831887602806,
|
|
0.10408417843282222,
|
|
0.08686433210968972,
|
|
0.0758298397064209,
|
|
0.0672867339104414,
|
|
0.039308654982596634,
|
|
0.011838322039693594,
|
|
0.006871764804236591,
|
|
0.0052734130760654805,
|
|
0.004198557254858315,
|
|
0.0034639336401596664,
|
|
0.002945988264400512,
|
|
0.0025588015676476063,
|
|
0.002250878850463778,
|
|
0.001987321622436866,
|
|
0.00176549835014157,
|
|
0.0015860479907132685,
|
|
0.0014257685572374613,
|
|
0.001284603984095156,
|
|
0.0011621303332503886,
|
|
0.001058747066417709,
|
|
0.0009622442419640719,
|
|
0.0008803784468909726,
|
|
0.0008031431585550308,
|
|
0.0007392621133476496,
|
|
0.0006830326892668381,
|
|
0.0006325942726107314,
|
|
0.0005879342090338469,
|
|
0.0005485323519678787,
|
|
0.0005097880377434195,
|
|
0.000475354035734199
|
|
],
|
|
"train_cos": [
|
|
-0.0006335609359666705,
|
|
-0.0006128832348622382,
|
|
0.00031340729037765414,
|
|
0.0010551628656685352,
|
|
-0.00047180199180729687,
|
|
-0.00012060831650160254,
|
|
-0.00025788332568481563,
|
|
-7.449511322192848e-05,
|
|
-0.000950003816979006,
|
|
0.00032683326862752437,
|
|
0.0010321048437617719,
|
|
0.001299273787299171,
|
|
-0.0007130704208975658,
|
|
0.000924542490975,
|
|
-0.0002920502389315516,
|
|
0.0004693918861448765,
|
|
0.003120499991928227,
|
|
0.04495907751843333,
|
|
0.1903381362557411,
|
|
0.2969889879226685,
|
|
0.3741989523172379,
|
|
0.4284930780529976,
|
|
0.4713885232806206,
|
|
0.5073551446199417,
|
|
0.537636449933052,
|
|
0.5838671892881393,
|
|
0.6395765393972397,
|
|
0.681765404343605,
|
|
0.7128141462802887,
|
|
0.7384995192289352,
|
|
0.7607674270868301,
|
|
0.7780507326126098,
|
|
0.7941315591335296,
|
|
0.8080481261014938,
|
|
0.8200115114450455,
|
|
0.8308481276035309,
|
|
0.839436200261116,
|
|
0.8489077925682068,
|
|
0.857072776556015,
|
|
0.8648102164268494,
|
|
0.8720859348773956,
|
|
0.8782071560621262,
|
|
0.8841082394123078,
|
|
0.8891082644462586,
|
|
0.8942194968461991,
|
|
0.8988852143287659,
|
|
0.9031313776969909,
|
|
0.9070431143045425,
|
|
0.9111765533685684,
|
|
0.9148575335741043
|
|
],
|
|
"epoch_time": [
|
|
0.31564807891845703,
|
|
0.31148481369018555,
|
|
0.31226205825805664,
|
|
0.3083987236022949,
|
|
0.30963873863220215,
|
|
0.31066060066223145,
|
|
0.3117384910583496,
|
|
0.3066434860229492,
|
|
0.3065643310546875,
|
|
0.305647611618042,
|
|
0.3098752498626709,
|
|
0.308699369430542,
|
|
0.30835938453674316,
|
|
0.3079249858856201,
|
|
0.35844945907592773,
|
|
0.3129396438598633,
|
|
0.31224703788757324,
|
|
0.30749011039733887,
|
|
0.3051295280456543,
|
|
0.304567813873291,
|
|
0.30544114112854004,
|
|
0.31043529510498047,
|
|
0.3090853691101074,
|
|
0.3085145950317383,
|
|
0.30823636054992676,
|
|
0.3080012798309326,
|
|
0.31002116203308105,
|
|
0.3099393844604492,
|
|
0.3117034435272217,
|
|
0.30851244926452637,
|
|
0.3104288578033447,
|
|
0.31025004386901855,
|
|
0.33881258964538574,
|
|
0.3169376850128174,
|
|
0.31180667877197266,
|
|
0.3054945468902588,
|
|
0.3066837787628174,
|
|
0.30525970458984375,
|
|
0.30773496627807617,
|
|
0.310072660446167,
|
|
0.3095667362213135,
|
|
0.3080751895904541,
|
|
0.3074307441711426,
|
|
0.30801868438720703,
|
|
0.310636043548584,
|
|
0.3092529773712158,
|
|
0.3074376583099365,
|
|
0.308363676071167,
|
|
0.31446313858032227,
|
|
0.32596468925476074
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"embed_dim": 768,
|
|
"num_neurons": 4096,
|
|
"num_steps": 128,
|
|
"param_count": 35792130,
|
|
"final_mse": 0.000240965629927814,
|
|
"final_cos": 0.9345790147781372,
|
|
"final_firing_rate": 0.006591401994228363,
|
|
"history": {
|
|
"train_mse": [
|
|
0.050383498845621946,
|
|
0.016759822145104408,
|
|
0.016319487243890762,
|
|
0.01632753717713058,
|
|
0.016577697033062576,
|
|
0.016722628194838763,
|
|
0.01638460415415466,
|
|
0.017303091753274203,
|
|
0.01562045537866652,
|
|
0.017654951894655824,
|
|
0.015393088851124049,
|
|
0.017090793093666436,
|
|
0.015247646160423756,
|
|
0.01802970711141825,
|
|
0.014849871164187789,
|
|
0.01768673602491617,
|
|
0.014522335072979332,
|
|
0.016660161968320607,
|
|
0.022777534509077667,
|
|
0.07960875257849694,
|
|
0.08738861903548241,
|
|
0.07619401700794697,
|
|
0.06750630661845207,
|
|
0.06111185327172279,
|
|
0.0561276288703084,
|
|
0.05258651487529278,
|
|
0.04961633887141943,
|
|
0.047137875109910965,
|
|
0.04483988843858242,
|
|
0.042448092438280585,
|
|
0.04012210033833981,
|
|
0.03815466444939375,
|
|
0.03579832632094622,
|
|
0.03362936414778232,
|
|
0.03179669212549925,
|
|
0.029551315866410733,
|
|
0.023225077847018837,
|
|
0.004381246119737625,
|
|
0.0018920901231467724,
|
|
0.0012780045217368752,
|
|
0.0008928824070608243,
|
|
0.0006806120480177924,
|
|
0.0005512076924787835,
|
|
0.0004670287104090676,
|
|
0.00040420792502118277,
|
|
0.00036108202184550466,
|
|
0.0003271194247645326,
|
|
0.00029680393636226653,
|
|
0.0002702482306631282,
|
|
0.0002493404463166371
|
|
],
|
|
"train_cos": [
|
|
0.0019518727553077043,
|
|
0.00023027235874906182,
|
|
-0.0008576239342801273,
|
|
0.0007587793108541519,
|
|
0.0017821344168623908,
|
|
-0.0004934875585604459,
|
|
0.001225034351227805,
|
|
-0.000645992430509068,
|
|
-6.0776164173148575e-05,
|
|
2.7863698778674007e-05,
|
|
0.00027794096677098424,
|
|
0.00044435825548134745,
|
|
-0.0003351354040205479,
|
|
-0.00031207134597934785,
|
|
0.0006706631334964186,
|
|
-0.0008817070163786411,
|
|
0.0017794579733163118,
|
|
0.009781408100388944,
|
|
0.13326355442404747,
|
|
0.307254721224308,
|
|
0.39986827224493027,
|
|
0.4722914919257164,
|
|
0.5215121805667877,
|
|
0.5602869123220444,
|
|
0.5894553273916244,
|
|
0.6161825060844421,
|
|
0.6383745968341827,
|
|
0.6602564662694931,
|
|
0.6810456424951553,
|
|
0.7007735282182693,
|
|
0.7183139681816101,
|
|
0.7357981413602829,
|
|
0.7517702579498291,
|
|
0.7652556300163269,
|
|
0.780607271194458,
|
|
0.7935797035694122,
|
|
0.8038456380367279,
|
|
0.8132355451583863,
|
|
0.8355513721704483,
|
|
0.855742734670639,
|
|
0.8713516771793366,
|
|
0.8839333206415176,
|
|
0.8941784411668777,
|
|
0.9019782781600952,
|
|
0.9090886652469635,
|
|
0.915064936876297,
|
|
0.9201614618301391,
|
|
0.9250260919332505,
|
|
0.929592365026474,
|
|
0.9332218527793884
|
|
],
|
|
"epoch_time": [
|
|
0.5978033542633057,
|
|
0.5644774436950684,
|
|
0.5648729801177979,
|
|
0.5609266757965088,
|
|
0.5661368370056152,
|
|
0.5674030780792236,
|
|
0.5654776096343994,
|
|
0.5649168491363525,
|
|
0.5635249614715576,
|
|
0.5665779113769531,
|
|
0.5884876251220703,
|
|
0.5703763961791992,
|
|
0.5692844390869141,
|
|
0.5662565231323242,
|
|
0.5670957565307617,
|
|
0.566314697265625,
|
|
0.5664417743682861,
|
|
0.6018545627593994,
|
|
0.5821633338928223,
|
|
0.5649609565734863,
|
|
0.5865209102630615,
|
|
0.5680630207061768,
|
|
0.5701131820678711,
|
|
0.6027038097381592,
|
|
0.5841753482818604,
|
|
0.5701019763946533,
|
|
0.5745871067047119,
|
|
0.5759179592132568,
|
|
0.6037595272064209,
|
|
0.5744614601135254,
|
|
0.6410343647003174,
|
|
0.5851333141326904,
|
|
0.5654141902923584,
|
|
0.5665855407714844,
|
|
0.5655779838562012,
|
|
0.5663223266601562,
|
|
0.5610270500183105,
|
|
0.5626001358032227,
|
|
0.5639331340789795,
|
|
0.5644669532775879,
|
|
0.5872030258178711,
|
|
0.5726428031921387,
|
|
0.5626821517944336,
|
|
0.5625336170196533,
|
|
0.5909669399261475,
|
|
0.6204497814178467,
|
|
0.5660824775695801,
|
|
0.5622656345367432,
|
|
0.5684542655944824,
|
|
0.5758025646209717
|
|
]
|
|
}
|
|
}
|
|
] |