-
Notifications
You must be signed in to change notification settings - Fork 6
Expand file tree
/
Copy pathSTORY.txt
More file actions
117 lines (89 loc) · 6.23 KB
/
Copy pathSTORY.txt
File metadata and controls
117 lines (89 loc) · 6.23 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
[20250427]
Corrected top-p issues: handle case when all tokens would be discarded and use bubblesort instead of quicksort - the stack is just not deep enough.
[20250424]
Tested on real hardware. Renamed weights.bin to weights.reu so that Ultimate2+ will easily recognize REU image and load it from context menu.
[20250421]
Detect REU, REU size and config magic bytes signature.
[20250410]
Optimized attn a bit, 62 token generation time went down from 9:31:23 to 9:25:43 (1%). Yay.
[20250326]
Corrected output printout. Display layers starting from 1 and visually indicate that top-p has no effect when temperature is 0.0.
[20250325]
Implemented top-p sampler, updated tests and ui.
Updated documentation to point to original repositories, including training scripts, model weights and tokenizer data.
Completed *.bin generating scripts, now old files from llama2.c can be removed (they will stay on `wrapped_debug` branch).
[20250324]
On (abandoned) branch feature-fastmult there is a not-so fast float32 multiplication based on uint8_t*uint8_t times table (64KB).
Makes token generation 2x slower though. It may come back if/when we try to add quantized weights.
Correct input/output ASCII/PETSCII conversion to get exact result for input 'Zoo' (4 tokens) and exactly the same output with
temp=0:
[Zoo was a little girl named Lily. She loved to play outside in the park. One day, she saw a big, red ball. She wanted to play with it, but she didn't want to play with]
[20250323]
Branch feature-fastmult has a uint8_t*uint8_t times table, to be put into REU (64K).
[20250322]
Some UI polish, initialize random number generator with seed value from CIA1 IRQ timer and VIC raster register read after user input,
so essentially random.
[20250321]
Startup screen almost ready with model parameter display and parameter settings.
Only top_p implementation is missing - need a sorting routine for that.
[20250320]
Simple text UI for prompt input and inference screen. Need also a welcome screen with name & title, model parameters (layers, heads, dims, vocab) info
and input settings: temperature (0.0-1.0 by 0.1 with +/- keys) and number of output tokens.
What should be the color scheme? Should the status go to the bottom?
Need correct ASCII<->PETSCII translation on both input (from prompt) and output (from decoded token). 'Zoo' should be 4 tokens, not 5. 'ZOO' is 5.
[20250319]
Optimized rope(), sin/cos values are calculated only once for each pos, so the calculation don't repeat for every layer/head. Worth maybe 24s of real time per token :)
Optimized rmsnorm(), matrix multiplications to have only one REU_getf/putf for whole row(column), no visible change.
One token every 37s (7m24 realtime). Feeling of on-premise 50B model :)
drmortalwombat already fixed the issue #207 with Oscar floats! That's awesome support.
[20250318]
Fixed rope() and now results with temp=1.0 exactly (up to 5 decimals) match x86 calculations.
We have tinystories LM running on C64!
There were two bugs involved:
- rope() calculates sin/cos of val, that is pos divided by 1, 10, 100 etc. for each iteration, up to head_size, then it wraps back to 1
original code chose to calculate it every time using pow(), while it's faster to divide by 10
on top of that pow(10000.0,0) somehow breaks the floating point result and some operations done on it (like multiplying by pos) will yield INF
(reported in https://github.com/drmortalwombat/oscar64/issues/207)
- my_sincos() is broken, returns wrong cos(2) value; since it doesn't save on calculations much I'll stick to my_sin/my_cos
Commited everything to wrapped_debug branch before cleaning up all that stuff for UI design.
[20250317]
Found out that sin/cos provided by Oscar math.c have worse accuracy than C64 BASIC.
Implemented my own functions for that using polynomials from C64 BASIC ROM and got much better results
Can't really figure out why it's still wrong. Implemented my_exp() (only slightly more accurate), but it didn't do much.
[20250314]
buffered_reu branch where we keep two input/output buffers so that we can read/write whole vectors (or weight matrix columns).
It's cleaner, but speedup is completely not visible.
[20250313]
With sampler64 and generate64 completed we can now run the full thing and read a story. It will take a while, we get token every 40s (on VICE's warp), that's 8m real time.
(reference PC does it 1.5MM faster)
At least that was the hope - it very quickly diverges from reference result in deterministic (temp=0.0) mode.
It's a calculation problem or some REU pointer arithmetic (addition and casting to uint32) where one thing overwrites another.
Hopefully not an intrinsic problem with Oscar64 floating point code, I won't be able to fix that.
[20250311]
Extract also attention calculation into a new function (VS Code handled that automatically), now we can continue.
Foward() is completed, can be used in generate() now.
[20250310]
Continue working on forward(). Got unreliable results, sometimes hangups because function is too big, with too many local variables.
Pieces can be extract, first rope().
[20250309]
Ported and tested rmsnorm() and softmax(). Was not too hard.
Tried to have REU_ptr as float*, but forgot that pointers are 16-bit here and we need at least 24.
[20250308]
Back to the project. Completed and verified matrix multiplication functions. Had to program 3 of those. Matrix of weights is always
remote (REU), but one (both) input/output vectors can be either in local memory or in remote.
On top of that only one multiplicatin works on vector with dimension > 256, so for the most part we can use uint8_t as indexes.
[20250112]
Briefly came back and started staring at nnet.c to work on nnet64.c. Need to figure out sizes of all these objects to assign smaller data types.
[202410xx]
Project starts. llama.2c is one giant C file, I split it into several functional blocks:
- tokenizer
- transformer
- nnet
- sampler
- generate
- util
I managed to write tokenizer and token decoder, ported transformer and checked array sizes to asses that it may work, it may fit within 2MB REU.
Then I got scared by amount of checking and verifications needed.
Excellent llama.2c README describes also int8 quantized version, but at the moment I want something working right away. Once float32 verison is working
we can play further.
-- Maciej 'YTM/Elysium' Witkowiak