Skip to content

chore(documentation): add context on why 48-vector element #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Warkanlock wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore(documentation): add context on why 48-vector element #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/llm/walkthrough/Walkthrough00_Intro.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,8 @@ and sort them in alphabetical order, i.e. to "ABBBCC".`;
breakAfter();

let c5 = commentary(wt)`In the 3d view, the each green cell represents a number being processed, and each blue cell is a weight. ${embed(GreenBlueCells)}
Each number in the sequence first gets turned into a 48 element vector. This is called an _embedding_.`;
Each number in the sequence first gets turned into a 48 element vector (number defined in the gpt-nano architecture). This is called an _embedding_.`;

breakAfter(c5);

{
Expand Down
2 changes: 2 additions & 0 deletions src/llm/walkthrough/Walkthrough02_Embedding.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ The ${c_dimRef('_C_', DimStyle.C)} stands for ${c_dimRef('_channel_', DimStyle.C

This matrix, which we'll refer to as the ${c_blockRef('_input embedding_', state.layout.residual0)} is now ready to be passed down through the model.
This collection of ${c_dimRef('T', DimStyle.T)} columns each of length ${c_dimRef('C', DimStyle.C)} will become a familiar sight throughout this guide.

For this particular model, we are using a 48-element vector for the embeddings given that it's based on the initial gpt-nano configurations and is a number that the developers should decided on in advance.
`;

cleanup(t9_cleanupInstant, [t3_moveTokenEmbed, t5_movePosEmbed, t6_plusSymAnim, t7_addAnim, t8_placeAnim]);
Expand Down