In a Llama 3.2 attention block the query and output projections map 3072 features to 3072, but the key and value projections map 3072 to 1024. Why are those two narrower?
Unlock detailed company stats for this questionUpgrade
In a Llama 3.2 model, the attention block's four linear projections all take 3072 input features. The query and output projections produce 3072 features, but the key and value projections produce only 1024. Explain in detail why the key and value projections are narrower.
Community answers
No answers contributed by the community yet.
Related courses

Course

Course
