Skip to main content
All Questions

In a Llama 3.2 attention block the query and output projections map 3072 features to 3072, but the key and value projections map 3072 to 1024. Why are those two narrower?

Unlock detailed company stats for this questionUpgrade
In a Llama 3.2 model, the attention block's four linear projections all take 3072 input features. The query and output projections produce 3072 features, but the key and value projections produce only 1024. Explain in detail why the key and value projections are narrower.

Community answers

No answers contributed by the community yet.

Related courses

Course

SQL Interview Prep Course

Prepare for SQL interviews with lessons on joins, aggregations, and window functions, then practice real SQL interview questions in an interactive editor with expert solutions.

Course

Data Science Interview Prep

Land your dream data science role at Google, Amazon, Microsoft, Meta, Apple, and other top companies. Learn from mock interviews, frameworks, and advice from senior candidates. Practice statistics, experimentation, coding, SQL, machine learning, behavioral interviews, and more.