The original attention mechanism and its scaling cost Self‑attention, introduced around 2017, allows each token in a sequence to interact with every other token. The mechanism builds a dense attention matrix whose size grows quadratically with the sequence length. This design is elegant but expensive: the time and memory
Word vs Sentence Encoders — Digging Deeper In machine learning, encoders convert raw text to vector embeddings that capture semantics. Historically, word-level encoders like Word2Vec and GloVe assign each token a single static vector. This is easy to train and interpret but it cannot differentiate the same word in different contexts.
News
This is DeepNeuro.AI Research Notes, a brand new site by Richard Young that's just getting started. Things will be up and running here shortly, but you can subscribe in the meantime if you'd like to stay up to date and receive emails when new content