The timeline below follows six milestones: early RNN examples in 1986 and 1990, LSTM in 1997, Seq2Seq in 2014, Transformer in 2017, then Switch Transformer and S4 in 2021. These cover the part of AI history concerned with processing sequences such as text. Some of the ideas began earlier than the milestones shown.
RNN: carry something forward
A recurrent neural network, or RNN, processes a sequence step by step. In the kind described by Jeffrey Elman in 1990, each step receives an internal state from the previous step. That state is a set of numbers carrying information about what came earlier. Michael Jordan had described a related approach in 1986.
Think of reading with a small scratchpad. Each new word changes your notes, and those notes help with the next word. The diagram shows the same network at two moments.
Basic RNNs can struggle to learn connections across long gaps. During training, the signals that tell the network how to adjust can shrink as they travel back through many steps. Researchers call this the vanishing-gradient problem.
LSTM: protect useful information
Long short-term memory, or LSTM, is a type of RNN. Sepp Hochreiter and Jürgen Schmidhuber's 1997 paper introduced memory cells with learned gates. The input gate controls what enters a cell; the output gate controls access to its contents.
A gate works like a numerical valve: the network learns how much information to let through. The memory path helps training signals survive across longer gaps, making some distant relationships easier to learn.
The scratchpad analogy still fits, with more control over what reaches the memory. An LSTM continues to update its state one step at a time, and it can still lose useful information.
Seq2Seq: turn reading into writing
Sequence to sequence, usually shortened to Seq2Seq, connects a reader to a writer. The reader, called the encoder, processes an input sequence. The writer, called the decoder, generates an output sequence. The two sequences can have different lengths. Translation is one example.
In a 2014 paper, Ilya Sutskever, Oriol Vinyals and Quoc Le used one LSTM to encode a sentence into a fixed-size set of numbers. Another LSTM used that set of numbers to produce a translation.
Squeezing a whole sentence into one fixed-size set of numbers can become a bottleneck when the sentence is long. Bahdanau, Cho and Bengio's 2014 work used a step called attention, which let the decoder draw on different parts of the input while translating. The writer could consult more of what the reader had processed.
Transformer: connect the context
The 2017 paper Attention Is All You Need introduced the Transformer. Self-attention lets each place in a sequence draw information from the other places it is allowed to use. Multi-head attention runs several learned attention calculations at the same time.
The model also uses feed-forward layers to process each position, and adds position information so word order matters. This diagram sketches the encoder side.
During training, the model can work on many positions in the sequence at the same time. When generating text, the original decoder still writes one element at a time. The original Transformer also has an encoder and decoder, so it is a Seq2Seq system too.
MoE: share the work
Mixture of experts, or MoE, has roots in a 1991 paper. In 2017, researchers used sparse expert layers with LSTMs, activating only some experts for each input. The 2021 milestone here is Switch Transformer, which puts expert layers inside a Transformer.
Each expert is a small network within the larger one. A learned router selects which expert receives an input. At a Switch layer, it selects one expert for each token, a piece of the input text.
This lets a model hold more learned numbers, called parameters, while using only some experts for each token. The extra parameters still need memory, and moving work between devices adds costs. The expert names describe network parts; they do not guarantee human-style specialties.
SSM: keep updating the state
A state-space model, or SSM, describes how a numerical state changes when new input arrives and how that state produces an output. State-space methods already existed in fields such as control theory. The 2021 milestone is the first preprint of S4, a structured state-space sequence model.
Like a running summary, the state carries earlier information forward. S4 uses mathematical structure to make long-sequence calculations more efficient. The diagram shows S4 working step by step, which researchers call its recurrent view. S4 also has a form that can work on different parts of a known sequence at the same time.
A later design called Mamba, described in a 2023 paper, made the state update depend on the current input. That lets the model select what information to carry forward. Its architecture works without attention. How well that approach performs depends on the task and implementation, so the timeline gives us no universal winner.