Long Short-Term Memory / LSTM
Advanced RNN type that can better learn dependencies in long series.
Long-short-term memory (Long Short-Term Memory, LSTM) is a more advanced RNN architecture developed at the end of 1990s in order to solve the problem of classic recurrent nerve networks (RNN). The basic innovation of LSTM is that there are three "door" (gate) mechanisms that control information on each cell: forget that the gate will be thrown from the memory cell of the old information, the entrance door decides which new information will be added to the memory, the output door will be transferred to the next step of what information in the memory. These doors allow the network to store important information for a long time, forget the junk or old information, which allows them to take away from disappearance throughout the long series of gradients.
Thanks to this ability, LSTM was used as a standard and most successful approach for a long time in tasks requiring sequential data such as language modeling, machine translation, speech recognition and handwriting recognition in 2017. Google’s early term translation systems and many speech assistants were based on LSTM-based architectures. Although today’s large-scale tasks can be paralleled and left to Transformers which can capture longer contexts, LSTM still continues to be used in smaller scale, resource restricted or real-time sequence data processing applications.
Thanks to this ability, LSTM was used as a standard and most successful approach for a long time in tasks requiring sequential data such as language modeling, machine translation, speech recognition and handwriting recognition in 2017. Google’s early term translation systems and many speech assistants were based on LSTM-based architectures. Although today’s large-scale tasks can be paralleled and left to Transformers which can capture longer contexts, LSTM still continues to be used in smaller scale, resource restricted or real-time sequence data processing applications.