tool
The process of splitting a long document into small meaningful parts that can be processed.
Shredding (chunking) is the process of splitting a long text or document into smaller, machinable and meaningal integrity as possible. This process is especially born from a practical need due to having a limited context window (context window) of large language models: the amount of text that a model can work at one time is limited, so hundreds of pages can not be given to the model directly. Shredding divides the document according to criteria such as sentence, paragraph or fixed character/token count, it works to protect as meaningful integrity as possible; leaving some overlap between the parts also helps prevent the content from being lost in part limits.
Shredding is a critical pre-processing step especially in the establishment of Retrieval-Augmented Generation (RAG) systems: documents are divided into pieces before, each piece is converted into a embedding vector and saved to a vector database; when a query comes, finding the most relevant parts in meaning and added to the context of the language model. The choice of parts size is an important engineering decision: very small parts can carry the context sufficiently, and the meaning can be shredded, while very large parts fill the context window unnecessary and reduce the sensitivity of the meaningal search.
Shredding is a critical pre-processing step especially in the establishment of Retrieval-Augmented Generation (RAG) systems: documents are divided into pieces before, each piece is converted into a embedding vector and saved to a vector database; when a query comes, finding the most relevant parts in meaning and added to the context of the language model. The choice of parts size is an important engineering decision: very small parts can carry the context sufficiently, and the meaning can be shredded, while very large parts fill the context window unnecessary and reduce the sensitivity of the meaningal search.