In advanced mode, the Chunking transformation splits large pieces of text into smaller segments, or chunks. This helps to increase the content's relevance before the Target transformation writes the embeddings and metedata to a vector database.
Pass output from a Chunking transformation to a Vector Embedding transformation to create vector embeddings for the text. For more information, see Vector Embedding transformation.
When you configure a Chunking transformation, choose a chunking method based on the text you want to split.To select the method, consider the following factors:
•Whether the text comes from a long document or short messages
•The length and complexity of user queries to the large language model (LLM)
•The use case of the application that uses the LLM, such as semantic search, question answering, or summarization
Note: The Chunking transformation can't run in a serverless runtime environment or on GPUs. If the transformation runs on a GPU-enabled cluster, GPUs are disabled and the transformation consumes CPUs.