Text processing functions make text data cleaner and semantically more consistent for vector embedding by focusing on words that are informative to the meaning of the text and by reducing variability to aid NLP. In RAG use cases, text processing ensures that text is clean, consistent, and easily comparable to user queries.
Text processing functions can clean text by removing noise such as whitespace and diacritics, and they can convert text to a standard format by lemmatizing words to their base forms.
You can use the following text processing functions:
Cleanse text
Cleanse the text by removing redundant whitespace and sets of dots and by converting letters to lowercase.
Remove diacritics
Removes diacritics including accents and other marks that change a letter's pronunciation. For example, café becomes cafe.
Check spelling
Checks for spelling errors based on the context of the data and corrects them.
Lemmatize
Converts words to their base form. For example, better becomes good and running becomes run.
Lemmatization preserves the semantic accuracy of the data, so it's useful for sentiment analysis and machine translation.
Remove stop words
Removes common stop words like pronouns, articles, prepositions, and conjunctions. For example, This is a sample text becomes sample text.
Converting words to lowercase and removing stop words is a simple and effective way to reduce data complexity that applies to most NLP tasks.