Implementing Text Chunking for LLMs

TL;DR

This article focuses on practical text-splitting strategies for Large Language Models (LLMs). It contains:

• Background of Text Chunking
• Chunking Approaches
• tiktoken and Its Considerations

Query Spelling Correction Evolution

TL;DR

Ever searched for a product online, only to be met with “no results found” because of a tiny typo? That’s the multi-billion dollar problem of query spelling correction. Spelling errors kill search results. This article reviews how query correction evolved from 🔡 edit distance

noisy channel → 📊 rankers → 🤖 transformers → 🌐 LLMs, and where the field is heading.