KB Configuration
Configure embedding model, chunk size, chunk overlap, and crawling limits for your Knowledge Base.
The Configuration page controls how your Knowledge Base content is processed and indexed. Changes here affect the quality and speed of knowledge retrieval across all collections. Access it via the Settings button in the Storage usage panel (bottom of the left sidebar), or navigate directly to Knowledge Base → Configuration.
Configuration is split into two tabs: Embeddings and Crawling.
Embeddings
The Embeddings tab controls how documents are converted into vector representations for semantic search.
AI model
Select the AI provider used for generating embeddings. This must match a provider you have configured under Configurations → AI Models.
Embedding model
The specific model used for embedding. Different models vary in quality, token limits, and cost. The default (text-embedding-3-small for OpenAI) is a strong general-purpose choice.
| Setting | Recommendation |
|---|---|
OpenAI text-embedding-3-small | Best for most use cases – fast, accurate, cost-efficient |
OpenAI text-embedding-3-large | Higher quality for large, complex knowledge bases |
| Azure equivalents | Use if you're running on Azure OpenAI infrastructure |
Changing the embedding model after documents have been indexed will cause a mismatch between old and new embeddings. Re-embed all documents after changing the model by re-saving each document, or re-importing your content. Until you do, retrieval quality for previously indexed documents may degrade.
Chunk size
Documents are split into chunks before embedding. The chunk size (in tokens) controls the maximum size of each chunk. Smaller chunks retrieve more precisely; larger chunks preserve more context per result.
| Chunk size | Best for |
|---|---|
| 500–800 tokens | Short, focused FAQs or product specs |
| 1000 tokens (default) | General-purpose content |
| 1500–2000 tokens | Long-form content where context continuity matters |
Chunk overlap
The number of tokens shared between adjacent chunks. Overlap ensures that information spanning a chunk boundary is not lost. The default of 200 tokens is appropriate for most content.
Increase overlap if you find that retrieved chunks often miss important context that appears just before or after the most relevant section.
Processing limit per batch
The maximum number of documents processed in a single embedding batch. Lower this value if you encounter rate-limit errors from your embedding provider.
Crawling
The Crawling tab controls the behaviour of the web crawler used for Website documents.
Crawling limit (parallel requests)
The maximum number of simultaneous HTTP requests the crawler makes when indexing a website. Default is 1 (sequential crawling).
Increase this to speed up crawling of large sites – but be aware that:
- Higher values may trigger rate limiting or temporary IP blocks on the target website
- Some websites explicitly disallow aggressive crawling in their
robots.txt
Applying changes
After editing any setting, click Edit to enter edit mode and Save to apply. Changes to embedding settings do not automatically re-embed existing documents – you must trigger re-indexing manually (re-save documents or re-import content) for the new settings to take effect on existing knowledge.
New documents added after the change will use the updated settings immediately.