ContextX

ContextX is a modular AI context-engineering platform that improves how LLMs retrieve, select, validate, and use information before generating an answer.

What it does

It takes a user query and combines it with relevant information from:

• Document collections
• Conversation context
• Stored user/project memory
• Metadata and source information

It then determines which information is actually useful, removes unnecessary or duplicate content, applies security checks, manages the model's token budget, and builds a grounded prompt for the LLM.

How it works

The core architecture is a staged pipeline:

Query → Retrieve → Rerank → Rank → Filter → Compress → Budget → Build → Validate → LLM

• Retrieval — Uses semantic vector search and BM25 lexical search to find candidate information.
• Reranking — Uses a cross-encoder to identify the most relevant candidates.
• Ranking — Combines relevance, similarity, keyword matching, recency, importance, conversation context, and preferences.
• Filtering — Removes duplicates, stale or contradictory information and applies tenant/ACL constraints.
• Compression — Reduces large context items while preserving useful information.
• Budgeting — Controls the amount of context sent to the LLM within a token budget.
• Context building — Constructs a structured prompt with sources and trust boundaries.
• Validation & security — Performs citation/provenance checks, prompt-injection scanning, and optional PII redaction.
• Generation — Sends the validated context to an interchangeable LLM backend such as Claude, OpenAI, Ollama, or a mock backend.
• Observability & memory — Tracks latency, retrieval statistics, cost estimates, sources, and updates long-term memory/cache where applicable.

Key technologies / capabilities

• Hybrid RAG: semantic embeddings + BM25 + Reciprocal Rank Fusion
• Cross-encoder reranking
• FAISS / NumPy / Qdrant / pgvector vector backends
• SQLite source-of-truth storage
• Claude / OpenAI / Ollama LLM providers
• Caching and long-term memory
• Tenant and ACL-aware retrieval
• PII redaction and prompt-injection detection
• Token budgeting and context compression
• Evaluation using Recall@K, MRR, nDCG, precision and faithfulness
• CI retrieval regression gates
• Configurable domain presets, including finance

Architecture

How the code is organized and how data flows through it.

High-level shape

contextx is a library (contextx/) with an optional service layer (contextx/service.py, FastAPI). The library is a staged pipeline orchestrated by ContextEngine (contextx/pipeline.py). The engine owns one instance of each stage component and threads a single ContextItem list + Trace through them.

                          ┌──────────────────────────────────────────┐
   HTTP (optional)        │              ContextEngine               │
   service.py  ─────────▶ │  ingest() / run() / run_stream() / a*()  │
   (auth, SSE)            └───┬───────────────────────────────────┬──┘
                             │ ingest                             │ query
                    ┌────────▼─────────┐              ┌───────────▼───────────┐
                    │   VectorStore    │              │  Collect→Retrieve→... │
                    │  (store.py)      │◀─────────────│  →Rerank→Rank→Filter→ │
                    │  backend + FTS5  │   search     │  Compress→Budget→Build│
                    │  + SQLite chunks │              │  →Validate→LLM        │
                    └────────┬─────────┘              └───────────┬───────────┘
                             │                                    │
                    ┌────────▼─────────┐              ┌───────────▼───────────┐
                    │  VectorBackend   │              │ Cache · Memory · Trace│
                    │ faiss/numpy/pg   │              │  (cross-cutting)      │
                    └──────────────────┘              └───────────────────────┘

Module map

Two entry phases

Ingest (engine.ingest(documents))

1. store.add_documents chunks each Document (chunking.py).
2. Embeds all new chunks in a batch (embeddings.py).
3. Persists: chunk text + metadata + tenant/ACL + the embedding blob into SQLite; the vector into the VectorBackend; the text into the FTS5 index.
4. Stamps the embedding-model name (so a later mismatch is caught, not silently wrong).

Also: update(documents) (delete-by-doc_id then re-add), delete(doc_id) (tenant-scoped), both invalidate the response cache.

Query (engine.run(request) / run_stream / arun)

_prepare() runs stages 1–9 and returns a _Prepared (prompt, validation report, confidence, scope). Then: - run() calls the LLM (through the semantic cache), writes memory, _finalize() records metrics/cost/audit, returns a PipelineResult. - run_stream() returns a StreamResult whose .stream yields text chunks; memory is written when the stream is fully consumed. Sources and confidence are known up front. - arun() / arun_stream() are the async wrappers — they offload the sync CPU/GPU-bound pipeline to a worker thread so an async host's event loop is never blocked (the correct async model for CPU-bound work). - arun_with_sources(request, fetchers) awaits async source fetchers concurrently (asyncio.gather) — N remote fetches cost ~one round-trip — then runs the pipeline. Embedding is batched within a request (embed_batch_size); cross-request batching is a serving-layer concern, out of the library's scope.

Request lifecycle

run() for one query:

Request
  → _prepare()                              # stages 1–9, in a worker thread under arun()
      1  collect ephemeral context + read memory (scope tenant:user)
      2  retrieve: vector ANN + FTS5 lexical → RRF, tenant/ACL filtered
      2b rerank recall set (cross-encoder) → confidence / abstention flag
      3  rank (weighted blend)   4 filter (dedup/stale/contradiction)
      6  compress oversized      7 budget to the window
      (redact PII if enabled)
      8  build prompt (fenced untrusted + numbered citations + cache marker)
      9  validate (ceiling, delimiters, injection scan)
      → _Prepared(prompt, report, confidence, scope)
  → 10 LLM (via semantic cache)             # blocked here if validation failed
  → 5  write memory (extract facts)
  → _finalize()                             # metrics, USD cost, audit, log
  → PipelineResult(answer, sources, low_confidence, trace)

run_stream() shares _prepare(), then streams stage 10 and writes memory when the stream is drained. run_with_tools() replaces stage 10 with an agentic tool loop.

Storage & data model

State lives under Config.index_dir plus the memory DB. Nothing is global.

The chunk store is the source of truth; the vector index is derived (a rebuild() reconstructs it from the stored embeddings and compacts tombstones).

Deployment

• Embedded (library): construct one ContextEngine and call it. State is on local disk; single process. Good for scripts, notebooks, one-box services.
• Service: contextx.service:create_app (FastAPI). One engine per worker; API-key auth maps callers to tenants; writes serialized with an async lock; the sync engine runs in a threadpool. Ships with a Dockerfile.
• Scaling: the CPU/GPU-bound work (embedding, reranking) scales with worker threads/processes; the vector index scales by swapping the VectorBackend (Qdrant or pgvector) off single-node FAISS. Chunk/memory SQLite is single-writer per process — move to Postgres/a shared store for multi-writer deployments.

The vector-store seam

VectorStore owns everything except the raw ANN index: chunk text, metadata, tenant/ACL, the FTS5 lexical index, tombstoning, and rebuild/compaction. The ANN index — the part with the single-node RAM ceiling — lives behind the narrow VectorBackend interface (add / search / count / reset / save / load). Swap Config.vector_backend to change it: * faiss — HNSW ANN, in-process (default). * numpy — brute-force cosine fallback. * qdrant — Qdrant (:memory:, local path, or server URL). * pgvector — Postgres + pgvector (experimental; needs a DB; not CI-covered).

Contract: the store assigns each vector a dense integer row id (rebuild renumbers 0..n-1), so FAISS/numpy can treat position == row; pgvector stores the ids explicitly.

Concurrency & persistence model

• SQLite (chunks + memory) uses WAL mode; write paths are guarded by an in-process RLock. Safe for concurrent readers and single-writer within a process.
• The service serializes index writes with an asyncio.Lock and offloads the sync engine to a threadpool — the correct model because the heavy work (embedding, reranking) is CPU/GPU-bound.
• The response cache, semantic-cache vectors, and memory are all bounded, so a long-running process does not leak.

Failure & degradation

Every external dependency is optional with a fallback, chosen at construction and reported in the trace footer (embed_backend, index_backend, rerank_backend, llm_backend):