update docs

docs/README.md

@@ -4,41 +4,40 @@ This folder contains the handover documentation for the Audit Assistant project.
 
 The architecture documentation follows the [arc42](https://arc42.org/) standard. Diagrams use the [C4 model](https://c4model.com/) conventions. The operations runbook follows the [Diátaxis](https://diataxis.fr/) "how-to guide" pattern.
 
-> **Status:** the documentation set is being built out as part of the project close-out. The Status column in each table below shows what is complete and what is in progress.
-
 ## How to read this documentation
 
 - **First-time readers.** Start with [Introduction, Goals & Constraints](architecture/01-introduction-goals-constraints.md) and [Context & Scope](architecture/02-context-and-scope.md). Together they take about 10 minutes and give you the system at a glance.
-- **Reviewers / auditors.** Go to the [Architecture Decision Records](architecture/adrs/) — each ADR explains one key design choice with its rationale, alternatives considered and consequences.
-- **Operators and future maintainers.** Start with the [Operations Runbook](runbook/) and [System Requirements](system-requirements.md).
+- **Reviewers / auditors.** Go to the [Architecture Decision Records](architecture/adrs/README.md) — each ADR explains one key design choice with its rationale, alternatives considered and consequences.
+- **Operators and future maintainers.** Start with the [Operations Runbook](runbook/README.md) and [System Requirements](system-requirements.md).
 - **Evaluators.** See the [Evaluation document](evaluation.md) — methodology, metrics, and reproducibility instructions.
 
 ## Documentation map
 
 ### Architecture (arc42)
 
-| # | Document | What it covers | Status |
-|---|---|---|---|
-| 1 | [Introduction, Goals & Constraints](architecture/01-introduction-goals-constraints.md) | arc42 §1, §2; quality goals summary | **done** |
-| 2 | [Context & Scope](architecture/02-context-and-scope.md) | arc42 §3; business + technical context diagrams | **draft** |
-| 3 | [Solution Strategy & Building Blocks](architecture/03-solution-strategy-and-building-blocks.md) | arc42 §4, §5; container + component diagrams | **draft** |
-| 4 | [Runtime View](architecture/04-runtime-view.md) | arc42 §6; sequence diagrams | **draft** (sections 4.1-4.5; resolver agent flow §4.7 pending) |
-| 5 | [Deployment View](architecture/05-deployment-view.md) | arc42 §7; HF Spaces topology + env vars | **draft** |
-| 6 | [Crosscutting Concepts](architecture/06-crosscutting-concepts.md) | arc42 §8; config, logging, errors, secrets, caching, concurrency | **draft** |
-| – | [Architecture Decision Records](architecture/adrs/) | arc42 §9; 6 ADRs on the key design choices | **draft** (all 6 ADRs done) |
-| 7 | [Quality, Risks & Glossary](architecture/07-quality-risks-glossary.md) | arc42 §10, §11, §12; quality scenarios + risk register + glossary | **draft** |
+| # | Document | What it covers |
+|---|---|---|
+| 1 | [Introduction, Goals & Constraints](architecture/01-introduction-goals-constraints.md) | arc42 §1, §2; quality goals summary |
+| 2 | [Context & Scope](architecture/02-context-and-scope.md) | arc42 §3; business + technical context diagrams |
+| 3 | [Solution Strategy & Building Blocks](architecture/03-solution-strategy-and-building-blocks.md) | arc42 §4, §5; container + component diagrams |
+| 4 | [Runtime View](architecture/04-runtime-view.md) | arc42 §6; agent-graph + retrieval sequence diagrams; resolver-agent dynamic-lookup flow |
+| 5 | [Deployment View](architecture/05-deployment-view.md) | arc42 §7; HF Spaces topology + env vars |
+| 6 | [Crosscutting Concepts](architecture/06-crosscutting-concepts.md) | arc42 §8; config, logging, errors, secrets, caching, concurrency, prompt-cache convention, MMR diversity, LLM-adapter resilience, observability, one-time-init lifecycle, test strategy |
+| – | [Architecture Decision Records](architecture/adrs/README.md) | arc42 §9; 7 ADRs on the key design choices (multi-agent, dense-only retrieval, reranker skip, LLM model choice, HF Spaces deployment, chunking, MMR-Jaccard) |
+| 7 | [Quality, Risks & Glossary](architecture/07-quality-risks-glossary.md) | arc42 §10, §11, §12; quality scenarios + risk register + glossary |
 
 ### Operations and references
 
-| Document | What it covers | Status |
-|---|---|---|
-| [Interfaces](interfaces.md) | External integration interfaces and internal module interfaces | **draft** |
-| [Operations Runbook](runbook/) | How-to guides for routine operational tasks (6 how-tos + index) | **draft** |
-| [Evaluation](evaluation.md) | Evaluation methodology, metrics, reproducibility | **draft** |
-| [System Requirements](system-requirements.md) | Hardware, software, network requirements | **draft** |
-| [Stack & Dependencies](stack-and-dependencies.md) | Dependency versions and recommended upgrade path | **draft** |
-| [Cost & Performance](cost-and-performance.md) | Cost analysis summary + link to the full analysis | **draft** |
-| [Accountability, Transparency, Limitations & Biases](accountability-transparency-limitations-biases.md) | WP5 final-stage assessments | **draft** |
+| Document | What it covers |
+|---|---|
+| [Interfaces](interfaces.md) | External integration interfaces and internal module interfaces |
+| [Operations Runbook](runbook/README.md) | How-to guides for routine operational tasks (7 how-tos + index) |
+| [Evaluation](evaluation.md) | Evaluation methodology, metrics, reproducibility instructions |
+| [System Requirements](system-requirements.md) | Hardware, software, network requirements |
+| [Stack & Dependencies](stack-and-dependencies.md) | Dependency versions and recommended upgrade path |
+| [Cost & Performance](cost-and-performance.md) | Per-query cost summary, hybrid model strategy, prompt-cache savings, monitoring; full analysis linked |
+| [Accountability, Transparency, Limitations & Biases](accountability-transparency-limitations-biases.md) | WP5 final-stage assessments |
+| [DEFERRED.md](DEFERRED.md) | Known open items + design notes carried forward post-handover |
 
 ## External references
 
@@ -46,9 +45,4 @@ The architecture documentation follows the [arc42](https://arc42.org/) standard.
 - [C4 model](https://c4model.com/) — diagram conventions used in the architecture views
 - [Diátaxis](https://diataxis.fr/) — documentation framework used for the runbook
 - [Project README](../README.md) — repository overview and quickstart
-
-## Deferred items
-
-Items intentionally deferred during this documentation pass will be listed here as they come up, so reviewers and future maintainers can see them at a glance.
-
-*(none yet)*
+- [CHANGELOG](../CHANGELOG.md) — chronological record of notable changes

docs/accountability-transparency-limitations-biases.md

@@ -29,7 +29,7 @@
 
 - The system runs **two to three LLM calls per turn** (analysis, optionally rewrite, answer generation). Token usage is logged server-side but not surfaced in the UI.
 - The system runs **Qdrant queries** (one or more `count()` calls for pre-validation + one vector search). Latency is hidden behind a "Thinking..." spinner.
-- Conversations are **logged server-side** to a Hugging Face dataset (`GIZ/spaces_logs`) for audit and analytics purposes. This is not currently disclosed in the UI — a privacy note should be added to the application (see DEFERRED items #21 and the runbook for the privacy-note draft).
+- Conversations are **logged server-side** to a Hugging Face dataset (`GIZ/spaces_logs`) for audit and analytics purposes. This is not currently disclosed in the UI — a privacy note should be added to the application footer before the system is exposed to end-users (tracked in [`DEFERRED.md`](DEFERRED.md) #21).
 - The system **infers filters from the user's natural-language query** (year, source category, district, entity type). These inferred filters drive what gets retrieved. The reasoning is captured in `resolution_notes` and logged server-side but not always surfaced to the user.
 
 ### Improving transparency — recommended next steps

docs/architecture/04-runtime-view.md

@@ -1,8 +1,6 @@
 # 4. Runtime View
 
 > *arc42 §6 — Runtime View. This document traces the system's behaviour during a typical user interaction, from the moment a query is submitted in the Streamlit UI through to the final answer being rendered.*
->
-> *Status: first draft, built from system walkthroughs. Will be cross-linked to the building-block view (§5) once that's drafted.*
 
 ## 4.1 Top-level interaction
 

docs/runbook/update-knowledge-base.md

@@ -68,28 +68,45 @@ comparable to old chunks).
 
 ### 3. Run the ingestion script
 
-> The exact script path may vary depending on the project's evolution.
-> Check `scripts/` and adjust if needed. The general invocation:
+The canonical batch-ingestion script is `maintanance/batch_ingest_folder.py`.
+It recursively walks a folder of PDFs, chunks them with Docling, embeds
+with `BAAI/bge-m3`, and uploads to Qdrant in batches. Resumable via a
+state file, retries on transient failures.
 
 ```bash
-python scripts/ingest_corpus.py \
-    --collection BAAI-bge-m3-full \
-    --model-name BAAI/bge-m3 \
-    --docs-dir reports/new_batch_2025/ \
-    --append   # don't recreate the collection, just add new docs
-```
+# Required environment variables
+export QDRANT_URL=https://...
+export QDRANT_API_KEY=...
 
-For documents that need explicit metadata:
-```bash
-python scripts/ingest_corpus.py \
+# Standard invocation: ingest all PDFs under a folder, append to existing collection
+python maintanance/batch_ingest_folder.py "/path/to/reports/new_batch_2025" \
     --collection BAAI-bge-m3-full \
-    --docs-dir reports/new_batch_2025/ \
-    --append \
-    --override-source "CSBAG" \
-    --override-year 2025 \
-    --override-district "Gulu"
+    --chunk-size 512 \
+    --batch-size 32
 ```
 
+Useful flags:
+
+| Flag | Purpose |
+|---|---|
+| `--collection BAAI-bge-m3-full` | Target Qdrant collection name (default already matches production) |
+| `--chunk-size 512` | Chunk size in tokens (default; matches the existing corpus) |
+| `--batch-size 32` | Embedding batch size (lower if RAM-constrained) |
+| `--pdf-batch-size 4` | How many PDFs Docling processes together (default 4) |
+| `--max-retries 5` | Retries per file on transient failure (default 3) |
+| `--no-skip-existing` | Re-upload chunks that already exist in Qdrant (default: skip) |
+| `--resume` (no value) | Auto-find the most recent state file for this folder and continue |
+| `--resume STATE_FILE.json` | Continue from a specific state file |
+| `--no-wait` | Use non-blocking Qdrant upserts (faster but less safe if interrupted) |
+
+State files (`ingestion_state_<batch_name>_<timestamp>.json`) are written
+to the working directory; they let you resume after a crash or network
+blip without re-uploading already-ingested chunks.
+
+For one-off cleanup after a partial ingest, use
+`maintanance/ingest_missing_files.py` instead — it scans Qdrant for missing
+chunks and only ingests those.
+
 ### 4. Wait for ingestion to complete
 
 For ~50 typical audit-report PDFs: