> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reflections.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ADR-0010: Ingestion orchestration, idempotency, and recovery

> Make source ingest retry-safe and recoverable while preserving evaluation-gated learning.

<Info>**Status:** Accepted **Date:** 2026-02-06 **Deciders:** Reflections Maintainers</Info>

## Context

Source ingestion has multiple side-effectful stages (fetch/chunk/embed/entity/fact/eval/apply). Failures and retries are expected; the system needs idempotency keys, resumable behavior, and explicit status telemetry.

## Decision

Use Inngest orchestration with ingest ledgering and patch-aware recovery:

* Create/update ingest run records keyed by `(reflection_id, source_id, content_hash)` and `idempotency_key`.
* Reuse existing patch batch where possible; skip duplicate work for terminal patch states.
* Recover from previously approved but unapplied patches by applying and marking source ingested.
* Record progress states and terminal outcomes (`completed`, `eval_rejected`, `failed`, `duplicate_skipped`).

## Alternatives considered

### Alternative 1: Fire-and-forget ingest with no run ledger

Pros:

* Minimal persistence overhead.

Cons:

* No reliable retry semantics.
* Poor observability and manual recovery burden.

### Alternative 2: Exactly-once queue semantics only

Pros:

* Cleaner mental model in theory.

Cons:

* Hard to guarantee across all external side effects.
* Still needs stateful compensation logic for partial failures.

### Alternative 3: Manual operator-driven retries only

Pros:

* Operationally explicit.

Cons:

* Slow recovery and low throughput.
* Error-prone repeated manual interventions.

## Consequences

**Benefits:**

* Controlled retries and deterministic duplicate handling.
* Better failure visibility and pipeline telemetry.
* Recovery logic tied to authoritative patch status.

**Costs:**

* Increased state-machine complexity in worker orchestration.
* More status transitions to test and maintain.

## Implementation notes

* End-to-end ingest orchestration is implemented in the worker functions layer using Inngest.
* Ingest run persistence and status handling live in the DB queries layer.
* Schema support for idempotency and statuses is defined across multiple migrations that add idempotency keys, entity constraints, and ingest run status refinements.

## Related ADRs

* [ADR-0003: Two-plane system architecture](/decisions/adr-0003)
* [ADR-0005: Temporal fact and patch lifecycle](/decisions/adr-0005)
* [ADR-0011: Observability and tracing strategy](/decisions/adr-0011)