JIDO_BEDROCK_PLAN.md

JIDO_BEDROCK_PLAN

Summary

Create a new package, jido_bedrock, that provides distributed durable persistence for Jido agents and Jido signals on Bedrock without requiring Postgres or a larger external DB.

The package will implement three adapter surfaces:

1. Jido.Storage for checkpoints + thread journals used by Jido.Persist.
2. Jido.Agent.Store for InstanceManager hibernate/thaw persistence.
3. Jido.Signal.Journal.Persistence for signal docs, causality edges, conversation indexes, checkpoints, and DLQ.

Default mode must be durability-strict and fail fast if Bedrock cluster configuration is below a durable profile.

Scope

In scope

1. New package at jido_bedrock with production-oriented adapters.
2. Shared key/codec/chunking internals.
3. Durability profile checks and telemetry.
4. Unit, contract, and distributed integration tests.
5. Operator-facing documentation and examples.

Out of scope (v1)

1. Changes to Jido/Jido Signal public APIs.
2. Bedrock storage engine redesign.
3. GUI/admin tooling.

Public Modules and Interfaces

1. Jido.Storage.Bedrock implementing Jido.Storage.
2. Jido.Agent.Store.Bedrock implementing Jido.Agent.Store.
3. Jido.Signal.Journal.Adapters.Bedrock implementing Jido.Signal.Journal.Persistence.
4. JidoBedrock.Config for common adapter options.
5. JidoBedrock.Durability for profile validation and diagnostics.
6. JidoBedrock.Codec behavior with default JidoBedrock.Codec.BERT.
7. JidoBedrock.Envelope for chunk manifests and checksum validation.
8. JidoBedrock.Keys for canonical tuple-key construction.

Shared Adapter Options

All adapters accept:

1. :repo (required): module using Bedrock.Repo.
2. :namespace (required): binary namespace isolation key.
3. :codec (default JidoBedrock.Codec.BERT).
4. :compress (default true).
5. :chunk_size_bytes (default 120_000).
6. :durability_mode (default :strict, alt :relaxed).
7. :retry_limit (default 20).

Data Model and Key Layout

Use tuple-packed keys rooted at:

• {"jb", namespace, ...}

Jido.Storage.Bedrock

1. Checkpoint manifest:

• {"jb", ns, "jido", "ckpt", key_hash}

2. Checkpoint chunks:

• {"jb", ns, "jido", "ckpt_chunk", key_hash, idx}

3. Thread meta:

• {"jb", ns, "jido", "thread", thread_id, "meta"}

4. Thread entry:

• {"jb", ns, "jido", "thread", thread_id, "entry", seq}

Jido.Agent.Store.Bedrock

1. Agent dump manifest:

• {"jb", ns, "agent_store", "dump", key_hash}

2. Agent dump chunks:

• {"jb", ns, "agent_store", "dump_chunk", key_hash, idx}

Jido.Signal.Journal.Adapters.Bedrock

1. Signal doc:

• {"jb", ns, "signal", "doc", signal_id}

2. Cause edge:

• {"jb", ns, "signal", "cause_edge", cause_id, effect_id}

3. Effect edge:

• {"jb", ns, "signal", "effect_edge", effect_id, cause_id}

4. Conversation edge:

• {"jb", ns, "signal", "conv_edge", conversation_id, signal_id}

5. Subscription checkpoint:

• {"jb", ns, "signal", "checkpoint", subscription_id}

6. DLQ by-subscription:

• {"jb", ns, "signal", "dlq_sub", subscription_id, inserted_at_us, entry_id}

7. DLQ pointer by-id:

• {"jb", ns, "signal", "dlq_id", entry_id}

Hashing for non-binary keys

key_hash = Base.url_encode64(:crypto.hash(:sha256, :erlang.term_to_binary(key)), padding: false)

Encoding and Chunking Strategy

1. Inline encode when serialized payload is <= 128 KiB.
2. Chunk when payload exceeds limit.
3. Manifest fields:

• version
• codec
• compressed
• chunk_count
• checksum
• original_size

4. Read path validates checksum and chunk continuity.
5. Corruption returns explicit adapter error, not silent fallback.

Adapter Behavior Mapping

Jido.Storage.Bedrock

1. get_checkpoint/2

• Resolve checkpoint manifest + chunks.
• Return :not_found if absent.

2. put_checkpoint/3

• One Bedrock transaction writing manifest and chunks.
• Overwrite semantics.

3. delete_checkpoint/2

• Atomic delete of manifest + chunk range.

4. load_thread/2

• Read meta and range-scan entry keys.
• Rebuild %Jido.Thread{} with consistent rev.

5. append_thread/3

• Read current rev.
• Enforce :expected_rev; on mismatch call rollback(:conflict).
• Write new sequential entries and update meta in one transaction.

6. delete_thread/2

• Clear thread entry range and meta atomically.

Jido.Agent.Store.Bedrock

1. get/2: load envelope/chunks by hashed store key.
2. put/3: atomic upsert manifest/chunks.
3. delete/2: atomic delete manifest/chunks.

Jido.Signal.Journal.Adapters.Bedrock

1. put_signal/2: upsert signal doc.
2. get_signal/2: lookup by id with {:error, :not_found} semantics.
3. put_cause/3: write cause/effect edge pair atomically.
4. get_effects/2: range-scan cause edges into MapSet.
5. get_cause/2: range-scan effect edges and return deterministic first cause id.
6. put_conversation/3: upsert conversation edge.
7. get_conversation/2: range-scan conversation edges into MapSet.
8. put_checkpoint/3: upsert subscription checkpoint.
9. get_checkpoint/2: lookup checkpoint with {:error, :not_found}.
10. delete_checkpoint/2: delete checkpoint key.
11. put_dlq_entry/5: write dlq_sub and dlq_id pointer atomically.
12. get_dlq_entries/2: range-scan dlq_sub keys, naturally sorted by key.
13. delete_dlq_entry/2: resolve pointer, delete both records atomically.
14. clear_dlq/2: scan subscription DLQ keys and delete in bounded batches.

Durability Contract

Default mode: :strict

Adapter init must validate runtime cluster profile before serving writes.

Required minimums:

1. desired_replication_factor >= 3
2. desired_logs >= 3
3. Persistent paths configured for coordinator, log, and storage roles.

Failure behavior:

1. Return {:error, {:durability_profile_failed, reason}}.
2. Emit telemetry failure event.

Relaxed mode: :relaxed

1. Allows startup with warnings and telemetry annotations.
2. Keeps semantics identical but does not claim strict profile compliance.

Telemetry

Emit:

1. [:jido_bedrock, :durability, :profile, :ok]
2. [:jido_bedrock, :durability, :profile, :failed]
3. [:jido_bedrock, :adapter, :operation, :stop]
4. [:jido_bedrock, :adapter, :operation, :exception]

Metadata includes adapter module, operation name, namespace, duration, and error class.

Implementation Phases

1. Bootstrap package skeleton, dependencies, CI matrix.
2. Build shared internals (Config, Keys, Codec, Envelope, tx helpers).
3. Implement Jido.Storage.Bedrock + contract tests.
4. Implement Jido.Agent.Store.Bedrock + tests.
5. Implement Jido.Signal.Journal.Adapters.Bedrock + tests.
6. Implement durability checks and strict/relaxed modes.
7. Build distributed integration harness (3-node Bedrock cluster).
8. Finish docs/examples/release notes.

Test Plan

Contract tests

1. Match behavior from existing ETS/File/Mnesia adapters for all callbacks.
2. Validate expected_rev conflict behavior and thread ordering.
3. Validate DLQ ordering/filtering and checkpoint semantics.

Data integrity tests

1. Chunk boundary and reassembly correctness.
2. Checksum mismatch and corruption handling.
3. Safe decode behavior for persisted terms.

Concurrency tests

1. Concurrent append_thread/3 calls with same expected rev.
2. Concurrent signal causality updates.

Distributed durability tests

1. 3-node happy path writes and reads.
2. Node restart survival for checkpoints, thread entries, and signals.
3. Leader failover during active workload.
4. Strict profile pass/fail verification.

Release Plan

1. 0.1.0-alpha: core adapters + local tests.
2. 0.2.0-beta: strict profile checks + distributed tests.
3. 1.0.0-rc: full docs and migration guidance.
4. 1.0.0: production gate after durability acceptance criteria validation.

Assumptions and Defaults

1. Package root will be /Users/mhostetler/Source/Jido/jido_bedrock.
2. Namespace is mandatory and per-environment.
3. Default codec is BERT with safe decode.
4. Compression defaults to enabled.
5. Chunk size defaults to 120_000 bytes.
6. Durability mode defaults to :strict.
7. get_cause/2 returns deterministic lexicographically first cause id.

Acceptance Criteria

1. All adapter callbacks implemented and tested.
2. Jido.Persist hibernate/thaw works end-to-end using Bedrock adapter.
3. Jido.Signal.Journal operations work end-to-end using Bedrock adapter.
4. Strict durability mode fails fast on under-provisioned cluster profiles.
5. Distributed restart/failover integration tests pass in CI.