# truf data design (v1)

## Entities and relationships

- `User`: account (email, password hash). Owns subscriptions and read state.
- `Feed`: one subscribed source URL (canonicalized). Stores fetch metadata (etag, last-modified, last fetch/success, failure count, last error, next poll time, host). Shared across users.
- `Item`: one normalized entry belonging to a `Feed`. Stored once, shared. Has title, link, sanitized content, author, published_at (clamped), fetched_at, dedup_key.
- `Subscription`: join of User <-> Feed (user subscribed to feed).
- `ReadState`: join of User <-> Item (whether read). Absence implies unread.

Relationships:
User 1--* Subscription _--1 Feed
Feed 1--_ Item
User 1--* ReadState *--1 Item

## Item identification and deduplication

Per feed, compute `dedup_key`:

1. If the feed provides a stable id/guid, use it.
2. Otherwise hash (SHA-256) of (canonical link, normalized title, published date ISO).
   On ingest, insert item only if no existing row with same (feed_id, dedup_key). Unique constraint enforces this. Re-polling never duplicates.

## Read state

Separate `read_state(user_id, item_id, read_at)` rows. Unread = subscribed items with no read_state row (or read_at null). Marking read inserts/updates row; unread deletes or nulls. This keeps item storage shared and per-user state small.

## Polling schedule

A single in-process scheduler ticks every 30s and selects feeds where `next_poll_at <= now` and not currently polling. Per-feed mutex prevents concurrent polls. A global concurrency cap and per-host cap (p-limit style) bound outbound requests. Conditional GET uses stored etag/last-modified; 304 skips parsing. Failures increment failure_count, apply exponential backoff to next_poll_at, store last_error. After threshold (e.g. 5 consecutive failures) feed marked failing but still retried with backoff.

## Live updates

In-process EventEmitter (event bus). When ingest adds N new items for a feed, emit `feed:new-items` with feed id + item ids. A per-user SSE endpoint subscribes and forwards only items whose feed is in that user's subscriptions. Clients use EventSource; reconnect is native. Heartbeat every 25s keeps proxies alive.

## Trade-offs and recommendations

- SQLite (node:sqlite) for v1 simplicity; repository module abstracts SQL so Postgres can replace later. Trade-off: single-writer; acceptable for v1.
- In-process scheduler + event bus: simple, no Redis. Trade-off: single instance only; documented. Redis deferred to later milestone if needed.
- Separate read_state table vs bitset: flexible and queryable; slightly more rows but fine.
- Cursor pagination by (published_at, id) descending: stable under inserts.