PDS proof of concept: x ipld sqlite driver importing CAR file => simple binary, two args x skeleton x env config: DB paths, port x commands: serve, import, inspect x integration test x atp db wrapper (with methods) schema in a single .sql file https://docs.rs/rusqlite_migration/latest/rusqlite_migration/ test version (in memory, per-thread) wrap in a mutex, unwrap and make new connection when desired x wrap both database in a struct with mutexes; have "get handle" helper that unlocks and returns a connection copy of the given type x repo store database wrapper (with methods) x response error handling (especially for XRPC endpoints) x basic crypto and did:plc stuff x did:key read/write helpers x test that did:plc generated as expected x signature read/write helpers x single shared signing key for all users (not what I expected) x sqlite schema (for application) - MST code to read and mutate tree state x check that empty tree works (eg, for account creation, and after deletes) x with in-memory tests => mutation batches - service-level config domain suffixes (eg, just ".test" for now) account registration allowed or not CLI account creation (?) PDS signing key - figure out auth JWT 'sub' situation (not a UCAN? CLI should ignore?) - switch to Argon2 for passwords? meh x IPLD objects to JSON value - implement basic non-authenticated CRUD on repository, test with CLI com.atproto x getAccountsConfig x createAccount x repoGetRecord x repoListRecords x syncGetRoot x repoDescribe x createSession x repoBatchWrite x repoCreateRecord x repoPutRecord x repoDeleteRecord syncGetRepo syncUpdateRepo x Did, Tid, Nsid types, with TID generator object? => xrpc_parse wrappers - push types back "down" (less String typing in function signatures) => eg Cid in repo - aturi canonicalization helper (re-writes usernames to DID?) x helper web methods - python XRPC API test script - PDS CLI helpers create-account reset-password list-repos list-accounts import-car [} - mutate wrapper which updates MST *and* updates other tables in transactions - JSON schema type generation (separate crate?) - HTTP API handler implementing many endpoints com.atproto getSession resolveName app.bsky updateProfile getProfile getHomeFeed getAuthorFeed getPostThread getUserFollowers getUserFollows getLikedBy getRepostedBy getNotifications getNotificationCount postNotificationsSeen getUsersSearch later: - TODO: why are the multiformat keys so long in did doc? - correct JWT helper stuff (?) - did:web handler? more PDS CLI helpers reindex => locally re-generate cache tables update-repo [] [] => or some full URI or something? update [] => does this work yet? update-all-dids => would re-fetch did cods for all non-local users (?) spider => collect a big list of DIDs => fetch all the remote repos for those DIDs other utils/helpers: - pack/unpack a repo CAR into JSON files in a directory tree (plus a commit.json with sig?) libraries: - `jsonschema` to validate requests and records (rich validation) - `schemafy` to codegen serde types for records (ahead of time?) - pretty_env_logger - no good published crate for working with CAR files... could rip out this code? https://github.com/n0-computer/iroh/tree/main/iroh-car - ??? for CBOR (de)serialization of MST, separate from the IPLD stuff? sync option: - `rouille` web framework - `rusqlite` with "bundled" sqlite for datastore - `rusqlite_migration` - `ipfs-sqlite-block-store` and `libipld` to parse and persist repo content async option: - `warp` as async HTTP service - `sqlx` for async pooled sqlite or postgresql db - `iroh-store` for async rocksdb IPFS blockstore ## concurrency (in warp app) note that there isn't really any point in having this be async, given that we just have a single shared sqlite on disk. could try `rouille` instead of `warp`? maybe good for remote stuff like did:web resolution? could try using sqlx instead of rusqlite for natively-async database stuff? for block store: - open a single connection at startup, store in mutex - handlers get a reference to mutex. if they need a connection, they enter a blocking thread then: block on the mutex, then create a new connection, unlock the mutex do any operations on connection synchronously exit the block ## system tables account did (PK) username (UNIQUE, indexed) email (UNIQUE) password_bcrypt signing_key did_doc did (PK) seen_at (timestamp) session did jwt ??? repo did head_commit record (should this exist? good for queries) did collection tid record_cid record_cbor (CBOR bytes? JSON?) password_reset did token ## atp tables what actually needs to be indexed? - post replies (forwards and backwards - likes (back index) - follows (back index) - usernames (as part of profile?) - mentions? hashtags? additional state - notifications bsky_post did tid (or timestamp from tid?) text reply_root (nullable) reply_parent (nullable) entities: JSON (?) bsky_profile did tid display_name description my_state (JSON) bsky_follow did tid target_did bsky_like did tid target_uri target_cid (what is this? the commit, or record CID?) bsky_repost did tid target_uri target_cid bsky_notification did created_at (timestamp) seen (boolean) reason target_uri TODO: - bsky_badge (etc) - bsky_media_embed ---- what is needed to be useful? - workflow to self-hosting a single user with did:web - host multiple users with did:plc and wildcard domain - allow users to bring their own did:web identifier, which is not controlled directly by the PDS => some method to register with existing DID => export did doc - some mechanism of automatically pulling from other PDS what might be interesting? - basic read-only web interface => tech "explorer" for MST etc - RSS feeds - activitypub export - PDS CLI that can inspect tree structure