PDS proof of concept: x ipld sqlite driver importing CAR file => simple binary, two args x skeleton x env config: DB paths, port x commands: serve, import, inspect x integration test x atp db wrapper (with methods) schema in a single .sql file https://docs.rs/rusqlite_migration/latest/rusqlite_migration/ test version (in memory, per-thread) wrap in a mutex, unwrap and make new connection when desired x wrap both database in a struct with mutexes; have "get handle" helper that unlocks and returns a connection copy of the given type x repo store database wrapper (with methods) x response error handling (especially for XRPC endpoints) x basic crypto and did:plc stuff x did:key read/write helpers x test that did:plc generated as expected x signature read/write helpers x single shared signing key for all users (not what I expected) x sqlite schema (for application) x fix did multibase key encoding: https://medium.com/asecuritysite-when-bob-met-alice/02-03-or-04-so-what-are-compressed-and-uncompressed-public-keys-6abcb57efeb6 x MST code to read and mutate tree state x check that empty tree works (eg, for account creation, and after deletes) x with in-memory tests x mutation batches x IPLD objects to JSON value x Did, Tid, Nsid types, with TID generator object? x push types back "down" (less String typing in function signatures) => eg Cid in repo x helper web methods x cli: aturi should accept trailing slash x implement basic non-authenticated CRUD on repository, test with CLI com.atproto x getAccountsConfig x createAccount x repoGetRecord x repoListRecords x syncGetRoot x repoDescribe x createSession x repoBatchWrite x repoCreateRecord x repoPutRecord x repoDeleteRecord x syncGetRepo x syncUpdateRepo x service-level config x domain suffixes (eg, just ".test" for now) x account registration allowed or not x PDS signing key x CLI account creation (including did:web accounts) - HTTP API handler implementing many endpoints com.atproto x getSession x resolveName app.bsky x updateProfile x getProfile x getHomeFeed x getAuthorFeed getPostThread getUserFollowers getUserFollows getLikedBy getRepostedBy getNotifications getNotificationCount postNotificationsSeen getUsersSearch x cli: 'bsky timeline' command (for self) x wire up basic bsky stuff to web interface x bsky app handler (new module) x batch mutation handler x high-level database/repo helpers x post threads (simple parent/children to start) - basic CLI testing of bsky stuff (against upstream version and mine) => follow => like => follow => repost => author feed (of their posts and reposts) => timeline feed - bit of polish x web error handler (request to request) x handlers should catch mutex poison error and exist process x /about x version in web footer x actually implement /.well-known/did.json, etc x check if CAR export works (size, blocks) x invite code validation x registration domain validation (including lack of domain blocking all registration) x update README x basic manpage for adenosine-pds (just a stub) x CHANGELOG - push demo => systemd unit files => sketch nginx file => get single-domain stuff working on adze => get multi-domain stuff working on adze => post some example dummy content => git/local 0.1.0 tag => build and push debs to adze, some archive.org mirror location => share links - v0.1.1 (or v0.2.0?) - make profile DID link an aturi (pink) - reverse order of author feed (?) - check compatibility with current atproto - CLI: update TID implementation - static compilation: rustls, musl - reduce clap and reqwest size? - PDS: RSS feed (and link to it) - PDS: export download link (on repository page?) - CLI: update profile - CLI: status actually connects to server, verifies auth x basic web handler (separate plan) - JSON schema type generation (separate crate?) - RSS, webfinger, maybe some other protocols https://lib.rs/crates/rss - improve updateRepo implementation - figure out auth JWT 'sub' situation (not a UCAN? CLI should ignore?) - switch to Argon2 for passwords? meh - aturi canonicalization helper (re-writes usernames to DID?) - python XRPC API test script - PDS CLI helpers create-account reset-password list-repos list-accounts import-car [} later: x TODO: why are the multiformat keys so long in did doc? - correct JWT helper stuff (?) - did:web handler? more PDS CLI helpers reindex => locally re-generate cache tables update-repo [] [] => or some full URI or something? update [] => does this work yet? update-all-dids => would re-fetch did cods for all non-local users (?) spider => collect a big list of DIDs => fetch all the remote repos for those DIDs other utils/helpers: - pack/unpack a repo CAR into JSON files in a directory tree (plus a commit.json with sig?) libraries: - `jsonschema` to validate requests and records (rich validation) - `schemafy` to codegen serde types for records (ahead of time?) - pretty_env_logger - no good published crate for working with CAR files... could rip out this code? https://github.com/n0-computer/iroh/tree/main/iroh-car - ??? for CBOR (de)serialization of MST, separate from the IPLD stuff? sync option: - `rouille` web framework - `rusqlite` with "bundled" sqlite for datastore - `rusqlite_migration` - `ipfs-sqlite-block-store` and `libipld` to parse and persist repo content async option: - `warp` as async HTTP service - `sqlx` for async pooled sqlite or postgresql db - `iroh-store` for async rocksdb IPFS blockstore ## concurrency (in warp app) note that there isn't really any point in having this be async, given that we just have a single shared sqlite on disk. could try `rouille` instead of `warp`? maybe good for remote stuff like did:web resolution? could try using sqlx instead of rusqlite for natively-async database stuff? for block store: - open a single connection at startup, store in mutex - handlers get a reference to mutex. if they need a connection, they enter a blocking thread then: block on the mutex, then create a new connection, unlock the mutex do any operations on connection synchronously exit the block ## system tables account did (PK) username (UNIQUE, indexed) email (UNIQUE) password_bcrypt signing_key did_doc did (PK) seen_at (timestamp) session did jwt ??? repo did head_commit record (should this exist? good for queries) did collection tid record_cid record_cbor (CBOR bytes? JSON?) password_reset did token ## atp tables what actually needs to be indexed? - post replies (forwards and backwards - likes (back index) - follows (back index) - usernames (as part of profile?) - mentions? hashtags? additional state - notifications bsky_post did tid (or timestamp from tid?) text reply_root (nullable) reply_parent (nullable) entities: JSON (?) bsky_profile did tid display_name description my_state (JSON) bsky_follow did tid target_did bsky_like did tid target_uri target_cid (what is this? the commit, or record CID?) bsky_repost did tid target_uri target_cid bsky_notification did created_at (timestamp) seen (boolean) reason target_uri TODO: - bsky_badge (etc) - bsky_media_embed ---- what is needed to be useful? - workflow to self-hosting a single user with did:web - host multiple users with did:plc and wildcard domain - allow users to bring their own did:web identifier, which is not controlled directly by the PDS => some method to register with existing DID => export did doc - some mechanism of automatically pulling from other PDS what might be interesting? - basic read-only web interface => tech "explorer" for MST etc - RSS feeds - activitypub export - PDS CLI that can inspect tree structure