summaryrefslogtreecommitdiffstats
path: root/adenosine-pds/plan.txt
blob: 56478a43345967dc8d8eb19713171d71b40337b4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309

PDS proof of concept:
x ipld sqlite driver importing CAR file
    => simple binary, two args
x skeleton
  x env config: DB paths, port
  x commands: serve, import, inspect
  x integration test
  x atp db wrapper (with methods)
        schema in a single .sql file
        https://docs.rs/rusqlite_migration/latest/rusqlite_migration/
        test version (in memory, per-thread)
        wrap in a mutex, unwrap and make new connection when desired
  x wrap both database in a struct with mutexes; have "get handle" helper that unlocks and returns a connection copy of the given type
  x repo store database wrapper (with methods)
  x response error handling (especially for XRPC endpoints)
x basic crypto and did:plc stuff
  x did:key read/write helpers
  x test that did:plc generated as expected
  x signature read/write helpers
x single shared signing key for all users (not what I expected)
x sqlite schema (for application)
x fix did multibase key encoding:
    https://medium.com/asecuritysite-when-bob-met-alice/02-03-or-04-so-what-are-compressed-and-uncompressed-public-keys-6abcb57efeb6
x MST code to read and mutate tree state
    x  check that empty tree works (eg, for account creation, and after deletes)
    x  with in-memory tests
    x  mutation batches
x IPLD objects to JSON value
x Did, Tid, Nsid types, with TID generator object?
x push types back "down" (less String typing in function signatures)
    => eg Cid in repo
x helper web methods
x cli: aturi should accept trailing slash
x implement basic non-authenticated CRUD on repository, test with CLI
    com.atproto
      x getAccountsConfig
      x createAccount
      x repoGetRecord
      x repoListRecords
      x syncGetRoot
      x repoDescribe
      x createSession
      x repoBatchWrite
      x     repoCreateRecord
      x     repoPutRecord
      x     repoDeleteRecord
      x syncGetRepo
      x syncUpdateRepo
x service-level config
      x domain suffixes (eg, just ".test" for now)
      x account registration allowed or not
      x PDS signing key
x CLI account creation (including did:web accounts)
- HTTP API handler implementing many endpoints
    com.atproto
      x getSession
      x resolveName
    app.bsky
      x updateProfile
      x getProfile

      x getHomeFeed
      x getAuthorFeed
        getPostThread

        getUserFollowers
        getUserFollows

        getLikedBy
        getRepostedBy

        getNotifications
        getNotificationCount
        postNotificationsSeen

        getUsersSearch

x cli: 'bsky timeline' command (for self)
x wire up basic bsky stuff to web interface
x bsky app handler (new module)
    x  batch mutation handler
    x  high-level database/repo helpers
x post threads (simple parent/children to start)

- basic CLI testing of bsky stuff (against upstream version and mine)
    => follow
    => like
    => follow
    => repost
    => author feed (of their posts and reposts)
    => timeline feed
- bit of polish
    x  web error handler (request to request)
    x  handlers should catch mutex poison error and exist process
    x  /about
    x  version in web footer
    x  actually implement /.well-known/did.json, etc
    x  check if CAR export works (size, blocks)
    x  invite code validation
    x  registration domain validation (including lack of domain blocking all registration)
    x  update README
    x  basic manpage for adenosine-pds (just a stub)
    x  CHANGELOG
- push demo
    => systemd unit files
    => sketch nginx file
    => get single-domain stuff working on adze
    => get multi-domain stuff working on adze
    => post some example dummy content

    => git/local 0.1.0 tag
    => build and push debs to adze, some archive.org mirror location

    => share links

- v0.1.1 (or v0.2.0?)
    - make profile DID link an aturi (pink)
    - reverse order of author feed (?)
    - check compatibility with current atproto
    - CLI: update TID implementation
    - static compilation: rustls, musl
    - reduce clap and reqwest size?
    - PDS: RSS feed (and link to it)
    - PDS: export download link (on repository page?)
    - CLI: update profile
    - CLI: status actually connects to server, verifies auth

x basic web handler (separate plan)
- JSON schema type generation (separate crate?)

- RSS, webfinger, maybe some other protocols
    https://lib.rs/crates/rss

- improve updateRepo implementation
- figure out auth JWT 'sub' situation (not a UCAN? CLI should ignore?)
- switch to Argon2 for passwords? meh
- aturi canonicalization helper (re-writes usernames to DID?)
- python XRPC API test script
- PDS CLI helpers
    create-account <username> <password> <email>
    reset-password <username> <password>
    list-repos
    list-accounts 
    import-car <car-file> [<did|alias>}

later:
x TODO: why are the multiformat keys so long in did doc?
- correct JWT helper stuff (?)
- did:web handler?

more PDS CLI helpers
    reindex <did>
        => locally re-generate cache tables
    update-repo <did> [<pds-server>] [<plc-server>]
        => or some full URI or something?
    update <did> [<plc-server>]
        => does this work yet?
    update-all-dids
        => would re-fetch did cods for all non-local users (?)
    spider
        => collect a big list of DIDs
        => fetch all the remote repos for those DIDs

other utils/helpers:
- pack/unpack a repo CAR into JSON files in a directory tree (plus a commit.json with sig?)

libraries:
- `jsonschema` to validate requests and records (rich validation)
- `schemafy` to codegen serde types for records (ahead of time?)
- pretty_env_logger
- no good published crate for working with CAR files... could rip out this code?
    https://github.com/n0-computer/iroh/tree/main/iroh-car
- ??? for CBOR (de)serialization of MST, separate from the IPLD stuff?

sync option:
- `rouille` web framework
- `rusqlite` with "bundled" sqlite for datastore
- `rusqlite_migration`
- `ipfs-sqlite-block-store` and `libipld` to parse and persist repo content 

async option:
- `warp` as async HTTP service
- `sqlx` for async pooled sqlite or postgresql db
- `iroh-store` for async rocksdb IPFS blockstore

## concurrency (in warp app)

note that there isn't really any point in having this be async, given that we
just have a single shared sqlite on disk. could try `rouille` instead of
`warp`?

maybe good for remote stuff like did:web resolution?

could try using sqlx instead of rusqlite for natively-async database stuff?

for block store:
- open a single connection at startup, store in mutex
- handlers get a reference to mutex. if they need a connection, they enter a blocking thread then:
    block on the mutex, then create a new connection, unlock the mutex
    do any operations on connection synchronously
    exit the block

## system tables

account
    did (PK)
    username (UNIQUE, indexed)
    email (UNIQUE)
    password_bcrypt
    signing_key

did_doc
    did (PK)
    seen_at (timestamp)

session
    did
    jwt
    ???

repo
    did
    head_commit

record (should this exist? good for queries)
    did
    collection
    tid
    record_cid
    record_cbor (CBOR bytes? JSON?)

password_reset
    did
    token


## atp tables

what actually needs to be indexed?
- post replies (forwards and backwards
- likes (back index)
- follows (back index)
- usernames (as part of profile?)
- mentions? hashtags?

additional state
- notifications

bsky_post
    did
    tid (or timestamp from tid?)
    text
    reply_root (nullable)
    reply_parent (nullable)
    entities: JSON (?)

bsky_profile
    did
    tid
    display_name
    description
    my_state (JSON)

bsky_follow
    did
    tid
    target_did

bsky_like
    did
    tid
    target_uri
    target_cid (what is this? the commit, or record CID?)

bsky_repost
    did
    tid
    target_uri
    target_cid

bsky_notification
    did
    created_at (timestamp)
    seen (boolean)
    reason
    target_uri

TODO:
- bsky_badge (etc)
- bsky_media_embed


----

what is needed to be useful?
- workflow to self-hosting a single user with did:web
- host multiple users with did:plc and wildcard domain
- allow users to bring their own did:web identifier, which is not controlled directly by the PDS
    => some method to register with existing DID
    => export did doc
- some mechanism of automatically pulling from other PDS

what might be interesting?
- basic read-only web interface
    => tech "explorer" for MST etc
- RSS feeds
- activitypub export
- PDS CLI that can inspect tree structure