diff options
| -rw-r--r-- | TODO | 37 | ||||
| -rw-r--r-- | notes/architecture.md | 56 | ||||
| -rw-r--r-- | notes/refs.md | 5 |
3 files changed, 95 insertions, 3 deletions
@@ -1,14 +1,44 @@ next: -- DatPeer wrapper API -- Synchronizer API +- DatConnection + => u32 for feed index + => change sockaddr argument... not sure to what, maybe ToSockAddr? + => rip out more crap ("receive some", no_delay, registration extras) + => how to accept() (from_stream() refactor?) +- PeerThread + => proper error propagation + => don't store state (bitfield) in PeerThread itself, leave to Synchronizer + => to start, just use feed index (don't keep key/index table) + => need "self peer id" passed around + => keep "peer handle"... in the future, should be based on remote handle returned +- Synchronizer + => probably an options builder + => generate "self peer id" + => "wanted" bitmap + => design statemachine (global and per-peer) + => actions for each message type + => PeerState helper, with bitfield + => when a peer is ready, find a mutal chunk option, mark as in progress + => some logic to do drive stuff (add new registers, etc) + => chan-signal to shutdown +- get everything to compile - clean up node bitfield implementation -- "insert" API for HyperRegister + => use bit_vec (bit-vec = "0.4.4") instead of bit_field (remove) +- "insert" API for HyperRegister (when we have a signed Data message) +- how to create a Data message in response to a Request - sparse register clone - 'geniza clone' command - 'geniza checkout' command - 'geniza init', 'geniza commit' commands +threaded sync: +- rust stdlib async TCP? +- event loop runner +- channel polling +- polling for connections (channels + async TCP) +- UNIX signal handling +- later: accepting inbound connections + specific tests: - drive children array: deep directory nesting (like 'tree' example) - sleep info a variety of bad sleep files, should error @@ -62,3 +92,4 @@ Backburner: https://github.com/ia0/data-encoding - sendfile() or splice() for disk-to-disk copying? - Rust FS API example: https://docs.rs/zbox/0.1.3/zbox/ +- "bytes" crate diff --git a/notes/architecture.md b/notes/architecture.md new file mode 100644 index 0000000..c9268c9 --- /dev/null +++ b/notes/architecture.md @@ -0,0 +1,56 @@ + +peers wrap connections with given clients (any transport type). each peer can +be talking about multiple feeds. state consists of which feeds, what the remote +has (feed, bitmap), and what chunks (feed, index) have been requested +("in-flight"). connection logic runs in threads, passes back and forth entire +messages in channels. + + ConnCommand + Send(DatNetMessage) + Terminate() + + ConnReply + Received(DatNetMessage) + Disconnected() + +A synchronizer event loop has local feeds (storage) and a bunch of peers. Extra +state includes mode configuration, feeds, "wanted" bitmap for each feed. +Callbacks for certain feed events allow for extension (eg, hyperdrive stuff). +Needs a timer event. + +Use chan and chan-signal for now (MPSC style)... tokio will be the future but +seems too unstable for now. mio is too low-level to use on it's own (eg, no +timers, no threads). Use two threads per peer: a "main" thread that does +connection setup, `chan_select`, and blocking writes, and a "reader" thread +that does blocking reads of entire messages (writing either into the main +thread or skipping straight to the output channel). Got this pattern from the +`rusty_torrent` package. Need to set some write timeout. Synchronizer handles +cleanup if peer isn't replying/heartbeating. + +Probably want a dedicated discovery thread/worker in the future, but for now +just do it synchronously and connect to up to N peers to start. + +Synchronize will select on: + + signal + graceful shutdown + unified_inbound + feed: depends on state; ignore for now + handshake: error, close connection? + have, info: update PeerState bitmap + unhave, info, want, unwant: ignore for now + request: depends on state; read from storage + cancel: ignore for now + data: write to storage, clear pending; send new request + tick + check peer states, kill if too slow + dump status to command line + +Synchronize needs state: +- "tried" peers, so we don't retry in a loop if they fail + +Synchronize options: +- whether or not to hook signals + +Where does drive logic live? Plugin, via callback, to Synchronize? Let's just +fold it in to synchronizer for now, as a mode flag. diff --git a/notes/refs.md b/notes/refs.md new file mode 100644 index 0000000..4995578 --- /dev/null +++ b/notes/refs.md @@ -0,0 +1,5 @@ + +Decent "Original" Bittorrent client description: + +https://github.com/mgp/coding-in-the-real-world/blob/master/manuscript/bittorrent-client-case-study.md + |