aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2018-01-21 23:28:46 -0800
committerBryan Newbold <bnewbold@robocracy.org>2018-01-21 23:28:51 -0800
commit0069b123b9bf787af46b01517aa27b1fb7ade9d2 (patch)
tree682736eedf5e3ba361b790d9670a62b48728d1bb
parent7989a5f2979361b0ebdcdaea41cccbd02c38fdd1 (diff)
downloadgeniza-0069b123b9bf787af46b01517aa27b1fb7ade9d2.tar.gz
geniza-0069b123b9bf787af46b01517aa27b1fb7ade9d2.zip
yet more notes
-rw-r--r--TODO37
-rw-r--r--notes/architecture.md56
-rw-r--r--notes/refs.md5
3 files changed, 95 insertions, 3 deletions
diff --git a/TODO b/TODO
index 84c0413..9a008ad 100644
--- a/TODO
+++ b/TODO
@@ -1,14 +1,44 @@
next:
-- DatPeer wrapper API
-- Synchronizer API
+- DatConnection
+ => u32 for feed index
+ => change sockaddr argument... not sure to what, maybe ToSockAddr?
+ => rip out more crap ("receive some", no_delay, registration extras)
+ => how to accept() (from_stream() refactor?)
+- PeerThread
+ => proper error propagation
+ => don't store state (bitfield) in PeerThread itself, leave to Synchronizer
+ => to start, just use feed index (don't keep key/index table)
+ => need "self peer id" passed around
+ => keep "peer handle"... in the future, should be based on remote handle returned
+- Synchronizer
+ => probably an options builder
+ => generate "self peer id"
+ => "wanted" bitmap
+ => design statemachine (global and per-peer)
+ => actions for each message type
+ => PeerState helper, with bitfield
+ => when a peer is ready, find a mutal chunk option, mark as in progress
+ => some logic to do drive stuff (add new registers, etc)
+ => chan-signal to shutdown
+- get everything to compile
- clean up node bitfield implementation
-- "insert" API for HyperRegister
+ => use bit_vec (bit-vec = "0.4.4") instead of bit_field (remove)
+- "insert" API for HyperRegister (when we have a signed Data message)
+- how to create a Data message in response to a Request
- sparse register clone
- 'geniza clone' command
- 'geniza checkout' command
- 'geniza init', 'geniza commit' commands
+threaded sync:
+- rust stdlib async TCP?
+- event loop runner
+- channel polling
+- polling for connections (channels + async TCP)
+- UNIX signal handling
+- later: accepting inbound connections
+
specific tests:
- drive children array: deep directory nesting (like 'tree' example)
- sleep info a variety of bad sleep files, should error
@@ -62,3 +92,4 @@ Backburner:
https://github.com/ia0/data-encoding
- sendfile() or splice() for disk-to-disk copying?
- Rust FS API example: https://docs.rs/zbox/0.1.3/zbox/
+- "bytes" crate
diff --git a/notes/architecture.md b/notes/architecture.md
new file mode 100644
index 0000000..c9268c9
--- /dev/null
+++ b/notes/architecture.md
@@ -0,0 +1,56 @@
+
+peers wrap connections with given clients (any transport type). each peer can
+be talking about multiple feeds. state consists of which feeds, what the remote
+has (feed, bitmap), and what chunks (feed, index) have been requested
+("in-flight"). connection logic runs in threads, passes back and forth entire
+messages in channels.
+
+ ConnCommand
+ Send(DatNetMessage)
+ Terminate()
+
+ ConnReply
+ Received(DatNetMessage)
+ Disconnected()
+
+A synchronizer event loop has local feeds (storage) and a bunch of peers. Extra
+state includes mode configuration, feeds, "wanted" bitmap for each feed.
+Callbacks for certain feed events allow for extension (eg, hyperdrive stuff).
+Needs a timer event.
+
+Use chan and chan-signal for now (MPSC style)... tokio will be the future but
+seems too unstable for now. mio is too low-level to use on it's own (eg, no
+timers, no threads). Use two threads per peer: a "main" thread that does
+connection setup, `chan_select`, and blocking writes, and a "reader" thread
+that does blocking reads of entire messages (writing either into the main
+thread or skipping straight to the output channel). Got this pattern from the
+`rusty_torrent` package. Need to set some write timeout. Synchronizer handles
+cleanup if peer isn't replying/heartbeating.
+
+Probably want a dedicated discovery thread/worker in the future, but for now
+just do it synchronously and connect to up to N peers to start.
+
+Synchronize will select on:
+
+ signal
+ graceful shutdown
+ unified_inbound
+ feed: depends on state; ignore for now
+ handshake: error, close connection?
+ have, info: update PeerState bitmap
+ unhave, info, want, unwant: ignore for now
+ request: depends on state; read from storage
+ cancel: ignore for now
+ data: write to storage, clear pending; send new request
+ tick
+ check peer states, kill if too slow
+ dump status to command line
+
+Synchronize needs state:
+- "tried" peers, so we don't retry in a loop if they fail
+
+Synchronize options:
+- whether or not to hook signals
+
+Where does drive logic live? Plugin, via callback, to Synchronize? Let's just
+fold it in to synchronizer for now, as a mode flag.
diff --git a/notes/refs.md b/notes/refs.md
new file mode 100644
index 0000000..4995578
--- /dev/null
+++ b/notes/refs.md
@@ -0,0 +1,5 @@
+
+Decent "Original" Bittorrent client description:
+
+https://github.com/mgp/coding-in-the-real-world/blob/master/manuscript/bittorrent-client-case-study.md
+