1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
Status: Undefined (as of 2018-01-XXX)
Type: Standard
Github PR: (add HTTPS link here after PR is opened)
Short Name: `0000-hyperdb`
# Summary
[summary]: #summary
HyperDB is an iteration on the hyperdrive directory tree implementation,
providing a general purpose key/value store on top of the hypercore append-only
log abstraction layer. Keys are path-like strings (eg, `/food/fruit/kiwi`), and
values are arbitrary binary blobs (with expected size of under a megabyte).
hyperdrive is expected to be re-implemented on top of HyperDB; the hyperdrive
API should be largely unchanged, but the metadata register format will be
backwards-incompatible.
# Motivation
[motivation]: #motivation
HyperDB is expected to drastically improve performance of dat clients when
working with archives containing tens of thousands of files in single
directories. This is a real-world bottleneck for several current users, with
basic local actions such as adding a directory taking an unacceptably long time
to complete.
A secondary benefit is to refactor the tree-structured key/value API out of
hyperdrive, allowing third party code to build on this abstraction layer.
# Usage Documentation
[usage-documentation]: #usage-documentation
TODO:
Document new features or processes as if this proposal has already been accepted and implemented. This section should introduce new concepts and terms, describe use cases or situations, and provide examples. If a user-facing feature is added, this should also include user-accessible documentation.
# Reference Documentation
[reference-documentation]: #reference-documentation
TODO: for now see [ARCHITECTURE.md][arch_md].
This section is a more formal description of the proposal, written as if it was a sub-section of the standard (for techincal proposals) or a formal process or "fine print" for process proposals.
It should be unambiguous and cover all known corner-cases. Formal language (such as protobuf schemas or binary header diagrams) are appropriate here.
# Drawbacks
[drawbacks]: #drawbacks
TODO:
Why should we *not* do this?
# Rationale and alternatives
[alternatives]: #alternatives
TODO:
- Why is this design the best in the space of possible designs?
- What other designs have been considered and what is the rationale for not choosing them?
- What is the impact of not doing this?
# Unresolved questions
[unresolved]: #unresolved-questions
TODO:
- What parts of the design do you expect to resolve through the RFC consensus process before this gets merged?
- What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers?
- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?
# Migration logistics
[migration]: #migration
HyperDB is not backwards compatible with the existing hyperdrive
implementation, meaning dat clients will need to support multiple on-disk
representations during a transition period.
a new abstraction layer between hypercore (replicated append-only
logs) and hyperdrive (versioned file system abstraction). HyperDB provides an
efficient key/value database API, with path-like strings as keys and arbitrary
binary data (up to a reasonable chunk size) as values. HyperDB will require
breaking changes to dat clients, but will not require changes to the network
wire protocol.
# Changelog
[changelog]: #changelog
As of January 2018, @mafintosh is leading development of a hyperdb nodejs
module on [github](https://github.com/mafintosh/hyperdb), which is the basis
for this RFC.
- 2017-12-06: @noffle publishes `ARCHITECTURE.md` overview in the
[hyperdb github repo][arch_md]
- 2018-01-XXX: First complete draft submitted for review
[arch_md]: https://github.com/mafintosh/hyperdb/blob/master/ARCHITECTURE.md
|
