1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
Title: **DEP-0000: Hyperdrive File Hashes**
Short Name: `0000-hyperdrive-hashes`
Type: Standard
Status: Undefined (as of YYYY-MM-DD)
Github PR: [Discussion](https://github.com/datprotocol/DEPs/pull/12)
Authors: [Bryan Newbold](https://github.com/bnewbold)
# Summary
[summary]: #summary
Full-file hashes are optionally included in hyperdrive metadata to complement
the existing cryptographic-strength hashing of sub-file chunks. Multiple
popular hash algorithms can be included at the same time.
# Motivation
[motivation]: #motivation
Naming, discovering, and cataloging data "by content" (aka, by a fixed-size
hashes of the data) is a powerful pattern for robust distributed systems. Dat
is one among several such systems. Unfortunately, interoperability between or
layering such systems on top of each other is difficult because each tends to
adopt it's own hashing norms and formats. Design variances can include hash
algorithm selection, hash configuration, salting, data chunking, and
intermediate Merkle tree data formats.
As a concrete example, the sha1sum command-line tool, the bittorrent P2P
protocol and the git code versioning software both use the SHA-1 algorithm to
hash file contents. However, one can not use the simple `sha1sum` hash of a
given file to check whether that file is the same as referenced in either a
bittorrent `.torrent` file or from git metadata, because each calculate the
hash in different ways. Bittorrent combines all files in the torrent into a
single stream, then splits into a fixed number of chunks and hashes those
separately; the chunk boundaries usually do not correspond to individual files.
git prepends the size of the file (in bytes) as a fixed header before hashing
and storing the file as a "blob". This makes comparison or interoperability
between these systems impossible without having either a universal cross-hash
table (infeasible to build in the general sense) or without having the full
file contents on-hand to compare or re-hash in all three formats.
The design decisions to adopt hash variants are usually well-founded, motivated
by security concerns (such as pre-image attacks), efficiency, and
implementation concerns.
By adding simple full-file hashes of files as optional complementary metadata
in our distributed data systems, we can make interoperability and powerful
efficiency gains possible.
For example, a large collection of files could be stored in a simple format on
disk, indexed by a popular hash format. Gateway clients to several P2P networks
could make the same files accessible by storing metadata (relatively small)
separately for each network, but accessing the file contents from the shared
store by a common hash.
In the case of Dat, a particular efficiency of this use case would be enabling
fast de-duplication of file storage between multiple Dat archives on a
full-file level, instead of at the chunk-level (which would be sensitive to
changes in chunking algorithm).
# Usage Documentation
[usage-documentation]: #usage-documentation
Implementations would include hashes as file-level metadata along with existing
"stat" fields.
Existing API methods would include options to control generation of hashes (and
which types) when creating a new drive or adding files.
# Reference Documentation
[reference-documentation]: #reference-documentation
Hashes would be stored as additional fields in hyperdrive's existing `Stat`
protobuf message, with the following structure:
```protobuf
message Stat {
message ExtraHash {
required uint32 type = 1;
required bytes value = 2;
}
required uint32 mode = 1;
optional uint32 uid = 2;
optional uint32 gid = 3;
optional uint64 size = 4;
optional uint64 blocks = 5;
optional uint64 offset = 6;
optional uint64 byteOffset = 7;
optional uint64 mtime = 8;
optional uint64 ctime = 9;
repeated ExtraHash hashes = 10;
}
```
`type` is a number representing the hash algorithm, and `value` is the
bytestring of the hash output itself. The length of the hash digest (in bytes)
is available from protobuf metadata for the value. This scheme, and the `type`
value table, is intended to be interoperable with the [multihash][multihash]
scheme from the IPFS community.
A subset of the multihash hash digest table includes:
```
md5 0x00D5
sha1 0x0011
sha2-256 0x0012
sha2-512 0x0013
blake2b-256 0xB220
```
Multiple hashes would be calculated in parallel with the existing
chunking/hashing process, in a streaming fashion. Final hashes would be
calculated when the chunking is complete, and included in the `Stat` metadata.
For 2018, recommended default full-file hash functions to include are `SHA1`
(for popularity and interoperability) and `blake2b-256` (already used in other
parts of the Dat protocol stack).
[multihash]: https://multiformats.io/multihash/
# Drawbacks
[drawbacks]: #drawbacks
The metadata storage overhead (on a per-file basis) should be minimal, but the
additional computational resources to hash a large file multiple times are
non-trivial on machines with a single (or few) cores, even when computed in a
parallel/streaming format.
# Security and Privacy Concerns
[privacy]: #privacy
Additional optional fields may leak additional bits of user-specific
configuration metadata, analogous to the "[evercookie][]" and
"[panopticlick][]" browser fingerprinting issues.
[evercookie]: https://en.wikipedia.org/wiki/Evercookie
[panopticlick]: https://panopticlick.eff.org/
# Rationale and alternatives
[alternatives]: #alternatives
Users wanting this metadata could instead maintain a manifest file (mapping
paths to hashes) inside the Dat archive itself. The Dat client could support
this with a special mode or flag. One downside of this is that for large
archives, the file would need to be updated and duplicated for every new or
modified file.
# Unresolved questions
[unresolved]: #unresolved-questions
What does the user-facing API look like, specifically?
Should we allow non-standard hashes, like the git "hash", or higher-level
references like (single-file) bittorrent magnet links or IPFS file references?
Modifying a small part of a large file would require re-hashing the entire
file, which is slow. Should we skip including the updated hashes in this case?
Currently mitigated by the fact that we duplicate the entire file when recoding
changes or additions.
# Changelog
[changelog]: #changelog
- 2018-03-17: First draft for comment.
|
