bad-hashish: a tool for recursively, remotely multi-hashing files "recursively" meaning that files inside archives (.zip, .tar.gz) are hashed without extracting everything to disk. "remotely" meaning that large remote (HTTP/HTTPS) files can be hashed in a streaming fashion without saving to disk. "multi-" meaning that mulitple hash algorithms are computed in a single pass. There are other ways to do most of these; in un-UNIX-y fashion (for now) this tool does them all together. ## Planned Features - sha1, sha256, sha512, md5, blake2b - support base64, base32, hex (upper/lower), etc - can recurse on .tar and .zip (and more?) without hitting disk - can stream files via HTTP(S) without hitting disk - variable output (json, tsv, etc) Someday? - dat, ipfs, zsync index computations - simhash/minhash/etc, for plain text https://github.com/bartolsthoorn/simhash-rs - support piping out to arbitary other commands (eg, for pdf extraction simhash, image hash...) https://github.com/abonander/img_hash ## Planned Libraries rust: - zip - tar + flate2 - tree_magic - rust-crypto - crc - clap - error-chain - reqwest - log (or slog?) - rayon (for parallelization?) - something json - csv (xsv?) - data-encoding ## Minimum Viable Version Parse arguments as local files or URLs. Either way, start reading/streaming data and hand off pipe to a thing that consumes 4MB chunks at a time and hashes. Next, add parallelization (rayon?) for hashes. Output as space-separated (default), csv, or json, one line per file. Examples: hashish some_file.txt cat zip_urls.txt | parallel -j8 hashish --recurse-only {} > all_hashes.txt Arguments: - chunk size - recurse into files or not - output format - cores to use? ## Later Thoughts Limited by {CPU, disk, network}? Where to parallelize? Data locality.