ribasushi / dagger

🗡️ stream-DAGger

an elegant slicer for a more civilized age

ALPHA - REPOSITORY WILL MOVE BEFORE MAY 1ST 2020

stream-dagger is a command line tool designed to:

• receive a single STDIN input stream
• efficiently split it based on a wide array of tunable criteria
• aggregate and emit detailed statistics about the resulting chunks
• possibly form a Directed Acyclic Graph (DAG) from the individual stream parts
• possibly stream the DAG on STDOUT in format understood by e.g. ipfs

Further documentation for individual use cases will be available in several days (ipfs-shipyard/DAGger#5). Until then some basic example snippets:

go get -v -u github.com/ribasushi/DAGger/cmd/stream-dagger

cat {{somefile}} | stream-dagger --ipfs-add-compatible-command="--cid-version=1"

find {{somedirectory}} -type f -exec sh -c '
perl -e "print( pack( q{q>}, -s \$ARGV[0]))" {} && cat {}
' \; | stream-dagger --multipart --ipfs-add-compatible-command="--cid-version=1"

Lead Maintainer

Peter 'ribasushi' Rabbitson

License

SPDX-License-Identifier: Apache-2.0 OR MIT

dagger's Issues

panic: runtime error: index out of range [-46]

Following panic occurs when running this:

< temp-2.txt go run ./cmd/stream-dagger --ipfs-add-compatible-command='--cid-version=1 --chunker=rabin' --chunkers=rabin_polynomial=17437180132763653_window-size=64_state-target=0_state-mask-bits=8_min-size=18_max-size=262144 --emit-stdout=chunks-jsonl | jq .
panic: runtime error: index out of range [-46]

On a following file
temp-2.txt