ysf / anewer

anewer appends lines from stdin to a file if they don't already exist in the file. This is a rust version of https://github.com/tomnomnom/anew

anewer

anewer appends lines from stdin to a file if they don't already exist in the file. You could also use it as uniq without sort. This is a rust version of tomnomnom/anew. It makes use of tkaitchuck/aHash to cut down runtime to ~50%. Since only hashed lines are held in memory, it cuts down memory usage for inputs with long lines. Which is similar how huniq works.

Usage

$ anewer -h
USAGE:
    anewer [FLAGS] [filename]

FLAGS:
    -n, --dry-run    Dry run, will leave the file as it is
    -h, --help       Prints help information
    -v, --invert     Invert the sense of matching
    -q, --quiet      Quiet, won't print to stdout
    -V, --version    Prints version information

ARGS:
    <filename>

Installation

cargo install anewer

Add unknown elements of newthings.txt to things.txt

$ cat things.txt
Zero
One
Two

$ cat newthings.txt
One
Two
Three
Four

$ cat newthings.txt | anewer things.txt
Three
Four

$ cat things.txt
Zero
One
Two
Three
Four

Or use it as simple uniq without sort

$ cat list.txt
One
One
Two
Two
Three
Four
Three
Four

$ cat list.txt | anewer
One
Two
Three
Four

License

GPLv3+

anewer's Issues

Add option to open [filename] with O_CREAT

by default, anew opens file with O_CREAT. despite liking anewer's behavior of failing i file doesn't exist, it would be nice to have a flag -c, --create to be able to ask anewer to create the file if it doesn't exist yet

Invert

@ysf It would be good if it have invert match functionality too which is not in anew.

Anewer fails to create and write into a new file.

cat things.txt | anewer newthings.txt

Error: Failed to open file: "newthings.txt"

Caused by:
    No such file or directory (os error 2)

[FEATURE] Implement flock(2) advisory locks on linux

Hi ! After posting this tweet, @sn0int mentioned anewer tool, which is an awesome, modern replacement for anew.

My feature request is about supporting advisory locks flock(2) on linux. Indeed, when 2 instances of anew/anewer try to write to the same file, both may end up using an outdated hashmap, and then allowing to add duplicate results to the file.

Implementing flock on the opened file would prevent this.

I understand that it may sound overkill, and i don't know if it's easy to implement in rust

Easily allow disregarding timestamps, etc.

Of course that can be done with filters before, but it would be easier to be able to specify an option to e.g., ignore changed numbers or timestamp-like data.

I assume simply regexp-changing the string to hash will allow that. It would only need to find nice regular expressions for some common filters.