Topic: deduplication Goto Github

👇 Here are 389 public repositories matching this topic...

alephdata / fingerprints
deduplication,Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Organization: alephdata
company-data names fingerprint entity clustering nlp deduplication

arsenetar / dupeguru
deduplication,Find duplicate files
User: arsenetar
Home Page: https://dupeguru.voltaicideas.net
python deduplication

borgbackup / borg
deduplication,Deduplicating archiver with compression and authenticated encryption.
Organization: borgbackup
Home Page: https://www.borgbackup.org/
python python-3 compression dedupe ssh deduplication backup borgbackup encryption cython

borgmatic-collective / borgmatic
deduplication,Simple, configuration-driven backup software for servers and workstations
Organization: borgmatic-collective
Home Page: https://torsion.org/borgmatic/
python backup borg borgbackup servers deduplication compression borgbase cronhub cronitor

cargo-limit / cargo-limit
deduplication,Productivity improvements for Rust ecosystem: warnings are skipped until errors are fixed, LSP-independent Neovim integration, etc.
Organization: cargo-limit
rust wrapper filter limit cargo cargo-wrapper deduplication crates neovim nvim

chinalu / imagedups
deduplication,图片查重、图片去重、Find/Delete duplicated images
User: chinalu
duplicate duplicated-files deduplication fdupes

cupcakearmy / autorestic
deduplication,Config driven, easy backup cli for restic.
User: cupcakearmy
Home Page: https://autorestic.vercel.app/
restic backup cli config pruning incremental-backup incremental deduplication config-driven

deajan / backup-bench
deduplication,Quick and dirty backup tool benchmark with reproducible results
User: deajan
backup benchmark benchmarking borgbackup compression deduplication duplicacy kopia restic bupstash

dm-vdo / kvdo
deduplication,A kernel module which provide a pool of deduplicated and/or compressed block storage.
Organization: dm-vdo
kernel-modules storage vdo linux-kernel deduplication compression

dm-vdo / vdo
deduplication,Userspace tools for managing VDO volumes.
Organization: dm-vdo
vdo storage deduplication compression

dpc / rdedup
deduplication,Data deduplication engine, supporting optional compression and public key encryption.
User: dpc
data-deduplication backup encryption deduplication

elemental-lf / benji
deduplication,Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices
User: elemental-lf
Home Page: https://benji-backup.me
backup deduplication ceph lvm block-based s3 b2 kubernetes iscsi

f483 / dejavu
deduplication,Quickly detect already witnessed data.
User: f483
Home Page: https://f483.github.io
golang go history memory deduplication duplicate-values duplicates command-line command-line-tool probabilistic

fake-name / intraarchivededuplicator
deduplication,Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
User: fake-name
postgresql image-search deduplication bk-tree python cython

fcorbelli / zpaqfranz
deduplication,Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
User: fcorbelli
zpaq compression deduplication backup solaris

fritshermans / deduplipy
deduplication,Python package for deduplication/entity resolution using active learning
User: fritshermans
Home Page: https://www.deduplipy.com
deduplication entity-resolution fuzzy-matching record-linkage

ghatage / horcrux
deduplication,The Dropbox for IPFS (without the icky stuff)
User: ghatage
ipfs file-sharing file-manager deduplication

j535d165 / data-matching-software
deduplication,A list of free data matching and record linkage software.
User: j535d165
data-matching record-linkage entity-resolution deduplication fuzzy-matching software awesome awesome-list machine-learning open-source

j535d165 / recordlinkage
deduplication,A powerful and modular toolkit for record linkage and duplicate detection in Python
User: j535d165
Home Page: http://recordlinkage.readthedocs.io/
record-linkage entity-resolution dedupe string-distance machine-learning privacy python python-library data-matching deduplication

jaskey / rocketmqdeduplistener
deduplication,RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用
User: jaskey
rocketmq rocketmq-client deduplication

jvirkki / dupd
deduplication,CLI utility to find duplicate files
User: jvirkki
Home Page: http://www.virkki.com/dupd
duplicates duplicatefilefinder c duplicate-files deduplication fdupes

kdeldycke / mail-deduplicate
deduplication,📧 CLI to deduplicate mails from mail boxes.
User: kdeldycke
Home Page: https://kdeldycke.github.io/mail-deduplicate
python mail dedupe cli maildir mailbox mbox deduplication cleanup babyl

kopia / kopia
deduplication,Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Organization: kopia
Home Page: https://kopia.io
deduplication backup google-cloud-storage encryption cloud hacktoberfest

lakshmipathi / dduper
deduplication,Fast block-level out-of-band BTRFS deduplication tool.
User: lakshmipathi
btrfs btrfs-progs btrfs-tools dedupe deduplication

lostatc / acid-store
deduplication,[UNMAINTAINED] A transactional and deduplicating virtual file system
User: lostatc
deduplication encryption acid redis s3 sqlite filesystem sftp rclone fuse

markusressel / py-image-dedup
deduplication,CLI utility to find near duplicate images and remove all but the best copy.
User: markusressel
python python3 python-3 image-analysis duplicate-detection duplicate-images deduplication dedup find-duplicates image-comparison

mattilyra / lsh
deduplication,Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
User: mattilyra
cython deduplication duplicate-documents lsh minhash

mhx / dwarfs
deduplication,A fast high compression read-only file system for Linux, Windows and macOS
User: mhx
gpl-license filesystem squashfs compression fuse-filesystem cpp lzma zstd lrzip zpaq

moj-analytical-services / splink
deduplication,Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
Organization: moj-analytical-services
Home Page: https://moj-analytical-services.github.io/splink/
record-linkage spark em-algorithm deduplication deduplicate-data entity-resolution data-matching fuzzy-matching data-science duckdb

netinvent / npbackup
deduplication,A secure and efficient file backup solution that fits both system administrators (CLI) and end users (GUI)
Organization: netinvent
backup cli deduplication gui restic vss compression healthcheck monitoring prometheus-metrics

nlfiedler / fastcdc-rs
deduplication,FastCDC implementation in Rust
User: nlfiedler
Home Page: https://crates.io/crates/fastcdc
rust deduplication chunking-algorithm

opengene / gencore
deduplication,Generate duplex/single consensus reads to reduce sequencing noises and remove duplications
Organization: opengene
sequencing ngs bioinformatics sequencing-error consensus deep-sequencing sequencing-noise duplication duplex deduplication

opensanctions / nomenklatura
deduplication,Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Organization: opensanctions
data-integration deduplication record-link

openvenues / libpostal
deduplication,A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Organization: openvenues
address-parser machine-learning nlp address international c deduplication record-linkage deduping natural-language-processing

openvenues / lieu
deduplication,Dedupe/batch geocode addresses and venues around the world with libpostal
Organization: openvenues
deduplication geocoding address venues international

prometheus / alertmanager
deduplication,Prometheus Alertmanager
Organization: prometheus
Home Page: https://prometheus.io
alertmanager deduplication email hacktoberfest monitoring notifications opsgenie pagerduty slack

restic / restic
deduplication,Fast, secure, efficient backup program
Organization: restic
Home Page: https://restic.net
go restic backup deduplication dedupe secure-by-default

rin-nas / postgresql-patterns-library
deduplication,Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)
User: rin-nas
sql postgresql recipes phone email validation parse syntax check function

ropeladder / record-linkage-resources
deduplication,Resources for tackling record linkage / deduplication / data matching problems
User: ropeladder
record-linkage deduplication data-matching entity-resolution python java javascript

rustic-rs / rustic
deduplication,rustic - fast, encrypted, and deduplicated backups powered by Rust
Organization: rustic-rs
Home Page: https://rustic.cli.rs
backup deduplication encryption rust restic

sahib / rmlint
deduplication,Extremely fast tool to remove duplicates and other lint from your filesystem
User: sahib
Home Page: http://rmlint.rtfd.org
c python filesystem lint duplicates fdupes deduplication

sreedevk / deduplicator
deduplication,Filter, Sort & Delete Duplicate Files Recursively
User: sreedevk
deduplication duplicate-detection duplicate-files duplicatefilefinder filesystem rust

tsileo / blobstash
deduplication,You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
User: tsileo
backup blob-store blobstash content-addressed deduplication document-store go storage

unreadablewxy / fs-curator
deduplication,Automation for the serious data hoarder that wants to have their data and use it
User: unreadablewxy
file-sorting file-renamer hard-links organizer directory-tree deduplication

usc-isi-i2 / rltk
deduplication,Record Linkage ToolKit (Find and link entities)
Organization: usc-isi-i2
linkage similarity similarity-metric string-similarity record-linkage entity-resolution deduplication

vintasoftware / entity-embed
deduplication,PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Organization: vintasoftware
Home Page: https://entity-embed.readthedocs.io/en/latest/
entity-resolution record-linkage representation-learning embeddings approximate-nearest-neighbors deduplication entity-matching data-matching deep-learning python

yomguithereal / talisman
deduplication,Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
User: yomguithereal
Home Page: https://yomguithereal.github.io/talisman/
natural-language-processing machine-learning fuzzy-matching clustering record-linkage information-retrieval deduplication

yornaath / batshit
deduplication,A batch manager that will deduplicate and batch requests for a certain data type made within a window. Useful to batch requests made from multiple react components that uses react-query
User: yornaath
Home Page: https://batshit-example.vercel.app/
async batch-processing concurrency deduplication fetch react-query typescript react tanstack

zinggai / zingg
deduplication,Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Organization: zinggai
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution

zouzias / spark-lucenerdd
deduplication,Spark RDD with Lucene's query and entity linkage capabilities
User: zouzias
spark lucene rdd spatial-search record-linkage deduplication entity-linking linkage hacktoberfest