internetarchive Goto Github PK

scrapy-warcio

Support for writing WARC files with Scrapy

scribe-redux

snakebite-py3

Pure python HDFS client: python3.x version

snowpack-files-hash

solr-ia

sparkling

Internet Archive's Sparkling Data Processing Library

strainer

Heritrix frontier files manipulation tool.

surt

Sort-friendly URI Reordering Transform (SURT) python module

tarb_gsoc23_content_drift

Content drift assessment tool for TARB project.

tarb_gsoc23_soft404

Soft-404 detction system for the TARB project.

tarb_insights

A Streamlit application to visualize Wikipedia IABot statistics

test

GitHub Actions test for CI/CD pipelines -- see full CI/CD at https://github.com/internetarchive/cicd

tnh

(T)he (N)ew (H)otness. Improved full-txt search of archival web data.

trendmachine

A mathematical model to calculate a normalized score to quantify the temporal resilience of a web page as a time-series data based on the historical observations of the page in web archives.

trough

Trough: Big data, small databases.

tvnews

ultrajson

Ultra fast JSON decoder and encoder written in C with Python bindings

umbra

A queue-controlled browser automation tool for improving web crawl quality

waimea

Full-text indexing pipeline based on Hadoop/Pig scripts.

warc

Python library for reading and writing warc files

warcprox

WARC writing MITM HTTP/S proxy

warctools

Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)

wayback

IA's public Wayback Machine (moved from SourceForge)

wayback-diff

React components to render differences between captures at the Wayback Machine

wayback-discover-diff

A Python 3.6+ application that calculates and returns simhash values for Internet Archive's snapshots

wayback-machine-android

wayback-machine-edge

wayback-machine-firefox

Reduce annoying 404 pages by automatically checking for an archived copy in the Wayback Machine. Learn more about this Test Pilot experiment at https://testpilot.firefox.com/

wayback-machine-safari

wayback-machine-versions