pde / tosback2 Goto Github PK
View Code? Open in Web Editor NEWReimplementing TOSBack using git as a database layer!
Home Page: http://tosback.org
License: GNU General Public License v2.0
Reimplementing TOSBack using git as a database layer!
Home Page: http://tosback.org
License: GNU General Public License v2.0
This is TOSBack version 2, a clean redesign & reimplementation of EFF's TOSBack project. It uses Git as an inherently and efficiently versioned backend storage database. After cloning the git repository, you need to execute this command: git submodule update --init --recursive That will fetch a recent version of the GitPython code, which we depend upon. *BUGS IN WGET* If you want to actually run the crawler yourself (not really necessary unless you're testing something), be aware that TOSBack2 also exposes a number of bugs in common versions of wget. As of December 2011, there are two bugs you might need to patch yourself! (FOR YOUR CONVENIENCE, a patched version of the wget source can be found in lib/wget-1.13.4/ . There is also a binary .deb that Debian and Ubuntu users can try in lib/. More hints on building from source below) 1. Versions of wget built against gnutls may suffer from fatal memory leaks https://lists.gnu.org/archive/html/bug-wget/2011-10/msg00050.html (so apply that patch, or build against openssl using ./configure --with-ssl=openssl). 2. You should also apply the following patch https://savannah.gnu.org/support/download.php?file_id=24473 to fix this bug: https://savannah.gnu.org/bugs/?21714 HINTS FOR BUILDING WGET FROM SOURCE ON DEBIAN OR UBUNTU sudo apt-get build-dep wget cd lib/wget-1.13.4/ fakeroot debian/rules binary an installable .deb file *should* be written to the lib/ directory
hi! as we know, tosback.org has not been updated for a couple of years now, and it looks like the site is now inaccessible altogether (there's an "OMG KITTENS" acl on there). but not to worry! :) as promised, we (the ToS;DR project) got tosback's back and will fully back its comeback! ;)
@JimmStout has been working hard to put tosback.org back online, and is nearly finished with that (staging URL: http://tosback-staging.michielbdejong.com/ ). some things that are missing:
There is https://github.com/pde/tosback2-data/blob/data/crawls/facebook.com/Privacy-Policy/raw/www.facebook.com/about/privacy/index.html but all the actual content, linked from this page, like ../../about/privacy/your-info.html are not archived.
A lot of websites maintains different set of pages and sometimes the links they display for their privacy policy and terms of service are redirection to other pages. For instance, Skype's main privacy policy is at http://www.skype.com/intl/en-us/legal/privacy/general/ But this page will typically be accessed from the footer with the link to http://www.skype.com/go/privacy/
For our purpose of tracking changes and archiving. Which links should we use?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.