opentermsarchive / engine Goto Github PK
View Code? Open in Web Editor NEWTracks contractual documents and exposes changes to the terms of online services.
Home Page: https://opentermsarchive.org
License: European Union Public License 1.2
Tracks contractual documents and exposes changes to the terms of online services.
Home Page: https://opentermsarchive.org
License: European Union Public License 1.2
Exécuter la récupération et l'enregistrement des changements à intervalle régulier et arbitraire.
Commençons par un intervalle à 60 minutes.
In OpenTermsArchive/contrib-versions@49832cdfa, Facebook removes a lot of content from its Commercial Terms towards a new “Data Transfer Addendum” that should be tracked.
The current sanitizers for Facebook's Developer Policy do not avoid the change of subdomain in URLs.
Un fichier JSON de description par fournisseur de service, dans le dossier providers
à la racine.
{
"serviceProviderName": "Facebook",
"langs": [ "en", "fr" ],
"langStrategy": "accept-header",
"documents": {
"tos": {
"url": "https://…",
"contentSelector": ".UI_selector",
"noiseSelectors": [
"script",
"style",
".csrf_token",
"#ad_unique_id"
],
"sanitizationPipeline": [
"convert-emojis",
"markdown"
]
}
}
}
langs
est-il à la racine ou dans chaque entrée de document ?noiseSelectors
et sanitizationPipeline
sont-ils une extension d'une base ou redéfinis intégralement ?langs
est à la racine, on verra si des fournisseurs font varier les langues par document.serviceProviderName
documents
url
contentSelector
For example, Vimeo renders its content with a script.
See https://github.com/ambanum/CGUs-versions/blob/master/Vimeo/Privacy%20Policy.md
Créer un commit pour chaque nouvelle version du HTML récupéré.
Hi,
A question has just surfaced about the robots.txt
files and whether CGUs project should honor them / keep track of the robots.txt
status.
So far, the following script does the job of testing whether the URLs can be crawled by a robot (called "CGUs Bot"):
import json
import os
import requests
import urllib.robotparser
from urllib.parse import urljoin
count = 0
for file in sorted(os.listdir('services')):
if not file.endswith('.json'):
continue
with open(os.path.join('services', file), 'r') as fh:
service = json.load(fh)
for document in service['documents']:
count += 1
document_url = service['documents'][document]['fetch']
req = requests.head(document_url)
if req.status_code == 301:
print('WARNING: Terms URL is a redirection: %s to %s.' % (document_url, req.headers['Location']))
robots_url = urljoin(req.headers['Location'], '/robots.txt')
else:
robots_url = urljoin(document_url, '/robots.txt')
req = requests.get(robots_url, headers={'User-Agent': 'CGUs Bot'})
if req.status_code != 200:
print('WARNING: Status Code %s for %s.' % (req.status_code, robots_url))
continue
rp = urllib.robotparser.RobotFileParser()
rp.parse(req.text.split())
if not rp.entries:
print('FAILED TO PARSE robots.txt for %s.' % robots_url)
if not rp.can_fetch('CGUs Bot', document_url):
print(document_url)
print('Scanned a total of %d terms URLs.' % count)
No issue detected at this time with the already existing services files.
Best,
We need to be able to regenerate versions from snapshots. As documents is aim to change over time (location or filters) we can't rely on the last version of the declaration to regenerate the version from an old snapshot. So we need a system to keep track of declaration changes, that's what we called declarations and filters versioning.
At this time, we see three solutions which have in common the following rules:
history
is optionalvalid_until
date is an inclusive expiration date. It should be the exact authored date of the last snapshot commit for which the declaration is still valid.history
field in service declarationIn services/ASKfm.json
:
{
"name": "ASKfm",
"documents": {
"Terms of Service": {
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": ".selection",
"filter": [ "add" ]
"history": [
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"filter": [ "add" ]
"valid_until": "2020-08-24T14:02:39Z"
},
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"valid_until": "2020-08-23T14:02:39Z"
}
]
}
}
}
Note: When no historisation is needed the file may have no mention of history.
Pros:
Cons:
serviceId.history.json
fileIn services/ASKfm.json
:
{
"name": "ASKfm",
"documents": {
"Terms of Service": {
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": ".selection",
"filter": [ "add" ]
}
}
}
In services/ASKfm.history.json
:
{
"name": "ASKfm",
"documents": {
"Terms of Service": [
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"filter": [ "add" ]
"valid_until": "2020-08-24T14:02:39Z"
},
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"valid_until": "2020-08-23T14:02:39Z"
}
]
}
}
Pros:
Cons:
services/history
folderIn services/ASKfm.json
:
{
"name": "ASKfm",
"documents": {
"Terms of Service": {
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": ".selection",
"filter": [ "add" ]
}
}
}
In services/history/ASKfm.json
:
{
"name": "ASKfm",
"documents": {
"Terms of Service": [
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"filter": [ "add" ]
"valid_until": "2020-08-24T14:02:39Z"
},
{
"fetch": "https://ask.fm/docs/terms_of_use/?lang=en",
"select": "body",
"valid_until": "2020-08-23T14:02:39Z"
}
]
}
}
Pros:
Cons:
The choice might have implication on the community that will grow around the project.
Option 1 shows everything to everyone, it might frightened some contributors with some apparent complexity (once there are history in the declaration file), but it might also encourage them to learn about it if they want or feel the need to. All contributors will share the same view and knowledge about the system. This might encourage collaboration between them to learn and improve together.
Option 2 and Option 3 hide the complexity of history management in separate files and only most adventurous contributors will find them by themselves. Contribution to those files will probably be done by specific contributors that will be taught to manage those file. Thus creating two different kind of contributors: those who will stay with the basic service declaration, not knowing that more complex options exist, and those who will have the knowledge of history management whose work might stay in the shadow or work as gatekeeper.
Reddit Privacy Policy is in a table format. The content in the table is not parsed correctly (e.g.
tags remain).
TikTok has a ToS and Privacy policy by jurisdiction consolidated in the same document, targetable with an ID selector (#terms-us
, #terms-eea
…). It could be good to split it by jurisdiction.
As described in #71 (comment)
Hi,
Just to keep track, I found this page https://policies.google.com/terms/service-specific?hl=en-GB listing all the terms of services which are applicable for Google services as well as the extra terms of services for specific services. Might be worth double-checking we are tracking all of them.
Best,
Hi,
Rakuten and Leboncoin have very strong bot detectors, hence preventing from automatically fetching their CGUs (at least on a regular OVH machine). See https://fr.shopping.rakuten.com/newhelp/conditions-generales/ or https://www.leboncoin.fr/dc/cgu. It is possible that #138 and having JS enabled will help here, but I think this won't be enough.
Best,
EDIT: Same for RueDuCommerce (see https://www.rueducommerce.fr/info/mentions-legales/cgv) or FNAC (https://www.fnac.com/Help/cgv-fnac#bl=footer), they all use the same system, powered by Datadome.
As discussed this morning with @michielbdejong and @Ndpnt: import the history of snapshots from https://github.com/tosdr/tosback2/tree/master/crawl.
For each document of each service declaration present after #58:
<html><body>
to transform the HTML fragment into an HTML document parseable by JSdom.This snapshot was imported from https://github.com/tosdr/tosback2/blob/5acac7abb5e967cfafd124a5e275f98f6ecd423e/crawl/4shared.com
.Obtenir les CGUs de Facebook dans leur zone de juridiction (donc en anglais).
On peut accéder aux CGU dans une langue particulière en changeant le sous-domaine : https://en-gb.facebook.com/terms.php
Une version imprimable existe, qui sera peut-être plus facile à parser : https://www.facebook.com/legal/terms/plain_text_terms
Pour cette version imprimable, on peut obtenir la version anglaise avec le header Accept-Language: en
.
https://parler.com/ seems to be more and more used, might be a good option to follow their CGU
Hi!
We're now running this code (without scheduler) on https://tosback.org (went live today!) and one thing we had to change was to add a timeout, so the process exits after one hour.
Did you have a similar experience or does npm start
neatly exit after it reaches the last service in the alphabet?
As discussed this morning with @michielbdejong and @Ndpnt:
Add a validation step in scripts/validation/validate.js
to check that the resulting string after filtering a document is at least n
characters long, where n
is either arbitrary or computed once as a fraction of the minimum size of all currently tracked documents.
Pour les fournisseurs déjà inscrits :
Voir #12 pour la forme du manifeste.
We might add Onavo and CrowdTangle services.
Reporting to https://faq.whatsapp.com/general/security-and-privacy/the-facebook-companies
Add ESlint to ease collaboration, importing code style preferences from existing @ambanum repositories.
Hi,
While looking at app stores terms and developer policies, I came across the ones from Apple App Store, which are only available as PDF. See https://developer.apple.com/terms/apple-developer-agreement/Apple-Developer-Agreement-French.pdf.
Not sure how common this is and whether it might be worth developing ad-hoc code to handle PDF files?
I'm thinking of basically running pdftotext
/ pdftohtml
from popler-utils
or something more elaborate such as https://www.npmjs.com/package/pdf2html. This should be a bit carefully designed to integrate nicely in the current CD setup.
Best,
In order to go from 50 to 5,000 services, we need to have and understand our system for picking a canonical service name.
If you look at the filenames in https://github.com/ambanum/CGUs/tree/master/services most of the names are spelt and capitalized:
Examples:
ask.fm (1) Ask.fm (2) Ask.FM (5) AskFM
https://www.apple.com/ios/app-store/ (1) App Store (3) Apple App Store (5) AppleAppStore
Facebook Payments Inc. (4) Facebook Payments (5) FacebookPayments
Foursquare City Guide (4) Foursquare
reddit.com (1) reddit (2) Reddit
Exceptions (fix these?):
This statement should be read together with our related Privacy Notices, in particular our Privacy Notice for the Public Network, Privacy Notice for Stack Overflow Teams Basic, Privacy Notice for Stack Overflow Teams Business, Privacy Notice for Stack Overflow Teams Enterprise, Privacy Notice for Stack Overflow Talent and Jobs and our Employee Privacy Notice.
If you interact with us through the Public Network In addition, we will collect and process your personal information in accordance with the Stack Overflow Privacy Notice for the Public Network. If you are a Stack Overflow for Teams, Basic customer In addition, we will collect and process your personal information in accordance with the Stack Overflow for Teams Basic Privacy Notice. If you are a Stack Overflow for Teams, Business customer In addition, we will collect and process your personal information in accordance with the Stack Overflow for Teams, Business Privacy Notice and any other agreement that we may have with you. If you are a Stack Overflow Talent customer This statement should be read together with our related Privacy Notices, in particular our Stack Overflow Talent Privacy Notice and any other agreement that we may have with you. If you are a Stack Overflow for Teams, Enterprise customer In addition, we will collect and process your personal information in accordance with the Stack Overflow for Teams, Enterprise Privacy Notice and any other agreement that we may have with you. If you visit or register with any of our websites We will collect and use your personal information in accordance with this Privacy Policy and the Privacy Notice for the Public website if applicable. In particular, we may see how you use our websites and what content you interact with and for how long. This may involve the use of cookies which is explained in our Cookie Policy. If you are a supplier and you or your company provides us with goods or services We may collect your individual contact information in order to communicate with you and may use other information that we need in order to manage our account with your business. We will process such data in accordance with this Privacy Policy and any other agreement that we may have with you. If you are employed by us We will process your personal information in accordance with our Employee Privacy Notice. This will be made available to you internally.
What is the license on the data? (I see licensing on the software)
I see fetch failures for URLs like:
Could it be they fail because node-fetch doesn't follow the 301 redirects?
By default node-fetch should have { redirects: 'follow' }
in its options,
and the code doesn't seem to override that but still it's odd I'm seeing
these URLs fail. I'll dig into the code and see what I can find out.
Follow-up from #71. I think some tasks hit their 10 seconds timeout but then still keep running in the background. Script run ends with this and then hangs for a long time:
Next task (1 tasks left, running 5 in parallel)
Sologig Privacy Policy start
Sologig Privacy Policy skip
Saving services/Washingtonpost.json
Washingtonpost Privacy Policy done
Pending: [
'Zimbio - Privacy Policy - http://www.livingly.com/privacy-policy/',
'Urbanoutfitters - Privacy Policy - http://www.urbanoutfitters.com/urban/help/privacy_security.jsp',
'Zynga - Privacy Policy - http://company.zynga.com/about/privacy-center/privacy-policy#sharing-information'
]
Could not fetch http://www.urbanoutfitters.com/urban/help/privacy_security.jsp
Urbanoutfitters Privacy Policy fail
Pending: [
'Zimbio - Privacy Policy - http://www.livingly.com/privacy-policy/',
'Zynga - Privacy Policy - http://company.zynga.com/about/privacy-center/privacy-policy#sharing-information'
]
Could not filter http://www.livingly.com/privacy-policy/
Zimbio Privacy Policy fail
Pending: [
'Zynga - Privacy Policy - http://company.zynga.com/about/privacy-center/privacy-policy#sharing-information'
]
Could not fetch http://company.zynga.com/about/privacy-center/privacy-policy#sharing-information
Zynga Privacy Policy fail
Pending: []
Could not fetch http://tracking.quisma.com/policy.htm
Could not fetch http://www.providecommerce.com/privacy.aspx
Could not fetch http://www.seattletimescompany.com/notices/notice2.html
Could not filter http://www.washingtonpost.com/privacy-policy/2011/11/18/gIQASIiaiN_print.html
Currently, images in base64 are stored while they are illustrative only. They could be removed to decrease the file size.
Faut-il également séparer le dépôt des données brutes du dépôt des données nettoyées ?
Hi,
It would be good to add Proctorio. They are an online exam company with certain interesting practices (e.g. telling people what they can/can not have in their rooms, enforced via a browser extension [1].
https://proctorio.com/terms
https://proctorio.com/privacy
They are also receiving some attention so their Ts and Cs may change.
[1] https://twitter.com/ejohnson99/status/1303142452233203714
Transformer le HTML brut en Markdown.
Hi,
I would be interested in getting notifications only for a subset of services (especially with the upcoming inclusion of the tosback declarations).
I could handle it on my own, running a separate copy of CGUs, but would it this feature be interesting more broadly? Seems notification code is in https://github.com/ambanum/CGUs/blob/d979a3ac789e071933b8c2940a24b2a4e1bca8de/src/notifier/index.js but I am not very familiar with your SendInBlue setup for notifications.
Depending on your feedback, I can help writing patch / code on this :)
Best,
This link https://www.whatsapp.com/legal/terms-of-service-eea
was last updated in 2018
The one we should use seems to be this one : https://www.whatsapp.com/legal/updates/terms-of-service-eea
Hi,
When starting with an empty cgus-data
git repository and running yarn start
, I get
[Fri Jul 03 2020 12:05:40] [ERROR] [Discord-Terms of service] Error: Error: Could not commit undefined for undefined (raw version) due to error: Error: fatal: Unable to create '/home/lverney/cgus/cgus-data/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
at file:///home/lverney/cgus/src/history/persistor.js:9:10
at /home/lverney/cgus/node_modules/async/dist/async.js:1249:46
at Array.forEach (<anonymous>)
at trigger (/home/lverney/cgus/node_modules/async/dist/async.js:1249:27)
at /home/lverney/cgus/node_modules/async/dist/async.js:1314:25
at /home/lverney/cgus/node_modules/async/dist/async.js:321:20
at invokeCallback (/home/lverney/cgus/node_modules/async/dist/async.js:179:13)
at /home/lverney/cgus/node_modules/async/dist/async.js:173:13
at processTicksAndRejections (internal/process/task_queues.js:97:5)
I did not yet dig much into this issue, sounds like a concurrent access to git repository in async promises.
Best,
In order to ease the process of adding a new service, I suggest we add a new attribute to types.json
{
"Terms of Service": {
"commitment": {
"writer": "service provider",
"audience": "end user",
"object": "end user’s service usage"
},
"translations": {
"fr": [
"Conditions de service",
"Conditions générales d'utilisation"
]
}
},
"Privacy Policy": {
"commitment": {
"writer": "service provider",
"audience": "end user",
"object": "end user’s personal data"
},
"translations": {
"fr": [
"Politique de confidentialité"
]
}
},
},```
This way, contributors will easily find the right document name
What do you think?
Do we want to crawl a document like https://www.foxnews.com/closed-captioning ?
To optimize http requests, it would be good to send If-Modified-Since and/or If-None-Match headers and use ETag and Last-Modified.
Hi,
So far, we are tracking the en-GB versions of the documents. However, Brexit might affect these versions, which might no longer be aligned with European reglementation.
See for instance this move by Facebook : https://www.ft.com/content/bde9c983-bfbd-4bb8-a524-2e93facfe36b. Few days ago, a change in the Microsoft policies (OpenTermsArchive/contrib-versions@151d6e2) with no other changes than changing from british english to american english tends as well to indicate that UK terms might get aligned on US terms soon.
I understand we want to track English versions of documents as main elements, to ease comparison with foreign services and prevents translations artifacts. I suggest we move from en-GB to en-IE (Ireland) as this last one is ensured to be in English and under European law.
Best,
I tried npm start
but after a couple of hours of good work, it suddenly exited:
2020-11-19 09:24:24 error Debka — Privacy Policy FetchError: Invalid response body while trying to fetch https://www.debka.com/tac/: incorrect header check
at Gunzip.<anonymous> (/home/tosback3/tosback-crawler/node_modules/node-fetch/lib/index.js:399:12)
at Gunzip.emit (events.js:326:22)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
/home/tosback3/tosback-crawler/node_modules/async/dist/async.js:181
setImmediate$1(e => { throw e }, err);
^
FetchError: Invalid response body while trying to fetch https://www.debka.com/tac/: incorrect header check
at Gunzip.<anonymous> (/home/tosback3/tosback-crawler/node_modules/node-fetch/lib/index.js:399:12)
at Gunzip.emit (events.js:326:22)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
type: 'system',
errno: 'Z_DATA_ERROR',
code: 'Z_DATA_ERROR'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node src/index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/tosback3/.npm/_logs/2020-11-19T14_24_24_862Z-debug.log
That should never happen, right? It should just catch that error and continue with its next task?
See https://www.redditinc.com/policies/user-agreement-october-15-2020 (FYI, I've been notified by email associated with my own account for the upcoming update).
It would be really interesting to be able to follow documents split into several pages.
For exemple, look at the community guidelines of Facebook
So, the fetch
attribute could take several values
"Terms of Service": {
"fetch": {
"https://www...",
"https://www...",
"https://www...",
"https://www...",
}
}
but we might need to define a different document property
on each page
"Terms of Service": {
"fetch": {
{
"url:" "https://www...",
"select": ".target",
"remove": ".child",
"executeClientScripts": "true"
},
{
"url:" "https://www...",
"select": "#container",
"filter": "removeIdFromParent"
},
}
}
Hi,
When the request fails for some reason (in my particular case, a filtering proxy returning an unauthorized error), the error is silenced in the logs. See:
lverney@peren-app-jupyter:~/cgus$ npm start "Airbnb"
> [email protected] start /home/lverney/cgus
> node src/index.js "Airbnb"
(node:2946) ExperimentalWarning: The ESM module loader is experimental.
2020-10-09 14:57:36 info Refiltering 3 documents from 1 services…
2020-10-09 14:57:36 info Refiltered 3 documents from 1 services.
2020-10-09 14:57:36 info Start tracking changes of 3 documents from 1 services…
2020-10-09 14:57:36 info Tracked changes of 3 documents from 1 services.
lverney@peren-app-jupyter:~/cgus$
(the "hint" to notice it is the absence of explicit "Recorded version" log, but it would probably be better to explicitly have a warning log on request returning anything else than a 200 HTTP code)
Hi,
Apparently Google has two query parameters for l10n and i18n, namely hl
(for locale) and gl
(for country).
There was issues recently with rapid and unexpected switching between two geographical regions on Google terms (possibly due to IP geolocation or something like this). This seems solved at the moment, so just noting this here in case the issue comes back.
It might be solved by passing something such as gl=ie
in the URL.
As an example, https://play.google.com/store/apps/details?id=com.hulu.plus&hl=fr&gl=us and https://play.google.com/store/apps/details?id=com.hulu.plus&hl=fr&gl=fr makes the distinction between these parameters clean in their footer.
Best,
OpenTermsArchive/contrib-versions@3af7bf439 results in a notification when the only modifications happening are changes between non-breakable spaces and regular ones.
As discussed this morning with @michielbdejong and @Ndpnt:
scripts/validation/validate.js
.
xpathContentSelector
key in service declarations, which would rely on document.evaluate
supposedly exposed by JSdom."importedFrom": "https://github.com/tosdr/tosback2/blob/5acac7abb5e967cfafd124a5e275f98f6ecd423e/rules/4shared.com.xml"
. Make sure to use stable URLs (with a commit ID) and not a branch reference 🙂 Adding this key will require changing the validation schema.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.