publicsuffix / list Goto Github PK

i really think these onion proxies should be added:

onion.cab
onion.city
onion.direct
onion.link
onion.lu
onion.nu
onion.sh
onion.to
tor2web.blutmagie.de
tor2web.fi
tor2web.org
torstorm.org

(list may be slightly out of date)

they are proxies for the tor hidden services and work like this:
http://berrycckr666acln.onion/ -> https://berrycckr666acln.onion.link/
so that users can visit these websites without tor.

the problem is that if people whitelist the proxies e.g. onion.link in their browser/addons then they allow every tor website which they visit on that suffix to be whitelisted. this would be unsafe and unwanted. addons/browsers should be able to distinguish every proxied onion site from each other

the other problem is that most of them have no proper way of contact or they have not replied to my emails. if you look at any of these websites, their sole purpose is to proxy onion hostnames to the clear web and they should be treated as suffixes. what can be done about this?

How about making system for handling automatic submissions for publicsuffix list for domain owners?

• page for submiting domain/subdomain (with captcha)
• scan new submitions immediately and all submited domains every ~30 days - check http://submited-domain/.well_known/public_suffix for value '1'
• if value is 0 then remove from list
• if scan fails retry 5 times, 24h apart. If value is still neither 1 or 0 remove from list

(by submiting domain/subdomain again and placing 0 in .well_known/public_suffix you basicly request quick delist)

alternatively you can do same thing but using dns txt record for lower load/traffic. edit: now I think of it dns txt record would be probably better but if immediate actions were to be supported system would have to do recursive lockups (to avoid cache) (just for those immediate actions)

without this publicsuffix system is basicly catalog for only tld prefixes - and it could be much more.

Lots of Russian domains lost special status 5 years ago (see issue #43). We have been specifically asked by the owners of magnitka.ru to remove them. We should be able to do this even before completing the larger investigation.

Similar to issue #88 for FreeDNS, the same is needed for no-ip.com and their domains to work with let's encrypt. Although I guess no-ip.com would not have that many entries.

As I'm a customer with them I opened a ticket asking for support...

Hi,

it would be easier for downstream packagers if there were release tarballs.

Best regards -- Dago

Although I could not find an explicit mission statement of your project, most of the information you publish sounds like it would be something like having the most accurate list of public suffixes possible, which in turn would help browsers (and other internet related software / services) deliver a better and more secure web experience to users.

Under that light, I would like an explanation for the policy that privately managed suffixes can only be submitted by authorized representatives of the domain registrant.

The only explanation I found for myself is that you have an overwriting goal of protecting those registrants' property sacrificing the security of internet users in the process.

If this is not the case, I would ask you to replace that policy with simply some kind of proof of that the added domains are in fact public suffixes (like for example a link to a registration page that shows every internet user can register a subdomain of them).

Please remove qld.gov.au. This bug fix has been actioned for other states within Australia.

// act.gov.au Bug 984824 - Removed at request of Greg Tankard
// nsw.gov.au Bug 547985 - Removed at request of [email protected]
// nt.gov.au Bug 940478 - Removed at request of Greg Connors [email protected]

As it is published but only in flowing text form, the
gov.hu is controlled by governmental agency.

Reference: http://www.domain.hu/domain/English/szabalyzat/specnev.html

thanks, Viktor Varga

Hi,

Please add testurl.cz domain to the PSL.

I have just added the TXT record for verification, it might take a couple of minutes to propagate:

dig -t txt +short psl.testurl.cz
"#173"

Thanks!

Kolar

I received a report about krakow.pl. Apparently, the site is up and running:
http://krakow.pl/aktualnosci/195027,34,komunikat,maraton_dla_programistow_i_designerow_zbliza_sie_do_krakowa.html

Although this is not a proof, I checked when the suffix was listed, and I noticed it is no longer considered a regional domain at https://www.dns.pl/english/dns-regiony.html

However, we grandfathered it in 7d3893d when we reorganized the .PL listings (see bug 1069069).

@gerv you were involved in the discussion. I don't see any specific mention to these grandfathered suffixes in the ticket. Do you have some more context?

Can you please add some documentation on how to run the tests.

The .lol TLD, one of the many newly approved TLDs, is not listed on publicsuffix.

It appears that many popular browsers including Google Chrome read from the publicsuffix list to decide whether to direct users to a website or to a search... I recently purchase http://flip.lol and as a result of .lol being omitted from publicusffix, .lol domains entered into Google Chrome without the http are sent to Google search instead of the appropriate website.

I believe it is the case that all private suffixes end in an ICANN suffix. Is that accurate? If so, is it a guaranteed property of the PSL, and is it part of the automated tests?

Thanks,
Jacob

From: https://raw.githubusercontent.com/publicsuffix/list/master/tests/test_psl.txt

// TLD with only 1 (wildcard) rule.
checkPublicSuffix('cy', null);
checkPublicSuffix('c.cy', null);
checkPublicSuffix('b.c.cy', 'b.c.cy');
checkPublicSuffix('a.b.c.cy', 'b.c.cy');

It looks to me like the current public_suffix_list.dat no longer has a *.cy wildcard, so these tests will fail. The dot-cy TLD currently lists thirteen explicit second-level domains.

I suggest replacing the cy TLD with bd.

Domain hosts multiple instances with mutually untrusted customers

dlinkddns.com is an Dynamic DNS server für DLINK Customer.
This can be checked under https://help.dyn.com/d-link-device-verification-faqs/

Please add the following domains:

ap-northeast-2.compute.amazonaws.com
s3.ap-northeast-2.amazonaws.com
s3-ap-northeast-2.amazonaws.com

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

// Amazon Elastic Compute Cloud: https://aws.amazon.com/ec2/
// Submitted by Luke Wells  2016-02-05
ap-northeast-1.compute.amazonaws.com
ap-northeast-2.compute.amazonaws.com
ap-southeast-1.compute.amazonaws.com
ap-southeast-2.compute.amazonaws.com
cn-north-1.compute.amazonaws.cn
compute.amazonaws.cn
compute.amazonaws.com
compute-1.amazonaws.com
eu-west-1.compute.amazonaws.com
eu-central-1.compute.amazonaws.com
sa-east-1.compute.amazonaws.com
us-east-1.amazonaws.com
us-gov-west-1.compute.amazonaws.com
us-west-1.compute.amazonaws.com
us-west-2.compute.amazonaws.com
z-1.compute-1.amazonaws.com
z-2.compute-1.amazonaws.com

// Amazon S3 : https://aws.amazon.com/s3/
// Submitted by Luke Wells  2016-02-05
s3.amazonaws.com
s3-ap-northeast-1.amazonaws.com
s3.ap-northeast-2.amazonaws.com
s3-ap-northeast-2.amazonaws.com
s3-ap-southeast-1.amazonaws.com
s3-ap-southeast-2.amazonaws.com
s3-external-1.amazonaws.com
s3-external-2.amazonaws.com
s3-fips-us-gov-west-1.amazonaws.com
s3-eu-central-1.amazonaws.com
s3-eu-west-1.amazonaws.com
s3-sa-east-1.amazonaws.com
s3-us-gov-west-1.amazonaws.com
s3-us-west-1.amazonaws.com
s3-us-west-2.amazonaws.com
s3.cn-north-1.amazonaws.com.cn
s3.eu-central-1.amazonaws.com

Hello publicsuffix team,

I am the maintainer of the publicsuffixlist module for Node.js.
By the way, it would have been great to see it at https://publicsuffix.org/learn.

One of its users has reported an issue while the module was attempting to download a recent version of the list - which is part of its installation routine.

Further research has shown, that on the users' OS (a debian-based distribution) download managers like cURL and wget reject the certificate, too. In contrast, browsers like Firefox and Google Chrome do not complain about the certificate as they seem to bundle more recent certificates.

In a recent MacOS X 10.10.5 environment, the file is downloaded flawlessly, but I couldn't verify this on "Windows".

When attempting to download the list with cURL, the debug options returns the following:

$ curl https://publicsuffix.org/list/public_suffix_list.dat
* Hostname was NOT found in DNS cache
*   Trying 63.245.217.20...
* Connected to publicsuffix.org (63.245.217.20) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

The same is valid for the former location: https://publicsuffix.org/list/effective_tld_names.dat?raw=1 - wget returns the following:

Connecting to publicsuffix.org (publicsuffix.org)|63.245.217.20|:443... connected.
ERROR: cannot verify publicsuffix.org's certificate, issued by ‘CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
To connect to publicsuffix.org insecurely, use `--no-check-certificate'.

At the moment, this issue forces me to disable certificate validation while downloading.
This should be regarded solely as temporary workaround for all users who want to deploy their applications on Debian-based servers.

TLDs_to_be_added.txt

Hi - you currently just have

// cy : http://en.wikipedia.org/wiki/.cy
*.cy

However there are multiple 2nd level domains here.

I am NOT affiliated with the cyprus ccTLD. However this is right off their website.

http://www.nic.cy/nslookup/online_database.htm
The dropdown shows you -

com.cy
biz.cy
ltd.cy
pro.cy
net.cy
org.cy
name.cy
tm.cy
ac.cy
ekloges.cy
press.cy
parliament.cy

Here's a screenshot

E.g. by prepending this line to the tests:

{ echo '<?php '; cat tests/test_psl.txt; } | php -l || echo "Invalid code in text file."

#21 (comment)

Hi,

Azote.org is a well known french free domain provider.
They provide .fr.cr, .fr.nf, .ze.cx, .asso.st, .infos.st and .xxl.st domains.

I contacted them and asked they add their own domains to this list, but they didn't answer back.

From what I've seen, you require validation of domains? Might we add their domains without validation on their side?

please add dev-myqnapcloud.com & alpha-myqnapcloud.com to publicsuffix list

Thanks

The domain servebeer.com is one of 80+ free DDNS domains, used by NO-IP.com.
Thank you

The list is missing some entries for the .PRO TLD.

They are: recht.pro, aaa.pro, acct.pro, and avocat.pro

They can be seen on the Wikipedia page for .PRO: https://en.wikipedia.org/wiki/.pro

While the file is transferred over HTTPS it would be benefitial for many cases to provide a checksum to ensure transfer and storage succeeded, meaning you could download https://publicsuffix.org/list/public_suffix_list.dat.sha256 as well. Or something similar.

SITA (sponsor of the .aero TLD) would like to remove these two domains from the list:

taxi.aero
marketplace.aero

Regards,
SITA .aero office

After cross government consultation, agreement and permission has been granted by the Domain Name Administrator at the Department of Finance and Deregulation to remove .gov.au from the public suffix. AUDA have been notified of our intent to remove the .gov.au from the public suffix list. Please remove at your earliest convenience.

// gov.au - Bug XXXXXX - Removed with permission from the Commonwealth of Australia at the request of Richard Denver

Lots of Russian domains lost special status 5 years ago:
http://cctld.ru/en/news/news_detail.php?ID=634&sphrase_id=128335

http://cctld.ru/en/domains/domens_ru/reserved/ now lists only ac.ru, edu.ru, int.ru, gov.ru, mil.ru and test.ru.

However, because they didn't cancel 3rd level registrations when that made e.g. com.ru not special any more, there are still many companies in com.ru. So I don't think we can simply reduce the PSL to the list above. We will need to do an investigation and make our best guess. It may be that we leave the generic ones and remove the regional ones, but we need to look at the situation carefully.

The issue was raised today in the Let's Encrypt forum. Apparently, it's possible to register second-level .ZM domains such as ischool.zm. However, our current rule is *.zm that totally denies this possibility.

According to Wikipedia the .ZM TLD has several different suffixes. However, there is no official documentation from the registry, and the website is not providing any hint.

I'm going to contact the registry to get a list of officially approved suffixes, and to get more hints about second-level TLDs. Specifically, I'd like to know if they are exceptions, and in that case how long is the exception list.

If you have .ZM contacts, feel free to share.

/cc @gerv @sleevi

The TLD xn--e1a4c appears to be missing in the PSL.

http://www.iana.org/domains/root/db/xn--e1a4c.html

I'm on qnap.com, where we have a public dynamic dns service.
Please let me know how to verify this and feel free to contact me (nickchang at qnap.com) if any need.

The letsencrypt project is using the public suffix list while checking for accidental and intentional abuse.(see https://community.letsencrypt.org/t/rate-limits-for-lets-encrypt/6769)

The result is that this prevents all users of the FreeDNS project(http://freedns.afraid.org/) from getting a free certificate due to said limits.

The problem is that we're talking about a list of ~90k domains which is being constantly updated with some being marked as 'private'(see http://freedns.afraid.org/domain/registry/). The addition of the public ones is still valid as they are in fact publicly used suffixes which are open for registration(e.g. mooo.com with ~400k subdomains).

Not adding them would prevent ~8 million sites(check 'records' at http://freedns.afraid.org/domain/registry/) from using letsencrypt.

The sheer number of domains and their fluctuation would require automation. I'm already in contact with one of their admins([email protected]) and they would provide an export link for this purpose.

agrica.za appears to be a typo in:

http://www.zadna.org.za/content/page/domain-information

That same page mentions AGRIC.ZA, which appears to be correct:

http://google.com/search?q=site:agric.za

I have notified the za registry.

I'm running the following test case with libpsl:

checkPublicSuffix('amber.museum', 'amber.museum');
checkPublicSuffix('aip.ee', 'aip.ee');
checkPublicSuffix('africa.com', 'africa.com');
checkPublicSuffix('amursk.ru', 'amursk.ru');
checkPublicSuffix('appspot.com', 'appspot.com');
checkPublicSuffix('ar.com', 'ar.com');
checkPublicSuffix('eu.org', 'eu.org');
checkPublicSuffix('nsk.ru', 'nsk.ru');

and it fails:

psl_registrable_domain(amber.museum)=NULL (expected amber.museum)
psl_registrable_domain(aip.ee)=NULL (expected aip.ee)
psl_registrable_domain(africa.com)=NULL (expected africa.com)
psl_registrable_domain(amursk.ru)=NULL (expected amursk.ru)
psl_registrable_domain(appspot.com)=NULL (expected appspot.com)
psl_registrable_domain(ar.com)=NULL (expected ar.com)
psl_registrable_domain(eu.org)=NULL (expected eu.org)
psl_registrable_domain(nsk.ru)=NULL (expected nsk.ru)

(full test log)

May I put this into the Wiki?
This script outputs only the SLD part from a list of domain names.
So if you have subdomain.example.co.uk on your list, the output will be example.co.uk.

# Prepare the list for finding second-level domains
Publicsuffix_regexp() {
    local LIST_URL="https://publicsuffix.org/list/public_suffix_list.dat"

    # Download list,
    #   remove empty lines and comments,
    #   escape dots, asterisks and add SLD regexp
    wget -qO- "$LIST_URL" \
        | grep -v "^\s*$\|^\s*//" \
        | sed -e 's/\./\\./g' -e 's/\*/.*/' -e 's/^\(.*\)$/[^.]\\+\\.\1$/'
}

Publicsuffix_regexp | grep -o -f - "$YOUR_DOMAINS"

I use it for registration expiry purposes.

3CX is now automatically provisioning certificates via Let's Encrypt for their users .

please add those subdomains

*.pbx.3cx.eu
*.pbx.3cx.us
*.pbx.3cx.asia

According to Wikipedia and ISOC-IL, there are eight second-level domains under .il. Given my limited understanding of the PSL, it seems like those eight¹ should be listed in addition to *.il. Otherwise the public suffix of, for example, clean-carpets.co.il will be identified as il when it should be co.il – as I understand it.

Please let me know if I'm totally wrong here.

¹ ac.il, co.il, org.il, net.il, k12.il, gov.il, muni.il, and idf.il

// Company : http://www.example.com/
// Submitted by John Doe <[email protected]> 2015-12-17
example.com

I propose to change it to

// Company : http://www.example.com/
// Submitted by John Doe <[email protected]>
example.com

I'm aware this comment field is unstructured, however it is becoming a kind of de-facto standard (especially for private suffixes) and I think we should try to find a rule.

The date is irrelevant, I can get it from git. The name and the email is still useful, especially to lookup the email in my account (or for future communications).

// Submitted by

can be kept as it is, or we can use a different word. We have used variants such as Submitted, Requested, Updated, etc. I propose to use just a single word that represents the last person who updated it. We can use

// Changed by John Doe <[email protected]>

that can potentially make sense also for multiple entries

// Changed by John Doe <[email protected]>
// Changed by Someone Else <[email protected]>

@gerv @sleevi feedback?

At the moment we have just a list, but no defined processing model for that list. Without such a defined processing model, it's impossible for standards to be accurate in their requirements.

E.g., HTML has "If new value matches a suffix in the Public Suffix List", but neither "matches" nor "suffix" is defined. And Public Suffix List is an opaque blob of data.

We could define this externally, e.g., some suggested it to be defined as part of the URL Standard: https://www.w3.org/Bugs/Public/show_bug.cgi?id=25865. However, it seems better to define this model at the source, no?

Please add co.cc to the PRIVATE section of the Public Suffix List.

// GoDaddy Operating Company, LLC
co.cc

Why is co.cc a Public Suffix? Because http://co.cc/ offers to register domains.

Why PRIVATE? Because .cc only registers directly under .cc. It does not offer registration under .co.cc.

Why GoDaddy? If you start from iana.org, find the registrar for cc, find their whois service, and look up cc, it will say GoDaddy.

Why GoDaddy Operating Company, LLC? That's what it says at the bottom of godaddy.com.

We should assemble the fragments of code in the tests/ directory into an actual runnable test suite which checks the file for common errors and makes it as unlikely as possible that we will break anyone's tree or parser.

This bug covers testing the things the existing tests cover. We should file additional bugs for other things we might like to test.

Please add co.dk, biz.dk, firm.dk, store.dk, and reg.dk to the Public Suffix List. I believe they belong in the PRIVATE section, sorted by company. The company is:

// Digital Marketing Support ApS

Ownership was confirmed with:

whois -h whois.dk-hostmaster.dk co.dk
(and so on, for each domain)

Here is the email:

Date: Fri, 26 Feb 2016 11:02:29 +0100
Message-ID: CAFybOX=9zOuqeN3eurFvjaMgiYD8ew3xkjcVShHbPMPRiDHnUA@mail.gmail.com
Subject: Re: co.dk and publicsuffix.org
From: Hostmaster Digital [email protected]
To: Erik van der Poel [email protected]
Cc: [email protected]
Content-Type: multipart/alternative; boundary=001a1140f86a9bfd19052ca96509

--001a1140f86a9bfd19052ca96509
Content-Type: text/plain; charset=UTF-8

Thanks for your e-mail.

We will be pleased if you could kindly add the following domain names to
publicsuffix.org as we offer to register domains under these:

co.dk, biz.dk, firm.dk, store.dk, reg.dk.

Best regards,

Anani Voule

Hostmaster

Dansk.net / Digital Marketing Support ApS

On 24 February 2016 at 22:15, Erik van der Poel [email protected] wrote:

Hello,

I got your email address from www.co.dk. I am trying to improve
publicsuffix.org, which is a list of domains used by browser developers,
search engines, etc to implement cookies, certificate checking, etc. I
noticed that co.dk offers to register domains but it is not included in
publicsuffix.org. I believe it belongs in the PRIVATE section of
publicsuffix.org.

Please let me know if you agree, or would like more information.

Thanks,
Erik van der Poel
Software Engineer
Google

Please remove “tp” from public suffix list. Earlier this year IANA decided to remove “tp” top level domain after a long deliberation process.
https://www.iana.org/reports/2015/tp-report-20150126.html
https://features.icann.org/removal-tp-top-level-domain-representing-portuguese-timor

I made a commit a few weeks ago that introduced a rule like *.*.private.domain and the commit caused the build to fail.

According to our website, that is supposed to be a valid format:

Wildcards are not restricted to appear only in the leftmost position, but they must wildcard an entire label. (I.e. ..foo is a valid rule: *bar.foo is not.)

@rockdaboot mentioned a potential incompatibility of Chromium if we allow multiple wildcards. libpsl is currently not compatible with multiple wildcards, and to be fair I haven't tested my Ruby implementation either.

@gerv @sleevi can we clarify whether multiple wildcard labels are accepted? Specifically, we should be more clear if the following rules are valid:

// multiple leading wildcards (common case)
*.*.foo.bar

// single wildcard, but inside the rule
foo.*.bar

// multiple willdcards, inside the rule
foo.*.*.bar

// multiple willdcards, inside the rule, non consecutive
foo.*.bar.*.baz

// I suppose this is invalid
foo.*

The current list definition doesn't explicitly deny these rules, they are supposed to be valid.

Once the decision is taken, I think we should:

1. Update the website to make it more clear
2. Add some corresponding tests in the test file
3. Should the rules be considered invalid, I'll also add the corresponding rules to the linter I'm working on,
4. Should the rules be considered valid, we should ping the various lib maintainers

Please add dnshome.de to the list.

dnshome.de is a German dDNS service. Before creating this issue I contacted the owner of the domain dnshome.de and asked for this approval to be added to the list. He has given the approval

Thanks :-)

Makes it easier to fix small bugs, such as the lack of HTTPS in the link to Mozilla Foundation. And add new standards that depend on this mechanism, such as https://storage.spec.whatwg.org/ (not explicit yet, see #27).

please add PRODUCTIONS.com.br to Public Suffix List in Private Section

Includes the following new ICANN gtlds:
bet, dds, grainger, metlife, pet, srl, theatre, app, apple, beats, blog, extraspace, hkt, nowtv, olayan, olayangroup, pccw, richardli, security, vig, volkswagen, xn--fzys8d69uvgm, xn--mgba7c0bbn0a, xperia, aarp, aetna, audi, dot, mlb, nikon, amica, bostik, game, games, dtv, lamborghini, locker, moto, ollo, ott, weber

For Search, the following are the U-Labels
xn--fzys8d69uvgm = 電訊盈科
xn--mgba7c0bbn0a = العليان

Domains with ICANN contract announcements between 05/05/2015 and 06/10/2015.