Meta has not updates PSL snapshot in whatsapp since november 2015 about list

[rubenskuhl]

Thanks for all of the testing. These platforms all have fundamentally different ways/APIs of identifying if something is a link, which is why you see these differences. We are still investigating this internally to determine what course of action (if any) we take in particular for WhatsApp.

While raising an issue with Android developers is totally in order considering its contribution to the issue, from a product standpoint it would make more sense to me to make this homogenous among platforms. But looking on the bright side, it's good that WhatsApp helped uncover the Android issue, even if it moves away from using Android libraries for this task.

[rambox1963]

@bedfordsean Thank you for linking the libraries, good to have tech people who actually know how to get stuff done around computers. And thank you for comments like this one "Shooting for eventual consistency feels like a sensible path forward however". Coming from you and META, should have a higher specific weight when addressing the root of the issues at ICANN.
Will try to find who can link the microsoft libraries also.

[rambox1963]

linkify.xlsx
The spreadsheet

[dnsguru]

@bedfordsean welcome back and thank you for looking further into this.

[dnsguru]

Yes... I've been looking a bit more in to our codebase for this today, and in summary... there is a lot of stuff going on ;-)

Android bakes in the IANA list as we've seen above. So does iOS/macOS, but it hides the TLDs being used behind a private API that just returns "True" when you give it a potential URL string.

We're going to keep investigating

Thank you @bedfordsean - I have no doubt that there is a "Matryoshka doll" of issues here as you start opening the "dolls" to look inside, especially where the are indications that the underlaying OS's are working from snapshots as old or older where they should not have.

[rubenskuhl]

This is the result for WhatsApp desktop app for macOS, version 2.2335.9 . macOS version is Ventura 13.5.2:
Darwin H615FHF3DW 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:17:35 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8112 arm64

[hank-nuss]

In regards to PSL and ICANN:
https://www.icann.org/en/system/files/files/octo-011-18may20-en.pdf

Page 6 quote:
"After a change is made in the PSL, it’s important to understand that some software and services
on the Internet that rely on the PSL get updated slowly or not at all. Although some developers
have their software regularly fetch the current PSL, others build a static list into their software or
update with lower frequency. Some of the default mobile and tablet browsers only update when
the underlying operating system updates; this can clearly cause problems for users."

You may find this ICANN Wiki of interest:
https://icannwiki.org/PSL

[bedfordsean]

The Android libraries in question have been linked above:

https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/util/Patterns.java;l=114?q=praxi

The relevant iOS libraries are these - you provide text to them and they return data type matches, including a "link" type.

https://developer.apple.com/documentation/foundation/nsdatadetector
https://developer.apple.com/documentation/foundation/nstextcheckingresult

For Windows I expect there's something similar however I'm not familiar with the platform so not sure where to look to find that information.

Note that in the iOS and Android cases these are core OS files so require an OS update and for people to then update their devices - so it's not a simple change (similar to what we see in the world of web). Shooting for eventual consistency feels like a sensible path forward however.

In the meantime we are still researching on our side at Meta if there is something we can do to handle this more effectively/consistently for our own applications and websites.

[dnsguru]

Glad to have another person with ICANN dents here @hank-nuss

Ultimately, there does exist a gap between dev world and that of ICANN/IANA - Not limited to a developer-friendly TLD list.

[rubenskuhl]

@bedfordsean thank you for the information. I still type nic.africa and nic.tube in whatsapp on android and is still not linkifying. Does it take time to propagate?. I have sent the text to my contact in Google, not sure if it was premature.

It will be used a subsequent code build, it's not something that's regularly download by phones. Best case scenario, but unlikely, it will be included in incremental updates to Android 11, 12, 12.1 and 13. Possibly on Android 14, due in the next few weeks. Worst case scenario would be to only be added in Android 15, due 2024Q3.

Note that after Google incorporates that, it only goes to phones when phone vendors decide updating/upgrading... so there is a long tail to navigate. Different from Apple, where they are both OS developer and phone vendor, and where they usually push new versions even to older phones.

[dnsguru]

thank you to @rambox1963 @bedfordsean @hank-nuss @rubenskuhl @nipoitra80 @case-fastly @FBNeal for real, material efforts around Universal Acceptance

[dnsguru]

@rambox1963 have you raised this to the attention of the ICANN UASG and gotten any response?

This seems like a matter that should be raised with the ICANN Universal Acceptance Steering Group, although there really is no manner in which they would be able to compel any developer to do anything.

I looked and it seems like Telegram has addressed the matter since you last tested, but I took some time and looked at some of the example domain names you listed, and it appears that Meta's recently launched threads.net platform also has this same URL behavior.

This makes me suspect that Meta may be using a common 2015 list for "URL or NOT" TLD logic.

We cover third party use/incorporation of PSL in the wiki, but I have to observe that an 7+ year span since last update is a very drastic lag, and a signal that it is a reasonable time to update.

Ultimately, there is not really much a way that the PSL volunteers can do to compel META or any party that incorporates the PSL to make changes... There are some folks from Meta that have been stepping in to diminish the impacts to their Ad revenues and a recent Pull Request was submitted from Meta.

Perhaps Meta can some restore volunteer karma they went deficit on via #1245, that extracted from the PSL volunteers and/or locate the person within their company that could update their TLD list.

The only other means of motivation to action by the PSL volunteers could suggest to Meta is a refresh of their TLD logic from 2015 PSL to the current one in threads and whatsapp being a prerequisite, required action that they have to perform before any Meta Pull Request would advance.

[rambox1963]

Of course I have alerted all parties, UASG, ICANN and Meta. All gave me the same standard copy/paste answer they give to everyone -we hear you but there is little we can do-, -this is out of our hands.-.
Telegram has addressed the matter, still not perfect but addressed. Not sure about threads, I don;t use threads, I know Facebook does not have the same problem. And yes, could be that Whatsapp has not updated the PSL in 8 years but seems crazy. Another alternative is that the information somehow comes corrupted from the source.
-0-0-0-
Whatever the problem one thing is for sure, this is probably the most emblematic Universal Acceptance issue I can remember and the fact that is happening now is a major embarrassment. Whatsapp in not a moms and pops website but the #1 instant messaging app in the world and its not treating all registries equally. After more than 8 years it does not even know .TUBE or .AMAZON or .AFRICA are registries. One in every three nTLD's is affected and unable to squeeze the full value of their asset and ICANN is clueless. The guardians of the DNS have no idea why NIC.MICROSOFT linkifies while NIC.AMAZON does not.
Universal Acceptance... UNIVERSAL comes from the Greek UNI, UNO, ONE. The ONE" that UNIVERSALIZES everything must be an ICANN REGISTRY list, an official database of the CONCESSIONARIES of the DNS and clear rules on how to connect to this database.
This issue proves that Universal Acceptance is foremost a technical issue and not a marketing strategy as is viewed by ICANN. It proves also that its an ASCII issue as much as a IDN issue contrary to ICANN's view that Universal Acceptance is equal to IDN Acceptance.

[dnsguru]

Of course I have alerted all parties, UASG, ICANN and Meta. All gave me the same standard copy/paste answer they give to everyone -we hear you but there is little we can do-, -this is out of our hands.-.

More of the same from PSL, unfortunately, as use of the PSL is intentionally non-prescriptive and flexible.

As a catalog/text file, PSL volunteers really have little to help here other than to encourage developers to keep their snapshots fresh if they opt to incorporate the PSL in a static manner in code.

The ICANN Security and Stability Advisory Committee did a review a few years ago (which robbed me of many, many volunteer cycles but on the upside allowed me the privilege of meeting many amazing infosec professionals) and the work product of that review was SAC-070, which made a number of recommendations, including "Recommendation 4c: Application developers should also replace proprietary PSLs with well-known and widely accepted PSL implementations such as the Mozilla PSL and the proposed IANA PSL", on page 27.

Additionally, the Office of the Chief Technical Officer (OCTO) at ICANN.org published OCTO-011 that explained how TLD Registries can interact with the PSL, oberving in conclusion that, "Some developers may choose to not include all PSL entries, or to introduce additional items. The PSL does not compel or prescribe how the list gets used or if it is used in its entirety. A developer is free to make their own determination of inclusion or exclusion of portions of the list at their own discretion or modify it for their own purpose. Therefore it is important to realise that while keeping entries in the PSL as up to date as possible is important, discrepancies may still be seen due to factors beyond your control.".

Telegram has addressed the matter, still not perfect but addressed.

Encouraging news. Sounds like SAC-070-R4c as above

Not sure about threads, I don;t use threads, I know Facebook does not have the same problem. And yes, could be that Whatsapp has not updated the PSL in 8 years but seems crazy.

OK so it sounds like some parts of Meta have updated their TLD logic, some have not. SAC-070 Recommendation 4a states, "Recommendation 4a: ICANN, as part of its initiatives on universal acceptance, should encourage the software development community (including the open source community) to develop and distribute programming and operating system libraries implementing robust (i.e. authenticated, timely, secure, accountable) distribution mechanisms for PSLs. These libraries should be written across all common platforms and operating systems in a way as to ensure consistent and standard interpretation of a given PSL across all platforms. ". If one were to interpret that recommendation to the scope of a company, it would deem it applicable that the recommendation might mean "consisitent accross Meta", but it is likely a far oversimplified and very liberal interpretation of something that is merely a recommendation.

In general "We hear you" is about the best that an under-resourced, volunteer-led project like the PSL can offer as a solution, but again, Meta do have requests sitting in the queue, so we'll note this issue with them in that pull request.

Developers don't follow the changes of ICANN/TLD lists too closely.

Your other observations are noted as well. In a utopian world, there would be a place for developers within ICANN's MSM, like a "internet developer stakeholder group", but there is not one.

Generally speaking, it is my observation that the ICANN multi-stakeholder model has no place for developers, nor is it very attractive for developers to participate in ICANN, as they are very different realms from each other. There is an IETF, but truly that is a standards-generation body, and not one representative of developers making internet applications. Similarly, developers are typically working to quickly address market needs and create solutions, for which any beaurocratic friction that adds scope has very low allure. So ICANN participation has the attraction of lead jogging shoes for developers.

The PSL ends up being the 'least awful' middle ground as a rainbow bridge between the worlds.

[dnsguru]

There is a Pull Request change from within Meta and have mentioned this issue to them requesting some aid in finding some form of solution, and there are some others that check in from time to time. They may or may not comment, but perhaps there might be some internal advocacy for updates that results.

[rambox1963]

There is a Pull Request change from within Meta and have mentioned this issue to them requesting some aid in finding some form of solution, and there are some others that check in from time to time. They may or may not comment, but perhaps there might be some internal advocacy for updates that results.

This is great news, hopefully this will be fixed. If that is the case KUDOS to you and I will make sure the registries give you all the credit you deserve for it.

[rambox1963]

More of the same from PSL, unfortunately, as use of the PSL is intentionally non-prescriptive and flexible.

As a catalog/text file, PSL volunteers really have little to help here other than to encourage developers to keep their snapshots fresh if they opt to incorporate the PSL in a static manner in code.

The ICANN Security and Stability Advisory Committee did a review a few years ago (which robbed me of many, many volunteer cycles but on the upside allowed me the privilege of meeting many amazing infosec professionals) and the work product of that review was SAC-070, which made a number of recommendations, including "Recommendation 4c: Application developers should also replace proprietary PSLs with well-known and widely accepted PSL implementations such as the Mozilla PSL and the proposed IANA PSL", on page 27.

Additionally, the Office of the Chief Technical Officer (OCTO) at ICANN.org published OCTO-011 that explained how TLD Registries can interact with the PSL, oberving in conclusion that, "Some developers may choose to not include all PSL entries, or to introduce additional items. The PSL does not compel or prescribe how the list gets used or if it is used in its entirety. A developer is free to make their own determination of inclusion or exclusion of portions of the list at their own discretion or modify it for their own purpose. Therefore it is important to realise that while keeping entries in the PSL as up to date as possible is important, discrepancies may still be seen due to factors beyond your control.

Perhaps this is excessive laxity and discression in the use of PSL is the reason why there are so many UA issues after so many years, This flexibility cannot be good, the Internet world must have certainty of WHO ARE THE CONCESSIONARIES OF THE DNS, the only parties authorized to do the character-number equivalency. This information is extremely important and mertis, at least, strict oversight of the public lists plus best practices on how to incorporate them. This is the core of UA. This whatsapp example proves that the policy of "Some developers may choose to not include all PSL entries, or to introduce additional items. The PSL does not compel or prescribe how the list gets used or if it is used in its entirety" runs counter to UA, is a major impediment for the Universal Acceptance. The ICANN UASG seems to be focusing on the IDN aspect of issues without placing weight on addressing core, fundamental issues that impact ALL nTLDs. Given the resources available to ICANN, it amazing that 90% of the effort goes towards IDN when 90% of the problem is fundamentals like this.

[rambox1963]

Telegram has addressed the matter, still not perfect but addressed.

OK so it sounds like some parts of Meta have updated their TLD logic, some have not. SAC-070 Recommendation 4a states, "Recommendation 4a: ICANN, as part of its initiatives on universal acceptance, should encourage the software development community (including the open source community) to develop and distribute programming and operating system libraries implementing robust (i.e. authenticated, timely, secure, accountable) distribution mechanisms for PSLs. These libraries should be written across all common platforms and operating systems in a way as to ensure consistent and standard interpretation of a given PSL across all platforms. ". If one were to interpret that recommendation to the scope of a company, it would deem it applicable that the recommendation might mean "consisitent accross Meta", but it is likely a far oversimplified and very liberal interpretation of something that is merely a recommendation.

In general "We hear you" is about the best that an under-resourced, volunteer-led project like the PSL can offer as a solution, but again, Meta do have requests sitting in the queue, so we'll note this issue with them in that pull request.

Developers don't follow the changes of ICANN/TLD lists too closely.

Your other observations are noted as well. In a utopian world, there would be a place for developers within ICANN's MSM, like a "internet developer stakeholder group", but there is not one.

Generally speaking, it is my observation that the ICANN multi-stakeholder model has no place for developers, nor is it very attractive for developers to participate in ICANN, as they are very different realms from each other. There is an IETF, but truly that is a standards-generation body, and not one representative of developers making internet applications. Similarly, developers are typically working to quickly address market needs and create solutions, for which any beaurocratic friction that adds scope has very low allure. So ICANN participation has the attraction of lead jogging shoes for developers.

The PSL ends up being the 'least awful' middle ground as a rainbow bridge between the worlds.

Fully agree, in a perfect world developers could work with registries, registrars and ICANN to create the "ONE WORLD ONE INTERNET" ideal which to me is synonym with UA. But it will remain an ideal as long as the regulator does not facilitate things. Which often leads to "least awful middle grounds" as you call them. The stakeholders most affected by these "least awful middle grounds" that cause this crippled UA are the Registries, who paradoxically are the CONCESSIONARIES OF THE DNS. Instead of being given preferential treatment as the concessionaries they are, ICANN allows developers "free to make their own determination of inclusion or exclusion of portions of the list at their own discretion or modify it for their own purpose". Imagine that, these concessionaries, they paid a lot of money for their concession, they keep paying quarterly, they must comply with a whole set of rules but developers can ignore them if they want, further, they can include other dot.something like https://blahblahblah.vladimirputin that is not even a registry and ICANN is OK with it". But then again, ICANN & UASG focus on IDN Acceptance not Universal Acceptance. Universal Acceptance are fundamental issues that impact ALL nTLDs including IDN's. No wonder after 8 years of thee launch of the first new TLD, the most important instant messaging application in the world is not UA Compliant.

[dnsguru]

I hear ya.

It is a lot for volunteers to track just this project. I will say that we prioritize ICANN / IANA changes and submissions / changes from the TLD operators, but only within our lane, which is documenting a list of eTLD+ into a shared file resource.

The UASG had been resourced with people doing outreach to developers when an issue like you are reporting would come up. I don't know if that's still done, but there were personnel that were tracking things like this.

Good luck - will leave this open so as to see what magic might happen from empathetic Meta-folk.

[rambox1963]

On the contrary, thank you for keeping the PSL and also for this effort to alert Meta that Whatsapp is not fully UA Compliant.
As for ICANN, the UASG, Global Support, etc. they are supposed to be the experts, the gate keepers of the DNS and at least in this case, they were all clueless. Somewhere a setting is wrong, it affects one in every three concessionaries but when it comes to UA, they hide behind their mantra and provide no help. UA is a technological problem, not a marketing/communications problem and at the core, in my opinion, half a dozen public lists with various degrees of trustworthiness and no clear guidance on how to use them.

[bedfordsean]

Hello, Sean from Meta checking in

Apologies for the long delay - I've been out on extended leave and just back.

For now, consider this an "ack", I'm going to start digging into the behaviour our platforms (not just WhatsApp) have here to find the right contact

[bedfordsean]

Hi,

We have done some investigation into the issue reported here and believe that our linkification behavior is working as intended. In our testing, with a registered “.africa” website (we used dataprotection.africa in our testing), we observed the following:

1. If you make a post with text such as “nic.africa” on Facebook, Instagram, Threads, Messenger, or WhatsApp, it will not automatically be detected as a link, nor will we attempt to generate a preview of the website at that link.
2. If you however prepend this with “http(s)://”, or “www.”, we will interpret this to be a valid web link, turn the text into a clickable link, and attempt to load a preview of the website to be displayed on the post on our platforms.
3. For a small number of very common website TLDs, like “com”, we will create a link irrespective of prepending the URL scheme or typical “www.” prefix.

The reason that we linkify a small number of TLDs more aggressively is in an attempt to balance the prevalence with which a particular TLD may be referenced in content with the reality that certain TLDs may be included in a larger body of text as a result of typos. For example someone may write

“Your photos from the trip look great.africa is our next place to go!”

In this case, people would likely be confused if we linkified “great.africa”. Similar concerns apply to TLDs like .amazon, .london, .microsoft, etc. Accordingly, we limit automatic linkification behaviour to specific TLDs in an attempt to balance such confusion and its impact on the user experience with the benefits of simpler linking.

We acknowledge that the heuristics we use today are somewhat simplistic. For instance, they do not take into account slashes, which might provide a stronger signal around the expectation that content should be treated as a URL. Although we may make changes and improve the heuristics here in the future, we are not committing to any specific changes at this time.

Meta updates our copy of the Public Suffix List regularly and these changes are incorporated and used across our website and applications. I can assure you that 2015 was not the last time we updated :-)

As mentioned, giving our platforms a link hint (http/www) will correctly create a clickable link to the website specified. We do not intend to change this behaviour.

If you have further examples or questions in terms of how our platforms function where the behaviour you see does not match what I have described, please can you let me know and we will further investigate?

[rambox1963]

Hello
(https://1drv.ms/x/s!Aojm-_cF3ePngdNU6Jop56sX_GlSSA?e=X8U3cQ)
This is a link to a database I made that clearly shows that TLD's delegated prior to November 2015 all "linkify" and those delegated after december 2015 don't with November being a gray period.
So it ain't as you say that "a small number of very common website TLD's like ".com" will create a link irrespective of the prefix". This is simply not true.
For instance .praxi is not popular at all, not even used as far as I know yet it linkifies. On the other hand .shop is one of the most popular new TLD's but it does not linkify.
Finally, www.iamanars.valimirputin or https://crazybomber.kimjongun bot linkify and they don't even exist which means that whatever you add the prefix https:// and www. will linkify regardless.
So to summarize, .microsoft delegated on 6/4/15 linkifies, .amazon delegated on 6/2/2020 does not. .Buzz delegated on 12/28/2013 linkifies, .food delegated on 11/10/2016 does not. .London 3/22/2014 linkifies, .boston 11/29/2016 does not. All in all 1 in every 3 new TLD's does not linkify.
Finally, it's fine if you want to linkify anything presided by https: or www. In my view, it opens the door to a lot of fraud. I would check the PSL first and if the termination (.vladimirputin) is not there, I wouldn't allow linkification. But TLD's that are legitimate such as .TUBE or .AFRICA or .WALMART have to linkify without the prefix, otherwise META is penalizing and discriminating against the TLD's that were later to market, a decision that was out of their control.
I am not urging you to reconsider changing the behavior of your platforms, just treat all TLD's equally.
I am not a frequent user of this platform so I have no idea what is customary or permitted and what isn;t, but feel free to contact me directly to discuss. rami - at - get.tube

[rambox1963]

Hello, the example of Great.africa is not a good example because there are other geographic TLD's that do linkify. Great.london, great.barcelona, great.amsterdam, great.paris all linkify because they were delegated prior to 11/2015. great.africa, great.boston, great.abudhabi, great.arab don't, they were delegated after.
I think this is very puzzling because other platforms, that are probably using the same PSL as Meta, don't have this issue. X or Telegram are two examples. So probably there is an identifier in the source or destination that is telling Whatsapp to view nic.volvo as text while nic.bmw as a link and I can't quite figure it out. But I have identified the problem so if you put a magnifying glass on the TLD's delegated between october 2015 and december 2015 you should be able to find the bug, because it is a bug. October is fully green, December is fully red, it is November there the bug entered either your system or the source of the database.

[rambox1963]

This image has all the examples needed, the result of the "simplistic heuristics" Meta uses today.
TLD's that linkify
TLD's that need prefix to linkify
BOGUS that linkify with prefix as if they were TLD's
I am not even sure this is META's fault, that behavior linked to specific ICANN dates (Delegation) is very suspicious. I wish we had the opinion of the experts at ICANN, what is their take on this mystery, on this ghost who appeared in the registry databases on November 2015. But so far I have not seen their take.
I have alerted them about this repository and invited them to read and participate.

[FBNeal]

Thanks for the screenshot: the context that you tested specifically on Android is helpful. :)

Are you perhaps observing core Android code like this being used?

https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/util/Patterns.java;l=114?q=praxi

The comment at the top mentions

List accurate as of 2015/11/24

[dnsguru]

I would like to update the link above to be https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/util/Patterns.java;l=114?q=wtf

[dnsguru]

Still, I notice that the same client issue exists across different platforms like Win MacOS or IOS and manifests similarly, so I am not quite certain the finger entirely points away to other libs. Though those default libraries and functions likely are not helping address the issue, individual apps do frequently override those defaults.

[bedfordsean]

For reference, here's what happens on our current iOS WhatsApp build. This matches what I stated above when I tested iOS yesterday (apologies for assuming we had consistency across our own iOS and Android applications!).

It seems we may be reliant on this Android platform library which includes a static list of IANA domains, but there is most likely more going on.

We will keep digging. No promises that we can easily change the behaviour, but I'd like to be able to fully explain what is happening here.

[rambox1963]

Hello and thank you for looking into this. I have 4 screen shots for you, two you have seen, android and IOS, but it turns out that web.whatsapp also has a different behavior when run in an apple computer with apple operating system and on a Windows PC with Bing.
So the plot keeps thickening, now I am not even sure if it's the operating systems that are using outdated lists of domains or if it's the interaction of Meta's apps with those operating systems. I can't find any logic and also, no logic on why the different behavior of TLD delegated prior or after NOC of 2015.




Once more, ICANN's help could be very valuable. Perhaps if @bedfordsean or @FBNeal invokes them...

On a related note, I strongly suggest you don't rely on the IANA list. A month ago I had to request that the record for .TUBE be updated because it had information that was outdated since 2017. I changed the information for .TUBE in ICANN years ago thinking PSL's checked that database constantly but apparently some do like PSL and some don't and ICANN does not even have a policy to use correct, updated databases.

[rambox1963]

Sorry, a few typos in the last paragraph of my previous post.
On a related note, I strongly suggest you don't rely on the IANA list. A month ago I had to request that the record for .TUBE be updated because it had information that was outdated since 2017. I changed the information for .TUBE in ICANN years ago thinking PSL's updated their databases constantly but apparently some do (like PSL) and some don't (Like IANA, don't know about MOZILA or others). ICANN does not have a policy to force PSL's to use correct, updated records to populate their databases

[bedfordsean]

Yes... I've been looking a bit more in to our codebase for this today, and in summary... there is a lot of stuff going on ;-)

Android bakes in the IANA list as we've seen above. So does iOS/macOS, but it hides the TLDs being used behind a private API that just returns "True" when you give it a potential URL string.

We're going to keep investigating

[rambox1963]

More screen shots...

sent from Android

Received in Iphone

Truly appreciate your digging deeper into this matter.

[rambox1963]

NEWS FLASH.
In 2019 the .ss ccTLD was delegated (SUDAN) and it's not linkifying. Attached screens how it's sent from android and received by IOS and also from Web.Whartsapp.com on WIndows 10/Firefox. Different results in all, inconsistencies seem to be happening across platforms and operating systems.

This one is very clear: It has both screenshots to compare, the sending from Windows and the receiving on Android. Can't get any clearer than this.

[rubenskuhl]

This is the result for WhatsApp desktop app for macOS, version 2.2335.9 . macOS version is Ventura 13.5.2: Darwin H615FHF3DW 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:17:35 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8112 arm64

And this is the exact same message, but as seen from Android WhatsApp. Note 2 TLDs that are linkfied by Mac desktop client (.tube and .shop) but not by Android client:

Also of notice are 3 TLDs that don't linkify in neither platforms: .amazon, .microsoft and .deloitte. Perhaps there is a list somewhere of TLDs that are not in use and that's is one of the dolls ?

[rubenskuhl]

NEWS FLASH. In 2019 the .ss ccTLD was delegated (SUDAN) and it's not linkifying.

nic.xj doesn't exist, but nic.ss does. So if a DNS resolution is being attempted, it's being treated differently in the 2 ccTLDs.

[rambox1963]

NEWS FLASH.
In 2019 the .ss ccTLD was delegated (SUDAN) and it's not linkifying. Attached screens how it's sent from android and received by IOS and also from Web.Whartsapp.com on WIndows 10/Firefox. Different results in all, inconsistencies seem to be happening across platforms and operating systems.


These two are is very clear: It has both screenshots to compare, the sending from Windows and the receiving

"nic.xj doesn't exist, but nic.ss does. So if a DNS resolution is being attempted, it's being treated differently in the 2 ccTLDs.". These screens prove this is incorrect.

[rambox1963]

Sorry, wrong highlight on the last image of the prevous post

[bedfordsean]

Thanks for all of the testing. These platforms all have fundamentally different ways/APIs of identifying if something is a link, which is why you see these differences. We are still investigating this internally to determine what course of action (if any) we take in particular for WhatsApp.

[rambox1963]

Thank you for your response, very enlightning
Here's a final test, summarizes all the inconsistencies in one screen. I understand this not being a priority for you, but the damage done to the domain name industry is enormous. As the uncle of Spiderman says, "with great powers come great responsibilities". META must recognize that to linkify https://nic.vladimirputinwarwar and not nic.africa when the second is Legit and the first not, in an issue that opens the doors to potential fraud.

WEB + ANDROID + IOS

[bedfordsean]

While raising an issue with Android developers is totally in order considering its contribution to the issue, from a product standpoint it would make more sense to me to make this homogenous among platforms.

I don't disagree, however this will just propagate the same issues we have on web to mobile ;-) There's already issues with the PSL in terms of update timing and acceptance in web browsers. Gaining consistency across all platforms requires (at the very least) Microsoft, Apple, and Google to commit to sensible update cycles of the PSL or some other "canonical list". All of that takes time...

We'll continue investigating in the meantime however and see if there's something we can do directly, even if it is Meta-only.

@rambox1963 can you enlighten me more on this statement? What real world fraud cases are enabled by linkifying something that is bogus (and therefore by definition isn't going to take users anywhere without a further compromise to either their DNS servers or their device itself)?

META must recognize that to linkify https://nic.vladimirputinwarwar/ and not nic.africa when the second is Legit and the first not, in an issue that opens the doors to potential fraud.

[rambox1963]

@bedfordsean
https://freename.io/
Many more to come. It's just a matter of time.
I could not agree more about the "Canonical list", that is what I have been proposing as the Universal solution, the one thing that will assure Universal Acceptance.
Here again, ICANN input would be valuable perhaps @rubenskuhl and @dnsguru can help explain why it's a risk that third parties use nomenclature that should be exclusive for the usage of the DNS "https:// or www." .
In my view,( (that is not of an engineer), the DNS is the specter of the Internet, the equivalent of the hertzian waves for Radio and TV but for data. Just as the VHF, UHF, AM, FM waves are concessioned, so is the DNS and just as no company can use 4VHF or 105.9FM at will, companies who are not concessionaries should not be able to trick people by using DNS only nomenclature.
In my view, this could be solved by checking the "canonical list" for ayes and nays. Aye (.africa) linkifies with or without prefix. Nay (.kimjungunrocketman) cannot linkify even if prefixes are added.
Thnx
R

[dnsguru]

There are a number of WEB3 and ALT-ROOT projects out there - the Public Suffix List gets occasional requests for those, but declines them as it adheres to ICP-3, which is the IANA/ICANN Root. The IANA/ICANN root is recognized by default in the ROOT HINTS shipped with every system. It is that strict adherence that has made the PSL the catalog of choice for browsers and the dozens of popular languages that are including it, because it is trusted and predictable, as the root zone database and the IANA have tenured, trusted operation.

Back to the point of the issue .TUBE has uncovered and reported... it really is not a PSL thing, because we are unable to prescribe how developers will use the PSL. or clearly IF, and at what frequency.

This has been an interesting dialog, and it sounds like there are a number of places that application developers and perhaps also OSes could update some stale reference lists of TLDs. Hoping to make sure domain name world and dev world have an 'elegance bridge' is why I have dedicated and volunteered thousands of hours over the last 15 years to help maintain the PSL.

This issue got written up in https://domainincite.com and is getting some attention beyond the typical set of eyeballs.

The crux of the issue here is that the world of ICANN/IANA and the world of developers are separate, and due to the root TLD changes appearing to be a static list it got treated as such on a scrum card, the dev ticked the box next to done, and moved on to the rest of their #dayjobassignments. Developers just being efficient absent being made aware of the impact of a choice.

For application / os etc devs... This handling of a static root TLD list drifting out of synch with the actual root is not a new issue, but it is kind of akin to Y2K bug, without the December 31, 1999 flag day. Eventually the issue flares up and needs attention.

If you are a developer looking at this thread and scratching your head about how to address this issue as you scroll through, here's some idears and the grade professor Jothan would give you, and you lose two grades if it is a one-time snapshot:

D-
Regular expression use (ex: \.[a-z][a-z] = ccTLD, \.[a-z][a-z][a-z] = gTLD) - still seen in the wild, .INFO has been in the penalty box for almost 25 years due to this approach.

C+ "Good Enough"
Basic: use https://data.iana.org/TLD/tlds-alpha-by-domain.txt but make sure you're checking with reasonable frequency (suggest not less than quarterly and recommend every 2 weeks) for deltas
(note: This approach requires punycode/IDNA as IDN TLDs are present but represented in A-Label format and not stored in Unicode in that file like they are in the PSL)

B "Better"
If you do not want to add punycode/IDNA "stuff" but want to have the string matches, use the upper ("#ICANN") section of the PSL,, and pull it at least quarterly for deltas. Higher grade here because you gain more elegance on the existence of subtleties (like co.uk vs just .uk)

A "Best"
Use the whole PSL, as it contains certain subspaces that have become TLD-esque and widely used. Some good, some bad, some ugly.

[hank-nuss]

Just to add that our IDN which appears in the PSL for close to a year is not recognized by Whatsapp in Android:

https://כולנו.ישראל/
which works of course in Chrome and Firefox

[rambox1963]

Oy vavoy.
Thank you for your input.
I apologize for not including IDN's in my analysis but it was on purpose. I wanted to prove that ICANN has not even solved UA for ASCII, how can they expect to tackle the much more complex adoption of ISDN's?. I also did not include ccTLDs. Thanks to your input, we now know this problem is extensive to IDN's and ccTLDs. Would be interesting to see if IDN's delegated prior to November 2015 are recognized by Whatsapp in Android.
I heard yesterday from another registry that X-Twitter is not linkifying newer TLD's. The registry requested I omit his string, but I he demonstrated and he is right, X has the same problem only they affect less TLD's.
I also provided examples on how IOS has the same problem as android, none recognizes all IDN's. The testing needed to find all the inconsistencies can be endless. At the end of all the testing, look at all the inconsistencies that happen across platforms and operating systems.
Eize Balagan, Javal al a Zman

[rambox1963]

Hello, any updates or progress to report on this? Anything else I may be of assistance. I spoke to ICANN, they thanked us for the database, it was very enlightening for them to learn about the 2015 threshold as it helped identify the problem with Android and the exact date when they las updated the PSL. I wonder how can we get Android involved in this as well as Microsoft and Apple. From the tests I made, the three platform show different linkabilities and it's all dependent on the operating system used by the recipient

[rubenskuhl]

I believe the first progress would be to know which criteria and OS libraries are used in each platform, so we know who we start complaining to...

[hank-nuss]

@rambox1963

Coming from you and META, should have a higher specific weight when addressing the root of the issues at ICANN.

As a member of ccNSO of ICANN I would think it will take 6-8 years for any change to be implemented as it gets discussed and reviewed in various sub-committees. You have a better chance to get things fixed directly by engaging with Meta, Microsoft and Google as you are doing here.

[rambox1963]

Yeah @hank-nuss welcome.
@FBNeal @bedfordsean @dnsguru... Today I got commitment from Google to escalate the android issue (their iOS libraries outdated since Nov 2015). If you can provide me with a few lines or pointers outlining the problem, I am pretty sure I can escalate it. I just don't want to waste this chance by not providing the right language. Apparently the timing is right because of some decisions Google has made regarding their presence in the gTLD space and they acknowledge they may have paid too little attention to this in the past years. Hopefully they can be convinced to remedy this.

[bedfordsean]

@rambox1963 I just checked the above Android list linked and it seems it has been updated very recently!

https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/util/Patterns.java;l=114?q=africa

You will now find Africa, Tube, etc on the list. The comment suggests it was updated on 11th September 2023.

This is a positive step and means that at some point of eventual consistency of Android OS releases, link parsing will start to correctly work for these top level domains.

I still think it is worth raising with Google directly though to ensure they have a means of regularly keeping this list up to date (I don't know if this was a one-time manual update).

[case-fastly]

The OS vendors really need some CI/CD to pull in changes from the IANA zones list at some regular interval. Nightly, weekly, whatever.

It would be straightforward to run some "outside in" monitoring (e.g. via jobs on GitHub Actions runners), to keep track of how current the iOS and Android "linking supported zones" lists are.

[rambox1963]

@bedfordsean thank you for the information. I still type nic.africa and nic.tube in whatsapp on android and is still not linkifying. Does it take time to propagate?. I have sent the text to my contact in Google, not sure if it was premature.

@case-fastly welcome and couldn't agree with you more, updates, trustworthy ZONE lists and at least best practices to incorporate those lists and keep them current. The Github Actions runners sounds like a great idea though I wouldn't know how to set those up. But agree completely OS Vendors must update ZOONE lists on regular intervals and there should be a tracking mechanism to assure it happens

[bedfordsean]

It will be used a subsequent code build, it's not something that's regularly download by phones. Best case scenario, but unlikely, it will be included in incremental updates to Android 11, 12, 12.1 and 13. Possibly on Android 14, due in the next few weeks. Worst case scenario would be to only be added in Android 15, due 2024Q3.

Yes, sadly due to the nature of mobile OS - and these are all core OS files, users will be required to update their devices after these code changes make it into an Android release. This will take time, and due to the incredibly complex nature of Android devices, some phones may never even get the update. I mentioned this above in one of my comments.

We are continuing to investigate if we can do something more specific on a per-application basis that would only necessitate e.g. a WhatsApp update vs the entire operating system.

[rambox1963]

@FBNeal and @bedfordsean any news? I am trying to attract Apple and Microsoft to this thread (and also Android even though they already responded by updating their database) so I did a press release yesterday telling the story of how I opened this thread, how Meta eventually responded and assigned professionals to this, how we all did a team effort and unveiled the fact that Android was outdated. Hopefully the press release will encourage participation from APPLE and Microsoft so they too pull in changes from the PSL zones list and commit to do so at some regular interval or as @bedfordsean put it, "Shooting for eventual consistency feels like a sensible path forward". In the meantime, I remain fully committed to this so I am here to assist in any way I can.

[bedfordsean]

We are still researching this internally to understand how/if we would like to make changes on our side. I'd look at it this way:

Two parallel tracks:

1. Pursue Apple/Google/Microsoft to ensure they have a robust process
2. Meta will internally continue to investigate and determine if we would like to take a more direct course of action

Even if we're all agreed on making changes to linkification (not a comprehensive given even internally at Meta), both of these things will take time due to multiple teams/groups being involved and needing to make changes, test "accidental" linkification, etc.

We're looking into it, it will take time though. I'm happy to help push on the OS side of things as well in an aim to drive consistency.

[hank-nuss]

It will be used a subsequent code build, it's not something that's regularly download by phones. Best case scenario, but unlikely, it will be included in incremental updates to Android 11, 12, 12.1 and 13. Possibly on Android 14, due in the next few weeks. Worst case scenario would be to only be added in Android 15, due 2024Q3.

Note that after Google incorporates that, it only goes to phones when phone vendors decide updating/upgrading... so there is a long tail to navigate. Different from Apple, where they are both OS developer and phone vendor, and where they usually push new versions even to older phones.

I use a Google Pixel 6a and just updated v13 to build #TQ3A.230901.001 and the issue in this thread has not been fixed.

[rambox1963]

@hank-nuss thank you for the information, I have sent it to my contact in Google. This contact is not in Android but rather in their domain division and many of their TLD's don't linkify either so I suspect it will have some priority. I also requested to share the link of this thread with someone in their Tech team to have someone from Google contribute to this forum. The more, the merrier, specially if they can make a difference or to say it better, without their participation, the issue in this thread will never be fixed.

[nipoitra80]

No problem share away

…

On Thu, Sep 28, 2023, 1:31 PM rambox1963 ***@***.***> wrote: @hank-nuss <https://github.com/hank-nuss> thank you for the information, I have sent it to my contact in Google. This contact is not in Android but rather in their domain division and many of their TLD's don't linkify either so I suspect it will have some priority. I also requested to share the link of this thread with someone in their Tech team to have someone from Google contribute to this forum. The more, the merrier, specially if they can make a difference or to say it better, without their participation, the issue in this thread will never be fixed. — Reply to this email directly, view it on GitHub <#1807 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKJTOVODIJICQBTZNEYMGBDX4XF75ANCNFSM6AAAAAA2TCC7N4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rambox1963]

here is an article that mentions the work we have done together on this thread
https://domainincite.com/29090-tube-registry-claims-victory-in-linkification-fight
Inside the article there is a link to the original press release
https://www.einnews.com/pr_news/657779910/tube-unveils-a-major-universal-acceptance-issue-and-detonates-exceptional-cooperation-between-meta-and-google
This ought to bring more talent to this pool

[bedfordsean]

Thanks for sharing @rambox1963 - hopefully this will bring more attention from the right groups!

[dnsguru]

Speaking of right groups, it occurs to me that very little of this is really a "PSL thing", and I have to put on the in-scope/out-of-scope focus hat and say we might be at some point be wanting to close this issue as the affected TLDs were already listed, so there was really nothing to do here other than assemble some dialog for things to happen elsewhere.

…

On Fri, Sep 29, 2023, 10:41 AM bedfordsean ***@***.***> wrote: Thanks for sharing @rambox1963 <https://github.com/rambox1963> - hopefully this will bring more attention from the right groups! — Reply to this email directly, view it on GitHub <#1807 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJNMFAL3GZ2QKM7GHATX44B6NANCNFSM6AAAAAA2TCC7N4> . You are receiving this because you were mentioned.Message ID: ***@***.***>