Comments (5)
Thanks for the comments!
Regarding the requirement for a handle on entities, this was for a number of reasons when I was testing the library with various RDAP servers:
- A suprising number of RDAP servers are not fully compliant with the specification and return weird formats
- The handle is most commonly used to look up the registrant to try and find details on the entity, for example, you look up an allocation which returns entities, ideally to get any information on the registrant you need then do a query for the registrant entity and this requires the handle for the API call as the entity RDAP queries are all referenced by handle
That limitation can be removed, however, results were "more useful" with it in place as almost all of the records that were caught by the "do you have a handle" check were junk or pointless (empty fields, results with just notes in, placeholders etc.). It's not totally accurate as per the spec as you've noticed, but it seemed to be more real world useful when attempting to actually query objects.
I could make it a config flag / opt-in param as well if that would be suitable. Can you give a use case where this check might filter out a record that shouldn't be filtered out? You can always just use raw=True
as well and bypass the whoisit
parser entirely.
Feel free to submit a PR to add the .ch
RDAP endpoint, the format is pretty straight forward in https://github.com/meeb/whoisit/blob/main/whoisit/overrides.py
from whoisit.
I experienced the same, that not all rdap providers are complient to the minimal rdap_level0 and do not follow the spec.
True, the parsing with raw=True
is a workaround but misses the point of a handy summary of the roles.
I'm curently re-reading the specification and I get the conclusion, that rdap from TLD-CH does not quite follow the spec, as the registrar role contact entity does not contain a handle.
In response to registrar
queries, the returned RDAP response MUST be an entity with registrar role, with a handle and valid elements fn, adr, tel, email
With the current code you get in the best case a registrar, tech, admin contact entity or in the worst case, nothing :)
Here an example:
>>> import whoisit
>>> whoisit.overrides.iana_overrides['domain'].update({'ch': ['https://rdap.nic.ch/']})
>>> whoisit.bootstrap(overrides=True)
True
>>> response = whoisit.domain('test.ch')
>>> import pprint
>>> pprint.pprint(response)
{'copyright_notice': '',
'description': [],
'entities': {},
'expiration_date': None,
'handle': 'TEST.CH',
'last_changed_date': None,
'name': 'test.ch',
'nameservers': ['ns1.cyon.ch', 'ns2.cyon.ch'],
'parent_handle': '',
'registration_date': datetime.datetime(1996, 11, 7, 0, 0),
'rir': '',
'status': ['active'],
'terms_of_service_url': '',
'type': 'domain',
'url': '',
'whois_server': ''}
>>> response = whoisit.domain('test.ch', raw=True)
>>> pprint.pprint(response)
{'entities': [{'objectClassName': 'entity',
'roles': ['registrar'],
'url': 'https://www.kreativmedia.ch',
'vcardArray': ['vcard',
[['version', {}, 'text', '4.0'],
['org', {}, 'text', 'Kreativ Media GmbH'],
['adr',
{},
'text',
['',
'',
'Höschgasse 45',
'Zürich',
'',
'8008',
'CH']],
['kind', {}, 'text', 'group']]]}],
'events': [{'eventAction': 'registration', 'eventDate': '1996-11-07'}],
'handle': 'test.ch',
'ldhName': 'test.ch',
'nameservers': [{'ipAddresses': {'v4': ['194.126.200.5'],
'v6': ['2a01:ab20::2']},
'ldhName': 'ns1.cyon.ch',
'objectClassName': 'nameserver'},
{'ipAddresses': {'v4': ['91.206.24.2'],
'v6': ['2001:67c:234::2']},
'ldhName': 'ns2.cyon.ch',
'objectClassName': 'nameserver'}],
'notices': [{'description': ['This information is subject to an Acceptable '
'Use Policy.'],
'links': [{'href': 'https://www.nic.ch/terms/aup/',
'rel': 'alternate',
'type': 'text/html'}],
'title': 'Acceptable Use Policy (AUP)'}],
'objectClassName': 'domain',
'rdapConformance': ['rdap_level_0'],
'secureDNS': {'delegationSigned': False},
'status': ['active'],
'switch_name': 'test.ch'}
in this example you see 'entities': {},
entities is empty, as in the raw response there is a entity with role registrar
.
Due to the swiss law: Since 1 January 2021, personal data associated with registered domain names is no longer disclosed. Information about holders of domain names can only be obtained in exceptional cases.
thanks for the discussion. I will probably stick to parsing the raw data.
and ping the TLD for the inclusion for the handle :)
from whoisit.
Thanks for the example. I've just released v2.4.2
which you can upgrade to now. This includes the following commits:
The behaviour now is:
>>> from pprint import pprint
>>> import whoisit
>>> whoisit.bootstrap(overrides=True)
True
>>> results = whoisit.domain('test.ch')
>>> pprint(results)
{'copyright_notice': '',
'description': [],
'entities': {'registrar': [{'name': 'Kreativ Media GmbH',
'type': 'entity',
'url': 'https://www.kreativmedia.ch'}]},
'expiration_date': None,
'handle': 'TEST.CH',
'last_changed_date': None,
'name': 'test.ch',
'nameservers': ['ns1.cyon.ch', 'ns2.cyon.ch'],
'parent_handle': '',
'registration_date': datetime.datetime(1996, 11, 7, 0, 0),
'rir': '',
'status': ['active'],
'terms_of_service_url': '',
'type': 'domain',
'url': '',
'whois_server': ''}
I re-ran some checks and the handle check was OK to remove, it was added quite early on and other checks to remove junk results were added afterwards so it wasn't doing a great deal other than filtering non-spec compliant entities.
from whoisit.
thank you.
just for clarification what i learned from my research:
rdapConformance
withrdap_level_0
specifies the conformance with RFC7483(JSON Responses for the Registration Data Access Protocol (RDAP)) and there is defined:
The entity object class can contain the following members
note: CAN. not only for the object entity
class, its "can" for all object members.
rdapConformance
withicann_rdap_response_profile_0
responds to the RDAP Response Profile (Version February 2019) which is mandatory for gTLDs, and I referenced earlier in my comments.
TLD registries and registrars are required to implement an RDAP service by 26 August 2019. ICANN org continues to work with gTLD registries and registrars to implement a service-level agreement and registry reporting requirements for RDAP.
https://www.icann.org/rdap
so I conclude that all gTLDs should have a well defined rdap response profile conforming to icann_rdap_response_profile_0
.
All the other TLDs, and in the most cases the ccTLDs implementations, have no requirements what they must or should return, only what properties/members are available and can be used.
from whoisit.
Thanks, that's generally what I'd discovered as well. The whoisit
parser likely won't ever be fully compliant given it has to handle some potentially invalid upstream responses. Feel free to report any other issues if you find any with data being over or under extracted.
from whoisit.
Related Issues (14)
- Some rdap servers seem to exhibit weak security, which throws errors HOT 2
- Parsing of address, telephone and individual name ("n") field HOT 8
- IP address lookup returns only the first CIDR on the list HOT 3
- No valid RDAP service URLs could be parsed HOT 2
- whoisit does not find the RDAP endpoint for UK domains HOT 4
- Mutable defaults parameters en request wrapper HOT 1
- Failure when sld require different url than tld HOT 7
- Implement an async mode HOT 3
- Issue with urllib3 using requests 2.31.0 HOT 5
- urllib3 2.x complains about DEFAULT_CIPHERS HOT 5
- RDAP server connection error handling (for discussion) HOT 6
- Feature request: Releases HOT 3
- setup.py itself depends on python-dateutil HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from whoisit.