port assignment for new registrations fails about oidctest

[rohe]

OK, I can fix this but I need some time and as I said in an earlier mail I don’t have any continuous time (more then an hour) before Thursday afternoon when I’m back home again. Now, I don’t think this is a major problem since the number of testers that will encounter a problem between now and then is probably not more then a handfull at the most. If you think otherwise tell me so.

…

11 juli 2017 kl. 13:57 skrev Hans Zandbelt ***@***.***>: I think I've got it: the migration script must have been off and even the backup files were wrong because they were created out of earlier migrations that were also using urlencoded keys. I don't think we noticed that before because no-one was using such an issuer/tag combo (yet) on new-op. Additionally I found a bug in displaying URL-unsafe tag names in the list of tags for an entity; that is fixed in 9ca443e <9ca443e> So I think we need to do the migration again, taking into account any new registrations that have been done in the mean time: ***@***.***:/usr/local/oidf/oidc_op$ cat conf_srv.log | grep "/action/create" | grep -v zmartzone | cut -d"?" -f2 | cut -d"&" -f1 | sort | uniq iss= iss=Gmail.well-known iss=http%3A%2F%2F31.18.108.162%3A8080%2F iss=http%3A%2F%2F95.91.228.235%3A8080%2F iss=http%3A%2F%2Fhref.synology.me%3A8080%2F iss=https%3A%2F%2F172.17.0.1%3A9443%2Fcarbon%2F iss=https%3A%2F%2Faccounts.google.com iss=https%3A%2F%2Fapi.developer.vodafone.com iss=https%3A%2F%2Fauth.freedom-id.de iss=https%3A%2F%2Fkrmidentity.azurewebsites.net iss=https%3A%2F%2Flocalhost%3A9443%2Fcarbon%2F iss=https%3A%2F%2Fpeterahl.auth0.com%2F iss=https%3A%2F%2Fsso.nguyenkim.com iss=https%3A%2F%2Ftest.veb.audi-connect.de%2F iss=https%3A%2F%2Fti-test.bbas.no%2Foidc%2F iss=sara.pixelite.nl — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGGPEHnWeTfFveeTkPbx-V0Etx7A7uYks5sM3EygaJpZM4OSoXd>.

[zandbelt]

Also: port various ports seem to have been assigned multiple times to different entities:

<port>    <nr-of-times-assigned>
60001     11
60219     55
60220     49
60813      2

and there was a "low port" 10002 assigned to

https://idcs-3db9795100ea41dbafdcc983db2a0d76.identity.oraclecloud.com/][oauth00": 10002

which was out of bounds and which I have given '61014' now.

Detailed report on duplicate port usage:

60001
   https://krmidentity.azurewebsites.net][default
   https://localhost:9443/carbon/][Test1
   http://href.synology.me:8080/][an-identity
   https://172.17.0.1:9443/carbon/][Test2
   https://accounts.google.com][default
   ][default
   Gmail.well-known][default
   https://peterahl.auth0.com/][default
   https://sso.nguyenkim.com][default
   sara.pixelite.nl][default
   https://idam.metrosystems.net][Code Flow
60219
   https://connectmls-api.mredllc.com/openid][RESO Web API
   https://websso.avencis.net][testting flo
   https://moros.gorill.at/openid/?user=00kaderabe&domain=default][123
   https://guarded-cliffs-8635.herokuapp.com/op][guarded-cliffs-8635
   https://idp-mcomm-dev.dsc.umich.edu/idp][idp-mcomm-dev
   https://id.stndrf.de][https://id.stndrf.de
   https://seed23.gluu.org][Cert seed23
   https://sandbox.biocryptology.net:8443/V1/auth][https://sandbox.biocryptology.net:8443/V1/openid-configuration
   https://beta-mynsid.2keys.ca/auth/oidc/roland/][first
   https://demo.trustedkey.com][foraws
   https://cb1.www3.netamia.net][ns test2
   https://b3dc2781.ngrok.io][Aq Dev
   https://cb1.www3.netamia.net][ns test1
   https://tenant1.idcs.internal.oracle.com:8943/][default00
   https://masdemo12.dev.ca.com:8443/DE298071][OIDC-OTK-4.1
   https://op.certification.openid.net:60000][https://op.certification.openid.net:60000/
   https://local.orchis.syntegrity.com/oauth/][ORCHIS_OIDC_CERTIFICATION_V1
   https://ec2-54-149-88-219.us-west-2.compute.amazonaws.com:14080/openam/oauth2][14.1.0-request-parameter
   https://ec2-54-213-25-148.us-west-2.compute.amazonaws.com:13081/openam/oauth2/][OpenAM 13.5.0 HTTPS
   https://mojeid.fred.nic.cz/][cznic-1
   https://www.linkedin.com/oauth/v2/accessToken][karim
   https://rp.certification.openid.net:8080/pf-cwang-test/private-key-jwt][private-key-jwt
   https://oidc.spomky-labs.com/app_dev.php/][OIDC
   https://gluu.dev.slbcloud.com][test-1
   https://idam-dev.metrosystems.net][default
   https://mojeid.fred.nic.cz/][Fred-mojeid
   https://andrew.onelogin.com/oidc][655622
   https://shib-sp-test.www.umich.edu/oidc-server/][shib-sp-test
   https://accounts.google.com/][Google
   https://plus.google.com/ LiqueanSua][blo
   https://ec2-54-149-88-219.us-west-2.compute.amazonaws.com:13080/openam/oauth2/openid-certification][openid-certification-am14-02-03-2017
   https://www.cbtks.com][Testing website
   https://plus.google.com/u/0/][https://plus.google.com/u/0/
   https://aaa-stage.nlsbph.org/auth/admin/NLS/console][AAA_Stage_001
   https://rp.certification.openid.net:8080/cwang.pf.test/private-key-jwt][private-key-jwt
   https://ec2-54-213-25-148.us-west-2.compute.amazonaws.com:13081/openam/oauth2][OpenAM 13.5.0
   https://accounts.google.com/][60218
   https://thawing-woodland-72588.herokuapp.com][thawing-woodland-72588
   https://github.com/jmakanjuola][demo-one
   https://accounts.solucaoadapta.com.br/feanor][Adapta
   https://ec2-54-213-25-148.us-west-2.compute.amazonaws.com:13080/openam/oauth2][OpenAM 13.0.0
   https://accounts.google.com/o/oauth2/v2/auth][karim
   https://connect.int-passeport.swisscom.ch][Metis+PreProd2
   https://isic.vzgcis.com:443/openam/oauth2]["https://isic.vzgcis.com:443/openam/oauth2
   https://190.180.0.200:8443][190_180_0_200 cert2
   https://yodata.me][test-2
   https://id.guldner.eu][Test
   https://idp.inid.int.inpoclab.com/op][test05072017
   https://gate.panva.cz][lambda-instance
   https://mojeid.fred.nic.cz/oidc/][cznic-5
   https://idp.inid.int.inpoclab.com/][Test_INId
   https://account.hoosierhometown.com:443/cas/oauth2][cas2
   https://evening-falls-86639.herokuapp.com][machined-oic
   https://ri-openidconnect-dev.eu-gb.mybluemix.net/openid-connect-server-webapp][1
   https://openid.cegepth.info/prof2g4][https://openid.cegepth.info/simpleid/
60220
   https://oidc.tanet.edu.tw][oidc-twedu
   https://bitminter.com/login][kader786
   https://smartpass.government.ae/secure/oauth2/authorize][SPA-OID-HIS
   https://oidc-provider-certification.herokuapp.com][oidc-provider-certification
   https://oidc.hwslabs.vmware.com/SAAS/auth][oidc.hwslabs.vmware.com
   https://www.guldner.eu/openid/][Test
   https://login.windows.net][test
   https://pod.afox.me][test1
   https://masdemo12.dev.ca.com:8443/DE298071][OIDC-OTK-4100
   https://sandbox.biocryptology.net:8443/V1/openid-configuration][https://sandbox.biocryptology.net:8443/V1/openid-configuration
   https://openid.worketc.com/Provider][1
   https://sandbox.biocryptology.net:8443/V1/openid-configuration][V1
   https://api.jp.office.doo.net][60988
   https://sso.unbugsolution.com][test1
   https://www.ytpak.com/watch?v=NsUFNU_HsR8][Future Salvation
   https://oidc.orchis.syntegrity.com/oauth/][ORCHIS_OIDC_CERTIFICATION_V1
   https://account.hoosierhometown.com/cas/oauth2/][cas
   https://aac-test.test.aeroflot.ru/][1
   https://op.certification.openid.net:60000/][https://op.certification.openid.net:60000/wowzerwize
   https://login.microsoftonline.com/b4ea3de6-839e-4ad1-ae78-c78e5c0cdc06/v2.0][Proxy+AAD
   https://jp.api.office.doo.net][doo-office-jp
   https://84.192.125.241/idhub/oidc][idhub
   https://www.bitium.com/bitium.com][test1
   https://edubase-de.test-weedu.ch/sts/openid][Test1
   https://masdemo12.dev.ca.com:8443/US336497][OIDC-OTK-4100
   https://mojeid.fred.nic.cz/oidc][cznic-4
   https://sts.windows.net/{tenantid}][1
   https://isic.vzgcis.com/openam/oauth2/][https://isic.vzgcis.com/openam/oauth2
   https://cf7afbd7.ngrok.io/][Aq Dev
   https://yodata.me/.well-know/openid-configuration][test-1
   https://isic.vzgcis.com/openam][https://isic.vzgcis.com/openam/
   https://stage.smartpass.government.net.ae:443/secure/oauth2/TRA][eidNationality
   https://login.xero.com/openid][Login-production
   https://accounts.google.com/o/oauth2/v2/auth.][Test ID
   https://dev-126996.oktapreview.com][oktaTestInstance
   https://connect.openid4.us:5443/phpOp][phpOp1
   https://atlasidentity.okta.com][Okta Test
   https://sandbox.biocryptology.net:8443/V1/openid-configuration][TESTISSWIPE
   https://cb1.www3.netamia.net][ns test3
   https://login.platform.absapp.net][prod
   https://connect-op.herokuapp.com][https://connect-op.herokuapp.com
   https://mojeid.fred.nic.cz/oidc/][cznic-6
   https://example.com][default
   https://facebook.com][karim
   https://oidc.spomky-labs.com/][OIDC Spomky-Labs
   https://openid-connect.onelogin.com][655622
   https://win10-testvm-cf.tdv.technodat.co.at/oidc][23
   https://example.com][CTTT
   https://test:[email protected]/auth][dev login
60813
   http://62.109.14.142:8080/openidservertest][123
   http://62.109.14.142:8080/OpenIdServerTest][123

[rohe]

Wonder what's so special with 60219 and 60220 ??

[zandbelt]

I do think I've found the correlation: it seems that the entries listed above have one thing in common: they have 2 entries, one with their "plain" name, and one with their "urlencoded" name. The first one has the new port. I guess 60219 was the first free port after the migration.

So it may just be a matter of finding where the url-encoding goes wrong, fix that and remove the entries with the "plain" keys.

Not sure if the issue is with the migration script or the current code.

[zandbelt]

It looks like unquoting is done very specifically here:
https://github.com/openid-certification/oidctest/blob/master/src/oidctest/ass_port.py#L19
so I guess the migration script should have created plain keys rather than url-encoded ones?
(I must admit that I'm confused since the assigned_ports.json.backup file does have url-encoded keys...)

[zandbelt]

I think I've got it: the migration script must have been off and even the backup files were wrong because they were created out of earlier migrations that were also using urlencoded keys. I don't think we noticed that before because no-one was using such an issuer/tag combo (yet) on new-op.

Additionally I found a bug in displaying URL-unsafe tag names in the list of tags for an entity; that is fixed in 9ca443e

So I think we need to do the migration again, taking into account any new registrations that have been done in the mean time:

hzandbelt@new-op:/usr/local/oidf/oidc_op$ cat conf_srv.log | grep "/action/create" | grep -v zmartzone | cut -d"?" -f2 | cut -d"&" -f1 | sort | uniq

iss=
iss=Gmail.well-known
iss=http%3A%2F%2F31.18.108.162%3A8080%2F
iss=http%3A%2F%2F95.91.228.235%3A8080%2F
iss=http%3A%2F%2Fhref.synology.me%3A8080%2F
iss=https%3A%2F%2F172.17.0.1%3A9443%2Fcarbon%2F
iss=https%3A%2F%2Faccounts.google.com
iss=https%3A%2F%2Fapi.developer.vodafone.com
iss=https%3A%2F%2Fauth.freedom-id.de
iss=https%3A%2F%2Fkrmidentity.azurewebsites.net
iss=https%3A%2F%2Flocalhost%3A9443%2Fcarbon%2F
iss=https%3A%2F%2Fpeterahl.auth0.com%2F
iss=https%3A%2F%2Fsso.nguyenkim.com
iss=https%3A%2F%2Ftest.veb.audi-connect.de%2F
iss=https%3A%2F%2Fti-test.bbas.no%2Foidc%2F
iss=sara.pixelite.nl

[zandbelt]

All done now with the new migration tools; deployed in RP 1.0.3 and OP 1.0.5:
https://github.com/openid-certification/oidctest/releases/tag/v1.0.5