jsdom / tr46

An implementation of the Unicode UTS #46: Unicode IDNA Compatibility Processing.

tr46

An JavaScript implementation of Unicode Technical Standard #46: Unicode IDNA Compatibility Processing.

API

toASCII(domainName[, options])

Converts a string of Unicode symbols to a case-folded Punycode string of ASCII symbols.

Available options:

• checkBidi
• checkHyphens
• checkJoiners
• ignoreInvalidPunycode
• transitionalProcessing
• useSTD3ASCIIRules
• verifyDNSLength

toUnicode(domainName[, options])

Converts a case-folded Punycode string of ASCII symbols to a string of Unicode symbols.

Available options:

• checkBidi
• checkHyphens
• checkJoiners
• ignoreInvalidPunycode
• transitionalProcessing
• useSTD3ASCIIRules

Options

checkBidi

Type: boolean Default value: false When set to true, any bi-directional text within the input will be checked for validation.

checkHyphens

Type: boolean Default value: false When set to true, the positions of any hyphen characters within the input will be checked for validation.

checkJoiners

Type: boolean Default value: false When set to true, any word joiner characters within the input will be checked for validation.

ignoreInvalidPunycode

Type: boolean Default value: false When set to true, invalid Punycode strings within the input will be allowed.

transitionalProcessing

Type: boolean Default value: false When set to true, uses transitional (compatibility) processing of the deviation characters.

useSTD3ASCIIRules

Type: boolean Default value: false When set to true, input will be validated according to STD3 Rules.

verifyDNSLength

Type: boolean Default value: false When set to true, the length of each DNS label within the input will be checked for validation.

tr46's Issues

Add a README and publish 1.0.0

0.xs make me sad

Work around V8's normalize failures

Per https://bugs.chromium.org/p/v8/issues/detail?id=4654 we need a version of normalize that doesn't suck. This blocks jsdom/whatwg-url#36 from passing its tests

Punycode dependency

Punycode is deprecated, please remove or add alternative.

https://nodejs.org/api/punycode.html

Use ES5 instead of ES6 for target

Including your npm package in other projects without babelification can break.

For example function findStatus(val, { useSTD3ASCIIRules }) { breaks on IE11.

See https://stackoverflow.com/questions/29738381/how-to-publish-a-module-written-in-es6-to-npm

Create LICENSE file

Would you please create a LICENSE file with your copyright information and the text of the MIT license? The MIT license actually requires that the license text accompany the source code. Thanks!

Punycode deprecated

The punycode package is deprecated, which generates a warning on start-up.
The suggested workaround on the package's repo is to import it as require('punycode/') instead of require('punycode').
In my testing this seems to resolve the warning, at least in my case.

toAscii and Unicode tests are failing

Description

Two tests fail.

✖ failing tests:

✖ /home/../tr46/test/toascii.js (95.585125ms)
  Error [ERR_TAP_LEXER_ERROR]: Unexpected character: ‍ at line 1, column 0
      at Socket.emit (node:events:511:28) {
    code: 'ERR_TAP_LEXER_ERROR'
  }

✖ /home/../tr46/test/unicode.js (288.905067ms)
  Error [ERR_TAP_LEXER_ERROR]: Unexpected character: ̀ at line 1, column 0
      at Socket.emit (node:events:511:28) {
    code: 'ERR_TAP_LEXER_ERROR'
  }

Reproduce

1. Clone tr46.
2. npm run test
3. Verify failing tests.

Check if usage of split fails in Unicode

Currently split is used to break apart the labels of the domain, however split might break apart at the wrong positions if a dot is position right after a surrogate character(?) maybe/probably.

Remove "punycode" dependency as it is deprecated.

Punycode is deprecated.

(node:53980) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

It's imported here:

tr46/index.js

Line 3 in d6cd9a7

const punycode = require("punycode/");

const punycode = require("punycode/");

It's also used a few times inside of the index.js file.

Stripping the FE0F Variation selector 16 modifier when resolving, causing major issues in the cryptocurrency space

I believe there is a discrepancy that is not allowing for standardizing normalisation between several large platforms and markets in the space

It appears someone is not applying the stand in compliance or doing things slightly different
most specifically the library strips FE0F code away when resolving displaying from a few native emojis with joiners and FE0F

https://unicode.org/Public/emoji/14.0/emoji-test.txt
According to this, FE0F should be preserved to be "fully qualified", which is theoretically the more adaptive version, this is currently causing major issues with decentralized emoji domains (.eth) in Opensea , Etherscan and Metawallet, everyone is using a different library and there is no standardization) more specifically Metawallet (which uses this library) resolves differently than Opensea and Etherscan

Taking the example of emoji (genie male) taken directly from unicode.org

1F9DE 200D 2642 FE0F ; fully-qualified # 🧞‍♂️ E5.0 man genie

If you were to take the man genie 🧞‍♂️ ( Hand held devices/Discord/Twitter etc etc will default to 1F9DE 200D 2642 FE0F) large markets like Opensea and Etherscan too. However after this script runs, the strip takes place, we are left with 1F9DE 200D 2642 It strips the FE0F from the man genie turning it into a minimally-qualified emoji. This is disrupting all of the emoji .eth domains, their connection with Metawallet addresses.

It would appear the wisest solution would hopefully try to apply the stand and more widely supported emoji version which in this case is the fully-qualified 1F9DE 200D 2642 FE0F the variation selector 16 modifier shouldn't be stripped away, as its responsible for current and likely future generations of aesthetically different emojis while being the arguably better choice for Unicode stand

Regards

Switch to ESM and require node v12.20

time to switch require for import?

Split out findStatus() to separate file (for browserify)

...so that it can be aliased to always return "valid" for lighter browser builds. I can PR this if you agree with the decision.

Wrong answer on "\u001fx", apparently

Continued from web-platform-tests/wpt#4504 (comment) . Repro:

"use strict";
const tr46 = require(".");
const util = require("util");

const input = "\u001fx";
const output = tr46.toASCII(input, false, tr46.PROCESSING_OPTIONS.TRANSITIONAL, false);
console.log(util.inspect(output));

outputs 'xn--\u001fx-' whereas it should output just '\u001fx' according to @annevk's reading in web-platform-tests/wpt#4504 (comment) . Updates to follow...

[Deprecation] `punycode` deprecation in Node 21+

The problem here is that punycode from standard library is now deprecated in node version 21+.

This shows the following error:

(node:24289) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)

Unicode test normalization might be wrong

This line looks very, very weird to me: https://github.com/Sebmaster/tr46.js/blob/f3cd434ae5997e3132c2214b41d0bc2ae5418c21/test/unicode.js#L16

Error on iPad Air 2, iOS 8 in safari

Get an error on iPad Air 2 iOS 8 in safari browser
Unexpected keyword 'const'. Const declarations are not supported in strict mode.

    "../../node_modules/tr46/index.js": /*!***************************************************************!*\
  !*** /PATH/node_modules/tr46/index.js ***!
  \***************************************************************/
    /*! no static exports found */
    /***/
    (function(module, exports, __webpack_require__) {

        "use strict";

        const punycode = __webpack_require__(/*! punycode */
        "../../node_modules/node-libs-browser/node_modules/punycode/punycode.js");
        const regexes = __webpack_require__(/*! ./lib/regexes.js */
        "../../node_modules/tr46/lib/regexes.js");
        const mappingTable = __webpack_require__(/*! ./lib/mappingTable.json */
        "../../node_modules/tr46/lib/mappingTable.json");

        function containsNonASCII(str) {
            return /[^\x00-\x7F]/.test(str);
        }

Deprecation issue with the 'punycode' module

Hello,

I am encountering a deprecation issue with the punycode module in your project. The warning message is as follows:

(node:19880) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead.

Could you consider updating your project to use a more recent version of punycode (e.g., [email protected])?

Thank you in advance for your help.

Best regards,
valorisa

488 failing tests on master?

Seems bad...

The `punycode` module is deprecated.

(node:163221) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead.

https://nodejs.org/api/punycode.html

https://www.npmjs.com/package/punycode

Use require('punycode/') to import userland modules rather than core modules.

Getting punycode deprecation warning

Hi I am getting punycode deprecation warning. Please fix this

It is not bundled with Rollup

tr46 is not bundled with Rollup. It seems that the generated JSON file is not imported correctly.

Here is the reproduction:
https://github.com/atom-ide-community/terminal/tree/rollup-bump

npm install
npm run build

[!] Error: 'default' is not exported by node_modules\tr46\lib\mappingTable.json, imported by C:\terminal\node_modules\tr46\lib\mappingTable.json?commonjs-proxy
https://rollupjs.org/guide/en/#error-name-is-not-exported-by-module
C:\terminal\node_modules\tr46\lib\mappingTable.json?commonjs-proxy (1:8)
1: export {default} from "C:\\terminal\\node_modules\\tr46\\lib\\mappingTable.json";
           ^
Error: 'default' is not exported by node_modules\tr46\lib\mappingTable.json, imported by C:\terminal\node_modules\tr46\lib\mappingTable.json?commonjs-proxy
    at error (C:\terminal\node_modules\rollup\dist\shared\rollup.js:5252:30)
    at Module.error (C:\terminal\node_modules\rollup\dist\shared\rollup.js:9811:16)
    at handleMissingExport (C:\terminal\node_modules\rollup\dist\shared\rollup.js:9700:28)
    at Module.getVariableForExportName (C:\terminal\node_modules\rollup\dist\shared\rollup.js:9987:24)
    at Module.traceVariable (C:\terminal\node_modules\rollup\dist\shared\rollup.js:10202:45)
    at ModuleScope.findVariable (C:\terminal\node_modules\rollup\dist\shared\rollup.js:8732:39)
    at FunctionScope.findVariable (C:\terminal\node_modules\rollup\dist\shared\rollup.js:2782:38)
    at ChildScope.findVariable (C:\terminal\node_modules\rollup\dist\shared\rollup.js:2782:38)
    at MemberExpression.bind (C:\terminal\node_modules\rollup\dist\shared\rollup.js:6436:49)
    at BinaryExpression.bind (C:\terminal\node_modules\rollup\dist\shared\rollup.js:2868:23)

npm ERR! code 1
npm ERR! path C:\terminal
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c "npm run clean && cross-env NODE_ENV=production cross-env BABEL_KEEP_MODULES=true rollup -c"

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\aminy\AppData\Roaming\npm-cache\_logs\2020-11-06T11_00_18_550Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] prepare: `npm run build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] prepare script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\aminy\AppData\Roaming\npm-cache\_logs\2020-11-06T11_00_18_595Z-debug.log

replace deprecated punycode dependency

Node v21 starts to loudly protest about punycode being deprecated:

(node:69587) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead.

Downstream dependencies would surely appreciate an update addressing this.

The `punycode` module is deprecated.

DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead
please use punycode.js

Missing LICENSE file in the repo of ver 0.0.3

Currently I'm using tr46 version 0.0.3, but can't find a LICENSE file in the repo of ver 0.0.3.
Could you add the LICENSE file in the repo of ver 0.0.3?
Or is it possible to use the LICENSE file of the latest version as the license for ver 0.0.3?
Thanks in advance.