iso8859-1 vs windows-1252

[hashseed]

This is somewhat related to #13722, but not quite.

Wikipedia contains the gist:

It is very common to mislabel Windows-1252 text with the charset label ISO-8859-1. [...] Most modern web browsers and e-mail clients treat the media type charset ISO-8859-1 as Windows-1252 to accommodate such mislabeling. This is now standard behavior in the HTML5 specification, which requires that documents advertised as ISO-8859-1 actually be parsed with the Windows-1252 encoding.

Chromium's ICU interprets "iso8859-1" to mean Windows-1252. Node.js does not. The WHATWG spec suggests Chromium's behavior to be correct.

The consequence of all of this is that when I build Node.js with Chromium's ICU, test/parallel/test-icu-transcode.js fails due to the character "€", which Windows-1252 includes, but ISO-8859-1 does not.

I propose:

1. Change the test to pass for both interpretations.
2. Conform to Chromium's behavior.

[hashseed]

@srl295

[joyeecheung]

The consequence of all of this is that when I build Node.js with Chromium's ICU, test/parallel/test-icu-transcode.js fails due to the character "€"

From a glance of the test I couldn't see how the encoding differences would come into play if the test itself was decoded properly. Was the test file stored in Windows-1252? And then got interpreted as utf8 by the CJS loader?

[hashseed]

The test loads the euro sign as utf8 and converts to latin1. With iso8859-1 it doesn't encode so it gets replaced by "?".

[richardlau]

Is this affected by full-icu?

[richardlau]

Also #13722.

[hashseed]

I can check again, but iirc full icu doesn't fix this issue. If it did, the full icu build would fail the above mentioned test.

[hashseed]

Same actually also applies to "ascii". According to WHATWG it is to be interpreted as Windows-1252 as well.

[mhdawson]

@srl295 as our ICU expert.

[TimothyGu]

We have a dedicated WHATWG text encoder/decoder util.TextDecoder and util.TextEncoder, which should strictly follow the semantics of the Encoding Standard. buffer.transcode() should indeed just fall back on whatever the linked ICU does. I believe #25866 is a good change because of this.

[hashseed]

Test fix landed.