## Toggle quote (43 lines)

> Timothy Sample <samplet@ngyro.com> writes:

>

> > I’m still looking into this, but I wanted to quickly post this

> > reproducer for the Guile bug:

> >

> > (use-modules (ice-9 regex))

> > (define str

> > "\U101514\U103ab0\U0f6e6e\U02e278\U01d9eb\U10b996\U1089b5\uea15\U0fa074\U101e41\U02e330\u0177\u2492")

> > (match:substring (string-match "[0-8]+" str))

> >

> > This triggers the out-of-range error when run with “LC_ALL=C”.

>

> It turns out that all that’s needed is the last code point, which is

> “Number Eleven Full Stop”, or ‘⒒’. When Guile converts this to an ASCII

> C string using ‘u32_conv_from_encoding’, it becomes “11.”. The regex

> (“[0-8]+”) matches the “11” part with start index 0 and end index 2.

> The ‘fixup_multibyte_match’ function does nothing (it only matters when

> the locale encoding is multibyte) [1]. Guile then builds the match

> vector with the original string but keeps the ASCII offsets. In other

> words, it thinks the match substring goes from 0 to 2 in a single code

> point string:

>

> ,use (ice-9 regex)

> (string-match "11" "\u2492")

> => #("\u2492" (0 . 2))

>

> I’m not sure there’s any way to solve this nicely in Guile. It would be

> clearer if the match vector included the string as libc matched it, but

> it’s still surprising that the match happens with a different string.

>

> In Disarchive, I can rewrite the generator without regex. I’ll do that

> and see what I can do about the “Gave up!” issue.

>

> [1] It works on the converted-to-ASCII C string, which means that the

> byte offsets and code point offsets are the same. Hence, it has nothing

> to do.

>

>

> -- Tim

>

>

>

What happens with these?

(code ppoints in decimal)

8554 _Ⅺ_ "ROMAN NUMERAL ELEVEN"

8570 _ⅺ_ "SMALL ROMAN NUMERAL ELEVEN"

9322 _⑪_ "CIRCLED NUMBER ELEVEN"

9342 _⑾_ "PARENTHESIZED NUMBER ELEVEN"

9362 _⒒_ "NUMBER ELEVEN FULL STOP"

9451 _⓫_ "NEGATIVE CIRCLED NUMBER ELEVEN"

13155 _㍣_ "IDEOGRAPHIC TELEGRAPH SYMBOL FOR HOUR ELEVEN"

13290 _㏪_ "IDEOGRAPHIC TELEGRAPH SYMBOL FOR DAY ELEVEN"

I would argue that none of these should be "decoded" into ascii polyglyphs

since they are atomic character glyphs. IMO It is over-eager transformation

to make them into ascii polyglyphs.

/Super/sub/-script placement metadata is another thing to consider --

"decode" to ascii art?? ;-)

Unicode characters representing mathematical values in

other languages are different. Those are subject to natural language

translation with locale-dependent semantics.

These might be candidates for that?:

(code points in decimal)

8544 _Ⅰ_ "ROMAN NUMERAL ONE"

8545 _Ⅱ_ "ROMAN NUMERAL TWO"

8546 _Ⅲ_ "ROMAN NUMERAL THREE"

8547 _Ⅳ_ "ROMAN NUMERAL FOUR"

8548 _Ⅴ_ "ROMAN NUMERAL FIVE"

8549 _Ⅵ_ "ROMAN NUMERAL SIX"

8550 _Ⅶ_ "ROMAN NUMERAL SEVEN"

8551 _Ⅷ_ "ROMAN NUMERAL EIGHT"

8552 _Ⅸ_ "ROMAN NUMERAL NINE"

8553 _Ⅹ_ "ROMAN NUMERAL TEN"

8554 _Ⅺ_ "ROMAN NUMERAL ELEVEN"

8555 _Ⅻ_ "ROMAN NUMERAL TWELVE"

8556 _Ⅼ_ "ROMAN NUMERAL FIFTY"

8557 _Ⅽ_ "ROMAN NUMERAL ONE HUNDRED"

8558 _Ⅾ_ "ROMAN NUMERAL FIVE HUNDRED"

8559 _Ⅿ_ "ROMAN NUMERAL ONE THOUSAND"

8560 _ⅰ_ "SMALL ROMAN NUMERAL ONE"

8561 _ⅱ_ "SMALL ROMAN NUMERAL TWO"

8562 _ⅲ_ "SMALL ROMAN NUMERAL THREE"

8563 _ⅳ_ "SMALL ROMAN NUMERAL FOUR"

8564 _ⅴ_ "SMALL ROMAN NUMERAL FIVE"

8565 _ⅵ_ "SMALL ROMAN NUMERAL SIX"

8566 _ⅶ_ "SMALL ROMAN NUMERAL SEVEN"

8567 _ⅷ_ "SMALL ROMAN NUMERAL EIGHT"

8568 _ⅸ_ "SMALL ROMAN NUMERAL NINE"

8569 _ⅹ_ "SMALL ROMAN NUMERAL TEN"

8570 _ⅺ_ "SMALL ROMAN NUMERAL ELEVEN"

8571 _ⅻ_ "SMALL ROMAN NUMERAL TWELVE"

8572 _ⅼ_ "SMALL ROMAN NUMERAL FIFTY"

8573 _ⅽ_ "SMALL ROMAN NUMERAL ONE HUNDRED"

8574 _ⅾ_ "SMALL ROMAN NUMERAL FIVE HUNDRED"

8575 _ⅿ_ "SMALL ROMAN NUMERAL ONE THOUSAND"

8576 _ↀ_ "ROMAN NUMERAL ONE THOUSAND C D"

8577 _ↁ_ "ROMAN NUMERAL FIVE THOUSAND"

8578 _ↂ_ "ROMAN NUMERAL TEN THOUSAND"

8579 _Ↄ_ "ROMAN NUMERAL REVERSED ONE HUNDRED"

8581 _ↅ_ "ROMAN NUMERAL SIX LATE FORM"

8582 _ↆ_ "ROMAN NUMERAL FIFTY EARLY FORM"

8583 _ↇ_ "ROMAN NUMERAL FIFTY THOUSAND"

8584 _ↈ_ "ROMAN NUMERAL ONE HUNDRED THOUSAND"

12321 _〡_ "HANGZHOU NUMERAL ONE"

12322 _〢_ "HANGZHOU NUMERAL TWO"

12323 _〣_ "HANGZHOU NUMERAL THREE"

12324 _〤_ "HANGZHOU NUMERAL FOUR"

12325 _〥_ "HANGZHOU NUMERAL FIVE"

12326 _〦_ "HANGZHOU NUMERAL SIX"

12327 _〧_ "HANGZHOU NUMERAL SEVEN"

12328 _〨_ "HANGZHOU NUMERAL EIGHT"

12329 _〩_ "HANGZHOU NUMERAL NINE"

12344 _〸_ "HANGZHOU NUMERAL TEN"

12345 _〹_ "HANGZHOU NUMERAL TWENTY"

12346 _〺_ "HANGZHOU NUMERAL THIRTY"

Just my intuitive reaction, no academic creds to back it up ;)

--

Regards,

Bengt Richter