mumi does not correctly display (some?) non-ascii characters

Done

Details

6 participants

Arun Isaac
Felix Lechner
Christopher Baines
Maxim Cournoyer
noe
Tomas Volf

Owner: unassigned

Submitted by: Tomas Volf

Severity: normal

Blocked by

Debbugs page

Tomas Volf wrote 1 years ago

Recipients:(address . bug-mumi@gnu.org)

Message-ID:Zds6yhPkZ0Id6SAT@ws

Hi,

when I compare mumi page[0] with debbugs page[1], the from field displays "???"

in mumi, but "宋文武" in debbugs.

Have a nice day,

Tomas Volf

0: https://issues.guix.gnu.org/57268

1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57268

There are only two hard things in Computer Science:

cache invalidation, naming things and off-by-one errors.

Felix Lechner wrote 10 months ago

[PATCH] Convert HTML to UTF-8 ourselves. (Closes: #69381)

Recipients:(address . 69381@patchwise.org)

Message-ID:20240514231249.18303-1-felix.lechner@lease-up.com

This fixes a host of encoding issues in Mumi, including the diff

problems that are not mentioned in the bug. An example is here:

https://issues.guix.gnu.org/63508#4

The procedure version may one day be more efficient but does not work.

Based on comments in the Guile source code, the procedure style may

one day enable more advanced response formats. The author is unclear

as to why the procedure does not work. There may be a complex

interaction involving the response headers.

A preview of this code is live at patchwise.org.

The solution of this bug may depend on the patch in Bug#70907. This

patch furthermore depends on the patch in Bug#70906, but the solution

of the bug may not.

---

mumi/web/render.scm | 9 +++++----

1 file changed, 5 insertions(+), 4 deletions(-)

Toggle diff (32 lines)diff --git a/mumi/web/render.scm b/mumi/web/render.scm
index 316ca4c..9b16f8d 100644
--- a/mumi/web/render.scm
+++ b/mumi/web/render.scm
@@ -28,6 +28,7 @@
   #:use-module ((ice-9 textual-ports)
                 #:select (get-string-all put-string))
   #:use-module (ice-9 match)
+  #:use-module (rnrs bytevectors)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -104,13 +105,13 @@
 (define* (render-html sxml #:key (extra-headers '()))
   (values (append extra-headers
                   '((content-type . (text/html (charset . "utf-8")))))
-          (lambda (port)
-            (sxml->html sxml port))))
+          (string->utf8
+           (sxml->html-string sxml))))
 
 (define (render-json json)
   (values '((content-type . (application/json (charset . "utf-8"))))
-          (lambda (port)
-            (scm->json json port))))
+          (string->utf8
+           (scm->json-string json))))
 
 (define (not-found uri)
   (values (build-response #:code 404)
-- 
2.41.0

Felix Lechner wrote 10 months ago

(no subject)

Recipients:(address . control@patchwise.org)

Message-ID:87a5ksvvcc.fsf@lease-up.com

block 69381 by 70906 70907
tags 69381 + patch
thanks

noe wrote 4 months ago

[PATCH] web: Use string to avoid losing unicode characters.

Recipients:(address . 69381@debbugs.gnu.org)(name . Noé Lopez)(address . noelopez@free.fr)

Message-ID:20241102000730.3330-1-noe@xn--no-cja.eu

I don’t really understand why the unicode characters were lost in the

first place, maybe something in the sanitize-response of (fibers web

server)? Specifically, strings and procedures don’t take the same

path there.

* mumi/web/render.scm (render-html): Return string instead of procedure.

---

mumi/web/render.scm | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (18 lines)diff --git a/mumi/web/render.scm b/mumi/web/render.scm
index 168f3bc..c28a26f 100644
--- a/mumi/web/render.scm
+++ b/mumi/web/render.scm
@@ -105,8 +105,9 @@
 (define* (render-html sxml #:key (extra-headers '()))
   (values (append extra-headers
                   '((content-type . (text/html (charset . "utf-8")))))
-          (lambda (port)
-            (sxml->html sxml port))))
+          (call-with-output-string
+	    (lambda (port)
+              (sxml->html sxml port)))))
 
 (define (render-json json)
   (values '((content-type . (application/json)))
-- 
2.46.0

Noé Lopez wrote 4 months ago

Recipients:(address . 69381@debbugs.gnu.org)

Message-ID:87ikt6pk4o.fsf@xn--no-cja.eu

Hi,

Wanted to send this patch separately but had this issue selected in mumi

so it sent it here, oops.

I recognize this solution is not optimal (a hack even), but it should be

heavily considered as the issue is rampant among international users.

I suspect the actual issue lies in fibers, as said in the commit message

and I’ll try to fix it there but this patch is still important in the

meanwhile.

Good night,

Noé

Noé Lopez wrote 4 months ago

Recipients:(address . 69381@debbugs.gnu.org)

Message-ID:87froape5f.fsf@xn--no-cja.eu

Small update,

I’ve investigated the issue in fibers and I now blame the guile web

library for the issue. Apparently it sets the port to ISO-8859-1

encoding each time you call read-request, but it acts like « yeah don’t

worry just use utf-8 for your body » in the docs.

That’s fine UNLESS you use chunked transfers (omitting content-length in

fibers), in which case it just decides to blow up :///// (it assumes one

character = one byte)

In the end I’m pretty sure any of this could have been avoided by just

not replacing every character with question marks. Had it kept the

invalid bytes intact they would have translated back with no issue.

Maxim Cournoyer wrote 1 months ago

Re: bug#69381: mumi does not correctly display (some?) non-ascii characters

Recipients:(name . Noé Lopez)(address . noe@xn--no-cja.eu)(address . 69381@debbugs.gnu.org)

Message-ID:87bjvdajpq.fsf_-_@gmail.com

Hi Noé,

Noé Lopez <noe@noé.eu> writes:

Toggle quote (15 lines)> Small update,
>
> I’ve investigated the issue in fibers and I now blame the guile web
> library for the issue.  Apparently it sets the port to ISO-8859-1
> encoding each time you call read-request, but it acts like « yeah don’t
> worry just use utf-8 for your body » in the docs.
>
> That’s fine UNLESS you use chunked transfers (omitting content-length in
> fibers), in which case it just decides to blow up :///// (it assumes one
> character = one byte)
>
> In the end I’m pretty sure any of this could have been avoided by just
> not replacing every character with question marks.  Had it kept the
> invalid bytes intact they would have translated back with no issue.

Nice investigation!  Did you create an issue at bug-guile@gnu.org?  
don't see it on the tracker.  Or perhaps this could be tackled from the
angle of fibers?  For example by adding a new failing test reproducing
the problem to its test suite, and going from there.

-- 
Thanks,
Maxim

Noé Lopez wrote 1 months ago

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:878qqhg57z.fsf@xn--no-cja.eu

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (25 lines)> Hi Noé,
>
> Noé Lopez <noe@noé.eu> writes:
>
>> Small update,
>>
>> I’ve investigated the issue in fibers and I now blame the guile web
>> library for the issue.  Apparently it sets the port to ISO-8859-1
>> encoding each time you call read-request, but it acts like « yeah don’t
>> worry just use utf-8 for your body » in the docs.
>>
>> That’s fine UNLESS you use chunked transfers (omitting content-length in
>> fibers), in which case it just decides to blow up :///// (it assumes one
>> character = one byte)
>>
>> In the end I’m pretty sure any of this could have been avoided by just
>> not replacing every character with question marks.  Had it kept the
>> invalid bytes intact they would have translated back with no issue.
>
> Nice investigation!  Did you create an issue at bug-guile@gnu.org?  
> don't see it on the tracker.  Or perhaps this could be tackled from the
> angle of fibers?  For example by adding a new failing test reproducing
> the problem to its test suite, and going from there.
>

I talked about this with Christopher Baines at FOSDEM and he seemed to

know much more about it than me, so maybe he can suggest a way forward?

Starting with a failing test seems like a good idea.

Have a nice day,

Noé

Christopher Baines wrote 1 months ago

Recipients:(name . Noé Lopez)(address . noe@xn--no-cja.eu)

Message-ID:87tt94lqp3.fsf@cbaines.net

Noé Lopez <noe@noé.eu> writes:

Toggle quote (32 lines)> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi Noé,
>>
>> Noé Lopez <noe@noé.eu> writes:
>>
>>> Small update,
>>>
>>> I’ve investigated the issue in fibers and I now blame the guile web
>>> library for the issue.  Apparently it sets the port to ISO-8859-1
>>> encoding each time you call read-request, but it acts like « yeah don’t
>>> worry just use utf-8 for your body » in the docs.
>>>
>>> That’s fine UNLESS you use chunked transfers (omitting content-length in
>>> fibers), in which case it just decides to blow up :///// (it assumes one
>>> character = one byte)
>>>
>>> In the end I’m pretty sure any of this could have been avoided by just
>>> not replacing every character with question marks.  Had it kept the
>>> invalid bytes intact they would have translated back with no issue.
>>
>> Nice investigation!  Did you create an issue at bug-guile@gnu.org?  
>> don't see it on the tracker.  Or perhaps this could be tackled from the
>> angle of fibers?  For example by adding a new failing test reproducing
>> the problem to its test suite, and going from there.
>>
>
> I talked about this with Christopher Baines at FOSDEM and he seemed to
> know much more about it than me, so maybe he can suggest a way forward?
>
> Starting with a failing test seems like a good idea.

I've raised a Pull Request which I think should help in fibers:

https://github.com/wingo/fibers/pull/116

I think this issue should be possible to work around in Mumi as well,

the encoding on the port needs to be set, and I think Guile 3.0.10 needs

to be used.

Felix Lechner wrote 4 weeks ago

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87mses2kry.fsf@lease-up.com

Hi,

On Sat, Feb 08 2025, Christopher Baines wrote:

Toggle quote (2 lines)

> this issue should be possible to work around in Mumi as well,

What is wrong with my proposed patch, please? Just because a lambda

will eventually save memory and enable chunking?

Kind regards,

Felix

Christopher Baines wrote 4 weeks ago

Recipients:(name . Felix Lechner)(address . felix.lechner@lease-up.com)(address . 69381@debbugs.gnu.org)

Message-ID:87tt90je7f.fsf@cbaines.net

Felix Lechner <felix.lechner@lease-up.com> writes:

Toggle quote (9 lines)

> Hi,

> On Sat, Feb 08 2025, Christopher Baines wrote:

>> this issue should be possible to work around in Mumi as well,

> What is wrong with my proposed patch, please? Just because a lambda

> will eventually save memory and enable chunking?

I'm late to this thread but looking at your patch that looks like it
should work, there's nothing wrong with it, I was just looking at fixing
the issue with chunked responses.

Maxim Cournoyer wrote 4 weeks ago

Recipients:(name . Felix Lechner)(address . felix.lechner@lease-up.com)

Message-ID:87y0ybri6d.fsf@gmail.com

Hi Felix,

Felix Lechner <felix.lechner@lease-up.com> writes:

Toggle quote (9 lines)

> Hi,

> On Sat, Feb 08 2025, Christopher Baines wrote:

>> this issue should be possible to work around in Mumi as well,

> What is wrong with my proposed patch, please? Just because a lambda

> will eventually save memory and enable chunking?

There's nothing wrong with it; I believe it could be pushed already,
while a more definitive fix in fibers or guile is pursued!  I just
haven't gotten round to it yet.  Anyone with commit access please feel
free to beat me to it, and update mumi on berlin with it.

-- 
Thanks,
Maxim

Arun Isaac wrote 4 weeks ago

Recipients:

Message-ID:87ldub2u6x.fsf@systemreboot.net

Hi Maxim,

I am redeploying mumi on berlin today for a bunch of other commits. I

can take this over, if you don't mind.

Regards,

Arun

Maxim Cournoyer wrote 4 weeks ago

Recipients:(name . Arun Isaac)(address . arunisaac@systemreboot.net)

Message-ID:87mserp5t9.fsf@gmail.com

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:

Toggle quote (5 lines)

> Hi Maxim,

> I am redeploying mumi on berlin today for a bunch of other commits. I

> can take this over, if you don't mind.

Yes, please :-). Thanks a lot.

Thanks,

Maxim

Closed

Arun Isaac wrote 4 weeks ago

Recipients:

Message-ID:87y0ya12ny.fsf@systemreboot.net

Hi all,

I have updated mumi on berlin. But, berlin needs to be rebooted for a

shepherd update before the mumi update can take effect. And, someone

needs to be at the data center when this happens, just in case there's

any trouble with the reboot. So, this is going to take a little while,

hopefully under a few days or a week.

Regards,

Arun

Closed

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 69381@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 69381

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date