From debbugs-submit-bounces@debbugs.gnu.org Sat May 04 05:34:02 2019 Received: (at 35350) by debbugs.gnu.org; 4 May 2019 09:34:02 +0000 Received: from localhost ([127.0.0.1]:50389 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hMr3R-0001lW-Li for submit@debbugs.gnu.org; Sat, 04 May 2019 05:34:01 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38307) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hMr3Q-0001lJ-0z for 35350@debbugs.gnu.org; Sat, 04 May 2019 05:34:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:47851) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMr3K-0006PP-PA; Sat, 04 May 2019 05:33:54 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35222 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hMr3J-0007KB-Tl; Sat, 04 May 2019 05:33:54 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#35350: Some compile output still leaks through with --verbosity=1 References: <87mukkfd2j.fsf@netris.org> <87r29v2jz2.fsf@gnu.org> <87ftq9silk.fsf@netris.org> <87imv5jai5.fsf@gnu.org> <87k1fgh9c0.fsf@netris.org> <874l6jh0bx.fsf@gnu.org> <87imuvme7g.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 =?utf-8?Q?Flor=C3=A9al?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 04 May 2019 11:33:51 +0200 In-Reply-To: <87imuvme7g.fsf@netris.org> (Mark H. Weaver's message of "Tue, 30 Apr 2019 16:26:32 -0400") Message-ID: <87r29e5zsw.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35350 Cc: 35350@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Mark, Mark H Weaver skribis: > Ludovic Court=C3=A8s writes: [...] >> So there are two things. To fix the issue you reported (build output >> that goes through), I think we must simply turn off UTF-8 decoding from >> =E2=80=98process-stderr=E2=80=99 and leave that entirely to =E2=80=98bui= ld-event-output-port=E2=80=99. > > Can we assume that UTF-8 is the appropriate encoding for > (current-build-output-port)? My interpretation of the Guix manual entry > for 'current-build-output-port' suggests that the answer should be "no". What goes to =E2=80=98current-build-output-port=E2=80=99 comes from builds = processes. It=E2=80=99s usually UTF-8 but it can be anything, including binary garbage, which should be gracefully handled. That=E2=80=99s why =E2=80=98process-stderr=E2=80=99 currently uses =E2=80= =98read-maybe-utf8-string=E2=80=99. > Also, in your previous message you wrote: > > The problem is the first layer of UTF-8 decoding that happens in > =E2=80=98process-stderr=E2=80=99, in the =E2=80=98%stderr-next=E2=80=99= case. We would need to > disable it, but only if the build output port is > =E2=80=98build-event-output-port=E2=80=99 (i.e., it=E2=80=99s capable o= f interpreting > =E2=80=9Cmultiplexed build output=E2=80=9D correctly.) > > It sounds like you're suggesting that 'process-stderr' should look to > see if (current-build-output-port) is a 'build-event-output-port', and > in that case it should use binary I/O primitives to write raw binary > data to it, otherwise it should use text I/O primitives and write > characters to it. Do I understand correctly? Yes. (Actually, rather than guessing if (current-build-output-port) is a =E2=80=98build-event-output-port=E2=80=99, there could be a fluid to ask = for the use of raw binary primitives.) > IMO, it would be cleaner to treat 'build-event-output-port' uniformly, > and specifically as a textual port of unknown encoding. (You mean =E2=80=98current-build-output-port=E2=80=99, right?) I think you=E2=80=99re right. I=E2=80=99m not yet entirely sure what the i= mplications are. There=E2=80=99s a couple of tests in tests/store.scm for UTF-8 interpretation that describe behavior that I think we should preserve. > I would suggest changing 'build-event-output-port' to create an R6RS > custom *textual* output port, so that it wouldn't have to worry about > encodings at all, and it would only be given whole characters. > Internally, it would be doing exactly what you suggest above, but those > details would be encapsulated within the custom textual port. > > However, I don't think we can use Guile's current implementation of R6RS > custom textual output ports, which are currently built on Guile's legacy > soft ports, which I suspect have a similar bug with multibyte characters > sometimes being split (see 'soft_port_write' in vports.c). > > Having said all of this, my suggestions would ultimately entail having > two separate places along the stderr pipeline where 'utf8->string!' > would be used, and maybe that's too much until we have a more optimized > C implementation of it. Yeah it looks like we don=E2=80=99t yet have custom textual output ports th= at we could rely on, do we? I support your work to add that in Guile proper! Thanks, Ludo=E2=80=99.