Hi Mark, Mark H Weaver skribis: > Ludovic Courtès writes: > >> The third read(2) call here ends on a partial UTF-8 sequence for LEFT >> SINGLE QUOTATION MARK (we get the first two bytes of a three byte >> sequence.) >> >> What happens is that ‘process-stderr’ in (guix store) gets that byte >> string from the daemon, passes it through ‘read-maybe-utf8-string’, >> which replaces the last two bytes with REPLACEMENT CHARACTER, which is >> itself a 3-byte sequence. > > It seems to me that what's needed here is to save the UTF-8 decoder > state between calls to 'process-stderr'. So there are two things. To fix the issue you reported (build output that goes through), I think we must simply turn off UTF-8 decoding from ‘process-stderr’ and leave that entirely to ‘build-event-output-port’. However, ‘build-event-output-port’ would still fail to properly decode split UTF-8 sequences, and for that we’d need to preserve decoder state as you describe. > Coincidentally, I also needed something like this a week ago, when I > tried implementing R6RS custom textual input/output ports on top of > R6RS custom binary input/output ports. > > To meet these needs, I've implemented a fairly efficient, purely > functional UTF-8 decoder in Scheme that accepts a decoder state and an > arbitrary range from a bytevector, and returns a new decoder state. > There's a macro that allows arbitrary actions to be performed when a > code point (or maximal subpart in the case of errors) is found. > > This macro is then used to implement a decoder (utf8->string!) that > writes into an arbitrary range of an existing string. Of course, it's > not purely functional, but it avoids heap allocation when compiled with > Guile. On my Thinkpad X200, it can process around 10 megabytes per > second. > > The state is represented as an exact integer between 0 and #xF48FBF > inclusive, which are simply the bytes that have been seen so far in the > current code sequence, in big-endian order, or 0 for the start state. > For example, #xF48FBF represents the state where the bytes (F4 8F BF) > have been read. The state is always either 0 or a proper prefix of a > valid UTF-8 byte sequence. Awesome! I think that’s something we should definitely add to Guile proper. We can use it in Guix before or after it’s included in Guile. Thank you! Ludo’.