'git-fetch' fails to delete files with non-ASCII names

i'm trying to build a golang package that i have just imported. its repo has a tag with unicode in it, namely v½.2.0, as observable at https://github.com/klauspost/pgzip/tags

(define-public the-pkg

(package

(name "go-github-com-klauspost-pgzip")

(version "1.0.2-0.20170402124221-0bf5dcad4ada")

(source

(origin

(method git-fetch)

(uri (git-reference

(url "https://github.com/klauspost/pgzip")

(commit "0bf5dcad4ada")))

(file-name (git-file-name name version))

(sha256

(base32 "0dgp2iljvhibzxia1g3lsfg4bjmfh4kf0bfrmfi7sd49hwhrvk7s"))))

(build-system go-build-system)

(arguments '(#:skip-build? #t #:import-path "github.com/klauspost/pgzip"))

(home-page "https://github.com/klauspost/pgzip")

(synopsis "pgzip")

(description

"Package pgzip implements reading and writing of gzip format compressed files, as

specified in @url{https://rfc-editor.org/rfc/rfc1952.html,RFC1952}.")

(license license:expat)))

i have attached the build log, but the essence is this:

guile: warning: failed to install locale

and i can't get rid of this^ warning. i installed glibc-locales to root and my user, reconfigured, restarted the guix-daemon.

which is probably the cause of the ultimate error:

warning: failed to delete .git/refs/tags/v??.2.0: No such file or directory

r:sha256 hash mismatch for...

the daemon starts from an empty env:

and then copies the env from the derivation, but it doesn't seem to contain any LANG value. i assume guile is also launched then without a LANG env. BTW, guile could be more informative in its warning, too.

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“The unexamined life is not worth living for a human being.”

— Socrates (c. 470–399 BC, tried and executed), 'Apology' (399 BC)

Attila Lendvai schreef op di 12-04-2022 om 19:47 [+0000]:

Some remarks:

* LANG should be set, because it is in #:leaked-env-vars (see

guix/git-download.scm). I don't know whose LANG it is though

-- the user's, or the daemon's?

* To install an UTF-8 locale, you need glibc-locales (or possibly

glibc-utf8-locales) (At least, for now. Upstream has some plans

for including a C.UTF-8 locale so maybe eventually we can fallback

to C.UTF-8.)

* This locale data needs to be in $GUIX_LOCPATH.

* GUIX_LOCPATH is not leaked.

* Even if it was, I don't think that /gnu/store/...glibc-locales

would be accessible from the build container (though you could give

it a try?).

* So perhaps GUIX_LOCPATH needs to be set in the gexp in

guix/git-download.scm, + some setlocale as done by

gnu-build-system.

* Long-term, it could be interesting to remove the

‘file name = string encoded in current locale's encoding’

assumption from Guile.

* svn-download, hg-download, bzr-download and cvs-download

probably have the same issue.

Greetings,

Maxime.

if i add this to the gexp:

(simple-format (current-error-port)

"LANG is '~A'~%"

(getenv "LANG"))

(setenv "LANG" "en_US.utf8")

(setenv "GUIX_LOCPATH" "/run/current-system/locale")

(setlocale LC_ALL (getenv "LANG"))

i see:

LANG is ''

Backtrace:

2 (primitive-load "/gnu/store/z4bis94jg0s0y0xj1xbmliv7xs8?")

In ice-9/eval.scm:

619:8 1 (_ #f)

In unknown file:

0 (setlocale 6 "en_US.utf8")

ERROR: In procedure setlocale:

In procedure setlocale: Invalid argument

it's the same if i add GUIX_LOCPATH to the #:leaked-env-vars and don't setenv it explicitly.

i didn't check this specifically, but i'm afraid you are right, and this is why my kludge doesn't work.

i don't understand why the setlocale call in gnu-build-system's install-locale works, but my setlocale kludge in git-download doesn't.

i even tried to add glibc-locale as native-inputs to the package in question, but it didn't help.

i'm not sure why the wrong locale breaks file-system walking and deleting, though.

i assume if every function in guile uses/assumes the same locale (character encoding), then both directions through the guile FFI should be idempotent, no? and i think both ASCII and UTF-8 are idempotent wrt C bytes <-> scheme string conversions. IOW, it's only the displaying of the chars that should be broken, not file operations.

or am i wrong to assume this?

or maybe the character encoding algo used in guile's FFI silently emits actual question marks in place of bytes that are outside the valid range of the encoding used? if so, that's not a very defensive way of coding, and it's eating up hours of my life...

hrm... this is not relevant here, only a related thought: things can go wrong in the GEXP serialization, too: if the writing side and the reading side doesn't use the same character encoding. locale should be set explicitly at the relevant entry points.

i'd appreciate if someone could help me come up with at least a kludge, so that i could make progress until it's fixed properly.

thanks for your insights Maxime,

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

If you never heal from what hurt you, you'll bleed on people who didn't cut you.

Attila Lendvai schreef op wo 13-04-2022 om 07:51 [+0000]:

I don't expect /run/current-system/locale to exist inside the build

container. Maybe try

(setenv "GUIX_LOCPATH" #+(file-append glibc-locales "/lib/locale"))

;; for testing

((@ (guix build utils) invoke)

#+(file-append coreutils "/bin/ls") (getenv "GUIX_LOCPATH"))

instead?

gnu-build-system has a (variant of) glibc-locales in its (implicit)

inputs, so there GUIX_LOCPATH can be set to the /gnu/store/.../locales

file name, in the 'set-paths' procedure.

Building the package and downloading the source code are separate steps

(derivations), they don't automatically have access to each other's

inputs.

Greetings,

Maxime

Attila Lendvai schreef op wo 13-04-2022 om 07:51 [+0000]:

The problem is that the default character encoding is ANSI_X3.4-1968

(US-ASCII) and any bytes above 127 makes things non-ASCII.

Also, the string procedures internally always use UTF-8 (or possibly

ISO-85519-1 as an optimisation?), they are not raw bytes instead they

can be consideres a vector of characters (string-ref returns

characters, not bytes, and doesn't use byte positions).

LANG=bogus guile

(guile-user)> (setlocale LC_ALL)

(guile-user)> (use-modules (ice-9 i18n))

(guile-user)> (locale-encoding)

(guile-user)> (locale-encoding)

$2 = "ANSI_X3.4-1968"

Apparently the fallback encoding is ‘ANSI_X3.4-1968’. Let's take a

look at this encoding. According to IANA

(https://www.iana.org/assignments/character-sets/character-sets.xhtml),

this character encoding can also be named ‘US-ASCII’ and is specified

in RFC2046. Some excerpts:

"US-ASCII" does not indicate an arbitrary 7-bit

character set[sic], but specifies that all octets in the body must

be interpreted as characters according to the US-ASCII character

set.

so it looks like, say, é cannot be encoded as US-ASCII, it does not

belong to the character set of the encoding. More generally, anything

beyond the 127 (Unicode) codepoint cannot be encoded in ANSI_X3.4-1968.

Let's test this (in a new REPL with an UTF-8 locale):

((@ (ice-9 iconv) string->bytevector) "é" "ANSI_X3.4-1968")

ice-9/boot-9.scm:1669:16: In procedure raise-exception:

Throw to key `encoding-error' with args `("put-char" "conversion to port encoding failed" 84 #<output: string 7fd5bbc23ee0> #\é)'.

((@ (ice-9 iconv) string->bytevector) "é" "ANSI_X3.4-1968" 'substitute)

$2 = #vu8(63)

((@ (rnrs bytevectors) utf8->string) #vu8(63))

$3 = "?"

and the other direction:

((@ (ice-9 iconv) bytevector->string) #vu8(128) "ANSI_X3.4-1968" 'substitute)

$5 = "?" ;; why #\? and not #\?? I don't know, I guess Guile is inconsistent

(FWIW, I would throw an decoding-error here instead of silently corrupting the

file names.)

Greetings,

Maxime.

Attila Lendvai schreef op wo 13-04-2022 om 07:51 [+0000]:

Serialisation is always done in UTF-8, search for UTF-8 in (guix gexp).

I don't know if deserialisation of the script in done in UTF-8, though

it should be done that way.

Also, using the same character encoding is not sufficient, the

character encoding must also encode all characters (in practice, that

means UTF-8).

Greetings,

Maxime.

thank you, this works indeed as a band aid:

(setenv "GUIX_LOCPATH" #+(file-append glibc-locales "/lib/locale"))

(setlocale LC_ALL "en_US.utf8")

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“If a nation expects to be ignorant and free, in a state of civilization, it expects what never was and never will be.”

— Thomas Jefferson (1743–1826)

Am Mittwoch, dem 13.04.2022 um 10:22 +0200 schrieb Maxime Devos:

You are first encoding a non-ASCII byte to ASCII, which has no glyph

for "I have no idea what this is", so a question mark (#\?) is used.

When converting from invalid ASCII to UTF-8 on the other hand, you do

have #\? as the WTF character, so that is used instead. This is

entirely consistent :)

Cheers

Liliana Marie Prikler schreef op wo 13-04-2022 om 12:40 [+0200]:

Right, makes sense.

i spoke too early. this works in a git checkout of guix, but it fails

to compile when i try to guix pull it.

even if i declare the dependency like this:

#:autoload (gnu packages base) (glibc-locales)

IIUC, this is due to a circular dependency: glibc-locales (and its

variants) depend on git-fetch, therefore i cannot refer to them from

the implementation of git-fetch.

i tried to set the locale to "C" or "POSIX", but it results in ASCII encoding.

i tried to set the locale to "en_US.iso-8859-1", hoping that it's

available, but it isn't.

all that is needed here is an encoding that is idempotent wrt a cycle

through bytes->string, string->bytes. i think the iso-8859-n encodings

are like that.

to verify that hypothesis:

$ mkdir -p /tmp/delme/v½.2.0

$ LANG=C guix repl

scheme@(guix-user)> (use-modules (guix build utils))

scheme@(guix-user)> (delete-file-recursively "/tmp/delme")

warning: failed to delete /tmp/delme/v??.2.0: No such file or directory

warning: failed to delete /tmp/delme: Directory not empty

$1 = #t

$2 = #<vhash 7fd60aef5540 1 pairs>

scheme@(guix-user)>

$ LANG=en_US.iso-8859-1 guix repl

scheme@(guix-user)> (use-modules (guix build utils))

scheme@(guix-user)> (delete-file-recursively "/tmp/delme")

$1 = #<vhash 7f7d7acc2040 2 pairs>

scheme@(guix-user)>

so, is such an idempotent locale available/embedded in glibc without any external dependencies? searching the web suggests that there isn't.

if not, then what would be a bird's eye view plan to make one

available for git-fetch?

should we create a new, ASCII-only git-fetch variant used in the bootstrap process?

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“The world is changed by your example, not by your opinion.”

— Paulo Coelho (1947–)

Attila Lendvai schreef op di 19-04-2022 om 11:38 [+0000]:

The module of the glibc-locales package depends on git-fetch, but I

don't think the package glibc-locales does. Anyway, circular imports

are messy and (guix build-system ...) and (guix git-download) use an

extra-lazy variant of #:autoload that doesn't load the module even when

compiling (*).

(*) Limitation: this method cannot be used to use macros.

Maybe the attached variant works?

Greetings,

Maxime.

Attila Lendvai schreef op di 19-04-2022 om 11:38 [+0000]:

Try:

$ LC_CTYPE=anything.ISO-8859-2 guix repl é

hint: Consider installing the `glibc-locales' package and defining `GUIX_LOCPATH', along these lines:

guix install glibc-locales

export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

See the "Application Setup" section in the manual, for more info.

;;; Stat of /home/[...]/?Š failed:

;;; In procedure stat: Bestand of map bestaat niet: "/home/[...]/?Š"

guix repl: fout: open-file: Bestand of map bestaat niet: "/home/regulator/source-code/rw/?Š"

IIUC, this causes lib/localcharset.c in Guile to run 'eviron_locale_charset',

which just uses environment variables and ignores glibc's locale data.

I don't know if this requires 'setlocale' or requires the absence of 'setlocale'.

Greetings,

Maxime.

Without this the git-fetch GEXP is run in an environment that uses ASCII

character encoding when strings are crossing the Guile - C boundary. It means

that e.g. tag names that have Unicode chars in them will cause problems,

e.g. when walking and deleting the .git directory.

An example in the wild: https://github.com/klauspost/pgzip/tags

For more details see: https://issues.guix.gnu.org/54893

---

thanks Maxime, this indeed seems to work! and i have successfully

guix pull'ed it, too.

guix/git-download.scm | 10 ++++++++++

1 file changed, 10 insertions(+)

Hi,

Attila Lendvai <attila@lendvai.name> skribis:

[...]

I changed this to ‘glibc-utf8-locales’, which is sufficient here, and

committed.

Thanks everyone for the investigation and fix!

Ludo’.

severity 54893 important

quit

Attila Lendvai <attila@lendvai.name> skribis:

I spoke a bit too fast and realized some adjustments were needed to

avoid a circular dependency on i586-gnu. Pushed as

8852f911ff506dd50b714274ba0e2143f0285f78!

Ludo’.

retitle 54893 'git-fetch' fails to delete files with non-ASCII names

quit