GuixSD /tmp cleaner fails to clean when Umlauts like "ä" are used in filenames

Hi,

the GuixSD /tmp cleaner fails to clean when Umlauts like "ä" are used in filenames. It will just leave them there.

For example I have an immortal file "/tmp/!x!home!dannym!scratchpost.org!www!mirror!science!physics!03._Relativitätstheorie!.webseealso~".

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

The problem is that the “activation scripts” run in the C locale and

thus Guile interprets file names in this locale encoding (i.e., ASCII),

which fails.

I believe the attached patch mostly fixes the problem. Could you try

and report back?

I say “mostly” because if /tmp contains a file in an encoding other than

that of the system locale, we still have a problem.

Once we’ve switched to Guile 2.2, we should probably force use of an

ISO-8859-1 locale to avoid file name decoding altogether.

Thanks,

Ludo’.

Hello,

Did you have a chance to look at this patch?

TIA,

Ludo’.

ludo@gnu.org (Ludovic Courtès) skribis:

tags 26353 patch

Hi Ludo,

I've applied it, but the system update is still running. Because of the massive number of patches I write I don't track master daily. I'm always behind 2 weeks (because that's the time until I can merge a patch). It seems lately a huge update got merged :)

Right now it's compiling qtbase from source locally (not Hydra - no idea why).

70 GiB non-home root partition seems also be too small for it all. I have to do guix gc quite often - I'll have to repartition somewhen.

texlive finally downloaded correctly *shrugs*. Texlive is really getting on my nerves - isn't it possible to modularize it more? Also, one shouldn't require 2 GiB for a word processor and DTP. *mumble mumble*

But I will test the tmp cleaner, it will just take some time.

On Sun, 23 Apr 2017 01:30:56 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Hmm, guix system reconfigure finished with the patch, I rebooted, and I get the same error message (No such file) and the file is still there.

My operating-system locale is en_US.UTF-8.

Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

TeX Live is getting on everybody’s nerves. :-)

There are ways to turn it into a zillion packages from CTAN, which is

what Nixpkgs did. Ricardo (I think?) had some thoughts as to how to

achieve this and I would really like to see it happen.

Ludo’.

Ludovic Courtès <ludo@gnu.org> writes:

That would be great. I miss this snippet from my ~/.nixpkgs/config.nix:

myTex = pkgs.texlive.combine {

inherit (texlive) scheme-small marginnote sectsty cm-super enumitem

xifthen ifmtarg unicode-math filehook collection-fontsrecommended

collection-fontsextra libertine gentium-tug ucharcat;

};

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Indeed, I just realized that the cleanup code runs before

/run/current-system has been created; thus it does not have access to

locale data and ‘setlocale’ fails.

I cannot think of a nice way to address this unfortunately. :-(

The problem of how to deal with file name encoding has been discussed on

the Guile side so hopefully the next release in the 2.2 series will have

a solution for this.

Ludo’.

Marius Bakke <mbakke@fastmail.com> skribis:

Yeah, I agree. Hopefully a profile hook could do what

pkgs.texlive.combine does, which would make it more convenient. Then we

also need a CTAN importer…

Ludo’.

Ludovic Courtès <ludo@gnu.org> writes:

Yeah, it’s true, I wanted to work on this, but … it hasn’t happened yet :)

I’d be happy if someone could help us out here.

--

Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC

severity 26353 important

Hi Ludo,

For what it's worth, I think the sane solution is the Plan 9 solution: Just represent file names as bytevectors. Programs which don't care about the actual name - for example programs that just want to do (for-each unlink (scandir (string->utf8 "."))) or something - have no reason to care about the encoding at all. And then use UTF-8 encoding everywhere (for the file names, also for everything else) throughout the operating system for the tools that do care.

There are also utf8 mount options in the Linux kernel to be able to present UTF-8 names to userspace even when the actual names on disk are something else - and we should use them. (I think we should even modify <file-system> flags to default to "utf8" or "iocharset=utf8" where possible)

This conversion of UTF-8 to UCS-4 especially is really just busywork. My opinion changed over the years - earlier I was all for UCS-4. But actually, most tools don't care about the actual content of the file names - it's just an opaque ID to them (similar to an UUID). Representing them as something else in userspace again (inviting another conversion failure) is just ... unnecessary.

In any case, it would be different if we had a non-UNIX kernel underneath. But as long as we do have UNIX the kernel VFS interface expects bytevectors, preferrably interpreted as UTF-8 (if interpreted at all).

I think this is also the consensus among the major Linux distributions and also among lowlevel libraries like glib: They assume one is using UTF-8 filenames and default to it whereever possible.

Ludovic Courtès transcribed 1.3K bytes:

Importers for the importer deity!

But seriously, whoever manages to split Texlive up gets a free non-alcoholic

drink if I should ever meet the person at a conference or somewhere else!

--

ng0

OpenPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

FWIW the problem has been discussed at length in Guile land, although I

don’t think anyone has come up with a complete solution yet.

I think it’s natural to represent file names as strings, but we made a

mistake in 2.0 when we assumed we’d basically always be able to decode

file names using the current locale “on sane systems”. So now we need a

way to represent file names that cannot be decoded while preserving

backward compatibility.

To be continued!

Ludo’.

Ricardo Wurmus <rekado@elephly.net> writes:

So… I already have a Texlive importer that fetches things from SVN

(because the tarballs on CTAN are not versioned).

The texmf-dist tarball actually seems to include a couple of generated

files (such as latex.ltx), which needs to be bootstrapped with initex

first. I’ve already made some progress on this end, but I need to first

build a few metafont fonts.

The hardest part here is to override search paths and figure out

dependencies. This is a very slow process right now, because it’s

mainly error-driven.

I’m close to finishing the bootstrap of latex-base. Once that’s done I

should be able to finish the texlive-build-system, and then I’ll try

building the other latex packages that are distributed with Texlive.

There’s more to Texlive (e.g. xetex packages), but I’ll take care of

that later.

One thing that’s still unknown at this point is how the profile hook

should work, but I’ll figure this out as I learn more about the search

paths and the like.

--

Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> skribis:

Awesome!

Yeah kpathsea and all that.

OK, we’ll see.

Thank you for this brave effort!

Ludo’.

Ludovic Courtès <ludo@gnu.org> writes:

I submitted a new bug for this:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=27217

…because this allows us the satisfaction of closing this bug once it’s

done; and because we can keep track of progress there.

--

Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC

Hmm, any news on this? I've again got some immortal files in /tmp ...

Danny Milosavljevic <dannym@scratchpost.org> skribis:

I’m afraid no. Months ago a solution was proposed on the Guile side but

not implemented.

I tried the attached workaround, which attempts to use are UTF-8-only

syscalls wrappers for the task. Unfortunately it doesn’t work because

the cleanup code runs on the initrd’s statically-linked Guile, where

‘dynamic-link’ calls used in (guix build syscalls) fail. :-/

Ideas?

Ludo’.

Hi Ludo,

+(define delete-file*

+ (let ((proc (syscall->procedure int "unlike" '(*))))

Typo. Should be "unlink".

Well, we could always include a special wrapper in guile-static - like we do

for load-linux-module/fd.

That way, it is included in the statically linked guile executable.

Danny Milosavljevic transcribed 249 bytes:

Did it ever work for you? I can't recall a single time in my years with

GuixSD when /tmp was cleaned. It was only when I started reading more

system specific code that I found out that the lack of /tmp cleaning

on shutdown is not a default.

Nils Gillmann <ng0@n0.is> skribis:

This bug report is about the specific case where it doesn’t work. :-)

Ludo’.

Hello!

Finally fixed with commit 76c321d8e85683091ecbcd3afe8c56fb7c45c00a.

I opted for a simpler approach (and I wonder why it didn’t come to mind

earlier than this…).

Thanks for your patience, and bye bye immortal files! :-)

Ludo’.