Setting TERMINFO_DIRS

Hi all,

I noticed foot did not behave normally whenever content would overflow it's current dimensions. when I installed alacritty in the same profile, this was fixed. Turned out, alacritty's declaration has a native-search-path field entry that fixed foot! Why not just search-paths, I can't tell. This might apply to further terminal emulators (kitty had problems starting emacs in some contexts for me but I would need to test that more), but for now, just foot!

search-path-specification for terminfo dirs

---

gnu/packages/terminals.scm | 4 ++++

1 file changed, 4 insertions(+)

florhizome schreef op vr 14-01-2022 om 14:02 [+0000]:

Canonically, search path are set in ‘consumers’, not ‘producers’

(though setting it in ‘producers’ sometimes works). Here,

‘consumer’ = ncurses, maybe screen (why doesn't screen have a native-

search-paths? An oversight?), and ‘producer’ = some terminal emulator.

What application were you running in foot that leaded to an overrun?

Maybe we should add TERMINFO_DIRS to the native-search-paths of the

application. Basically all applications using ncurses need it, I

think?

The following seems relevant:

though personally that doesn't seem a bug to me.

Were you running emacs in the terminal? If so, maybe TERMINFO_DIRS

need to be added to 'emacs'?

Greetings,

Maxime.

Hi Maxime,

I understand this kinda differently so far:

We want to expose the path where foot installs it's terminfo via

TERMINFO_DIRS, so that

every program running in that Terminal can look it up. foot without the

patch emits the following env vars inside it's process:

~ $ env | grep TERM

COLORTERM=truecolor

TERM=foot-direct

(i configured it to use foot-direct, and not foot-xterm, which is the

other terminfo it installs, bc that should give me more colors in emacs)

now what happens is that neither foot nor programs running from it seem

to know what foot-direct is.

As I tried to describe, foot renders problematic stuff, whenever the

text it displays reaches it's borders, simply a long line does it, but

also looking up manpages (i would get a

WARNING: terminal is not fully functional

Press RETRUN to continue

when typing "git send-email --help" f.e.

Comparing to kitty: Running in kitty:

COLORTERM=truecolor

TERMINFO=/gnu/store/zhmzdniycjykb6igrx4avs9vsn4ngk5q-kitty-0.20.3/lib/kitty/terminfo

TERM=xterm-kitty

So, kitty exposes the exact path to it's terminfo files within it's

process. so, we are fine, most of the time. But when i try to edit a

file with elevated right's, emacs complains:

~ $ sudo emacs -nw .config/guix/home.scm

Passwort:

emacs: Terminal type xterm-kitty is not defined.

If that is not the actual type of terminal you have,

use the Bourne shell command 'TERM=...; export TERM' (C-shell:

'setenv TERM ...') to specify the correct type. It may be necessary

to do 'unset TERMINFO' (C-shell: 'unsetenv TERMINFO') as well.

an interesting example with guix shell and nano:

~ $ guix shell nano -- nano ./.config/guix/home.scm

Folgende Ableitung wird erstellt:

/gnu/store/azcaj49div66k43afiiiq6njsjk8s5iv-profile.drv

Zertifikatsbündel der Zertifikatsautoritäten wird erstellt …

Liste der Emacs-Unterverzeichnisse wird erzeugt …

Schriftartenverzeichnis wird erstellt …

Verzeichnis von Info-Handbüchern wird erstellt …

Profil mit 1 Paket wird erstellt …

~ $ guix shell nano --pure -- nano ./.config/guix/home.scm

Error opening terminal: xterm-kitty.

(so, the first one worked, the second not.)

kitty works as long as i "preserve context" i would say, without

understanding in depth what happens.

So, to me it seems setting search-paths should make sense whenever you

want to expose a path in the store directory of a certain package, but

you cannot be sure to which package (otherwise, a wrapper would make

sense, and that's where your definiton of producer and consumer applies

better i think as it would be a much directer and clearer definition. ). In gnu guix we would want to take advantage of that rather more

then less i think, but it's poorly documented.

By my understanding, if emacs installs a terminfo file, yes, we could

set it's search-path field similar to alacritty. But it most definitely

makes sense for any terminal emulator that defines an own TERMINFO

variable and ships a terminfo file with that (and so a TERMINFO_DIR that

we want to expose to the environment).

Now what i don't understand is when I would set search-paths, but not

native-search-paths --- as i said, in this example search-paths would

make more sense to me, if I understood the two fields right.

Greetings,

Florian.

Maxime Devos <maximedevos@telenet.be> writes:

Florian schreef op vr 14-01-2022 om 22:52 [+0000]:

I don't see what this has to do with your example, since you aren't

cross-compiling. There's only a difference between 'search-paths' and

'native-search-paths' when cross-compiling:

When a cross-compiler 'cross-gcc' and a native compiler 'gcc' are in

the native-inputs of some package, a C library 'x' is in the

native-inputs and a C library 'y' is in the (non-native) inputs, then:

/gnu/store/[...]-x/include

(the cross-compiled 'y' library is ignored!)

set to:

/gnu/store/[...]-y/include

(the native (not cross-compiled) 'x' library is ignored!)

Packages that don't cross-compile anything usually only have

native-search-paths and AFAIK this works.

Greetings,

Maxime.

Florian schreef op vr 14-01-2022 om 22:52 [+0000]:

What I suggested, is that emacs would have a search-path because it can

be used _inside_ a terminal as a terminal application,and hence might

be a ‘consumer’ of TERMINFO_DIR.

That said, from your examples it seems a good idea to (also?) include

TERMINFO_DIR in the producer -- I'll send a separate mail for that.

Florian schreef op vr 14-01-2022 om 22:52 [+0000]:

I think the basic idea here is that the following should work:

$ guix shell --pure some-terminal-emulator -- some-terminal-emulator

# e.g., kitty

# from inside the terminal: (e.g. nano)

$ ~/.config/guix/current/bin/guix shell some-terminal-app -- some-terminal-app

# Variation: run under sudo

This works for kitty+nano, as you note, because kitty sets TERMINFO.

It also works for kitty+'emacs -nw'. However, in some combinations things won't work:

An example you noted is kitty+sudo+'emacs -nw'. This doesn't work for me either.

This seems to be caused by 'sudo' clearing too much. As I understand it, this is

only an illustrative counter-example, and not something that setting

'native-search-paths' could fix?

This patch is about 'foot', so I tried running foot+nano, but that failed because

I don't use wayland. Anyway, there seem to be a few options here:

1. patch the source code of 'foot' to set "TERMINFO_DIRS" or "TERMINFO"

appropriately (I'm not sure about the exact difference) (or use wrap-program

in 'foot').

The benefit of this above adding search paths, is that running 'foot'

directly from the store could work (somewhat niche though).

It would need to be investigated whether this works for your use case of

setting 'foot-direct' instead of 'foot-xterm'.

2. modify 'nano', 'emacs', ... to add a "TERMINFO_DIRS" search path.

A benefit of this approach above adding it to terminal emulators,

is that this how search paths are supposed to be used AFAIK

(adding an INFOPATH, LIBRARY_PATH, CROSS_LIBRARY_PATH, CPATH, ...

search path to every C library that has 'info' documentation is

rather cumbersome, instead INFOPATH ‘belongs’ to info readers,

LIBRARY_PATH etc. to compilers, ...).

A downside is that setting it here can be ‘too late’.

E.g., in the "guix shell terminal" then "guix shell editor"

example, the editor and terminal application are in separate profiles

so guix doesn't set TERMINFO_DIRS

3. Like (2), but add it to the terminal apps instead.

My order of preference: (1) > (3) > (2).

Here, (3) would be a lot easier than (1) but almost as good,

so maybe we could do (3) first and leave (1) for later?

AFAICT It's not only 'foot' that is affected, should we do (3) for all terminal

emulators that don't do (1) like kitty?

Greetings,

Maxime

Florian schreef op vr 14-01-2022 om 22:52 [+0000]:

I think the basic idea here is that the following should work:

$ guix shell --pure some-terminal-emulator -- some-terminal-emulator

# e.g., kitty

# from inside the terminal: (e.g. nano)

$ ~/.config/guix/current/bin/guix shell some-terminal-app -- some-terminal-app

# Variation: run under sudo

This works for kitty+nano, as you note, because kitty sets TERMINFO.

It also works for kitty+'emacs -nw'. However, in some combinations things won't work:

An example you noted is kitty+sudo+'emacs -nw'. This doesn't work for me either.

This seems to be caused by 'sudo' clearing too much. As I understand it, this is

only an illustrative counter-example, and not something that setting

'native-search-paths' could fix?

This patch is about 'foot', so I tried running foot+nano, but that failed because

I don't use wayland. Anyway, there are a few options here:

1. patch the source code of 'foot' to set "TERMINFO_DIRS" or "TERMINFO"

appropriately (I'm not sure about the exact difference) (or use wrap-program

in 'foot').

The benefit of this above adding search paths, is that running 'foot'

directly from the store could work (somewhat niche though).

It would need to be investigated whether this works for your use case of

setting 'foot-direct' instead of 'foot-xterm'.

2. modify 'nano', 'emacs', ... to add a "TERMINFO_DIRS" search path.

A benefit of this approach above adding it to terminal emulators,

is that this how search paths are supposed to be used AFAIK

(adding an INFOPATH, LIBRARY_PATH, CROSS_LIBRARY_PATH, CPATH, ...

search path to every C library that has 'info' documentation is

rather cumbersome, instead INFOPATH ‘belongs’ to info readers,

LIBRARY_PATH etc. to compilers, ...).

A downside is that setting it here can be ‘too late’.

E.g., in the "guix shell terminal" then "guix shell editor"

example, the editor and terminal application are in separate profiles

so guix doesn't set TERMINFO_DIRS

3. Like (2), but add it to the terminal apps instead.

My order of preference: (1) > (3) > (2).

Here, (3) would be a lot easier than (1) but almost as good,

so maybe we could do (3) first and leave (1) for later?

AFAICT It's not only 'foot' that is affected, should we do (3) for all terminal

emulators that don't do (1) like kitty?

Greetings,

Maxime

Florian schreef op za 15-01-2022 om 15:19 [+0000]:

I don't, but I presume it would work -- at least 'ncurses' has a

TERMINFO_DIRS native-search-path, 'info-reader' and 'emacs' have an

INFOPATH ...

It doesn't matter that nano and emacs don't ship terminfo files --

if a package in the profile (or build environment) has some search

path, then the search path is set with corresponding directories from

_every_ package (*), irrespective of whether they have that search

path.

So as long as 'nano/emacs/vi/...' and 'kitty/foot/gnome-terminal-

emulator/...' are in the _same_ profile, the environment variable

of the search path is set.

They aren't always in the same profile though,

(*) with some restrictions w.r.t. cross-compilation, native-search-

path/search-path, native-inputs/inputs ...

Greetings,

Maxime.

Florian schreef op za 15-01-2022 om 15:19 [+0000]:

About documentation, you might be interested in

https://issues.guix.gnu.org/22138#8.

Not sure what you mean here, when building a package it is unknown

what plug-ins will be used in advance. The point of search paths is

having some kind of extensibility, setting the search path environment

variables in qt-wrap would undo that.

Greetings,

Maxime.

Florian schreef op za 15-01-2022 om 15:19 [+0000]:

About documentation, you might be interested in

https://issues.guix.gnu.org/22138#8.

Not sure what you mean here, when building a package it is unknown

what plug-ins will be used in advance. The point of search paths is

having some kind of extensibility, setting the search path environment

variables in qt-wrap would undo that.

Greetings,

Maxime.

Hey Maxime,

the comment about native-search-paths vs search-paths was just for

better understanding.I thought they work like native inputs vs inputs,

but it seems like it's quite the opposite.

If my understanding holds, setting TERMINFO_DIRS with

native-search-paths could help all terminals that have a non-generic

terminfo (not each would). But i couldn't tell you which ones those are.

As I see major problems f.e. with QT applications, that seem related to

not setting search paths, a proposal of mine for the future would be to improve/ actually introduce good documentation to the search-patch

variables (and their relationship to profiles and propagated inputs) but

maybe simplify setting a search-path correspanding to a more distinct

group of programs. I could imagine this as something like a search-path

"type", working out like this:

(package

...

(native-search-paths

(list

(search-path-specification

(type terminal-emulator))

...)

)

at first this would just be the same as the code added to foot, but it

could be refined (people might be able to see overlaps, and integrate

some filtering or other optimization for a specific group. Also you could maybe eventually say "install a

package but without these search-paths"). For groups like qt plugins,

it could maybe be included in the qt-wrap functionalities to look for

the paths where plugins are installed and add the corresponding search paths automatically.

Have you tried to add a search path to emacs or nano?

I kinda doubt this would do much; neither nano or emacs ship a terminfo

for kitty, or for foot. What I think could work is wrapping nano and

emacs and every terminal application that has problems to add a path

derived from generically set env vars to point to the places where

terminfo files might be installed a la

TERMINFO_DIRS=$TERMINFO_DIRS:$XDG_DATA_DIRS/

this could actually be helpful on foreign distros, but i don't know if

apps don't already trythis by themselves.

So, to get more specific on what to do: Of your suggestions I would say

why not do 3 for foot immediately like my patch suggests, it fixes the

most immediate problems in a comprehensible and by that easily revertible way. I don't

think 2 helps much, maybe still misunderstanding things.

For 1) I don't know if I could do this, and I'm not really

motivated. It also doesn't really clash with 3. There also might be

usecases like programs

that need info about installed terminfos without being run from the

respective terminal emulator, where having the env var in the shell

could come handy, esp for usecases like guix on a foreign distro. but

i'm not sure if these exist here.

I also remember reading, when researching the problem with kitty sudo

emacs, possibilites for preserving chosen environment variables when switching

between different contexts, which should help.

oh, and i just set foot-direct recently, when i was investigating

this. the problems have stuck with me.

Maxime Devos <maximedevos@telenet.be> writes:

Hello Maxime,

to me, the functionality of search-paths seemed exactly communicating

paths *beyond* a single profile. It would make more sense to set

them with the package that installs the decisive files to me, at least

as a thumb-of-rule. Your examples don't really counter that, though, i

think ;)

In other words, what next?

Maxime Devos <maximedevos@telenet.be> writes:

Florian schreef op za 15-01-2022 om 18:46 [+0000]:

Search paths can be used for that (though that's rather fragile), but I

don't quite see what this has to do with foot, which can be patched or

wrapped to set TERMINFO_DIR or TERMINFO_DIRS as appropriate.

It would seem to me that adding:

* LIBRARY_PATH, CROSS_LIBRARY_PATH, C_INCLUDE_PATH, etc.

to every C library package

* INFOPATH to every package with an info manual

* MANPATH to every package with man pages

* EMACSLOADPATH to every emacs package

* GUILE_LOAD_PATH / GUILE_LOAD_COMPILED_PATH to every Guile library

or package with Guile bindings

* GUIX_PYTHONPATH to every Python package or package with Python

bindings

* PKG_CONFIG_PATH to every package with pkg-config files

* GI_TYPELIB_PATH for ... I'm not sure exactly

* similar things for R, Haskell, Minetest, OCaml, Tcl, Lua, Coq ...

would be rather tedious and it easy to forget.

How about keeping things simple and not creating much more work,

by instead adding these search paths to consumers

* GCC for LIBRARY_PATH etc.

* info-reader and emacs for INFOPATH

* man-db for MANPATH

* emacs for EMACSLOADPATH

* guile for GUILE_LOAD_PATH / GUILE_LOAD_COMPILED_PATH

* etc.

like we're doing currently? The number of packages in the list below

is much smaller than the number of packages in the list above

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22138would help, but

doesn't address the cross-profile issue.  If 22138 is considered

notabug, then

For addressing the cross-profile issue:

There was some terminal emulator setting TERMINFO_DIR (!=

TERMINFO_DIRS) I think, maybe that would be worth investigating as

well.

Greetings,

Maxime.

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

An option we’ve used before, in fact to work around

https://issues.guix.gnu.org/22138, is to define certain variables

globally; on Guix System, you’ll see this in /etc/profile:

We should probably do the same for ‘TERMINFO_DIRS’.

WDYT?

(Of course that that doesn’t help on foreign distros, but maybe

‘TERMINFO_DIRS’ is already set there?)

Thanks,

Ludo’.

retitle 53257 Setting TERMINFO_DIRS

quit

Ludovic Courtès schreef op vr 28-01-2022 om 23:34 [+0100]:

IIUC, that only works if 'foot' is installed in in the system profile,

and not if it is installed in the user profile.  Unlike, say,

'gnome-terminal', 'foot' doesn't seem like something that would be

in the system profile. The variable would need to be added to

$GUIX_PROFILE/etc/profile instead.

... (me notices .guix-profile in the example above) ...

Ok, this could work with things in the user profile, but it's a rather

ad-hoc solution and won't work with "guix shell", so I would prefer

adding it to 'etc/profile'.

One option is to modify 'build-etc/profile' to always add

'TERMINFO_DIRS' (and maybe DICPATH, GUIX_LOCPATH and

SSL_CERT_DIR/FILE?) to the list of search paths.

I'm currently on Debian+Guix. $TERMINFO_DIRS is not set here.

Even if it was, I doubt you'd find the files for 'foot' in there

unless 'foot' is installed with the foreign distro's installer

-- AFAICT, 'foot' is not present in ncurses' terminfos.

If I do "grep -RF foot /usr/share/terminfo/", I don't find any results.

I do find results for 'kitty' though, which makes it odd that some

previous tests I did with 'kitty' failed ...

Greetings,

Maxime

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

[...]

[...]

Yeah; like I wrote, it’s a workaround, but it solves the most common

issues.

That’s not appealing to me; the way I see it, that’s going a bit too far

in the direction of dismissing per-package search paths in favor of

hard-coded lists of search paths.

Oh wait, I just realized that ‘foot’ comes with its own terminfo files.

In that case, the solution would be to wrap it so that it systematically

adds itself to TERMINFO_DIRS. That’s the go-to solution in such

situations.

WDYT?

Ludo’.

Ludovic Courtès schreef op do 10-02-2022 om 21:30 [+0100]:

That would be (a suboption of) option (1) I suggested:

1. patch the source code of 'foot' to set "TERMINFO_DIRS" or

"TERMINFO" [...] (or use wrap-program in 'foot').

There is no disagreement from me here.

Greetings,

Maxime.

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

Perfect, sorry for overlooking it.

Florian, do you want to give it a try? You need to look at

‘wrap-program’.

Ludo’.

As discussed in https://issues.guix.gnu.org/53257,it appears the

preferred way to expose TERMINFO_DIRS to programs running under a

terminal is to wrap the terminal with a script that sets TERMINFO_DIRS

before exec'ing it.

This is less invasive than unconditionally setting TERMINFO_DIRS in the

profile (via native-search-paths) as this particular environment

variable is only necessary when running the terminal itself.

Tested:

./pre-inst-env guix install foot

cat "$(which foot)"

#!/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash

export TERMINFO_DIRS="/gnu/store/6dwhps0cgzk0z7c3q2q596l52ibzdl5c-foot-1.11.0/share/terminfo${TERMINFO_DIRS:+:}$TERMINFO_DIRS"

exec -a "$0" "/gnu/store/6dwhps0cgzk0z7c3q2q596l52ibzdl5c-foot-1.11.0/bin/.foot-real" "$@"

---

gnu/packages/terminals.scm | 14 +++++++++++++-

1 file changed, 13 insertions(+), 1 deletion(-)

I hope you don't mind, I went ahead and used wrap-program as discussed

(I was encountering this issue and had a very similar patch as the

OP's). Did I get the idea that was discussed in this thread right? If

yes, should I send another patch to fix the other terminals (e.g.:

On Sun, 27 Feb 2022 at 19:32, Kevin Boulain <kevinboulain@gmail.com> wrote:

Kevin Boulain schreef op zo 27-02-2022 om 19:34 [+0100]:

You'll need to add 'bash-minimal' to 'inputs', such that this works

even when cross-compiling. I think "./pre-inst-env guix lint foot"

would warn about this.

Also, this can be simplified a bit, to

,#(modify-phases %standard-phases

(add-after 'install 'wrap-program

(lambda _

(let ((out #$output))

;; footclient executes programs under the server process,

;; there is no need to wrap it too.

(wrap-program (string-append out "/bin/foot")

`("TERMINFO_DIRS" ":" prefix

(,(string-append out "/share/terminfo"))))))))))

or, reducing the whitespace:

,#(modify-phases %standard-phases

(add-after 'install 'wrap-program

(lambda _

(define out #$output)

;; footclient executes programs under the server process,

;; there is no need to wrap it too.

(wrap-program (string-append out "/bin/foot")

`("TERMINFO_DIRS" ":" prefix

(,(string-append out "/share/terminfo")))))))))

(you'll need to add (guix gexp) to the list of imports to do this)

YMMV on whether this is an improvement.

I hope you don't mind, I went ahead and used wrap-program as

discussed (I was encountering this issue and had a very similar patch

as the OP's). Did I get the idea that was discussed in this thread

right?

Yes, this was the idea, though to be 100% sure it would need to be tested (by running "guix shell --pure foot -- foot" and then running

~/.guix-profile/bin/SOME-NCURSES-APP in the terminal, or something like that.

If yes, should I send another patch to fix the other terminals

(e.g.:

)?

That would be nice, but keep in mind that this might not be needed

for every terminal app -- some apps set TERMINFODIR automatically (I forgot which) and hence don't need any wrapping.

Greetings,

Maxime.

Thank you, I'll send a follow-up patch to resolve the concerns (that's

greatly appreciated, I'm new at both Guix and Scheme).

I did test it with an ncurses program so I can confirm it works (no

complaints about an unknown terminal anymore).

I only have a single concern about this approach: native-search-paths

made the terminfo database somewhat 'global' (as long the shell

sourced the Guix profile) and it made stuff like SSH less awkward: by

installing foot on both the client and the server, ncurses

applications could 'just work'. Now, by making the terminfo database

only visible inside the terminal itself, SSH is pretty much never

guaranteed to work seamlessly (however, foot appears to be compatible

with xterm so it's not that much of an issue:

I agree this has always been a pain point with ncurses programs and

the proper solution would be to upstream the terminal's description to

terminfo so it's easily available everywhere (or people can always

install it manually), but I prefer following the recommendations of

the maintainers here :)

On Sun, 27 Feb 2022 at 20:22, Maxime Devos <maximedevos@telenet.be> wrote:

As discussed in https://issues.guix.gnu.org/53257,it appears the

preferred way to expose TERMINFO_DIRS to programs running under a

terminal is to wrap the terminal with a script that sets TERMINFO_DIRS

before exec'ing it.

This is less invasive than unconditionally setting TERMINFO_DIRS in the

profile (via native-search-paths) as this particular environment

variable is only necessary when running the terminal itself.

---

gnu/packages/terminals.scm | 18 ++++++++++++++++--

1 file changed, 16 insertions(+), 2 deletions(-)

It does warn about it, I only ran './pre-inst-env guix style foot'

(but I limited the reformatting to my changes), my bad.

Yup, thanks!

I confirm it works (for example tput doesn't complain anymore and I

can start emacs -nw).

Tested on my aarch64 machine. foot is now able to open curses

programs without needing to manually set TERM. Thanks!

This patch doesn't cleanly apply against master, but does cleanly apply

against commit deaa322963bc06b820e82d5945e6c496ac6bedf1 "guix: Index the

man-db database via man pages names.", which then cleanly cherry-picks

onto master.