.[PATCH] WIP gnu: add stack.

Hi Guix,

I looked at the wishlist on libreplanet and saw stack was on the list.

Also after the recent thread on haskell and clojure development I

realized stack would help out there.

I tried to package it which seems straightforward. I got the

dependencies built and linted but stack itself is failing with the

following:

gcc: error trying to exec

'/gnu/store/...-gcc-7.4.0/libexec/gcc/x86_64-unknown-linux-gnu/7.4.0/collect2':

execv: Argument list too long

This looks a lot like the nix issue here:

I thought I would share my work and maybe someone could help.

Thanks,

John

Fixing the subject line.

Hi Tim,

Just wanted to CC you because this issue is hard to find thanks to my

mistaken subject line.

- John

Hi John,

John Soo <jsoo1@asu.edu> writes:

This seems to be due to https://github.com/haskell/hsc2hs/issues/22.

Briefly, the “hsc2hs” program bundled with GHC 8.6.5 accepts response

files, but then goes ahead and passes arguments directly to GCC. Newer

versions of “hsc2hs” use response files for GCC, which solves the

“Argument list too long” problem.

My first suggestion would be to make a separate package for “hsc2hs”

using the version listed on Stackage (which is 0.68.6). Then, add it as

a native input to the “stack” package and get it to shadow the version

shipped with GHC.

Failing that, we could consider patching our GHC package. It would be

best to avoid that, though! :)

(Also, there’s a duplicate “ghc-hspec-discover” in the patch set. That

is, we already have an “hspec-discover” package.)

-- Tim

Hi Tim,

I added hsc2hs@0.68.6 as a native input to stack but I'm not sure how

to make it shadow the ghc package.

Used as a native input hsc2hs does not fix the issue.

I suppose another option would be to use the #:haskell key to use a

locally-patched version.

I am not familiar with ghc internals aside from having built from

source a few times so I can look into that option.

Agree! I think we should try shadowing hsc2hs first. I might even

suggest bumping lts/ghc versions before we patch our version.

Oops! Sorry!

It's definitely the importer. I know I'm not the first to have the problem.

Maybe we should open an issue to fix that?

Thanks!

John

Hi Tim,

Also here's an updated patch set with the native input for hsc2hs and

no ghc-hspec-discover.

- John

I tried using this patch and it still fails. Could I be missing something?

Hi John,

John Soo <jsoo1@asu.edu> writes:

I’m not sure, but I got the shadowing to work. In fact, you don’t need

shadowing at all. You can just pass “--with-hsc2hs=/gnu/store/...” as a

configure flag:

(arguments

`(#:configure-flags

(let ((hsc2hs (assoc-ref %build-inputs "ghc-hsc2hs")))

(list (string-append "--with-hsc2hs=" hsc2hs "/bin/hsc2hs")))))

Unfortunately, it doesn’t help. Looking at the source code, “hsc2hs”

*is* calling GCC with a response file. AIUI, GCC is supposed to

continue to use response files for all of its subprograms. Hence,

everything should work. I’m stumped. :/

I’ll take another look with a fresh mind sometime, and see if I can make

any progress.

-- Tim

Hello again,

Timothy Sample <samplet@ngyro.com> writes:

I don’t have a solution, but I have a bit more info. Cabal generates

extremely long argument lists for GCC. For whatever reason, it repeats

the same flags many times (maybe once for each Haskell dependency).

When building Stack, it passes Gawk as an include path 164 times. In

fact, there are only a handful of distinct include paths, each passed

164 times. The linker flags are similar but much worse.

This should be fine since everything is using a response file, right?

Nope. Like the bug you linked discusses, GCC writes all of its

arguments into an environment variable, the length of which is limited

by “MAX_ARG_STRLEN” (which is 128K). It would be nice to fix GCC here,

and it’s almost a very small fix, but there is minor complication. The

normal response file handling in GCC expects “argc” and “argv”

parameters, but the environment variable doesn’t come with an “argc”.

Rather, it has to be parsed (in the same way the shell would do it).

It’s still doable, but a it would be a larger change. Not to mention

that patching GCC is even less fun than patching GHC.

There was some work to deduplicate the flags that Cabal generates:

https://github.com/haskell/cabal/pull/5356/. This would actually fix

the build, but the call to “hsc2hs” doesn’t do the same deduplication

(see “ppHsc2hs” in “Cabal/Distribution/Simple/PreProcess.hs”).

At this point, it might be worth joining the Nix folks on that issue,

and suggesting a patch to Cabal to call “hsc2hs” with deduplicated

flags. It will help that the other deduplication code was written by

the same person who filed the bug you linked to earlier. Once the patch

lands, we can apply it to our Cabal and then Stack will build. Yay!

-- Tim

Hi Tim,

Sorry I'm a bit confused by all this. I guess if we can rely on a

patch into cabal that would be the least effort, right?

Is there already a patch to deduplicate flags passed to hsc2hs? Do we

need to comment anywhere or express support?

- John

Hi John,

John Soo <jsoo1@asu.edu> writes:

Exactly. Also, sorry if I’m being a little unclear. I’ve been kind of

thinking out loud with these messages, as you’ll see below. ;)

I couldn’t find a patch. Expressing support was definitely what I was

suggesting. I managed to find a note saying what I just said:

https://github.com/NixOS/nixpkgs/issues/49206#issuecomment-470324743

I don’t think anything has become of it since then, though.

There is one other thing we could do, but I’m still thinking it through.

The reason we hit the limit is because of the way we use the

“extra-lib-dirs” and “extra-include-dirs” flags when configuring. Every

time we pass Cabal a list of directories via these flags, Cabal notes

the directories in the package DB, and uses them when calling GCC. It

doesn’t just use the directories specified when configuring a given

package. It uses all the directories specified for all of that

package’s dependencies, too.

Right now, we pass in every “lib” and “include” directory from every

input, including the “standard-packages” from the GNU build system.

This means that Gawk’s “lib” directory (for example) is included every

time we configure any package. In turn, when GHC calls GCC as part of

compiling a Haskell package, Gawk gets included on the command line once

for every node on that package’s dependency tree. Packages with a large

dependency tree hit the 128K limit imposed by the kernel Linux.

What’s funny is that not a single Haskell package even uses shared

libraries from Gawk!

The other thing that’s funny is that most Haskell packages build just

fine without “extra-lib-dirs” and “extra-include-dirs”. Even those that

call out to C code work, because Guix sets “LIBRARY_PATH” for us.

Unfortunately, packages that depend on packages that call out to C code

fail. This is because Guix doesn’t set “LIBRARY_PATH” for the

transitive input, but GHC still wants to make use of the shared library.

This can be solved by adding back the “extra-lib-dirs” and

“extra-include-dirs” flags as-needed.

With this approach, I was able to compile Stack (the tests failed,

though).

The part that I’m still thinking through is how to make this whole

system work. I would really like something that works automatically,

but I don’t see a way to do it. This means we would have to add the

flags manually to the handful of packages that need them. The part that

I really don’t like is that we don’t know a package needs the flags

until a package that depends on it fails to build. Having everything

one step removed like that is not ideal. On the other hand, there are

probably less than a dozen Haskell packages that need flags – it’s not

that common to mix Haskell and C code. I will make a patch that does

this and see what the damages are.

This feels like a step in the right direction, because we end up with a

cleaner package graph as a result. The added work and delayed feedback

will be tricky, but it should only come up infrequently. What do you

think?

-- Tim

Hello again,

Timothy Sample <samplet@ngyro.com> writes:

Turns out there are thirteen!

I’ve attached the patch. The name of the keyword is not great, so

suggestions there are welcome. I managed to build all of our Haskell

packages (except for some usual suspects that fail for other reasons).

It’s not so bad, but it’s a bit of a hack.

I’m now wondering why Guix’s treatment of “LIBRARY_PATH” is not just

solving this outright without the need for those flags. Before I

consider pushing the patch, I’m going to answer that question. Ideally,

Guix could do more of what it’s good at: understanding the complete

package graph. :)

-- Tim

Hi Tim,

Sorry for my delay. I think I’m finally starting to understand your patch.

Yeah it would be nice if it were automatically tracked or if the env vars were respected.

I like the idea of offering more cabal file semantics to package authors. In that regards I have no issues with your patches. My only thought is we should make the flags lists instead of booleans. Not only would lists match the cabal file specification, but I think having the extra detail would be a nice way to verify against existing cabal files. I can’t imagine just yet how but I could see wanting to be able to specify which paths were being used and their order.

I am not opposed to the names of the fields either, I like that they match the cabal fields.

Thanks again!

John

Hi John,

John Soo <jsoo1@asu.edu> writes:

My first attempt did exactly that, but I ran into some problems with it.

IIRC, I had to worry about paths for transitive dependencies, which made

it really hard to get working (and would make it really hard to

maintain). This patch was take two. :) It’s a little less specific

but much easier to work with.

Good to hear. Maybe they’re not so bad.

I still need to take another look at this, but I’m not sure when I’ll

get the chance. I’ll try to carve out some time soon. Thanks again to

you for the patches. :)

-- Tim

Hi Tim,

Seem totally reasonable to me.

Then for the gio package problems with gi-gtk and custom Setup.hs? :D

Ok no rush. My pleasure :)

- John

Hi Tim and Guix!

Stack now builds but some tests fail!

Here are the updated patches :)

- John

Well it looks like cryptonite-conduit was added since the last time I

worked on stack. The patch that adds it should be ignored.