guix cannot download via an http proxy

OpenSubmitted by Joshua Randall.
Details
4 participants
  • Claes Wallin (韋嘉誠)
  • Joshua Randall
  • Ludovic Courtès
  • Mark H Weaver
Owner
unassigned
Severity
normal
J
J
Joshua Randall wrote on 22 Apr 2015 00:56
(address . submit@debbugs.gnu.org)
CAFBLGSNKzhWGjCiheo8W5sc=tzmcjw6PkczxsDizVw-nKwnSzQ@mail.gmail.com
Package: guixVersion: 0.8.1
I am attempting to use guix from within a network that does not allowoutbound http connections except via an http proxy. I am using Guilev2.0.11, which supports http proxies, so my expectation would be that sinceI have http_proxy and https_proxy environment variables set, guix would usethe specified proxy for outbound http connections, but instead it appearsto ignore the proxy settings and attempts to contact the http serverdirectly, which results in a timeout.
For example, when doing a `guix pull` I got the following:$ guix pullstarting download of `/tmp/guix-file.l1zwZ7' from `http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz'...ERROR: In procedure connect: Connection timed outfailed to download "/tmp/guix-file.l1zwZ7" from "http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz"guix pull: error: failed to download up-to-date source, exiting
It appears that Guile has had support for http proxies in the web clientpackage since v2.0.10, and although guix is using the http-get method fromGuile, it isn't using the open-socket-for-uri method, which is the one thatimplements making a proxy connection. Instead, guix seems to have copiedand modified the code from an older version of open-socket-for-uriinto open-connection-for-uri(http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/download.scm?id=v0.8.1#n153)and uses that instead. I suspect what has happened is that the Guileversion of open-socket-for-uri has added proxy support since the code wascopied into open-connection-for-uri. One fix would be to port over thechanges to open-socket-for-uri that were made in Guile 2.0.10.
However, it appears from the code comment that that the reasonopen-connection-for-uri copies the functionality of open-socket-for-uri isto avoid NSS lookups for symbolic port arguments, and it looks to me thatsince version 2.0.7 of Guile, its open-socket-for-uri can be convinced notto do NSS lookups as long as (uri-port uri) is not #f (seehttp://git.savannah.gnu.org/cgit/guile.git/tree/module/web/client.scm?id=v2.0.7#n53).
Rather than porting the new code from Guile's open-socket-for-uri, it mightmake more sense to just call open-socket-for-uri with a uri that always hasa port (i.e. implement the same hard-coding for http and https in thehttp-fetch function to make sure that uri has the default port set - Inotice for some reason Guile's string->uri parser does not set the port forhttp and https even though it has the default ports for both set in thecode. I suppose one could use the existing post-2.0.7? test to keep callingopen-connection-for-uri for backwards compatibility with old versions ofGuile (which in any case don't have proxy support, so for my use case itdoesn't matter).
I can try to put together a patch that implements this fix, although Ihaven't written scheme in quite a while, so someone else may be bettersuited for it.
Cheers,
Josh.
Attachment: file
J
J
Joshua Randall wrote on 22 Apr 2015 03:15
(address . 20402@debbugs.gnu.org)
CAFBLGSN8Gu5-=NsFuHumomtG80jNO94Sjc9AsuSZvkT9R46eEg@mail.gmail.com
Tags: patch
Please find a proposed patch that should fix this issue. I haven't writtenscheme in many years, so please excuse me if I have failed to followconvention in any way - I won't be offended at all if you want to refactorit. This patch should apply cleanly on the current git master branch andalso on the v0.8.1 release.
This patch modifies http-fetch (in build/download.scm) such that it callsGuile's open-socket-for-uri after fixing up the uri so that it always has aport specified. I'm not sure how to test the bootstrapping NSS issue thatrequired open-connection-for-uri, but my expectation based on reading thesource is that this alternative should work for Guile > 2.0.7, and I'veleft the original open-connection-for-uri in there for backwardscompatibility with Guile < 2.0.7. If someone can test this against asituation known to have needed the NSS workaround, that would be great.
I've also changed the only other call to open-connection-for-uri, which isin the probe-uri function in scripts/lint.scm - my suspicion is that won'tbe an issue because I'm guessing the lint scripts are not used whilebootstrapping, so the open-socket-for-uri function will probably be finefor the purpose of probing whether a URL is valid.
After applying this patch, guix should use the http proxy support built into Guile >= 2.0.10. This appears to be working for me - I've just started abootstrap and it has now started to successfully download packages - sofar, so good.
Please let me know if there is anything else I can do to assist withgetting this issue fixed.
Cheers,
Josh.
Attachment: file
J
J
Joshua Randall wrote on 22 Apr 2015 11:08
(address . 20402@debbugs.gnu.org)
CAFBLGSOaeXcbZpWAUByAkfe+kanD_2O+x97WW6A02yNPApVKTQ@mail.gmail.com
After more testing this morning, I see that although the core functionalityis working after applying the proposed patch (i.e. the guix daemon itselfcan download things), the derivations are still failing because theenvironment is being squashed and so the http_proxy and https_proxyenvironment variables are not available to the build. It looks like nix hasa way around this involving setting imputeEnvVars = "http_proxyhttps_proxy" so that those variables get copied into the build environment(for fixed-output derivations only, which makes sense to me).
I'll try to come up with a patch that inserts the imputeEnvVars, but I'mgetting farther from the code I understand so I will probably need somehelp in getting it in the right place.
Cheers,
Josh.

On Wed, Apr 22, 2015 at 2:15 AM, Joshua Randall <jcrandall@alum.mit.edu>wrote:
Toggle quote (36 lines)> Tags: patch>> Please find a proposed patch that should fix this issue. I haven't written> scheme in many years, so please excuse me if I have failed to follow> convention in any way - I won't be offended at all if you want to refactor> it. This patch should apply cleanly on the current git master branch and> also on the v0.8.1 release.>> This patch modifies http-fetch (in build/download.scm) such that it calls> Guile's open-socket-for-uri after fixing up the uri so that it always has a> port specified. I'm not sure how to test the bootstrapping NSS issue that> required open-connection-for-uri, but my expectation based on reading the> source is that this alternative should work for Guile > 2.0.7, and I've> left the original open-connection-for-uri in there for backwards> compatibility with Guile < 2.0.7. If someone can test this against a> situation known to have needed the NSS workaround, that would be great.>> I've also changed the only other call to open-connection-for-uri, which is> in the probe-uri function in scripts/lint.scm - my suspicion is that won't> be an issue because I'm guessing the lint scripts are not used while> bootstrapping, so the open-socket-for-uri function will probably be fine> for the purpose of probing whether a URL is valid.>> After applying this patch, guix should use the http proxy support built in> to Guile >= 2.0.10. This appears to be working for me - I've just started a> bootstrap and it has now started to successfully download packages - so> far, so good.>> Please let me know if there is anything else I can do to assist with> getting this issue fixed.>> Cheers,>> Josh.>>
Attachment: file
J
J
Joshua Randall wrote on 22 Apr 2015 16:22
updated patch to pass http_proxy and https_proxy env vars to derivation builds
(address . 20402@debbugs.gnu.org)
CAFBLGSMGXGz5CXDz6xmz7xez=1c7Vi17b=gY5uyOY80n0WfE=g@mail.gmail.com
Tags: patch
I've updated my patch to also modify guix/derivations.scm to add a settingfor nix's impureEnvVars, such that it will pass http_proxy and https_proxyvariables from the guix daemon to the build processes (which nix will doonly for fixed-output derivations).
I believe this is working, but I've now run into a third problem: thebootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxysupport. What would I need to do to use a later version of bootstrap guile?
Cheers,
Josh.
Attachment: file
L
L
Ludovic Courtès wrote on 1 May 2015 00:07
Re: bug#20402: guix cannot download via an http proxy
(name . Joshua Randall)(address . jcrandall@alum.mit.edu)(address . 20402@debbugs.gnu.org)
87r3r1feww.fsf@gnu.org
Hi,
Sorry for the delay, and thanks for the investigation and patch!
Joshua Randall <jcrandall@alum.mit.edu> skribis:
Toggle quote (9 lines)> This patch modifies http-fetch (in build/download.scm) such that it calls> Guile's open-socket-for-uri after fixing up the uri so that it always has a> port specified. I'm not sure how to test the bootstrapping NSS issue that> required open-connection-for-uri, but my expectation based on reading the> source is that this alternative should work for Guile > 2.0.7, and I've> left the original open-connection-for-uri in there for backwards> compatibility with Guile < 2.0.7. If someone can test this against a> situation known to have needed the NSS workaround, that would be great.
To name lookup with the bootstrap Guile, one way is to run this:
$ guix gc -d $(guix build -S -e '(@@ (gnu packages commencement) glibc-final)') $ ./pre-inst-env guix build -S \ -e '(@@ (gnu packages commencement) glibc-final)' --no-substitutes
The second command here uses the bootstrap Guile.
Another approach is this:
Toggle snippet (17 lines)scheme@(guile-user)> ,use(gnu packages bootstrap)scheme@(guile-user)> ,enter-store-monadstore-monad@(guile-user) [1]> (mlet %store-monad ((guile (package->derivation %bootstrap-guile))) (gexp->derivation "foo" #~(begin (mkdir #$output) (pk (getaddrinfo "www.gnu.org" "http"))) #:hash-algo 'sha256 #:hash (base32 "0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73") #:guile-for-build guile))$5 = #<derivation /gnu/store/5prfiblj6ddziavk9nz31bkgy8jzaghx-foo.drv => /gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo 2ae7960>store-monad@(guile-user) [1]> (built-derivations (list $5))building path(s) `/gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo'
;;; ((#(0 2 1 6 #(2 3497454484 80) #f) #(0 2 2 17 #(2 3497454484 80) #f) #(0 10 1 6 #(10 42541952298791455573290623124440612874 80 0 0) #f) #(0 10 2 17 #(10 42541952298791455573290623124440612874 80 0 0) #f)))
Here ‘built-derivations’ fails but the build log shows that‘getaddrinfo’ succeeded.
Lastly, one can extractgnu/packages/bootstrap/x86_64-linux/guile-2.0.9.tar.xz and run:
strace -o log ./bin/guile -c '(getaddrinfo "www.gnu.org" "http")'
Here the log shows that /etc/nsswitch.conf, /etc/services, and/etc/hosts are accessed and things just work; it does not attempt toconnect to the nscd.
A bit of archeology shows the following timeline:
1. d14ecda introduces the ‘open-connection-for-uri’ hack (Oct. 2012).
2. d3b5972 changes libc used to make bootstrap tarballs to use static NSS modules (Jan. 2013).
3. 0621349 updates the bootstrap guile-2.0.9.tar.xz tarballs (Nov. 2013), meaning that our current bootstrap Guile indeed uses static NSS modules and doesn’t attempt to talk to nscd.
In other words, the hack is no longer needed.
Thus, ‘open-connection-for-uri’ is almost (see below) unneeded now,which simplifies the solution to the problem you raise.
Toggle quote (6 lines)> I've also changed the only other call to open-connection-for-uri, which is> in the probe-uri function in scripts/lint.scm - my suspicion is that won't> be an issue because I'm guessing the lint scripts are not used while> bootstrapping, so the open-socket-for-uri function will probably be fine> for the purpose of probing whether a URL is valid.
‘open-connection-for-uri’ also handles TLS connections, which are alsouseful for ‘guix lint’, so we cannot completely get rid of it.
Commit d17551d simplifies it so that it is just a wrapper around‘open-socket-for-uri’. After that, ‘guix download’ honors $http_proxy.
$https_proxy is not handled yet because that requires more work, and Ido not fully understand how that is supposed to work. Patch welcome,though. :-)
Thanks!
Ludo’.
L
L
Ludovic Courtès wrote on 1 May 2015 00:12
Re: bug#20402: updated patch to pass http_proxy and https_proxy env vars to derivation builds
(name . Joshua Randall)(address . jcrandall@alum.mit.edu)(address . 20402@debbugs.gnu.org)
87bni5fep1.fsf@gnu.org
Joshua Randall <jcrandall@alum.mit.edu> skribis:
Toggle quote (5 lines)> I've updated my patch to also modify guix/derivations.scm to add a setting> for nix's impureEnvVars, such that it will pass http_proxy and https_proxy> variables from the guix daemon to the build processes (which nix will do> only for fixed-output derivations).
I took a slightly different approach in commit c046815 (and also notpassing “impureEnvVars” by default since that would lead to a fullrebuild.)
With commit 0d88313, $http_proxy is honored, but note that it’s the$http_proxy value in the daemon’s environment that is used, not in theclient’s environment.
Toggle quote (4 lines)> I believe this is working, but I've now run into a third problem: the> bootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxy> support. What would I need to do to use a later version of bootstrap guile?
We will need to update the bootstrap Guile, but that won’t happen untila later full-rebuild cycle.
You should be able to use substitutes as a workaround in the meantime.How does that sound?
Thank you!
Ludo’.
C
C
Claes Wallin (韋嘉誠) wrote on 9 Jun 2015 03:22
Re: guix cannot download via an http proxy
(address . 20402@debbugs.gnu.org)
CAGv_=BrefcGVvV160y278n00-N1Pf10dX+uOUygwAm2hwK2NNw@mail.gmail.com
Would support for $ftp_proxy as well require the implementation of newfunctionality, or would it be a case of just letting the environmentvariable through?
M
M
Mark H Weaver wrote on 10 Jun 2015 04:26
Re: bug#20402: guix cannot download via an http proxy
(name . Claes Wallin)(address . gnu@clacke.user.lysator.liu.se)(address . 20402@debbugs.gnu.org)
87vbewz4bl.fsf@netris.org
Claes Wallin (韋嘉誠) <gnu@clacke.user.lysator.liu.se> writes:
Toggle quote (4 lines)> Would support for $ftp_proxy as well require the implementation of new> functionality, or would it be a case of just letting the environment> variable through?
It would require new functionality. Our FTP client would have to beenhanced to support proxies.
Mark
L
L
Ludovic Courtès wrote on 13 Jun 2015 19:44
Re: HTTP and FTP proxies for guix package / guix-daemon
(name . Claes Wallin (韋嘉誠))(address . gnu@clacke.user.lysator.liu.se)(address . 20402@debbugs.gnu.org)
87d20zbj03.fsf@gnu.org
(Please always keep the bug email address Cc’d.)
"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
Toggle quote (5 lines)> I suppose if I have a freshly built guix with "guix package -i guix"> or "guix package -u guix" it should be using guile 2.0.11 then?> Actually looking at the head of .../bin/guix it's referring to a> guile-2.0.11.
Yes, so that should be OK. And it’s Guix > 0.8.2, right?
Toggle quote (2 lines)> But guix-daemon still doesn't seem to be honoring http_proxy.
As a first test, could you check whether ‘guix download’ honors it?
A second test would be:
1. Define ‘http_proxy’ in your user shell. 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.
Does that honor ‘http_proxy’?
And a third test is for substitutes: Are they also downloaded via theproxy? Here you need to:
1. Define ‘http_proxy’ in a root shell. 2. Run ‘guix-daemon’ from this shell. 3. From a user shell, run ‘guix build gcc’, say, which should normally download a substitute from hydra.gnu.org rather than trying to build things.
Does the substitute download honor ‘http_proxy’?
Thanks in advance!
Ludo’.
C
C
Claes Wallin (韋嘉誠) wrote on 17 Jun 2015 18:46
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 20402@debbugs.gnu.org)
CAGv_=BqVzDp6GGVE=J7Ej0pg3=tVEPU2u3DiwCVPgwNaRuyLsA@mail.gmail.com
Finally got around to this.
On Sat, Jun 13, 2015 at 7:44 PM, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (9 lines)> "Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:>>> I suppose if I have a freshly built guix with "guix package -i guix">> or "guix package -u guix" it should be using guile 2.0.11 then?>> Actually looking at the head of .../bin/guix it's referring to a>> guile-2.0.11.>> Yes, so that should be OK. And it’s Guix > 0.8.2, right?
# /root/.guix-profile/bin/guix --versionguix (GNU Guix) 0.8.3
# ls -l /root/.guix-profile/bin/guixlrwxrwxrwx 3 root 999 71 Jan 1 1970 /root/.guix-profile/bin/guix ->/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f/bin/guix

Toggle quote (4 lines)>> But guix-daemon still doesn't seem to be honoring http_proxy.>> As a first test, could you check whether ‘guix download’ honors it?
It does.
Toggle quote (7 lines)> A second test would be:>> 1. Define ‘http_proxy’ in your user shell.> 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.>> Does that honor ‘http_proxy’?
http_proxy is set (and exported) in user shell and in the daemon process.
No.
Toggle quote (11 lines)> And a third test is for substitutes: Are they also downloaded via the> proxy? Here you need to:>> 1. Define ‘http_proxy’ in a root shell.> 2. Run ‘guix-daemon’ from this shell.> 3. From a user shell, run ‘guix build gcc’, say, which should normally> download a substitute from hydra.gnu.org rather than trying to build> things.>> Does the substitute download honor ‘http_proxy’?
No. Whenever the daemon is doing the downloading, http_proxy isignored and I get "Temporary failure in name resolution" because theVM doesn't have an internet DNS.
As I said earlier, guix has guile-2.0.11 in the hashbang. Butguix-daemon seems to be making use of guile-bootstrap.
If I provoke it a bit I get this:
# find /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/-name '*.scm' -exec touch {} +# guix package -i texinfo . . .;;; note: source file/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/eval.scm;;; newer than compiled/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/lib/guile/2.0/ccache/ice-9/eval.go;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0;;; or pass the --no-auto-compile argument to disable.. . .;;; it seems /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/receive.scm;;; is part of the compiler; skipping auto-compilation;;; WARNING: compilation of/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scmfailed:;;; ERROR: failed to create path for auto-compiled file"/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scm"starting download of`/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...ERROR: In procedure getaddrinfo: Temporary failure in name resolution
... so it's pretty clear that *something* in there is stillreferencing the bootstrap guile. But maybe that's expected? Are theguile packages built using only the bootstrap and not themselves?
/gnu/store/0aq1kq37b6kr562330xz61rg08l6y5yg-guix-0.8.2.a43b55f.drv
has:Derive([("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f","","")]. . . ("/gnu/store/4lkrvcnh2y923la3ivsbyv2m2nv3xpjy-guile-2.0.9.tar.xz.drv",["out"]). . . ("/gnu/store/d3x2skfxhvqj21w39f6bckmxni1q3bbp-guile-2.0.11.tar.xz.drv",["out"]). . . ("/gnu/store/hvl3gwa57p7k1qnliyy6ymbmz2509azh-guile-2.0.11.drv",["out"]). . . ("/gnu/store/p4i1inysxx5ra0hvhcrd9liw3a5skcbx-guile-2.0.9.tar.xz.drv",["out"]). . . ("/gnu/store/wlivhx78f1a05dx0blqy209cb1nqwgf5-guile-2.0.9.tar.xz.drv",["out"]). . . ["/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],"x86_64-linux","/gnu/store/cnqmkmj40jmssnx6fkf9n0n3bqj5x426-guile-2.0.11/bin/guile",["--no-auto-compile","-L","/gnu/store/x2nfxrqhdsvir8l65x19bhnrwd4h41rs-module-import","/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],[("GUILE_LOAD_COMPILED_PATH","/gnu/store/07n184fk33b7gqkyrbn6b2ckdfisp09i-module-import-compiled"),("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f")])
Why so many different guile drivers? Is that expected?
Toggle quote (2 lines)> Thanks in advance!
No problem!
C
C
Claes Wallin (韋嘉誠) wrote on 17 Jun 2015 18:51
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 20402@debbugs.gnu.org)
CAGv_=BqCEHSVJzUtSMsqeyjHL8o+qr-bGaUr2NWPjfJ=7SG5rQ@mail.gmail.com
On Wed, Jun 17, 2015 at 6:46 PM, Claes Wallin (韋嘉誠)<gnu@clacke.user.lysator.liu.se> wrote:
Toggle quote (3 lines)> Why so many different guile drivers? Is that expected?

... derivations. Sorry.
L
L
Ludovic Courtès wrote on 1 Jul 2015 10:19
(name . Claes Wallin (韋嘉誠))(address . gnu@clacke.user.lysator.liu.se)(address . 20402@debbugs.gnu.org)
87pp4cxpbx.fsf@gnu.org
"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
Toggle quote (15 lines)>>> But guix-daemon still doesn't seem to be honoring http_proxy.>>>> As a first test, could you check whether ‘guix download’ honors it?>> It does.>>> A second test would be:>>>> 1. Define ‘http_proxy’ in your user shell.>> 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.>>>> Does that honor ‘http_proxy’?>> http_proxy is set (and exported) in user shell and in the daemon process.
Oops, my instructions were incorrect: ‘http_proxy’ needs to be definedin the daemon process.
To test, I first run the daemon like this:
sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'
and then I run:
Toggle snippet (12 lines)$ ./pre-inst-env guix build -S gcc --no-substitutesaccepted connection from pid 14399, user ludoLa jenaj derivoj estos konstruataj: /gnu/store/s32s498y88sidfnrppsn33nfmhlacpzx-gcc-5.1.0.tar.xz.drv /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv@ build-started /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv - x86_64-linux /var/log/guix/drvs/z1//n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv.bz2starting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `http://ftpmirror.gnu.org/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'...ERROR: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #fstarting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `ftp://ftp.cs.tu-berlin.de/pub/gnu/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'...ftp://ftp.cs.tu-berlin.de/.../gcc-5.1.0.tar.bz2 3.9% of 92728.9 KiB (610. KiB/s) C-c C-cSIGPOLL
Here the error message comes from the fact that $http_proxy is amalformed URL, showing that it *is* honored. Since the http URL fails,the download logic falls back to the next URL, which happens to be anftp URL, which is unaffected by ‘http_proxy’.
So it’s working as expected here.
Toggle quote (11 lines)>> And a third test is for substitutes: Are they also downloaded via the>> proxy? Here you need to:>>>> 1. Define ‘http_proxy’ in a root shell.>> 2. Run ‘guix-daemon’ from this shell.>> 3. From a user shell, run ‘guix build gcc’, say, which should normally>> download a substitute from hydra.gnu.org rather than trying to build>> things.>>>> Does the substitute download honor ‘http_proxy’?
This test also works as expected here:
Toggle snippet (26 lines)$ ./pre-inst-env guix build -S inkscapeaccepted connection from pid 15118, user ludoLa jena derivo estos konstruata: /gnu/store/fa6ss1ljwb97ns75332nshskj2bv0dh4-inkscape-0.91.tar.gz.drvLa jenaj derivoj estos elŝutataj: /gnu/store/acmzy3nkms38g6vrphh07bczf5s5mbjd-gnutls-3.4.1 /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1@ substituter-started /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 /home/ludo/src/guix/nix/scripts/substitutefound valid signature for '/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1', from 'http://hydra.gnu.org/nar/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1'downloading `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' (1.7 MiB installed)...
[...]
In guix/scripts/substitute.scm: 182: 2 [#<procedure 1523210 at guix/scripts/substitute.scm:165:9 ()>]In guix/build/download.scm: 212: 1 [open-socket-for-uri #]In web/client.scm: 73: 0 [open-socket-for-uri #]
web/client.scm:73:16: In procedure open-socket-for-uri:web/client.scm:73:16: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f@ substituter-failed /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 256 fetching path `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' failed with exit code 1guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/2v6lls3f8dj1rkpram4jnl8ckqs77r6m-gnutls-3.4.1.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
The backtrace again stems from the bogus $http_proxy that I chose (notnice, but shows that $http_proxy is honored.)
Please let me know if you have evidence that something works differentlyfor you.
Toggle quote (4 lines)> No. Whenever the daemon is doing the downloading, http_proxy is> ignored and I get "Temporary failure in name resolution" because the> VM doesn't have an internet DNS.
[...]
Toggle quote (4 lines)> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...> ERROR: In procedure getaddrinfo: Temporary failure in name resolution
That’s another, unrelated issue, probably in your network or on yourmachine.
Thanks,Ludo’.
C
C
Claes Wallin (韋嘉誠) wrote on 1 Jul 2015 15:08
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 20402@debbugs.gnu.org)
CAGv_=Bqgs-C=SeHf9zWfSPWWoFW+0-qUByyp=DhaL2BzJrsk5Q@mail.gmail.com
On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (4 lines)> To test, I first run the daemon like this:>> sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'
Could there be a difference because you are testing with thechecked-out, semi-manually built guix and I was checking with afreshly packaged (pull, package -i guix) guix? Like I said inhttp://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html,guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybethat only happens due to guix rewriting hashbangs etc, and on a buildoutside guix (even when using guix environment) it uses whatever guilehappens to be in the paths?
When I have the time again I will check it exactly the way you are doing it.

Toggle quote (5 lines)> Here the error message comes from the fact that $http_proxy is a> malformed URL, showing that it *is* honored. Since the http URL fails,> the download logic falls back to the next URL, which happens to be an> ftp URL, which is unaffected by ‘http_proxy’.
Ok, that's maybe a clearer way of testing it.

Toggle quote (3 lines)> Please let me know if you have evidence that something works differently> for you.
I would say that both of us have evidence that it's working and notworking. The conclusion must be that we are not testing the exact sameprogram.

Toggle quote (13 lines)>> No. Whenever the daemon is doing the downloading, http_proxy is>> ignored and I get "Temporary failure in name resolution" because the>> VM doesn't have an internet DNS.>> [...]>>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution>> That’s another, unrelated issue, probably in your network or on your> machine.
Yes. This is why the proxy is required. There is no internet DNS forthe VM, because it is on an isolated lab network.
When it does go through the proxy, like when running guix download, noname lookup is done and everything works.
L
L
Ludovic Courtès wrote on 1 Jul 2015 16:45
(name . Claes Wallin (韋嘉誠))(address . gnu@clacke.user.lysator.liu.se)(address . 20402@debbugs.gnu.org)
87h9poq6mc.fsf@gnu.org
"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
Toggle quote (15 lines)> On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:>>> To test, I first run the daemon like this:>>>> sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'>> Could there be a difference because you are testing with the> checked-out, semi-manually built guix and I was checking with a> freshly packaged (pull, package -i guix) guix? Like I said in> http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html ,> guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe> that only happens due to guix rewriting hashbangs etc, and on a build> outside guix (even when using guix environment) it uses whatever guile> happens to be in the paths?
Could you send the store file name of Guix that is being used?For instance, I have:
Toggle snippet (15 lines)$ sudo deco status guix-daemonPasvorto:Status of guix-daemon: It is started. Running value is 31316. It is enabled. Provides (guix-daemon). Requires (user-processes). Conflicts with (). Will be respawned.$ ps 31316 PID TTY STAT TIME COMMAND31316 ? Ss 0:00 /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon --build-users-group guixbu
That means I’m using commit 684bf7c of Guix, which has the fix, whichcame up after the http_proxy commit (commit 0d88313.)
Toggle quote (16 lines)>>> No. Whenever the daemon is doing the downloading, http_proxy is>>> ignored and I get "Temporary failure in name resolution" because the>>> VM doesn't have an internet DNS.>>>> [...]>>>>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'>>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...>>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution>>>> That’s another, unrelated issue, probably in your network or on your>> machine.>> Yes. This is why the proxy is required. There is no internet DNS for> the VM, because it is on an isolated lab network.
Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects tothe proxy (so its name must be available in DNS lookup), not to thetarget server. The code in (web client) goes like this:
Toggle snippet (16 lines)(define (open-socket-for-uri uri-or-string) "Return an open input/output port for a connection to URI." (define http-proxy (current-http-proxy)) (define uri (ensure-uri (or http-proxy uri-or-string))) ;← proxy (define addresses (let ((port (uri-port uri))) (delete-duplicates (getaddrinfo (uri-host uri) (cond (port => number->string) (else (symbol->string (uri-scheme uri)))) (if port AI_NUMERICSERV 0)) ;; ...
Do you see the same DNS issue when running:
http_proxy=http://... guix download http://...
?
Thanks,Ludo’.
C
C
Claes Wallin (韋嘉誠) wrote on 1 Jul 2015 16:53
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 20402@debbugs.gnu.org)
CAGv_=BoJ76W5znwb0A845r2r1GSpw4ij46GU7T+KrEbpGcESiA@mail.gmail.com
Ludovic, sorry for duplicate mail again.
On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:
Toggle quote (7 lines)> > Yes. This is why the proxy is required. There is no internet DNS for> > the VM, because it is on an isolated lab network.>> Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to> the proxy (so its name must be available in DNS lookup), not to the> target server. The code in (web client) goes like this:
The proxy is in the DNS.
Toggle quote (4 lines)> Do you see the same DNS issue when running:>> http_proxy=http://... guix download http://...
Nope. guix download works. It successfully looks up and connects to theproxy.
When I next have the chance I will report:
1) Results with guix-compiled guix-daemon2) Results with dev-compiled guix-daemon3) /gnu/store hash of 1)On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:
Toggle quote (88 lines)> "Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:>> > On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:> >> >> To test, I first run the daemon like this:> >>> >> sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon> --build-users-group=guixbuild'> >> > Could there be a difference because you are testing with the> > checked-out, semi-manually built guix and I was checking with a> > freshly packaged (pull, package -i guix) guix? Like I said in> > http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html ,> > guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe> > that only happens due to guix rewriting hashbangs etc, and on a build> > outside guix (even when using guix environment) it uses whatever guile> > happens to be in the paths?>> Could you send the store file name of Guix that is being used?> For instance, I have:>> --8<---------------cut here---------------start------------->8---> $ sudo deco status guix-daemon> Pasvorto:> Status of guix-daemon:> It is started.> Running value is 31316.> It is enabled.> Provides (guix-daemon).> Requires (user-processes).> Conflicts with ().> Will be respawned.> $ ps 31316> PID TTY STAT TIME COMMAND> 31316 ? Ss 0:00> /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon> --build-users-group guixbu> --8<---------------cut here---------------end--------------->8--->> That means I’m using commit 684bf7c of Guix, which has the fix, which> came up after the http_proxy commit (commit 0d88313.)>> >>> No. Whenever the daemon is doing the downloading, http_proxy is> >>> ignored and I get "Temporary failure in name resolution" because the> >>> VM doesn't have an internet DNS.> >>> >> [...]> >>> >>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'> >>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...> >>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution> >>> >> That’s another, unrelated issue, probably in your network or on your> >> machine.> >> > Yes. This is why the proxy is required. There is no internet DNS for> > the VM, because it is on an isolated lab network.>> Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to> the proxy (so its name must be available in DNS lookup), not to the> target server. The code in (web client) goes like this:>> --8<---------------cut here---------------start------------->8---> (define (open-socket-for-uri uri-or-string)> "Return an open input/output port for a connection to URI."> (define http-proxy (current-http-proxy))> (define uri (ensure-uri (or http-proxy uri-or-string))) ;← proxy> (define addresses> (let ((port (uri-port uri)))> (delete-duplicates> (getaddrinfo (uri-host uri)> (cond (port => number->string)> (else (symbol->string (uri-scheme uri))))> (if port> AI_NUMERICSERV> 0))> ;; ...> --8<---------------cut here---------------end--------------->8--->> Do you see the same DNS issue when running:>> http_proxy=http://... guix download http://...>> ?>> Thanks,> Ludo’.>
Attachment: file
?