From debbugs-submit-bounces@debbugs.gnu.org Fri May 01 09:36:02 2020 Received: (at 40993) by debbugs.gnu.org; 1 May 2020 13:36:02 +0000 Received: from localhost ([127.0.0.1]:48452 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jUVph-0007N7-L3 for submit@debbugs.gnu.org; Fri, 01 May 2020 09:36:02 -0400 Received: from mail-wm1-f42.google.com ([209.85.128.42]:52595) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jUVpe-0007Md-H1 for 40993@debbugs.gnu.org; Fri, 01 May 2020 09:35:59 -0400 Received: by mail-wm1-f42.google.com with SMTP id 188so5971677wmc.2 for <40993@debbugs.gnu.org>; Fri, 01 May 2020 06:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=nPx3f8qRTuuusvdvoSKde1n43sJ99f97tSp4VCb8pMU=; b=MJ3I3o1UzIsbQoWYOjdv4lhJNPfq6b0XwD9AsfqS//BFnMqjAEnfwYvzQVeXC2gsQo Qwq4s0NMBmv7zQvyc89PoY8DqMLPz/qQIS+nB59EW85+EyNGoxC7qwucx2w4OJkCyliM ppdKWJVN/TTnVlWdAMPprr5Nv7E5IhuWwhFfFb4rJDmvNDHisBkE84nZbLSifOHEB83F DNQ8AsY+BMyRkswRmi9uKFshjhuI8Jb3j0+0l7ZPFIriaVrqT5OVvcSM49WPB3Dq1Ykf WeYc8lHAoRr2rGHkfSbNYzYBMVpLUhm8ZnkDsjwEWcSrDbOC6B3dCWu7RS9iLq6vcATg YUoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=nPx3f8qRTuuusvdvoSKde1n43sJ99f97tSp4VCb8pMU=; b=Nlr1p2UowmLwiANBBiJvEG7DAHjTIDhTwX1BJ3CQsQpCeSN8eoNAvpFAmV9ywZ8law wwDKfCEcdfct5BGXxVBItJQaHfBdvkj2aGcH6zjWzbXN5uI18YkNlT2PcVA7H4m+9iMt Ksxp2KX7kIsSVLK6qwbAmXUpA+42LLz9wKEJyQByenXHV5nmc8vawHjpZRUH4hq7nR34 axMbTSVhusIughL83Nb5jYLf5IcMaw++w3izccfDR7zcHikGN/qHFeFVh1DE9y7mzH63 INSCvpnwQDt0mS4Z6hcZ6D/LADacTaSfm+u5mXaSRvJcj5/4Ey9bC/K2XMLtrkEWzfPw R4MA== X-Gm-Message-State: AGi0PubPc69e/O/PqcE4mt/EFOteHxcBQ/T8j4oJZC/Y5lLfdKKHu3mI xhcGBDa5HGqMaykV9MbmBnE= X-Google-Smtp-Source: APiQypIJz8gvWh68Cqe5Fnv7arBByLp374a8OWcyac2Iug40B8+xZrzMKEdiJI4wg6MIBsqZBFhwLg== X-Received: by 2002:a05:600c:2314:: with SMTP id 20mr4294486wmo.118.1588340152544; Fri, 01 May 2020 06:35:52 -0700 (PDT) Received: from meru ([2a01:cb18:832e:5f00:5142:5b8b:1d3f:ecc5]) by smtp.gmail.com with ESMTPSA id r2sm3896324wmg.2.2020.05.01.06.35.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 May 2020 06:35:51 -0700 (PDT) From: Mathieu Othacehe To: Danny Milosavljevic Subject: Re: [bug#40993] cuirass: Add build products download support. References: <87ees4uja7.fsf@gmail.com> <20200501120914.606ffe02@scratchpost.org> Date: Fri, 01 May 2020 15:35:50 +0200 In-Reply-To: <20200501120914.606ffe02@scratchpost.org> (Danny Milosavljevic's message of "Fri, 1 May 2020 12:09:14 +0200") Message-ID: <874ksz4w21.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 40993 Cc: ludo@gnu.org, 40993@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Hey Danny, > very cool! Thanks :) > Though I agree using sendfile would be much better, especially since the user > can download 800 MB image files there. > > The guile (web server) module allows passing a procedure as the #:body, but > then it makes a bytevector out of the result and hard-codes the content-type :P. > > Eventually (web server http) http-write is reached, which only supports encoding > bytevectors and #f, that's it. No files. > > So we'd have to overwrite http-write. > > But we are using our own (web server fiberized) impl already. > > So our impl chould be extended to be able to get and process FDs. > > client-loop there has > > (lambda (response body) > (write-response response client) > (when body > (put-bytevector client body)) > > which means the "when body" part should be extended to also handle files, not just bytevectors. The problem is that even with our fiberized implementation, what we pass as "body" is checked in "sanitize-response" procedure of Guile's (web server) module. With the (very) hacky patch attached, I fool sanitize-response, by sending the file name as a bytevector. This allows me to save gigabytes of RAM when downloading disk images. WDYT? Thanks, Mathieu --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=0001-cuirass-Use-sendfiles-instead-of-raw-copies.patch From 0c5e91c170639d50d1cc339fa0b0e68ea4fba68c Mon Sep 17 00:00:00 2001 From: Mathieu Othacehe Date: Fri, 1 May 2020 15:03:12 +0200 Subject: [PATCH] cuirass: Use sendfiles instead of raw copies. * src/cuirass/http.scm (respond-file): Send the file name as an UTF8 bytevector, instead of the raw file content, (respond-gzipped-file): ditto. Also set 'content-disposition header. * src/web/server/fiberized.scm (client-loop): Check if 'content-disposition is set. If it's the case, assume that the bytevector is the file name, and use sendfiles to send it. Otherwise, keep the existing behaviour and send directly the received bytevector. --- src/cuirass/http.scm | 25 ++++++++++--------------- src/web/server/fiberized.scm | 21 +++++++++++++++++++-- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm index 79fa246..bdc780c 100644 --- a/src/cuirass/http.scm +++ b/src/cuirass/http.scm @@ -40,7 +40,8 @@ #:use-module (web uri) #:use-module (fibers) #:use-module (fibers channels) - #:use-module ((rnrs bytevectors) #:select (utf8->string)) + #:use-module ((rnrs bytevectors) #:select (utf8->string + string->utf8)) #:use-module (sxml simple) #:use-module (cuirass templates) #:use-module (guix utils) @@ -246,19 +247,14 @@ Hydra format." "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd") (sxml->xml body port)))) - (define* (respond-file file - #:key name) + (define* (respond-file file) (let ((content-type (or (assoc-ref %file-mime-types (file-extension file)) '(application/octet-stream)))) (respond `((content-type . ,content-type) - ,@(if name - `((content-disposition - . (form-data (filename . ,name)))) - '())) - ;; FIXME: FILE is potentially big so it'd be better to not load - ;; it in memory and instead 'sendfile' it. - #:body (call-with-input-file file get-bytevector-all)))) + (content-disposition + . (form-data (filename . ,(basename file))))) + #:body (string->utf8 file)))) (define (respond-static-file path) ;; PATH is a list of path components @@ -273,10 +269,9 @@ Hydra format." (define (respond-gzipped-file file) ;; Return FILE with 'gzip' content-encoding. (respond `((content-type . (text/plain (charset . "UTF-8"))) - (content-encoding . (gzip))) - ;; FIXME: FILE is potentially big so it'd be better to not load - ;; it in memory and instead 'sendfile' it. - #:body (call-with-input-file file get-bytevector-all))) + (content-encoding . (gzip)) + (content-disposition . (form-data (filename . ,file)))) + #:body (string->utf8 file))) (define (respond-build-not-found build-id) (respond-json-with-error @@ -521,7 +516,7 @@ Hydra format." (('GET "download" id) (let ((path (db-get-build-product-path id))) - (respond-file path #:name (basename path)))) + (respond-file path))) (('GET "static" path ...) (respond-static-file path)) diff --git a/src/web/server/fiberized.scm b/src/web/server/fiberized.scm index 308b642..68ae132 100644 --- a/src/web/server/fiberized.scm +++ b/src/web/server/fiberized.scm @@ -37,6 +37,7 @@ #:use-module (web request) #:use-module (web response) #:use-module (web server) + #:use-module ((rnrs bytevectors) #:select (utf8->string)) #:use-module (ice-9 binary-ports) #:use-module (ice-9 match) #:use-module (fibers) @@ -92,6 +93,8 @@ ((0) (memq 'keep-alive (response-connection response))))) (else #f))))) +(define extend-response (@@ (web server) extend-response)) + (define (client-loop client have-request) ;; Always disable Nagle's algorithm, as we handle buffering ;; ourselves. @@ -119,9 +122,23 @@ #:headers '((content-length . 0))) #vu8())))) (lambda (response body) - (write-response response client) (when body - (put-bytevector client body)) + (let* ((headers (response-headers response)) + (file? (assq-ref headers 'content-disposition)) + (file (and file? (utf8->string body))) + (file-size (and file? (stat:size (stat file))))) + (cond + (file? + (call-with-input-file file + (lambda (port) + (write-response + (extend-response response 'content-length + file-size) + client) + (sendfile client port file-size)))) + (else + (write-response response client) + (put-bytevector client body))))) (force-output client) (if (and (keep-alive? response) (not (eof-object? (peek-char client)))) -- 2.26.0 --=-=-=--