Progress bar in Org source blocks

  • Open
  • quality assurance status badge
Details
2 participants
  • excalamus
  • zimoun
Owner
unassigned
Submitted by
excalamus
Severity
normal
E
E
excalamus wrote on 28 Oct 2021 04:10
(address . bug-guix@gnu.org)
Mn3eLkd--3-2@tutanota.com
I'm trying to document my work with Guix using Emacs Org. If I run a shell/sh/bash source block in org-mode, the result has control characters in it.

To reproduce, open a clean instance of Emacs with emacs -q, switch to the scratch buffer and insert:

;; eval this first so that source blocks can run sh
(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)))

#+begin_src sh :results output :session test
guix package -u
#+end_src

#+RESULTS:
: guix package: warning: nothing to do

Switch to org-mode, eval the lisp, and C-c C-c the source block. If your system is like mine, you'll see control non-printable characters.

When I check how org sees the shell, I get:

#+begin_src sh :results output :session test
echo $TERM
#+end_src

#+RESULTS:
: dumb

I've tried renaming .bashrc and .bash_profile to .bashrc.old and .bash_profile.old, starting a new shell and running through the steps above. The result is the same.

Thoughts and suggestions?
Z
Z
zimoun wrote on 29 Oct 2021 14:51
(address . excalamus@tutanota.com)(address . 51454@debbugs.gnu.org)
CAJ3okZ1P9LL1t5QzUQcvCtq3QYFT=wzoxbiNReM5TeP-AB8WKQ@mail.gmail.com
Hi,

On Fri, 29 Oct 2021 at 14:38, excalamus--- via Bug reports for GNU
Guix <bug-guix@gnu.org> wrote:

Toggle quote (2 lines)
> I'm trying to document my work with Guix using Emacs Org. If I run a shell/sh/bash source block in org-mode, the result has control characters in it.

[...]

Toggle quote (5 lines)
> #+RESULTS:
> : guix package: [1;35mwarning: [0mnothing to do
>
> Switch to org-mode, eval the lisp, and C-c C-c the source block. If your system is like mine, you'll see control non-printable characters.

You should try the environment variable INSIDE_EMACS with turns off
some fancy shell things.

Cheers,
simon
E
E
excalamus wrote on 30 Oct 2021 06:55
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51454@debbugs.gnu.org)
MnEYGl5--3-2@tutanota.com
Toggle quote (3 lines)
> try the environment variable INSIDE_EMACS with turns off
> some fancy shell things.
>
This is a good suggestion.  Maybe I'm misunderstanding how INSIDE_EMACS works. The docs say:

"Emacs sets the environment variable INSIDE_EMACS in thesubshell to ‘version,comint’, where version is theEmacs version (e.g., ‘24.1’). Programs can check this variableto determine whether they are running inside an Emacs subshell."


When I check it in a session set up as described above, I get the Emacs version:

#+begin_src sh :results output :session unset
echo $INSIDE_EMACS
#+end_src

#+RESULTS:
:
: 27.2,comint

I understand this to mean that INSIDE_EMACS *is* being set and that it's the guix command's responsibility to do something with it (if it wants to).

If instead I set INSIDE_EMACS to something falsey (e.g. 1) and run through things, I see the same result with color and control characters.
Z
Z
zimoun wrote on 30 Oct 2021 18:43
(address . excalamus@tutanota.com)(address . 51454@debbugs.gnu.org)
867ddujwee.fsf@gmail.com
Hi,

On Sat, 30 Oct 2021 at 06:55, excalamus@tutanota.com wrote:
Toggle quote (6 lines)
>> try the environment variable INSIDE_EMACS with turns off
>> some fancy shell things.
>>
> This is a good suggestion.  Maybe I'm misunderstanding how
> INSIDE_EMACS works. The docs say:

Thanks for the pointing the Emacs doc. From my understanding, the
environment variable INSIDE_EMACS is used by Guix to turn on/off
hyperlink support. It reads,

Toggle snippet (10 lines)
(define* (supports-hyperlinks? #:optional (port (current-output-port)))
"Return true if PORT is a terminal that supports hyperlink escapes."
;; Note that terminals are supposed to ignore OSC escapes they don't
;; understand (this is the case of xterm as of version 349, for instance.)
;; However, Emacs comint as of 26.3 does not ignore it and instead lets it
;; through, hence the 'INSIDE_EMACS' special case below.
(and (isatty?* port)
(not (getenv "INSIDE_EMACS"))))

Toggle quote (5 lines)
> "Emacs sets the environment variable INSIDE_EMACS in thesubshell to
> ‘version,comint’, where version is the Emacs version (e.g., ‘24.1’).
> Programs can check this variableto determine whether they are running
> inside an Emacs subshell."

Indeed, Emacs sets automatically this variable. That’s why it is used
by Guix. :-)


Toggle quote (4 lines)
> I understand this to mean that INSIDE_EMACS *is* being set and that
> it's the guix command's responsibility to do something with it (if it
> wants to).

Yes.


Toggle quote (4 lines)
> If instead I set INSIDE_EMACS to something falsey (e.g. 1) and run
> through things, I see the same result with color and control
> characters.

Thanks for reporting. I though tweaking that variable would be enough to
fix the odd behaviour you are seeing,

Toggle snippet (3 lines)
: guix package: [1;35mwarning: [0mnothing to do

which are related to shell fancy “controllers”. If you have not tried
yet, then NO_COLOR=1 should do the job. Sorry to not have thought
before. :-)


Cheers,
simon
E
E
excalamus wrote on 1 Nov 2021 03:03
(name . zimoun)(address . zimon.toutoune@gmail.com)(name . 51454)(address . 51454@debbugs.gnu.org)
MnOEBp9--3-2@tutanota.com
Toggle quote (12 lines)
>
> Indeed, Emacs sets automatically this variable. That’s why it is used
> by Guix. :-)
>
>
>> I understand this to mean that INSIDE_EMACS *is* being set and that
>> it's the guix command's responsibility to do something with it (if it
>> wants to).
>>
>
> Yes.
>
I was thinking there would be a catch-all, but it looks like I need to ensure that various environment variables are set.  It looks like we need to identify a few more.

Toggle quote (4 lines)
>
> If you have not tried yet, then NO_COLOR=1 should do the job.
>
>
It looks like NO_COLOR removes colors as expected (which is great). However, there are other characters which appear that are undesirable.  They are characters which copy correctly in email, but don't work within org or it's html export.  They render in Emacs as ^M^[[K and ^H.  Here's a kill-yank of a session (removed some progress bars, but I'm sure you get the point):

Toggle snippet (118 lines)
(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

#+begin_src sh :results output :session test
echo $TERM
#+end_src

#+RESULTS:
:
: dumb

#+begin_src sh :results output :session test
echo $INSIDE_EMACS
#+end_src

#+RESULTS:
: 27.2,comint

#+begin_src sh :results output :session test
export NO_COLOR=1
#+end_src

#+RESULTS:

#+begin_src sh :results output :session test
echo $NO_COLOR
#+end_src

#+RESULTS:
: 1

#+begin_src sh :results output :session test
guix pull
#+end_src

#+RESULTS:
#+begin_example
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
[                                                      ]receiving objects   2% [#                                                     ]receiving objects   5% [##                                                    ]receiving objects   7% [Kindexing objects  95% [####################################################   ]indexing objects  98% [#####################################################  ]Authenticating channel 'guix', commits 9edb3f6 to f3d5d57 (3 new commits)...
########################                                                     ][###################################################                           ][##############################################################################]Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git    f3d5d57
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
#################] 100.0% compute-guix-derivation  1004B       132KiB/s 00:00 [##################] 100.0%
Computing Guix derivation for 'x86_64-linux'...  -\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   8.3%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  16.7%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  25.0%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  33.3%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  41.7%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  50.0%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  58.3%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  66.7%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  75.0%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  83.3%substitute: updating substitutes from 'https://ci.guix.gnu.org'...  91.7%substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/6s9dsrraqil9h3c1y3wjsd03wl1vkms0-profile.drv
   /gnu/store/1giiiwjmj4wnqnss4snagrxc8q6dhhm9-guix-home-modules.drv
   /gnu/store/ar8pb4p6lgqyrhkqcj36vdpzxblz6879-guix-system-modules.drv
   /gnu/store/dj196mg5l18bfslkrmjnnx7bakhzr4m8-guix-system-tests-modules.drv
   /gnu/store/ibb6dqgfv5yzgyxhcnvljyh27qjvh4zr-guix-cli-modules.drv
   /gnu/store/jin7ydbcbrzzwz7h1z4fxlvic1qmj52n-guix-packages-base-modules.drv
   /gnu/store/lchb46366ydz2m54zi4i1v35z9clra05-guix-cli-core-modules.drv
   /gnu/store/ik9393vzwfa88hkvg7srddh0468y6yyj-guix-command.drv
   /gnu/store/is3f35mxggdcnlhqrlv7jz0hb0qj3bqr-guix-daemon.drv

44.7 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
  guix-package-cache                           1.2MiB/s 00:00 | 64KiB transferred guix-package-cache                          615KiB/s 00:00 | 192KiB transferred guix-package-cache                          1.1MiB/s 00:01 | 576KiB transferred guix-package-cache                          1.2MiB/s 00:01 | 700KiB transferred
  guix-cli                                     444KiB/s 00:00 | 64KiB transferred guix-cli                                    635KiB/s 00:00 | 253KiB transferred guix-cli                                    597KiB/s 00:00 | 253KiB transferred
  guix-cli-core                                470KiB/s 00:00 | 64KiB transferred guix-cli-core                               637KiB/s 00:00 | 256KiB transferred guix-cli-core                               1.0MiB/s 00:01 | 640KiB transferred guix-cli-core                               1.5MiB/s 00:01 | 1.1MiB transferred
#################] 100.0% guix-cli-modules  261B                89KiB/s 00:00 [##################] 100.0%
#################] 100.0% guix-cli-core-modules  265B           84KiB/s 00:00 [##################] 100.0%
  guix-home                                    437KiB/s 00:00 | 64KiB transferred guix-home                                   471KiB/s 00:00 | 192KiB transferred guix-home                                   564KiB/s 00:01 | 323KiB transferred
 guix-packages-base  11.9MiB          640KiB/s 00:00 [                  ]   2.1% guix-packages-base  11.9MiB          1.3MiB/s 00:01 [#                 ]   6.3% guix-packages-base  11.9MiB          2.0MiB/s 00:01 [##                ]  13.6% guix-packages-base  11.9MiB          2.8MiB/s 00:01 [####              ]  23.6% guix-packages-base  11.9MiB          3.3MiB/s 00:01 [######            ]  33.5% guix-packages-base  11.9MiB          2.7MiB/s 00:02 [######            ]  36.7% guix-packages-base  11.9MiB          2.9MiB/s 00:02 [########          ]  45.1% guix-packages-base  11.9MiB          3.2MiB/s 00:02 [#########         ]  55.0% guix-packages-base  11.9MiB          3.4MiB/s 00:02 [###########       ]  64.5% guix-packages-base  11.9MiB          3.6MiB/s 00:02 [#############     ]  74.4% guix-packages-base  11.9MiB          3.8MiB/s 00:03 [###############   ]  84.9% guix-packages-base  11.9MiB          4.0MiB/s 00:03 [################# ]  94.8% guix-packages-base  11.9MiB          4.0MiB/s 00:03 [##################] 100.0%
#################] 100.0% guix-home-modules  265B               98KiB/s 00:00 [##################] 100.0%
#################] 100.0% guix-packages-base-modules  270B      94KiB/s 00:00 [##################] 100.0%
  guix-system                                  3.2MiB/s 00:00 | 64KiB transferred guix-system                                 2.8MiB/s 00:00 | 640KiB transferred guix-system                                 3.9MiB/s 00:00 | 1.7MiB transferred guix-system                                 4.3MiB/s 00:01 | 2.7MiB transferred guix-system                                 4.4MiB/s 00:01 | 3.7MiB transferred guix-system                                 4.5MiB/s 00:01 | 4.7MiB transferred guix-system                                 4.6MiB/s 00:01 | 5.8MiB transferred guix-system                                 4.7MiB/s 00:01 | 6.8MiB transferred guix-system                                 4.7MiB/s 00:01 | 7.0MiB transferred
  guix-system-tests                            488KiB/s 00:00 | 64KiB transferred guix-system-tests                           660KiB/s 00:00 | 256KiB transferred guix-system-tests                          1018KiB/s 00:01 | 559KiB transferred
#################] 100.0% guix-system-modules  264B             98KiB/s 00:00 [##################] 100.0%
#################] 100.0% guix-system-tests-modules  270B       89KiB/s 00:00 [##################] 100.0%
 guix-f3d5d572d-modules  22.8MiB      624KiB/s 00:00 [                  ]   1.1% guix-f3d5d572d-modules  22.8MiB      944KiB/s 00:01 [                  ]   2.5% guix-f3d5d572d-modules  22.8MiB      1.5MiB/s 00:01 [                  ]   5.5% guix-f3d5d572d-modules  22.8MiB      1.8MiB/s 00:01 [#                 ]   8.5% guix-f3d5d572d-modules  22.8MiB      2.1MiB/s 00:01 [##                ]  11.5% guix-f3d5d572d-modules  22.8MiB      2.4MiB/s 00:01 [##                ]  15.4% guix-f3d5d572d-modules  22.8MiB      2.5MiB/s 00:02 [###               ]  18.7% guix-f3d5d572d-modules  22.8MiB      2.6MiB/s 00:02
#################] 100.0% guix-f3d5d572d  14KiB                881KiB/s 00:00 [##################] 100.0%
 building CA certificate bundle...
 listing Emacs sub-directories...
 building fonts directory...
 building directory of Info manuals...
 building database for manual pages...
[#########################################################################]100% [#########################################################################]100% [#########################################################################]building profile with 1 package...
 New in this revision:
  1 new package: binwalk
  1 package upgraded: pipe-viewer@0.1.5
#+end_example

#+begin_src sh :results output :session test
guix package -u
#+end_src

#+RESULTS:
#+begin_example
guix package: warning: package 'python-plover-python-dictionary' no longer exists
The following package will be upgraded:
   python-language-server (dependencies or package changed)

The following derivation will be built:
   /gnu/store/aa1096zp388n4xbxxw8ra7lp7c2pgbkm-profile.drv

 applying 30 grafts for /gnu/store/iw3ndz2qlim3f5a4qh86lm01gfvsgp14-python-language-server-0.36.2.drv ...
 \building CA certificate bundle...
 listing Emacs sub-directories...
 building fonts directory...
 generating GLib schema cache...
 creating GTK+ icon theme cache...
 building cache files for GTK+ input methods...
 building directory of Info manuals...
 building database for manual pages...
[                                                                         ]  0% [                                                                         ]  0% [                                                                         ]  1% [                                                                         ]  1% [                                                                         ]  1% [                                                                         ]  1% [#                                                                        ]  2% [#                                                                        ]  2% [#                                                                        ]  2% [#                                                                        ]  2% [#                                                                        ]  3% [#                                                                        ]  3% [##                                                                       ]  3% [##                                                                       ]  3% [##                                                                       ]  3% [##                                                                       ]  3% [##                                                                       ]  4% [##                                                                       ]  1% [                                                                         ]  4% [##                                                                       ]  4% [##                                                                       ]  4% [###                                                                      ]  4% [##                                                                       ]  4% [###                                                                      ]  2% [#                                                                        ]  5% [#########################################################################]100% [#########################################################################]building XDG desktop file cache...
 building XDG MIME database...
 building profile with 89 packages...
#+end_example

Toggle quote (3 lines)
>
> Sorry to not have thought before. :-)
>
How dare you. /s (Seriously, thank you for your help. I've ready learned a lot and the output is looking much better.)
Z
Z
zimoun wrote on 3 Nov 2021 15:14
(address . excalamus@tutanota.com)(name . 51454)(address . 51454@debbugs.gnu.org)
CAJ3okZ3jfQLaicFzEQd=ia+XYqS+HM1ZrO=QamMrcPdTHHpQhA@mail.gmail.com
Hi,

On Mon, 1 Nov 2021 at 03:03, <excalamus@tutanota.com> wrote:

Toggle quote (2 lines)
> It looks like NO_COLOR removes colors as expected (which is great). However, there are other characters which appear that are undesirable. They are characters which copy correctly in email, but don't work within org or it's html export. They render in Emacs as ^M^[[K and ^H. Here's a kill-yank of a session (removed some progress bars, but I'm sure you get the point):

I miss what you would like or what you are expecting. And I do not
think it is an issue about Guix. Instead, it is an issue about
dealing with outputs and Org-mode (Babel). The results should be
formatted to be displayed correctly after being catched, for instance,
something along these lines:

Toggle snippet (66 lines)
#+begin_src emacs-lisp :results none
(defun color-region (start)
(save-excursion
(goto-char start)
(when (looking-at org-babel-result-regexp)
(let ((end (org-babel-result-end))
(ansi-color-context-region nil))
(ansi-color-apply-on-region beg end)))))

(defun replace-region (start)
(save-excursion
(goto-char start)
(when (looking-at org-babel-result-regexp)
(let ((end (org-babel-result-end)))
(while (re-search-forward " " end t)
(replace-match "
:"))))))

(add-hook 'org-babel-after-execute-hook
#'(lambda ()
(let ((beg (org-babel-where-is-src-block-result nil nil)))
(when beg
(mapc (lambda (f) (apply f (list beg)))
(list 'color-region
'replace-region))))))
#+end_src

#+begin_src sh :results output
echo -e "\e[31mSomething\e[0m"
#+end_src

#+RESULTS:
: -e Something

#+begin_src sh :results output :session test
guix environment --ad-hoc hello -- hello
#+end_src

#+RESULTS:
: The following derivation will be built:
: /gnu/store/wkhxrm33smq3mbc6f50fivyd1yc9kk6z-profile.drv
:
: 10.0 MB will be downloaded
: #################] 100.0%
: hello-2.10 51KiB 907KiB/s 00:00
[##################] 100.0%
:
:building database for manual pages...
: [#########################################################################]
:100% [#########################################################################]
:100% [#########################################################################]
:
:building profile with 1 package...
:
:Hello, world!
:building database for manual pages...
: [#########################################################################]
:100% [#########################################################################]
:100% [#########################################################################]
:
:building profile with 1 package...
:
:Hello, world!


For example, this part,

Toggle snippet (12 lines)
Updating channel 'guix' from Git repository at
'https://git.savannah.gnu.org/git/guix.git'...
[ ] [Kreceiving
objects 2% [# ]
[Kreceiving objects 5% [##
] [Kreceiving objects 7% [Kindexing objects 95%
[#################################################### ] [Kindexing
objects 98% [##################################################### ]
[KAuthenticating channel 'guix', commits 9edb3f6 to f3d5d57 (3 new
commits)...

is not managed by Guix but it comes from the dependency 'guile-git'
and even probably libgit,



Cheers,
simon
E
E
excalamus wrote on 3 Nov 2021 18:05
(name . zimoun)(address . zimon.toutoune@gmail.com)(name . 51454)(address . 51454@debbugs.gnu.org)
MnakshN--3-2@tutanota.com
Toggle quote (2 lines)
> I miss what you would like or what you are expecting.
>
My goal has been to clean up the guix command output as seen in an org-babel results block.  I'm blogging about Guix using org-babel and Org's export functionality. I want the shell output as seen in the final html to be a reasonable approximation of the shell output a user would see if the command were run in something like xterm.  Based on feedback from people in the Org community, I thought there might be a way to toggle off certain characters.

Toggle quote (3 lines)
> And I do not think it is an issue about Guix. Instead, it is an issue about
> dealing with outputs and Org-mode (Babel).
>
I agree that handling this in org makes sense. When I asked the Org people, they had said it was a Guix issue. And with what we found with NO_COLOR, fair enough. You also raise a good point, however, that the characters originate upstream. Anything Guix might do would be a work around and work arounds are probably best handled in Emacs.

Toggle quote (2 lines)
> for instance, something along these lines...
>
Thank you for the example. I was unaware of the Emacs function ansi-color-apply-on-region.  The rest can probably be handled with replace-match.

I appreciate all your help. Thank you, Simon!
Z
Z
zimoun wrote on 3 Nov 2021 18:52
(address . excalamus@tutanota.com)(name . 51454)(address . 51454@debbugs.gnu.org)
CAJ3okZ3fx_OMRGodejpK+-QUD1G+227vqY_Fh=jn_ZLvapu-iA@mail.gmail.com
Hi,

On Wed, 3 Nov 2021 at 18:05, <excalamus@tutanota.com> wrote:

Toggle quote (2 lines)
> My goal has been to clean up the guix command output as seen in an org-babel results block. I'm blogging about Guix using org-babel and Org's export functionality. I want the shell output as seen in the final html to be a reasonable approximation of the shell output a user would see if the command were run in something like xterm. Based on feedback from people in the Org community, I thought there might be a way to toggle off certain characters.

I understand. Somehow, the output of Guix CLI is dynamic where Org
babel captures static text (the last output). Therefore, maybe you
could demo using Gifs or something in this flavor.

Toggle quote (5 lines)
> > And I do not think it is an issue about Guix. Instead, it is an issue about
> > dealing with outputs and Org-mode (Babel).
> >
> I agree that handling this in org makes sense. When I asked the Org people, they had said it was a Guix issue. And with what we found with NO_COLOR, fair enough. You also raise a good point, however, that the characters originate upstream. Anything Guix might do would be a work around and work arounds are probably best handled in Emacs.

Some special characters (as coloring) are from Guix and it should be
possible to turn them off. However, others are from progress-bars;
some from Guix itselfs, other from guile-git. Therefore, I am not
sure what you are asking, I mean, are you asking to be able to turn
off all the progress-bars?


Toggle quote (4 lines)
> > for instance, something along these lines...
> >
> Thank you for the example. I was unaware of the Emacs function ansi-color-apply-on-region. The rest can probably be handled with replace-match.

Yes, my example was just... an example. I am sure one could come with
more complex filters for formatting adequately.

Cheers,
simon
E
E
excalamus wrote on 4 Nov 2021 01:43
(name . zimoun)(address . zimon.toutoune@gmail.com)(name . 51454)(address . 51454@debbugs.gnu.org)
MncOgPF--3-2@tutanota.com
Toggle quote (2 lines)
> are you asking to be able to turn off all the progress-bars?
>
Okay, I see the confusion. No, I'm not looking to remove the progress bars, but instead have them display as much like the guix command output would look after execution completes in a standard terminal. At this point that probably means replace ^H with something like space, ^M with something like \n, etc.
Toggle quote (2 lines)
> I am sure one could come with more complex filters for formatting adequately.
>
That's my goal after having talked with you. :) Thank you for your time and help.
?