cbqn currently targets AVX2

  • Done
  • quality assurance status badge
Details
4 participants
  • Liliana Marie Prikler
  • Liliana Marie Prikler
  • Ludovic Courtès
  • Christopher Rodriguez
Owner
unassigned
Submitted by
Christopher Rodriguez
Severity
normal
C
C
Christopher Rodriguez wrote on 26 Sep 2022 21:03
(address . guix-patches@gnu.org)
878rm6hs4d.fsf@gmail.com
Hey All,

Came to my attention today that cbqn (which I recently submitted the
package for) was assuming a CPU with AVX2 extensions when built as I had
originally packaged it.

I have a patch to fix this; cbqn now will target a generic CPU, and the
AVX2 optimizations have been moved to their own package, cbqn-singeli.

Let me know what I can improve!

--
Christopher Rodriguez
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEJMQbvYVxvZ0eF/84XZ6FgaGVz3sFAmMx+GIACgkQXZ6FgaGV
z3tonQ/+JJntJaGScho+sNBFRf7DP+su03MBnORSlmXPkw4MucagtMPIldBqTr2e
q5SwCg8yVFtdFmaiY3nZSu9U8N3uQHdu05n/wbWQkPWFEO2vQQkCI7+YJgHFeJlo
5bu05EJ58Qr6/bsfdhUJbnRECRyif4kmj697Xk/rjx7yuM3WIdpAapAw6vcuMIU7
vx5wp2eHkOlK9m+orDdhfdhJlimFJ13oXJLtzO/SXdKxmtcsgR1+XTgwToW4i/f4
GiXIp9USlrSHLAdIpo1PmxvwSaI1PnA1m3cUdl4wlsqCUiXP9CD/V7x2rWS4n9ZS
KqT/qy/VNCURBr+ck+cCJv3rt61hKKjTqYSd7/JA5XrYCi0Zv+d+OdVBI6iBWntD
hdOxZuGiM5x79OAvEURJRqI5SAu9z9Fvzlq/YGanzXjZ6RmIPjq+gd/v1QdBOQ3z
RUeb+flk8GU/Gez8pBkGjg51DRxT2fR0a6zjN0neg1jRFi7oToBvJbUCbHO8x9YI
yj6T0uV444lVsDFUJ4FwWVKNyzsy1nF1PbC4PX40/KbO7g9+A3h8M3FHbh4C1ASo
svy3yxSTYTQ4Qfd6ojoqavv4kUWXAd/ZBgOrp2dE3lUEKZAZ4ZY19Pw532yG2jTh
KJ6bR/29bAb9wRGK5WUS8N34E+a9Yeia4L6xt2jHDGwGc9kjVvw=
=FTne
-----END PGP SIGNATURE-----

C
C
Christopher Rodriguez wrote on 26 Sep 2022 21:12
[PATCH] gnu: cbqn: factor out singeli into derivative package.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
20220926191212.30609-1-yewscion@gmail.com
Persuant to a recent conversation[1] with upstream, I've factored the
AVX2-dependent builds into their own package (defaulting to a generic build
instead).

+ gnu/packages/bqn.scm: cbqn-singeli
~ gnu/packages/bqn.scm: cbqn: changed make flags, inputs, and commit.
---
gnu/packages/bqn.scm | 45 ++++++++++++++++++++++++++++++++++++++------
1 file changed, 39 insertions(+), 6 deletions(-)

Toggle diff (78 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index c4e0f39efb..56705beede 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -126,7 +126,7 @@ (define bqn-sources
(define cbqn-bootstrap
(let* ((revision "1")
- (commit "9c1cbdc99863b1da0116df61cd832137b196dc5c"))
+ (commit "bd823839feaf42af4013e5a245981f58f563e659"))
(package
(name "cbqn-bootstrap")
(version (git-version "0" "1" commit))
@@ -138,7 +138,7 @@ (define cbqn-bootstrap
(file-name (git-file-name name version))
(sha256
(base32
- "0w38fhwf20drkyijy6nfnhmc5g5gw0zmzgmy1q605x57znlj85a2"))))
+ "0735mixvpgp2g1jsmbg7aaw35r4xs348y7djkwdvb0193ais6xz2"))))
(build-system gnu-build-system)
(arguments
(list #:tests? #f ;skipping tests for bootstrap
@@ -182,12 +182,9 @@ (define-public cbqn
(name "cbqn")
(outputs '("out" "lib"))
(arguments
- (list #:make-flags '(list "shared-o3" "o3n-singeli")
+ (list #:make-flags '(list "shared-o3" "o3")
#:phases #~(modify-phases %standard-phases
(delete 'configure)
- (add-before 'build 'link-singeli
- (lambda* (#:key inputs #:allow-other-keys)
- (symlink #+singeli-sources "Singeli")))
(add-before 'build 'generate-bytecode
(lambda* (#:key inputs #:allow-other-keys)
(system (string-append #+dbqn
@@ -222,3 +219,39 @@ (define-public cbqn
libffi
clang-toolchain
linux-libre-headers))))
+(define-public cbqn-singeli
+ (package
+ (inherit cbqn)
+ (name "cbqn-singeli")
+ (outputs '("out"))
+ (arguments
+ (list #:make-flags '(list "singeli=1 f='-mavx2 -mbmi2 -O3' c")
+ #:phases #~(modify-phases %standard-phases
+ (delete 'configure)
+ (add-before 'build 'link-singeli
+ (lambda* (#:key inputs #:allow-other-keys)
+ (symlink #+singeli-sources "Singeli")))
+ (add-before 'build 'generate-bytecode
+ (lambda* (#:key inputs #:allow-other-keys)
+ (system (string-append #+dbqn
+ "/bin/dbqn ./genRuntime "
+ #+bqn-sources))))
+ (replace 'check
+ (lambda* (#:key inputs tests? #:allow-other-keys)
+ (when tests?
+ (system (string-append "./BQN -M 1000 \""
+ #+bqn-sources
+ "/test/this.bqn\""))
+ (map (lambda (x)
+ (system (string-append "./BQN ./test/" x
+ ".bqn")))
+ '("cmp" "equal" "copy" "random")))))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((bin (string-append (assoc-ref outputs
+ "out")
+ "/bin")))
+ (mkdir-p bin)
+ (copy-recursively "BQN"
+ (string-append bin "/bqn"))))))))))
+
--
2.37.3
C
C
Christopher Rodriguez wrote on 26 Sep 2022 21:17
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
874jwuhric.fsf@gmail.com
Christopher Rodriguez <yewscion@gmail.com> writes:

Toggle quote (4 lines)
> Persuant to a recent conversation[1] with upstream, I've factored the
> AVX2-dependent builds into their own package (defaulting to a generic build
> instead).

Sorry, forgot to actually include the link.

This change came from this[1] discussion with upstream, and this[2]
discussion on IRC.


--
Christopher Rodriguez
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEJMQbvYVxvZ0eF/84XZ6FgaGVz3sFAmMx+3sACgkQXZ6FgaGV
z3tPCRAAy4Sc4GGiNCzsm/NZsri+G2M8M6Ch71M2nfaJMG12LiyGZJJDRwbBTRgg
ihPvy9sWflsgzWQXkeEyq+w4+e6mTg6YCFQbNKOpxGpGpwN0SUEFUgNIGRAZQdn9
LdpNu6KKAuu7fs97Q3plflN0ovD/kOIf+mbnlKPQvAPuP/bFDXUq0huIIgyg1GUv
cWbtT3wQ1U4hpqD2DNay81X2rbUEwDcj9LKX5X9Np63EZEpsEtA/38g33FQ0aGIp
R1Z7VE2gJfDYbuZFA++B5NdD6rcRWdSgsGIxZ/F+ihJTs1/gvsbtVrLzyF11k6qO
Y38HwU6cpQQuBJ9iHIZZxfeN+j2M25p+tNh0sXv0hRozkDvzyZFeym8VdVfqnm4E
qQxAvG/gK6poUjU4+hjQ6PdalwOQLT8XK9OKn28c+xMvIjBLUISC+2mg0f9xT1qh
cnQluk+9NTk7PT3mobOM55IxO/3tMnAbwttjzKo61vsVbZXq/u1MY/a8zosCfA6r
L8AqRDIgZFcId9Rjj3raRRWgoQipwe6R1sjYiFbhJEsJNt5qx5fPl04AaqbepBSU
Uxz8wXUdbDk/26Fs1C/2ZHx7SX6AZHffGHumqsRfhq7CVRiBej0Q/7kmR/Jn0RdS
Ozov0WVZn/QMpBg5rIXGi1KA8vXv7rOpntg9SNizboCEFm5h4F0=
=npee
-----END PGP SIGNATURE-----

L
L
Liliana Marie Prikler wrote on 27 Sep 2022 09:26
982732ff807d152d4e8c181e3c33ca03ea72ac4b.camel@ist.tugraz.at
Am Montag, dem 26.09.2022 um 15:12 -0400 schrieb Christopher Rodriguez:
Toggle quote (7 lines)
> Persuant to a recent conversation[1] with upstream, I've factored the
> AVX2-dependent builds into their own package (defaulting to a generic
> build
> instead).
>
> + gnu/packages/bqn.scm: cbqn-singeli
> ~ gnu/packages/bqn.scm: cbqn: changed make flags, inputs, and commit.
This is not how to write a ChangeLog. See [1] or infer from other
commits.

Toggle quote (15 lines)
> ---
>  gnu/packages/bqn.scm | 45 ++++++++++++++++++++++++++++++++++++++----
> --
>  1 file changed, 39 insertions(+), 6 deletions(-)
>
> diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
> index c4e0f39efb..56705beede 100644
> --- a/gnu/packages/bqn.scm
> +++ b/gnu/packages/bqn.scm
> @@ -126,7 +126,7 @@ (define bqn-sources
>  
>  (define cbqn-bootstrap
>    (let* ((revision "1")
> -         (commit "9c1cbdc99863b1da0116df61cd832137b196dc5c"))
> +         (commit "bd823839feaf42af4013e5a245981f58f563e659"))
Why are you changing the bootstrap commit? This looks suspicious.
Toggle quote (21 lines)
>      (package
>        (name "cbqn-bootstrap")
>        (version (git-version "0" "1" commit))
> @@ -138,7 +138,7 @@ (define cbqn-bootstrap
>                  (file-name (git-file-name name version))
>                  (sha256
>                   (base32
> -                 
> "0w38fhwf20drkyijy6nfnhmc5g5gw0zmzgmy1q605x57znlj85a2"))))
> +                 
> "0735mixvpgp2g1jsmbg7aaw35r4xs348y7djkwdvb0193ais6xz2"))))
>        (build-system gnu-build-system)
>        (arguments
>         (list #:tests? #f                         ;skipping tests for
> bootstrap
> @@ -182,12 +182,9 @@ (define-public cbqn
>      (name "cbqn")
>      (outputs '("out" "lib"))
>      (arguments
> -     (list #:make-flags '(list "shared-o3" "o3n-singeli")
> +     (list #:make-flags '(list "shared-o3" "o3")
Okay
Toggle quote (59 lines)
>             #:phases #~(modify-phases %standard-phases
>                          (delete 'configure)
> -                        (add-before 'build 'link-singeli
> -                          (lambda* (#:key inputs #:allow-other-keys)
> -                            (symlink #+singeli-sources "Singeli")))
>                          (add-before 'build 'generate-bytecode
>                            (lambda* (#:key inputs #:allow-other-keys)
>                              (system (string-append #+dbqn
> @@ -222,3 +219,39 @@ (define-public cbqn
>                           libffi
>                           clang-toolchain
>                           linux-libre-headers))))
> +(define-public cbqn-singeli
> +  (package
> +    (inherit cbqn)
> +    (name "cbqn-singeli")
> +    (outputs '("out"))
> +    (arguments
> +     (list #:make-flags '(list "singeli=1 f='-mavx2 -mbmi2 -O3' c")
> +           #:phases #~(modify-phases %standard-phases
> +                        (delete 'configure)
> +                        (add-before 'build 'link-singeli
> +                          (lambda* (#:key inputs #:allow-other-keys)
> +                            (symlink #+singeli-sources "Singeli")))
> +                        (add-before 'build 'generate-bytecode
> +                          (lambda* (#:key inputs #:allow-other-keys)
> +                            (system (string-append #+dbqn
> +                                                   "/bin/dbqn
> ./genRuntime "
> +                                                   #+bqn-sources))))
> +                        (replace 'check
> +                          (lambda* (#:key inputs tests? #:allow-
> other-keys)
> +                            (when tests?
> +                              (system (string-append "./BQN -M 1000
> \""
> +                                                     #+bqn-sources
> +                                                    
> "/test/this.bqn\""))
> +                              (map (lambda (x)
> +                                     (system (string-append "./BQN
> ./test/" x
> +                                                           
> ".bqn")))
> +                                   '("cmp" "equal" "copy"
> "random")))))
> +                        (replace 'install
> +                          (lambda* (#:key outputs #:allow-other-
> keys)
> +                            (let* ((bin (string-append (assoc-ref
> outputs
> +                                                                 
> "out")
> +                                                       "/bin")))
> +                              (mkdir-p bin)
> +                              (copy-recursively "BQN"
> +                                                (string-append bin
> "/bqn"))))))))))
> +
Instead of providing a singeli variant, would just tuning the package
suffice? If not, I think we should try to properly unbundle singeli
(as in build an actual singeli package) before adding another package
variant. Then, you could use existing patterns to decide whether to
use singeli by making it an input or not.

Cheers

C
C
Christopher Rodriguez wrote on 27 Sep 2022 19:27
(name . Liliana Marie Prikler)(address . liliana.prikler@ist.tugraz.at)
87r0zwhez2.fsf@gmail.com
Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> writes:

Toggle quote (3 lines)
> This is not how to write a ChangeLog. See [1] or infer from other
> commits.

Thank You for pointing me in the right direction; I will ensure a
standardized changelog in my revised patch after reading all of the
linked info.

Toggle quote (4 lines)
>> -         (commit "9c1cbdc99863b1da0116df61cd832137b196dc5c"))
>> +         (commit "bd823839feaf42af4013e5a245981f58f563e659"))
> Why are you changing the bootstrap commit? This looks suspicious.

I am bumping the commit from 9c1cbdc99863b1da0116df61cd832137b196dc5c to
46501ac819c8f21c69d7d2ba4b0457a7356f5e42 (another commit was made when
closing the above-linked issue) as the new commit is the most recent one
for this un-tagged project.

There is no 'release' to package, and so in order to stay up to date
regular updates of this package's commit will likely be necessary.

I can see why the bootstrap commit might stay the same, however. Would
it be better to solely amend the commit on the actual installable
package instead?

Toggle quote (5 lines)
>>      (arguments
>> -     (list #:make-flags '(list "shared-o3" "o3n-singeli")
>> +     (list #:make-flags '(list "shared-o3" "o3")
> Okay

I have actually just re-read the documentation after the abovementioned
most recent commit, and noticed a surviving line in the upstream
`README.md`[1]:

`make PIE=""` on ARM CPUs (incl. Android & M1)

Perhaps I can incorporate this into the package to allow it to build on
aarch64 as well as x86_64? Though as mentioned in the original upstream
issue, I think there is a problem with a dependent package… perhaps this
should be saved for a separate issue, then.

Toggle quote (2 lines)
> Instead of providing a singeli variant, would just tuning the package
> suffice?
This was actually my biggest question when making the patch. I chose a
singeli variant because there is no architecture detection at all in the
makefile; it relies entirely on the specified targets ("o3", "c", etc)
to decide what to build for.

As an example: Switching the target from "o3n-singeli" to "o3"
immediately changed the entire build, preventing it from looking at all
for Singeli sources, even though I had yet to unlink them from the
source directory.

Is there a preferred method for this kind of build structure in a Guix
package? I suppose I could do a (cond *) in the make flags… maybe
referencing a variable for the target? I don't know what variable that
might be, though, as we would not only be looking for an x86_64 target,
but specifically that the underlying system supports AVX2…

In general, I would much prefer to keep it as one package. I think it is
much easier to maintain that way, and makes the user experience much
easier as well. But I'm sorry to report that I'm unaware of how best to
implement this, and would greatly appreciate some advice.

Toggle quote (5 lines)
> If not, I think we should try to properly unbundle singeli (as in
> build an actual singeli package) before adding another package
> variant. Then, you could use existing patterns to decide whether to
> use singeli by making it an input or not.

As for unbundling singeli: Running singeli requires a version of cbqn
built with or without singeli support. Building a version of cbqn with
singeli support requires the /source/ for singeli to be present in the
build directory at build time, not a precompiled binary. Singeli itself
is actually just a BQN script[2], and not a compiled binary at all, and
is a transpiler from BQN to IR/C. It's used in the optimized version to
transpile/compile the SIMD algorithms (sic, I am unfamiliar with this
concept).

In short, to unbundle singeli, we can just avoid including singeli in
the build, as in the revised cbqn package in the patch. We could make a
package for singeli that uses an installed bqn binary from any cbqn
package, but we would still need the sources present at build time due
to the way they are used and called in the build script. Fully
decoupling the optimized cbqn from singeli would require rewriting the
parts of the build that locate and run the singeli script, and (I think)
is more suited to an upstream patch than a package definition. I have
opened an issue regarding this possibility with upstream as of this
email[3].

(FWIW, I've also opened an issue regarding release tagging[4], though
from my previous correspondence with upstream I'm fairly certain a
release tag is not yet something they are comfortable with).

Toggle quote (3 lines)
>
> Cheers

Thank You (as always) for Your guidance, Liliana. Sorry for these
issues, I am still learning best practices and standards for this
project and those of GNU in general. I hope You have a great day!


--
Christopher Rodriguez
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEJMQbvYVxvZ0eF/84XZ6FgaGVz3sFAmMzOvEACgkQXZ6FgaGV
z3vUthAAlX0NKqeaIi40GSuvVfH16hwbHJhP9Cnzaw192e3LZPeoDfF/U03ydUcR
nOkk2ZSxUhv6Ptrkw6wCdpBtm4zTe1ZKj0Q5+h6UCxWpWtYRgyHTnSYBauonibas
wk831wvM1WnUVEnrPvS2pH/OjkP+01K0perOPq01O/SJJIwvhEJ0sOLk58gYX8Mi
pIX36R4aAXckAGEQnd3ruun+276EliTjjeG7iKJGelQReHWSvA0bKrr3bShQxYOG
hNHx4L+cIwQbpZdyWgsiCLkxJXmVmBQrRGQ7PLZmXLLGJGNgHcUpCb305uu7WD/i
agW9vGpH8wU2wCE7hkMnVVWHWmOt9PyaBgMLlBI1EiJm6MHUiAi5PNW9zkzjWzDj
1Twv21cNwpJuQj60JYE4BCosU7Dd8I8bWIvNBrisqoTCKSD1Qwe8GKxedHg1Seaz
arSh2GbaHnQoOHY09kmhj6zzfv4f+hNhFaWQ25PneuEHUM+60Qm0n7RKEK+O5cId
ltG/hx0ku3bRY6FCB8sv/mOssdAFxBdPlP7/Jm83xFIUFLOJgqr2QUusQxZmRo/i
fXT+MGoyvUtGqxFutTwqVYLnPm4o1cqMjAqmRCvrcpZLU6+zuoSyP+C+n2k1Bjpa
aQcmKVdLjLiBY9rMPy9vUxPQr1fpJlG8gYNox0uL2ht4UVJAILk=
=gVvX
-----END PGP SIGNATURE-----

L
L
Liliana Marie Prikler wrote on 28 Sep 2022 09:41
(name . Christopher Rodriguez)(address . yewscion@gmail.com)(address . 58100@debbugs.gnu.org)
2d539eb0ba2583fba5c2ce9c339b73d7606603a3.camel@ist.tugraz.at
Am Dienstag, dem 27.09.2022 um 13:27 -0400 schrieb Christopher
Rodriguez:
Toggle quote (4 lines)
> I am bumping the commit from 9c1cbdc99863b1da0116df61cd832137b196dc5c
> to 46501ac819c8f21c69d7d2ba4b0457a7356f5e42 (another commit was made
> when closing the above-linked issue) as the new commit is the most
> recent one for this un-tagged project.
There is no policy in Guix to tail untagged projects. For personal
experimentation, use transformations, e.g. --with-branch. But even if
there is a good reason to bump the package, use a separate commit for
this.

Toggle quote (2 lines)
> There is no 'release' to package, and so in order to stay up to date
> regular updates of this package's commit will likely be necessary.
Staying up to date is not good in and of itself, you have to evaluate
costs and benefits. Right now, you provided neither.

Toggle quote (3 lines)
> I can see why the bootstrap commit might stay the same, however.
> Would it be better to solely amend the commit on the actual
> installable package instead?
I have no hard stance on this. While one could argue that it'd be
better for bootstrap binaries to be unmoving, this doesn't seem to
apply here. Other than that, it's better for bootstrap paths to be
short, see e.g. Rust.

Toggle quote (10 lines)
> > >      (arguments
> > > -     (list #:make-flags '(list "shared-o3" "o3n-singeli")
> > > +     (list #:make-flags '(list "shared-o3" "o3")
> > Okay
>
> I have actually just re-read the documentation after the
> abovementioned most recent commit, and noticed a surviving line in
> the upstream `README.md`[1]:
>
> `make PIE=""` on ARM CPUs (incl. Android & M1)
I'm pretty sure we'd like to specify e.g. -shared instead.

Toggle quote (4 lines)
> Perhaps I can incorporate this into the package to allow it to build
> on aarch64 as well as x86_64? Though as mentioned in the original
> upstream issue, I think there is a problem with a dependent package…
> perhaps this should be saved for a separate issue, then.
It ought to at very least be a separate commit.

Toggle quote (6 lines)
> > Instead of providing a singeli variant, would just tuning the
> > package suffice?
> This was actually my biggest question when making the patch. I chose
> a singeli variant because there is no architecture detection at all
> in the makefile; it relies entirely on the specified targets ("o3",
> "c", etc) to decide what to build for.
That would be handled by the tuning compiler.

Toggle quote (16 lines)
> As an example: Switching the target from "o3n-singeli" to "o3"
> immediately changed the entire build, preventing it from looking at
> all for Singeli sources, even though I had yet to unlink them from
> the source directory.
>
> Is there a preferred method for this kind of build structure in a
> Guix package? I suppose I could do a (cond *) in the make flags…
> maybe referencing a variable for the target? I don't know what
> variable that might be, though, as we would not only be looking for
> an x86_64 target, but specifically that the underlying system
> supports AVX2…
>
> In general, I would much prefer to keep it as one package. I think it
> is much easier to maintain that way, and makes the user experience
> much easier as well. But I'm sorry to report that I'm unaware of how
> best to implement this, and would greatly appreciate some advice.
My question is: what does singeli even do for cbqn? If it's a
compiler, can you not simply run that compiler in a separate phase?

Toggle quote (16 lines)
> > If not, I think we should try to properly unbundle singeli (as in
> > build an actual singeli package) before adding another package
> > variant.  Then, you could use existing patterns to decide whether
> > to use singeli by making it an input or not.
>
> As for unbundling singeli: Running singeli requires a version of cbqn
> built with or without singeli support. Building a version of cbqn
> with singeli support requires the /source/ for singeli to be present
> in the build directory at build time, not a precompiled binary.
> Singeli itself is actually just a BQN script[2], and not a compiled
> binary at all, and is a transpiler from BQN to IR/C. It's used in the
> optimized version to transpile/compile the SIMD algorithms (sic, I am
> unfamiliar with this concept).
>
> In short, to unbundle singeli, we can just avoid including singeli in
> the build, as in the revised cbqn package in the patch.
Good to know.

Toggle quote (4 lines)
> We could make a package for singeli that uses an installed bqn binary
> from any cbqn package, but we would still need the sources present at
> build time due to the way they are used and called in the build
> script.
I'm pretty sure that finding the right arguments to copy-build-system
would be the first problem here. As for the comment from dzaima, I
don't quite understand why you'd need two versions of cbqn. Assume you
already have one built without singeli, how is it expected to change
once singeli is added?

Toggle quote (5 lines)
> Fully decoupling the optimized cbqn from singeli would require
> rewriting the parts of the build that locate and run the singeli
> script, and (I think) is more suited to an upstream patch than a
> package definition. I have opened an issue regarding this possibility
> with upstream as of this email[3].
On that note, do you even need CBQN to run singeli or does any
(conforming) BQN implementation suffice? For instance, you could try
building BQN with CBQN and then use that to run singeli, or use the
Java BQN.

Toggle quote (3 lines)
> (FWIW, I've also opened an issue regarding release tagging[4], though
> from my previous correspondence with upstream I'm fairly certain a
> release tag is not yet something they are comfortable with).
That's fair and well, but to come back to the point made earlier, we
probably won't go to a newer commit unless there's an important
feature, bug fix, etc. You can spend a lot of time optimizing without
substantially changing anything and trailing every single step on this
way would be a fool's errand.

Cheers

L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:23
[PATCH v2 1/8] cbqn: Build without singeli.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
15a1bde90dc5a23d31dca8f7139ecb1d3cffbaed.camel@gmail.com
* gnu/packages/bqn.scm (cbqn)[inputs]: Remove singeli-sources.
[arguments]<#:make-flags: Replace “o3n-singeli” with “o3”.
<#:phases>: Remove “link-singeli”.
---
gnu/packages/bqn.scm | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)

Toggle diff (30 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index c4e0f39efb..02f539bb8b 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -182,12 +182,9 @@ (define-public cbqn
(name "cbqn")
(outputs '("out" "lib"))
(arguments
- (list #:make-flags '(list "shared-o3" "o3n-singeli")
+ (list #:make-flags '(list "shared-o3" "o3")
#:phases #~(modify-phases %standard-phases
(delete 'configure)
- (add-before 'build 'link-singeli
- (lambda* (#:key inputs #:allow-other-keys)
- (symlink #+singeli-sources "Singeli")))
(add-before 'build 'generate-bytecode
(lambda* (#:key inputs #:allow-other-keys)
(system (string-append #+dbqn
@@ -218,7 +215,6 @@ (define-public cbqn
(install-file "libcbqn.so" lib)))))))
(native-inputs (list dbqn
bqn-sources
- singeli-sources
libffi
clang-toolchain
linux-libre-headers))))

base-commit: bb2701b9111a3d82a82ceaaf2b22b51ecd8ac21f
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:43
[PATCH v2 5/8] gnu: cbqn: Build using GCC.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
943f0bc6e7e697f5d02d18e8d6fe0931dc8ff196.camel@gmail.com
* gnu/packages/bqn.scm (cbqn-bootstrap, cbqn)[#:make-flags]: Set CC via
‘cc-for-target’.
[inputs]: Remove clang-toolchain.
---
gnu/packages/bqn.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Toggle diff (29 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index bc714c3d49..fcf2e98bc7 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -142,6 +142,7 @@ (define cbqn-bootstrap
(build-system gnu-build-system)
(arguments
(list #:tests? #f ;skipping tests for bootstrap
+ #:make-flags #~(list (string-append "CC=" #$(cc-for-target)))
#:phases #~(modify-phases %standard-phases
(delete 'configure)
(add-before 'build 'generate-bytecode
@@ -156,7 +157,7 @@ (define cbqn-bootstrap
(copy-recursively "BQN"
(string-append #$output
"/bin/bqn")))))))
- (native-inputs (list dbqn clang-toolchain bqn-sources))
+ (native-inputs (list dbqn bqn-sources))
(inputs (list icedtea-8 libffi))
(synopsis "BQN implementation in C")
(description "This package provides the reference implementation of
@@ -202,5 +203,4 @@ (define-public cbqn
(native-inputs (list dbqn
bqn-sources
libffi
- clang-toolchain
linux-libre-headers))))
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:25
[PATCH v2 2/8] gnu: Remove singeli-sources.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
2ac3a9fa2bb4acf33edab88639f68484a767a7d2.camel@gmail.com
They aren't currently used to build a package, nor a package themselves.

* gnu/packages/bqn.scm (singeli-sources): Delete variable.
---
gnu/packages/bqn.scm | 11 -----------
1 file changed, 11 deletions(-)

Toggle diff (24 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index 02f539bb8b..be634d6dfb 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -165,17 +165,6 @@ (define cbqn-bootstrap
(home-page "https://mlochbaum.github.io/BQN/")
(license license:gpl3))))
-(define singeli-sources
- (let ((commit "fd17b144483549dbd2bcf23e3a37a09219171a99"))
- (origin
- (method git-fetch)
- (uri (git-reference
- (url "https://github.com/mlochbaum/Singeli")
- (commit commit)))
- (file-name (git-file-name "singeli-sources" commit))
- (sha256
- (base32 "1rr4l7ijzcg25n2igi1mzya6qllh5wsrf3m5i429rlgwv1fwvfji")))))
-
(define-public cbqn
(package
(inherit cbqn-bootstrap)
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:34
[PATCH v2 4/8] gnu: cbqn: Rewrite in terms of cbqn-bootstrap.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
69d947544bdbcec3c1eb06ff349c7b467b5b759b.camel@gmail.com
* gnu/packages/bqn.scm (cbqn)[arguments]: Use substitute-keyword-arguments
to preserve options from cbqn-bootstrap.
---
gnu/packages/bqn.scm | 59 +++++++++++++++++++++-----------------------
1 file changed, 28 insertions(+), 31 deletions(-)

Toggle diff (72 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index 8a1b1dfba2..bc714c3d49 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -171,37 +171,34 @@ (define-public cbqn
(name "cbqn")
(outputs '("out" "lib"))
(arguments
- (list #:make-flags '(list "shared-o3" "o3")
- #:phases #~(modify-phases %standard-phases
- (delete 'configure)
- (add-before 'build 'generate-bytecode
- (lambda* (#:key inputs #:allow-other-keys)
- (system (string-append #+dbqn
- "/bin/dbqn ./genRuntime "
- #+bqn-sources))))
- (replace 'check
- (lambda* (#:key inputs tests? #:allow-other-keys)
- (when tests?
- (system (string-append "./BQN -M 1000 \""
- #+bqn-sources
- "/test/this.bqn\""))
- (map (lambda (x)
- (system (string-append "./BQN ./test/" x
- ".bqn")))
- '("cmp" "equal" "copy" "random"))
- (system "make -C test/ffi"))))
- (replace 'install
- (lambda* (#:key outputs #:allow-other-keys)
- (let* ((bin (string-append (assoc-ref outputs
- "out")
- "/bin"))
- (lib (string-append (assoc-ref outputs
- "lib")
- "/lib")))
- (mkdir-p bin)
- (copy-recursively "BQN"
- (string-append bin "/bqn"))
- (install-file "libcbqn.so" lib)))))))
+ (substitute-keyword-arguments (strip-keyword-arguments
+ (list #:tests?)
+ (package-arguments cbqn-bootstrap))
+ ((#:make-flags flags #~(list))
+ #~(cons* "shared-o3" "o3" #$flags))
+ ((#:phases phases #~%standard-phases)
+ #~(modify-phases #$phases
+ (replace 'check
+ (lambda* (#:key inputs tests? #:allow-other-keys)
+ (when tests?
+ (system (string-append "./BQN -M 1000 \""
+ #+bqn-sources
+ "/test/this.bqn\""))
+ (map (lambda (x)
+ (system (string-append "./BQN ./test/" x
+ ".bqn")))
+ '("cmp" "equal" "copy" "random"))
+ (system "make -C test/ffi"))))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((bin (string-append (assoc-ref outputs "out")
+ "/bin"))
+ (lib (string-append (assoc-ref outputs "lib")
+ "/lib")))
+ (mkdir-p bin)
+ (copy-recursively "BQN"
+ (string-append bin "/bqn"))
+ (install-file "libcbqn.so" lib))))))))
(native-inputs (list dbqn
bqn-sources
libffi
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:28
[PATCH v2 3/8] gnu: cbqn-bootstrap: Use let-bound revision.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
ba98387ad4d0e23c88413b0d291e0965386ea83e.camel@gmail.com
* gnu/packages/bqn.scm (cbqn-bootstrap)[version]: Use let-bound revision.
---
gnu/packages/bqn.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index be634d6dfb..8a1b1dfba2 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -129,7 +129,7 @@ (define cbqn-bootstrap
(commit "9c1cbdc99863b1da0116df61cd832137b196dc5c"))
(package
(name "cbqn-bootstrap")
- (version (git-version "0" "1" commit))
+ (version (git-version "0" revision commit))
(source (origin
(method git-fetch)
(uri (git-reference
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 21:01
[PATCH v2 7/8] gnu: cbqn-bootstrap: Install regular files rather than copying them recursively.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
8948ad36694f74da7604fd0451ded04e775cdbe8.camel@gmail.com
* gnu/packages/bqn.scm (cbqn-bootstrap)[arguments]: Reindent.
<#:phases>{install}: Use ‘install-file’ rather than ‘copy-recursively’ for
“bqn”.
---
gnu/packages/bqn.scm | 33 +++++++++++++++++----------------
1 file changed, 17 insertions(+), 16 deletions(-)

Toggle diff (46 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index 700175db16..2b59d5b6ac 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -140,22 +140,23 @@ (define cbqn-bootstrap
"0w38fhwf20drkyijy6nfnhmc5g5gw0zmzgmy1q605x57znlj85a2"))))
(build-system gnu-build-system)
(arguments
- (list #:tests? #f ;skipping tests for bootstrap
- #:make-flags #~(list (string-append "CC=" #$(cc-for-target)))
- #:phases #~(modify-phases %standard-phases
- (delete 'configure)
- (add-before 'build 'generate-bytecode
- (lambda* (#:key inputs #:allow-other-keys)
- (system (string-append #+dbqn
- "/bin/dbqn ./genRuntime "
- #+bqn-sources))))
- (replace 'install
- (lambda* (#:key outputs #:allow-other-keys)
- (mkdir-p (string-append #$output "/bin"))
- (chmod "BQN" #o755)
- (copy-recursively "BQN"
- (string-append #$output
- "/bin/bqn")))))))
+ (list
+ #:tests? #f ; skipping tests for bootstrap
+ #:make-flags #~(list (string-append "CC=" #$(cc-for-target)))
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'configure)
+ (add-before 'build 'generate-bytecode
+ (lambda* (#:key inputs #:allow-other-keys)
+ (system (string-append #+dbqn
+ "/bin/dbqn ./genRuntime "
+ #+bqn-sources))))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (mkdir-p (string-append #$output "/bin"))
+ (chmod "BQN" #o755)
+ (rename-file "BQN" "bqn")
+ (install-file "bqn" (string-append #$output "/bin")))))))
(native-inputs (list dbqn bqn-sources))
(inputs (list icedtea-8 libffi))
(synopsis "BQN implementation in C")
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 21:01
[PATCH v2 8/8] gnu: cbqn: Install regular files rather than copying them recursively.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
691c381ee820cbb9c45f1056192682fa0e8354a2.camel@gmail.com
* gnu/packages/bqn.scm (cbqn)[#:phases]<install>: Use ‘install-file’ rather
than ‘copy-recursively’ for “bqn”.
---
gnu/packages/bqn.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Toggle diff (17 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index 2b59d5b6ac..e2dc8f85dd 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -197,8 +197,8 @@ (define-public cbqn
(lib (string-append (assoc-ref outputs "lib")
"/lib")))
(mkdir-p bin)
- (copy-recursively "BQN"
- (string-append bin "/bqn"))
+ (rename-file "BQN" "bqn")
+ (install-file "bqn" bin)
(install-file "libcbqn.so" lib))))))))
(native-inputs (list dbqn
bqn-sources
--
2.38.0
L
L
Liliana Marie Prikler wrote on 22 Oct 2022 20:59
[PATCH v2 6/8] gnu: dbqn: Install regular files rather than copying them recursively.
(address . 58100@debbugs.gnu.org)(name . Christopher Rodriguez)(address . yewscion@gmail.com)
cf2bb57d2460c15b0818c2c271837fc1cb24beee.camel@gmail.com
* gnu/packages/bqn.scm (dbqn)[#:phases]<install>: Use ‘install-file’ rather
than ‘copy-recursively’ for “dbqn”.
---
gnu/packages/bqn.scm | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)

Toggle diff (18 lines)
diff --git a/gnu/packages/bqn.scm b/gnu/packages/bqn.scm
index fcf2e98bc7..700175db16 100644
--- a/gnu/packages/bqn.scm
+++ b/gnu/packages/bqn.scm
@@ -92,9 +92,8 @@ (define-public dbqn
(dest-jar (string-append out "/share/java")))
(mkdir-p dest-bin)
(mkdir-p dest-jar)
- (copy-recursively "BQN"
- (string-append dest-bin
- "/dbqn"))
+ (rename-file "BQN" "dbqn")
+ (install-file "dbqn" dest-bin)
(install-file "BQN.jar" dest-jar)
(substitute* (string-append dest-bin "/dbqn")
(("BQN.jar")
--
2.38.0
L
L
Ludovic Courtès wrote on 6 Nov 2022 18:39
Re: bug#58100: cbqn currently targets AVX2
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
8735awgext.fsf_-_@gnu.org
Hi Liliana,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

Toggle quote (4 lines)
> * gnu/packages/bqn.scm (cbqn)[inputs]: Remove singeli-sources.
> [arguments]<#:make-flags: Replace “o3n-singeli” with “o3”.
> <#:phases>: Remove “link-singeli”.

This series of patches that you sent LGTM!

However, how does it relate to AVX2-specific builds? Anyhow, as you
suggested in the thread, it’d be a good idea to add a ‘tunable?’
property.

Thanks,
Ludo’.
L
L
Liliana Marie Prikler wrote on 6 Nov 2022 19:25
(name . Ludovic Courtès)(address . ludo@gnu.org)
f86773e61bf0252c05534b2a2831f64f44996c8b.camel@gmail.com
Am Sonntag, dem 06.11.2022 um 18:39 +0100 schrieb Ludovic Courtès:
Toggle quote (11 lines)
> Hi Liliana,
>
> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
>
> > * gnu/packages/bqn.scm (cbqn)[inputs]: Remove singeli-sources.
> > [arguments]<#:make-flags: Replace “o3n-singeli” with “o3”.
> > <#:phases>: Remove “link-singeli”.
>
> This series of patches that you sent LGTM!
>
> However, how does it relate to AVX2-specific builds?  
IIUC, singeli is some tool that emits avx2-specific opcodes and the
singeli builds of cbqn use that somehw; I admit, I do not completely
understand what's going on on their side.

Toggle quote (2 lines)
> Anyhow, as you suggested in the thread, it’d be a good idea to add a
> ‘tunable?’ property.
Will simply adding (tunable? . #t) as a property be enough? Also
related to the signeli thing, do we have the means to create special
build flows for tuned packages?

Cheers
L
L
Ludovic Courtès wrote on 7 Nov 2022 14:35
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
87a652ubsx.fsf@gnu.org
Hi,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

Toggle quote (16 lines)
> Am Sonntag, dem 06.11.2022 um 18:39 +0100 schrieb Ludovic Courtès:
>> Hi Liliana,
>>
>> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
>>
>> > * gnu/packages/bqn.scm (cbqn)[inputs]: Remove singeli-sources.
>> > [arguments]<#:make-flags: Replace “o3n-singeli” with “o3”.
>> > <#:phases>: Remove “link-singeli”.
>>
>> This series of patches that you sent LGTM!
>>
>> However, how does it relate to AVX2-specific builds?  
> IIUC, singeli is some tool that emits avx2-specific opcodes and the
> singeli builds of cbqn use that somehw; I admit, I do not completely
> understand what's going on on their side.

Oh, I see.

Toggle quote (6 lines)
>> Anyhow, as you suggested in the thread, it’d be a good idea to add a
>> ‘tunable?’ property.
> Will simply adding (tunable? . #t) as a property be enough? Also
> related to the signeli thing, do we have the means to create special
> build flows for tuned packages?

Yes, ci.guix builds tunable packages for several x86_64 ISA variants,
which are listed in (gnu ci).

Thanks,
Ludo’.
L
L
Liliana Marie Prikler wrote on 7 Nov 2022 22:03
(name . Ludovic Courtès)(address . ludo@gnu.org)
01c4d98fab0c6f58fcedab8fc7eb717161d15b5f.camel@gmail.com
Am Montag, dem 07.11.2022 um 14:35 +0100 schrieb Ludovic Courtès:
Toggle quote (30 lines)
> Hi,
>
> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
>
> > Am Sonntag, dem 06.11.2022 um 18:39 +0100 schrieb Ludovic Courtès:
> > > Hi Liliana,
> > >
> > > Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
> > >
> > > > * gnu/packages/bqn.scm (cbqn)[inputs]: Remove singeli-sources.
> > > > [arguments]<#:make-flags: Replace “o3n-singeli” with “o3”.
> > > > <#:phases>: Remove “link-singeli”.
> > >
> > > This series of patches that you sent LGTM!
> > >
> > > However, how does it relate to AVX2-specific builds?  
> > IIUC, singeli is some tool that emits avx2-specific opcodes and the
> > singeli builds of cbqn use that somehw; I admit, I do not
> > completely understand what's going on on their side.
>
> Oh, I see.
>
> > > Anyhow, as you suggested in the thread, it’d be a good idea to
> > > add a ‘tunable?’ property.
> > Will simply adding (tunable? . #t) as a property be enough?  Also
> > related to the signeli thing, do we have the means to create
> > special build flows for tuned packages?
>
> Yes, ci.guix builds tunable packages for several x86_64 ISA variants,
> which are listed in (gnu ci).
Done and pushed.

Cheers
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 58100@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 58100
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch