[PATCH] gnu: Add c-lightning.

Good morning Guix developers,

C-Lightning is one of the earliest living implementations of the Lightning

Network protocol.

I am one of the developers of this implementation.

I tested this as follows:

check for reproducability issues.

`lightningd --help`, and the developer-secret undocumented

`lightningd --test-daemons-only` and checked they work correctly.

--bind-addr=127.0.0.1:9735 --addr=statictor:127.0.0.1:9051

--always-use-proxy=true` on a server with Bitcoin and Tor installed,

and checked that it indeed created an LN wallet and started operating

the node.

* This also tests the SQLITE database backend and that the sub-daemons

are accessible and runnable.

to the new node running on the above command, and confirmed that the

above new node was connected to, and that it started downloading the

gossip map (the first thing every new node does when it initially gets

connected to the network).

the `c-lightning.scm` file and reran `guix build --rounds=4 -f

c-lightning.scm` as well.

machine as on the first test server.

`find ${GNU_STORE_PATH} -type f | sort | xargs -n1 sha256sum`.

`c-lightning-postgresql.scm` file.

server to check for reproducibility issues.

`c-lightning-postgresql` had references to `PQ*` functions, and that

the `c-lightning` version had none, meaning that `c-lightning-postgresql`

indeed included PostgreSQL support.

Thus, PostgreSQL support is not fully tested yet, but I believe the

produced binaries will work correctly as a C-Lightning instance.

Therefore I would like to submit this patch for inclusion into Guix.

Some notes:

several additional python packages, some of which (e.g.

`python-bitcoinlib`) are not yet in Guix.

The tests are extensive and *very long*, running the entire test

suite can take an hour or more depending on exact settings.

relative-path `BINTOPKGLIBEXECDIR`, making it an absolute path, as

it also incidentally solves the issue of "upgrading C-lightning may

cause strange problems in a running instance".

See patch for more details.

fix those upstream.

Regards,

ZmnSCPxj

From 490aa608765228fbb6bb64b9fa679bb4c8145001 Mon Sep 17 00:00:00 2001

---

gnu/packages/finance.scm | 153 +++++++++++++++++++++++++++++++++++++++

1 file changed, 153 insertions(+)

--

2.30.1

Hello!

Thanks for the patch!

I built it and it was successful, however, few comments:

The various hacks in the recipe to get a working build seem troublesome

maintenance-wise, is it possible you think since you are a developer

that you make the build scripts a bit more standard in the project?

I see there's some vendored libraries: gheap, jsmn, libbacktrace,

libsodium, libwally-core ; Is it possible to unvendor those and create

separate packages for each of them? Or is there a requirement of strict

behavior here, either way, you could also create separate packages for

each of them even if those are specific to c-lightning. It would make

the main recipe cleaner, since as far as I can see it's also why the

build system is so stubborn to GNU Guix? That it also attempts to build

vendored libraries?

Léo

Additionally, please do enable the test suite it's really valuable in

GNU Guix.

I can help packaging the necessary Python dependencies, also we have a

Python importer, e.g. "$ guix import pypi -r <pkg>" to help us go

faster at it.

Good morning Leo,

Already working on that, but 0.10.0 will release soon and I doubt my changes will make it by then, so Guix might get c-lightning to 0.10.1 or later if we wait for it.

I think it is more the custom `configure` script.

The custom `configure` script also compiles a ***C*** program that then generates the configuration for the `ccan` library of the project, meaning cross-compilation requires special work for C-Lightning.

And `configure` does not call `configure` of the included libraries as well.

Yes, it is true that there is something of a requirement of a strict behavior here, I suppose it is possible to use `git-fetch` instead of `url-fetch` for our external libraries.

How do I generate `guix hash` for `git-fetch`ed `source`s?

However it also means that every release of C-Lightning I have to go double-check what git commit to use for each library (though `jsmn` and `libbacktrace` at least seem very stable).

But it looks to me that unvendoring will require more extensive patching of the `Makefile` and an even larger maintenance burden on Guix side?

`libwally-core` itself depends on another library `libsecp256k1`, so I suppose it must transitively be unvendored as well.

The test suite is fairly large and can take a significant amount of time to run in full.

In addition, part of the test includes checks which take advantage of `BINTOPKGLIBEXECDIR` relative path we normally use, which I want to disable for Guix at least since the relative path only has an advantage if the user wants to move the installation after-the-fact to a different location (and on Guix the "installation" path cannot be moved anyway).

Using an absolute `BINTOPKGLIBEXECDIR` gives an advantage as mentioned in the comments that a profile being upgraded from one version of C-Lightning to another does not cause problems for daemons currently running off the profile.

The test can be disabled (but not easily), I suppose.

Please do, I am not very familiar with any Python infrastructure and am primarily a C programmer here, I just barely hack together some kind of test in Python.

Regards,

ZmnSCPxj

On Tue, 2021-03-16 at 11:27 +0000, ZmnSCPxj wrote:

Awesome!

Yes you can use git-fetch, to make sure we are on the same page, are

you speaking of strict behavior requirements like for Bitcoin Core's

consensus code?

Actually what I do is put a wrong hash in and then copy the "actual

hash" from the error. I havent found another way but this definitely

feels subpar and prevents much verification before putting in the hash,

better suggestions welcome.

Unvendoring is more or less a policy because we must be able to audit

each piece of software separately for freedom issues (licenses,

violations to the GNU FSDG), and it eases work for security-patching

also.

Probably, we already have a libsecp256k1 specially for Bitcoin Core.

That's fine.

Problematic or unrealistically expensive tests can be disabled yes.

If you can list the Python dependencies and their version I can look at

packaging them.

Léo

Good morning Leo,

Well we need to produce signatures and transactions that pass Bitcoin Core signature validation at least, so it is best to use a version of `libsecp256k1` (which produces signatures) that we know works, as well as `libwally-core` (which produces transactions).

I would personally use the `libsecp256k1` version that `libwally-core` vendors in as well, since there may be specific interactions between `libwally-core` and `libsecp256k1` that may be different if we use the Bitcoin Core version of `libsecp256k1`.

For `libsodium`, at least the hashing has to work correctly, but I think it is simple enough that strict behavior requirements are not so onerous.

Indeed, we usually get this from the OS (but we need a later feature than that available on some old Ubuntu versions, which is why it got vendored in), so I should probably "just" add it as an input.

Haha I shall do so as well.

I understand.

This will require a largish amount of work I think.

Would this technique be acceptable?

* `rm -rf` the vendored externals.

* `ln -s` the needed `.h` and `.la`/`.a`/`.so` files from the `inputs` to the expected paths within the `external/` directory.

?

We have a `requirements.txt` file which contains this, I duplicate below:

```

# Dependencies required to build and test c-lightning

https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp36-cp36m-linux_x86_64.whl;'linux' in sys_platform and python_version == '3.6'

https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-linux_x86_64.whl;'linux' in sys_platform and python_version == '3.7'

https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl;sys_platform == 'darwin' and python_version == '3.7'

mrkd ~= 0.1.6

Mako ~= 1.1.3

# Dependencies from pyln-client

Sphinx ~= 3.4.0

flake8==3.7.8

recommonmark>=0.7.*

sphinx-rtd-theme==0.4.2

sphinxcontrib-websupport==1.1.0

tqdm==4.32.2

# Dependencies from pyln-testing

Flask==1.1.*

cheroot==8.5.*

ephemeral-port-reserve==1.1.1

filelock==3.0.*

flaky ~= 3.7.0

psutil==5.7.*

psycopg2-binary==2.8.*

pytest-rerunfailures==9.1.1

pytest-timeout ~= 1.4.2

pytest-xdist ~= 2.2.0

pytest==6.1.*

python-bitcoinlib==0.11.*

# Dependencies from pyln-proto

base58 ~= 2.0.1

bitstring ~= 3.1.6

coincurve ~= 13.0.0

cryptography ~= 3.2

mypy ~= 0.790

pysocks ~= 1.7.1

# Dependencies from pyln-spec

# None

```

Incidentally, we also install some Python modules.

How do I "properly" export the Python modules within Guix?

Regards,

ZmnSCPxj

On Wed, 2021-03-17 at 03:42 +0000, ZmnSCPxj wrote:

Maybe simply add an option like --with-system-libbacktrace etc. in the

build system? And yes remove externals, I think that you should use a

snippet in the origin field instead of a phase for that.

Thank you, I am going to go through them.

I will ask someone else here but you might have to mix the python-

build-system in.

Léo

Good morning Leo,

That would not be simple I think, but let me see what can be done.

For one, it would be fairly difficult to test it outside of a patched Guix, as the packages are not usually available in most distros (which is why we ship the source build with them); e.g. we do not know how to actually find the "system" libbacktrace as no existing system I know of actually ships the libbacktrace we use (Debian ships a libbacktrace for android development, and is not the same package as what we use).

Of the external packages we use, only libsodium is available on Debian (and older Debian/Ubuntu, as mentioned, have an older version that does not have a functionality we require, which is why it is included in our source build as well), so that is the only external we have that we can plausibly say "we can use the system version", because all the others are not available on most systems.

Also --- we would be somewhat wary of taking the "system" `libsecp256k1`, as this is very consensus-critical, and we know that our program works with a very specific version, but cannot assure this to be true if the "system" `libsecp256k1` is not the exact version.

(Note that we cannot take the `libsecp256k1` from `bitcoin-core` because (1) the `bitcoin-core` package uses a vendored `libsecp256k1`, the separate Guix `libsecp256k1` is actually only used by `electrum` and related projects and (2) the interface `bitcoin-core` uses may be different from the interface we use, it would require review before we would be able to assure that the interfaces they use are exactly the same; this would be mitigated by running the full test suite as it also tests consensus-criticality.)

Is there an easy way to ensure that any `libsecp256k1` that gets fed into the `c-lightning` build as an `inputs` is of a specific `git` commit?

Hmm do you mean something like:

(origin

; whatever...

(snippet

'(begin

(delete-file-recursively "external/gheap")

(delete-file-recursively "external/jsmn")

(delete-file-recursively "external/libbacktrace")

(delete-file-recursively "external/libsodium")

(delete-file-recursively "external/libwally-core"))))

?

How do you mix in a build system?

Would `python-pyqt` be a good example of such "mix in"?

Regards,

ZmnSCPxj

On Thu, 2021-03-18 at 16:54 +0000, ZmnSCPxj wrote:

We can package every of those in GNU Guix, there's not as much

bureaucracy as Debian when it comes to inclusion, as long as Free

Software and freedom-non-controversial. The "system" thing is just so

handy for us build-system wise we can just provide it as an input and

it will find things automatically, I don't say that "system" lookup

should imply to take any available library in general, in the context

of GNU Guix we can give it a particular version and stick to it, unlike

Debian's who can't have or with great difficulty, multiple versions of

the same library installed.

Yes we can package yet another variant like Bitcoin Core at a specific

commit for c-lightning.

Yes!

Maybe? I'm no expert here.

FYI, I'm not as available for GNU Guix as I'd like, I am looking at the

Python deps, just it will take me some time to actually do it (not at

home now). I see python-mako is already packaged, python-wallycore I am

packaging it, python-mrkd also looking.

Also if we want to provide the Python bindings (?) to c-lightning we

also have the option of creating a separate package based on the same

sources but with different build-system (like python-build-system), and

also it can have c-lightning as an input.

Léo

Good morning Leo,

Okay.

It is somewhat unclear to me however how to discover where the library is located.

As far as I understand how autotools does it, it looks at some standard locations on `/`, but I am uncertain if that is how it actually works.

For example if `libwally-core` is separately packaged, obviously it would install into a `/gnu/store/*` via a `--prefix` override.

And if the dependent package were using autotools, it is somehow able to magically find where the `lib*.a` file is.

However, C-Lightning does not actually use autotools (its build system predated my participation, I offered to port to autotools but this was declined), so I somehow need to emulate what autotools does in `--use-system-libraries` mode.

I probably have to go hack some autotools package that trivially depends on `libwally-core` and see what the exact shell script is generated.

Note that AFAIK the Python bindings do not actually invoke any of the C-Lightning binaries, they just open the UNIX socket to the c-lightning daemon, so I think there is no need for `c-lightning` itself as an input.

Regards,

ZmnSCPxj

Hello!

pkg-config if your deps provide .pc files! Otherwise I am actually not

sure of the lookup strategies.

Alright!

Léo

Just for your information, I am quite busy on a GNOME upgrade now, we

have the 1.2.1 release soon also, some security issues to solve, I told

you earlier I would help but at the end I think it may come a bit later

rather than sooner. Hang out on the IRC channel #guix on Freenode for

more real-time collaboration if that's of interest to you!