Matterbridge contained a lot of vendored code

OpenSubmitted by Denis 'GNUtoo' Carikli.
Details
3 participants
  • Denis 'GNUtoo' Carikli
  • Leo Famulari
  • Liliana Marie Prikler
Owner
unassigned
Severity
normal
D
D
Denis 'GNUtoo' Carikli wrote on 23 Oct 16:57 +0200
(address . bug-guix@gnu.org)
20211023165702.1e518f56@primarylaptop
Hi,
When I sent the patch adding matterbridge to Guix, I only notified thatI didn't know if it contained vendored code or not at the last moment(after the patch was sent, during the discussion about it, and beforeit was merged).
The issue is that I didn't know go at all and more specifically I didn'tknow its the compilation system worked. So I managed to create apackage for matterbridge by looking at how it was done for other gopackages.
After learning more about how go compilation worked, I found out thatmatterbridge contained a lot of vendored code.
And Guix explicitly wants to avoid bundles code. In the "16.6 SubmittingPatches" section of the manual[1], we have:
Toggle quote (2 lines)> 6. Make sure the package does not use bundled copies of software> already available as separate packages.
And here while most dependencies are not already packaged, some are,and I guess that I should read between the lines and conclude that allthe matterbridge dependencies should rather be packaged.
So the question is what should we do about that.
As I understand with the go build system, or you vendor alldependencies, or you vendor none, and I've not yet managed to find away to workaround that yet in Guix (to do a progressive unvendoring).
So instead I've started working on unvendoring matterbridge[2]completely, but if we go this route, there are more than 500dependencies.
To do that I first used the following command: guix import go -r github.com/42wim/matterbridge
I then started looking at each package definition that Guix didn'tmanage to detect the license of, and I read the licenses to find ifthey were free software. All the licenses I read were FSDG compliant.Usually they had some extra text indicating the provenance of the codeor they would have multiple free software licenses.
Then I started adding packages for the dependencies that guix import godidn't manage to find.
Theses are repositories that are being forked from the official onesfor a reason or another.
I've not finished that yet, but I still think it was a good idea toopen a bug report as I've now more understanding of the problem.
Given the huge amount of dependencies I was wondering what was the bestapproach here:- Would it makes sense to remove matterbridge from Guix, or should we fix it instead?- If we fix it by packaging each dependencies, would it be ok if that is done step by step, like if dependencies are packaged and patches for them are sent, without necessarily a way to seriously test if the packaged dependency work until they are used by other software (like matterbridge)?
Also when I'll manage to update matterbridge[3] how should we deal withsuch amount of packages? Would I need to send one (generated) patch forthe upgrade of each package?
I also guess that sticking as much as possible to what Guix import gogenerates would help in situations like that as it would make themaintenance faster.
References:-----------[1]https://guix.gnu.org/manual/en/guix.html#Submitting-Patches[2]https://git.replicant.us/contrib/GNUtoo/infrastructure/guix/log/?h=matterbridge-unvendor[3]Right now there is a compilation issue that I didn't manage to fix, even with help from #guix).
Denis.
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEeC+d2+Nrp/PU3kkGX138wUF34mMFAmF0IsAACgkQX138wUF34mOH+Q//ZpZ+CbsJyQfHQmfgy33cT01st4p8UWP4/+1NIyw43bS/A+vtrqNv46B+P/uN9p7OhZ2XG5FsdQ/HrcB2akuOMDhSR+qPsgAFr1uvVajJz9lB0uS1gUZ8za9q++KdwXxV4/LxBi/5aBCxirLKP3F5Yn4S2tXifJyMvC3lI2w1gF3boy0sGx2oIvupUvjSew7aKVa60rZkk2GpdoU3S7Xk+RxyirowDeS2kvkcf/VidJK/BFbv2JVCkmk5rFh6g+b3ewU8+kBv0axNz9taNz8kcZ7YzfiZRSzxJEwANSF62Sy4CFK9c+7TDAD0Q7+6Yhj/N8Cur7kwPKSrJVyO/8O45CN3OsmA6zld09FofPrfIXQ2DCJNNF8gwFl9pNven+IikcmE03780pVJwdbrqCAkMEbiZ+ydP2j4uegiGNWAeNsAaydQVBSpm4GAqlOdrI9dleszAkGlfZmkYdpSfB6ULgCTzwTUsHIu6ftwVlEK+05yGVOkETvNYiA8Fw+dVkw3QTP8Rcq3zzXj6nhso8xzKgmhlvWSAvO1X7gjV6TTQ15o97eP4dlLLqIAeCtWPaDT0mDqmPsKbfl99nYYBZMLHlYQdk+h+hKNXzs3/6YHJ3oCPqa3uDa3fcIEe71584IUdFgU/nChHATwfhmlBls3tHYqblNei0XT5FRhD2nikC8==1Rqb-----END PGP SIGNATURE-----

L
L
Liliana Marie Prikler wrote on 23 Oct 18:43 +0200
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
02da99762b4b0ac01d05e22f974d85737a351815.camel@gmail.com
Hi,
Am Samstag, den 23.10.2021, 16:57 +0200 schrieb Denis 'GNUtoo' Carikli:
Toggle quote (14 lines)> Hi,> > When I sent the patch adding matterbridge to Guix, I only notified> that I didn't know if it contained vendored code or not at the last> moment (after the patch was sent, during the discussion about it, and> before it was merged).> > The issue is that I didn't know go at all and more specifically I> didn't know its the compilation system worked. So I managed to create> a package for matterbridge by looking at how it was done for other go> packages.> > After learning more about how go compilation worked, I found out that> matterbridge contained a lot of vendored code.
Judging from the original issue, you did note that guideline and thatyou were unsure about the "under the hood" stuff, which imo should haveprompted a closer look. CC'd Mathieu to check whether pushing itregardless was intentional.
Toggle quote (29 lines)> And Guix explicitly wants to avoid bundles code. In the "16.6> Submitting Patches" section of the manual[1], we have:> > 6. Make sure the package does not use bundled copies of software> > already available as separate packages.> And here while most dependencies are not already packaged, some are,> and I guess that I should read between the lines and conclude that> all the matterbridge dependencies should rather be packaged.> > So the question is what should we do about that. > > As I understand with the go build system, or you vendor all> dependencies, or you vendor none, and I've not yet managed to find a> way to workaround that yet in Guix (to do a progressive unvendoring).> > So instead I've started working on unvendoring matterbridge[2]> completely, but if we go this route, there are more than 500> dependencies.> > To do that I first used the following command:> guix import go -r github.com/42wim/matterbridge> > I then started looking at each package definition that Guix didn't> manage to detect the license of, and I read the licenses to find if> they were free software. All the licenses I read were FSDG compliant.> Usually they had some extra text indicating the provenance of the> code or they would have multiple free software licenses.> > Then I started adding packages for the dependencies that guix import> go didn't manage to find.
It's good that you've started. Perhaps you might want to illustratethe dependency "tree" (assuming it is one) or graph to roughly indicatehow much needs to be done and at which points we could cut off portionsof the DAG to process as batches.
Toggle quote (10 lines)> Theses are repositories that are being forked from the official ones> for a reason or another.> > I've not finished that yet, but I still think it was a good idea to> open a bug report as I've now more understanding of the problem.> > Given the huge amount of dependencies I was wondering what was the> best approach here:> - Would it makes sense to remove matterbridge from Guix, or should we> fix it instead?
I'll let Mathieu decide that one.
Toggle quote (5 lines)> - If we fix it by packaging each dependencies, would it be ok if that> is done step by step, like if dependencies are packaged and patches> for them are sent, without necessarily a way to seriously test if> the packaged dependency work until they are used by other software> (like matterbridge)?
Ideally, those packages would at least have a working test suite, but Iunderstand that such concerns are not always valid in the industry :)As noted before, cutting off branches of the dependency DAG atreasonable points and committing them in batches sounds like a goodidea to me. No one wants to review a 500 commit bomb.
Toggle quote (3 lines)> Also when I'll manage to update matterbridge[3] how should we deal> with such amount of packages? Would I need to send one (generated)> patch for the upgrade of each package?
One patch per package, plus one to fix the matterbridge package.
Toggle quote (3 lines)> I also guess that sticking as much as possible to what Guix import go> generates would help in situations like that as it would make the> maintenance faster.
The importers exist to make your life easier, but they're notomniscient. Please do exercise caution when dealing with them :)
Cheers,Liliana
L
L
Leo Famulari wrote on 23 Oct 21:49 +0200
Re: bug#51352: Matterbridge contained a lot of vendored code
(name . Denis 'GNUtoo' Carikli)(address . GNUtoo@cyberdimension.org)(address . 51352@debbugs.gnu.org)
YXRnSKLCiClkxvaC@jasmine.lan
On Sat, Oct 23, 2021 at 04:57:02PM +0200, Denis 'GNUtoo' Carikli wrote:
Toggle quote (5 lines)> Given the huge amount of dependencies I was wondering what was the best> approach here:> - Would it makes sense to remove matterbridge from Guix, or should we> fix it instead?
There's no need to remove it.
Vendoring, or bundling, is not a matter of software freedom, but ratherabout maintainability or transparency.
With a bundled dependency graph, the Guix tools for inspecting andmanipulating the dependency graph do not work. But, we already bundlethe dependencies of some other Go packages, and every Rust package doesnot work with the Guix dependency graph tooling.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 51352@debbugs.gnu.org