[PATCH 00/47] gnu: Add go-github-com-prometheus-prometheus.

Hi there. Long-time Guix user, first-time contributor.

I've been chipping away at packaging Prometheus for quite some time now.

The dependency tree turned out to be sizeable, apologies for the long

list of patches. Most of these are new packages.

I've had to resort to some tricks to reduce this effort:

* Most discovery plugins are patched out for now using a lightweight

patch, as they pulled in far more dependencies than was manageable

in a single patch set.

* I'm targeting the previous version. The latest version released a

few weeks ago (2.34.0) uses OpenTelemetry for tracing, pulling in

another complex set of dependencies, and I didn't want to patch out

tracing entirely.

Some additional notes:

* Alertmanager and other components are not included, nor the web UI

written in JavaScript. My goal was to get the base server running.

* The version of go-golang-org-x-net currently in Guix is slightly

older, and doesn't include the HTTP/2 ConfigureTransports function.

I thought I'd rather not update this package to avoid rebuilding its

dependents, and resorted to another patch.

* A couple of the Prometheus packages already existed under (gnu

packages syncthing), although unused. These look like remnants of

older packages. I kept them in the same module, but added new

packages under (gnu packages golang).

There might be appetite to consolidate all of these Prometheus

dependencies under (gnu packages golang), I'm willing to put in the

work if that's the case.

I've tested this package on x86_64 and aarch64, and attempted to follow

the relevant style guides as best I could. Let me know if (when) there's

anything I've missed.

Looking forward to your feedback!

Leo Nikkilä (47):

gnu: Add go-google-golang-org-protobuf.

gnu: go-github-com-golang-protobuf-proto: Update to 1.5.2.

gnu: Add go-github-com-golang-protobuf-ptypes.

gnu: go-github-com-prometheus-client-golang: Update to 1.12.1.

gnu: Add go-github-com-asaskevich-govalidator.

gnu: Add go-github-com-dennwc-varint.

gnu: Add go-github-com-edsrzf-mmap-go.

gnu: Add go-github-com-go-logfmt-logfmt.

gnu: Add go-github-com-go-kit-log.

gnu: Add go-github-com-go-stack-stack.

gnu: Add go-github-com-josharian-intern.

gnu: Add go-github-com-julienschmidt-httprouter.

gnu: Add go-github-com-mailru-easyjson.

gnu: Add go-github-com-modern-go-concurrent.

gnu: Add go-github-com-modern-go-reflect2.

gnu: Add go-github-com-montanaflynn-stats.

gnu: Add go-github-com-mwitkow-go-conntrack.

gnu: Add go-github-com-oklog-run.

gnu: Add go-github-com-oklog-ulid.

gnu: Add go-github-com-opentracing-contrib-go-stdlib.

gnu: Add go-github-com-puerkitobio-urlesc.

gnu: Add go-github-com-puerkitobio-purell.

gnu: Add go-github-com-shurcool-httpgzip.

gnu: Add go-github-com-shurcool-httpfs.

gnu: Add go-github-com-shurcool-vfsgen.

gnu: Add go-github-com-simonpasquier-klog-gokit.

gnu: Add go-github-com-simonpasquier-klog-gokit-v3.

gnu: Add go-github-com-tidwall-pretty.

gnu: Add go-github-com-uber-jaeger-lib.

gnu: Add go-github-com-uber-jaeger-client-go.

gnu: Add go-github-com-youmark-pkcs8.

gnu: Add go-go-uber-org-goleak.

gnu: Add go-gopkg-in-alecthomas-kingpin.v2.

gnu: Add go-github-com-json-iterator-go.

gnu: Add go-go-mongodb-org-mongo-driver.

gnu: Add go-github-com-go-openapi-errors.

gnu: Add go-github-com-go-openapi-swag.

gnu: Add go-github-com-go-openapi-jsonpointer.

gnu: Add go-github-com-go-openapi-jsonreference.

gnu: Add go-github-com-go-openapi-spec.

gnu: Add go-github-com-go-openapi-strfmt.

gnu: Add go-github-com-go-openapi-analysis.

gnu: Add go-github-com-go-openapi-loads.

gnu: Add go-github-com-go-openapi-validate.

gnu: Add go-github-com-prometheus-alertmanager-api.

gnu: Add go-github-com-prometheus-exporter-toolkit.

gnu: Add go-github-com-prometheus-prometheus.

gnu/packages/golang.scm | 1248 ++++++++++++++++-

gnu/packages/monitoring.scm | 83 ++

...etheus-common-remove-readidletimeout.patch | 28 +

...om-prometheus-prometheus-remove-deps.patch | 33 +

gnu/packages/syncthing.scm | 182 +--

5 files changed, 1489 insertions(+), 85 deletions(-)

create mode 100644 gnu/packages/patches/go-github-com-prometheus-common-remove-readidletimeout.patch

create mode 100644 gnu/packages/patches/go-github-com-prometheus-prometheus-remove-deps.patch

base-commit: d05fcc21cb9509084a0424e6808b84b58dc52d62

--

2.34.0

---

gnu/packages/golang.scm | 43 +++++++++++++++++++++++++++++++++++++++++

1 file changed, 43 insertions(+)

to 1.5.2.

[arguments]: Enable tests.

---

gnu/packages/golang.scm | 10 +++++-----

1 file changed, 5 insertions(+), 5 deletions(-)

variable.

---

gnu/packages/golang.scm | 8 ++++++++

1 file changed, 8 insertions(+)

New patch, avoid updating go-golang-org-x-net just yet.

Update to 1.12.1.

(go-github-com-prometheus-client-model): Update to 0.2.0.

(go-github-com-prometheus-common): Update to 0.32.1.

[patches]: Use patch.

(go-github-com-prometheus-procfs): Update to 0.7.3.

{unpack-test-data}, {delete-test-data}: New phases.

[arguments]: Enable tests.

---

...etheus-common-remove-readidletimeout.patch | 28 +++

gnu/packages/syncthing.scm | 182 ++++++++++--------

2 files changed, 130 insertions(+), 80 deletions(-)

create mode 100644 gnu/packages/patches/go-github-com-prometheus-common-remove-readidletimeout.patch

variable.

---

gnu/packages/golang.scm | 33 +++++++++++++++++++++++++++++++++

1 file changed, 33 insertions(+)

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

---

gnu/packages/golang.scm | 26 ++++++++++++++++++++++++++

1 file changed, 26 insertions(+)

---

gnu/packages/golang.scm | 27 +++++++++++++++++++++++++++

1 file changed, 27 insertions(+)

---

gnu/packages/golang.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

variable.

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

variable.

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

variable.

---

gnu/packages/golang.scm | 32 ++++++++++++++++++++++++++++++++

1 file changed, 32 insertions(+)

variable.

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

variable.

---

gnu/packages/golang.scm | 22 ++++++++++++++++++++++

1 file changed, 22 insertions(+)

variable.

---

gnu/packages/golang.scm | 28 ++++++++++++++++++++++++++++

1 file changed, 28 insertions(+)

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

variable.

---

gnu/packages/golang.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

variable.

---

gnu/packages/golang.scm | 28 ++++++++++++++++++++++++++++

1 file changed, 28 insertions(+)

New variable.

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

---

gnu/packages/golang.scm | 57 +++++++++++++++++++++++++++++++++++++++++

1 file changed, 57 insertions(+)

variable.

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

---

gnu/packages/golang.scm | 28 ++++++++++++++++++++++++++++

1 file changed, 28 insertions(+)

variable.

---

gnu/packages/golang.scm | 27 +++++++++++++++++++++++++++

1 file changed, 27 insertions(+)

---

gnu/packages/golang.scm | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)

variable.

---

gnu/packages/golang.scm | 35 +++++++++++++++++++++++++++++++++++

1 file changed, 35 insertions(+)

New variable.

---

gnu/packages/golang.scm | 17 +++++++++++++++++

1 file changed, 17 insertions(+)

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

variable.

---

gnu/packages/golang.scm | 12 ++++++++++++

1 file changed, 12 insertions(+)

variable.

---

gnu/packages/golang.scm | 31 +++++++++++++++++++++++++++++++

1 file changed, 31 insertions(+)

variable.

---

gnu/packages/golang.scm | 27 +++++++++++++++++++++++++++

1 file changed, 27 insertions(+)

---

gnu/packages/golang.scm | 28 ++++++++++++++++++++++++++++

1 file changed, 28 insertions(+)

variable.

---

gnu/packages/golang.scm | 67 +++++++++++++++++++++++++++++++++++++++++

1 file changed, 67 insertions(+)

---

gnu/packages/golang.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

variable.

---

gnu/packages/golang.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

---

gnu/packages/golang.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

variable.

---

gnu/packages/golang.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

variable.

---

gnu/packages/golang.scm | 25 +++++++++++++++++++++++++

1 file changed, 25 insertions(+)

---

gnu/packages/golang.scm | 26 ++++++++++++++++++++++++++

1 file changed, 26 insertions(+)

variable.

---

gnu/packages/golang.scm | 30 ++++++++++++++++++++++++++++++

1 file changed, 30 insertions(+)

variable.

---

gnu/packages/golang.scm | 32 ++++++++++++++++++++++++++++++++

1 file changed, 32 insertions(+)

New variable.

---

gnu/packages/golang.scm | 36 ++++++++++++++++++++++++++++++++++++

1 file changed, 36 insertions(+)

variable.

---

gnu/packages/golang.scm | 37 +++++++++++++++++++++++++++++++++++++

1 file changed, 37 insertions(+)

New variable.

---

gnu/packages/golang.scm | 37 +++++++++++++++++++++++++++++++++++++

1 file changed, 37 insertions(+)

variable.

---

gnu/packages/golang.scm | 26 ++++++++++++++++++++++++++

1 file changed, 26 insertions(+)

New patch, reduce go-github-com-prometheus-prometheus dependency tree.

variable.

---

gnu/packages/monitoring.scm | 83 +++++++++++++++++++

...om-prometheus-prometheus-remove-deps.patch | 33 ++++++++

2 files changed, 116 insertions(+)

create mode 100644 gnu/packages/patches/go-github-com-prometheus-prometheus-remove-deps.patch

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

(A part of) this is already in Guix, see go-github-com-golang-protobuf-

proto. Possibly relevant: https://issues.guix.gnu.org/48259.

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

I'm not seeing go-golang.org-x-sync-errgroup here. So my question is:

does this package build?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

go-google-golang-org-protobuf (which presumably contains the ptypes

stuff) has been packaged in 01/47, so is this separate package

still necessary?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

What are ‘glob paths’ here? They don't appear to be used anywhere else

in Guix. What's the problem with the default build phase?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Why is this phase deleted? I know it is currently being done for a few

other go packages, but AFAIK these deletals are pointless. I'm

wondering if removing them from other Go packages would cause any

problems ...

Greetings,

Maxime

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Is ignoring tests still necessary?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Returning #t from phases is not necessary anymore. I'm wondering, did

you find this in some documentation floating around somewhere or

something? If so, I'd like to know where such that it can be updated.

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Is this still necessary?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

This looks Linux-specific. Is it? If so, please remove non-linux

things from supporetd-systems:

(supported-systems (filter target-linux? %supported-systems))

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Preferably, this description could be expanded a bit to make it more

useful. ‘(guix)Synopses and Descriptions’ has some tips.

This does not seem relevant to the user?

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

From (guix)Synopses and Descriptions:

Keep in mind that the synopsis must be meaningful for a very wide

audience. For example, “Manipulate alignments in the SAM format” might

make sense for a seasoned bioinformatics researcher, but might be

fairly unhelpful or even misleading to a non-specialized audience. It

is a good idea to come up with a synopsis that gives an idea of the

application domain of the package. In this example, this might give

something like “Manipulate nucleotide sequence alignments”, which

hopefully gives the user a better idea of whether this is what they are

looking for.

I guess it is an implementation of bignums and bignum arithmetic?

Maybe "Arithmetic on arbitrarily large numbers"?

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

To me, the note about portability doesn't seem very relevant to people

using Guix, though I suppose there is an argument to be made for

developers looking for sufficiently portable libraries. Guix only

supports GNU targets (GNU/Linux and GNU/Hurd), and I doubt that mmap-go

supports DOS. Everything is unportable, it's the degree of portability

that matters, and the description doesn't qualify the statement ...

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

'mmap' is a C-ism, maybe "Map files into virtual memory", with a

mention of 'mmap' in the description?

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

I think the less fancy words ‘encode’ and ‘decode’ are more widely

understood and just as accurate.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

This is a rather confusing synopsis. If these strings are internal to

Go, why aren't they part of the 'go' package? I know that's not what

is meant here, but I'd recommend a different description ...

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

Can be simplified a bit:

This package is thread-safe.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

While it seems that HttpRouter actually has some evidence for these

claims about lightweightness and performance

(https://github.com/julienschmidt/go-http-routing-benchmark),

it still leans rather close to marketing, and AFAIK the conclusions

have not been independently verified.

This is an implementation detail and hence doesn't seem relevant to me.

Greetings,

Maxime

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

As per (guix)Synopses and Descriptions, no marketing (‘fast and easy’)

please.

This seems better suited for a bug report or patches to encoding/json

and the other JSON packages --- putting this in the Guix description

isn't going to improve encoding/json and friends ...

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

The Go in Guix is >1.9.

Leo Nikkilä schreef op vr 25-03-2022 om 15:31 [+0200]:

It would be nice to actually describe what this package provides

instead of mentioning some symbols.

Leo Nikkilä schreef op vr 25-03-2022 om 15:32 [+0200]:

Why?

Leo Nikkilä schreef op vr 25-03-2022 om 07:55 [+0200]:

In which patch is this?

Anyway, Go packages are relatively cheap to build, so maybe the number

of dependents isn't a problem, and it will have to updated sometime

anyway ...

Leo Nikkilä schreef op vr 25-03-2022 om 07:55 [+0200]:

The (... golang) module is already big (~10000) lines (*), so I would

recommend not putting it in (gnu packages golang) and instead looking

into creating a new module -- maybe (gnu packages prometheus)?

Non-Go prometheus-related packages could be put there as well

Alternatively, maybe the already-existing (gnu packages monitoring)

would be a good place?

(*) IIUC, crates-io becoming big has caused building guix to become

slower because bignums had to be used?

Leo Nikkilä schreef op vr 25-03-2022 om 15:32 [+0200]:

IIUC, the result is some [...]/bin/prometheus binary that can be run or

something like that and not a library, so I think that the 'go-github-

com-prometheus) prefix can be dropped here. I would go with

"prometheus" such that the user can just do "guix shell prometheus".

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 15:32 [+0200]:

I doubt that all of these are test dependencies, probably most of them

need to be in 'inputs'.

Maxime, thank you for the speedy review.

This is in go-github-com-prometheus-common-remove-readidletimeout.patch.

According to `guix refresh --list-dependent' there are 34 dependents, that's less than I expected. I'll remove the patch and attempt an upgrade here.

That might be better for now. It's not as large yet, and it already houses existing packages like go-github-com-prometheus-node-exporter.

Would it be appropriate to update the existing variables in (gnu packages syncthing) to use `deprecated-package'?

Regarding your comments on several patches about descriptions: this is entirely fair, I didn't spend much time on these at all and mostly copied what was described upstream.

I'll update these.

Great point, will update this.

This as well.

The errgroup dependency was dropped in 1.3.1:

Later in 1.4.0, the package was rewritten in terms of google.golang.org/protobuf, thus the added dependency:

I clarified the difference between github.com/golang/protobuf and google.golang.org/protobuf under 03/47.

Should https://issues.guix.gnu.org/48259 be applied, the separate `-proto` and `-ptypes` packages could certainly be replaced by the merged package.

There's no top-level package in google.golang.org/protobuf, instead the repository consists of individual packages which depend on each other.

Looking at (gnu packages golang), I see two ways this has been dealt with before: splitting each one into multiple Guix packages, or building all of the packages together in one Guix package.

There are many of these packages in the repository [0], and another list of "internal" packages [1]. (This listing doesn't cover all of them, some are deeper within the tree.) In order to use this library you'll need most of them, and splitting each one into a separate Guix package feels unnecessary.

There are examples in (gnu packages golang) where a single Guix package contains many Golang packages, e.g. go-github-com-blanu-dust, redefining build phases to reduce a list of packages instead.

However, the number of these packages makes this approach difficult. The list of internal packages is not considered public nor stable; these are added and removed between releases. Maintaining a list of these within Guix seems prone to breaking during the update process.

`go build' and others support wildcards [2], which are also used within build scripts of the library [3]. Since the upstream build makes use of these, I thought they would be fair game to build it.

However, due to how go-build-system correlates these import paths with directory paths, supporting wildcards in `#:import-path` directly would require further changes to the build system.

Adapting how go-github-com-blanu-dust and others call these build phases directly, I found this to be the simplest way to use the wildcard capability, and avoid having to maintain a list of the packages within Guix directly.

Happy to try out other approaches, this is just what I came up with earlier.

I should've also specified `wildcard' instead of `glob' when naming this.

[0]: https://github.com/protocolbuffers/protobuf-go#package-ind

[1]: https://github.com/protocolbuffers/protobuf-go/tree/v1.27.1/internal

[2]: https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns

[3]: https://github.com/protocolbuffers/protobuf-go/blob/v1.27.1/integration_test.go#L118-L120

Looks like you're right, I was able to use the default build phases with these packages locally, after minor tweaks. I believe I followed the pattern from the existing packages earlier, seems it's unnecessary.

Old habit, I learned this when this was still common. Will remove these.

Thank you, it is.

I copied this over from the packages in (... syncthing), this is unnecessary as well. Will remove.

Leo Nikkilä schreef op vr 25-03-2022 om 22:53 [+0000]:

Looking at "git grep -F deprecated-package", 'deprecated-package' seems

to be used to indicate that a package has been renamed, so I guess it

isn't.

Greetings,

Maxime

Leo Nikkilä schreef op vr 25-03-2022 om 22:55 [+0000]:

Looking at go-build-system.scm, it seems that 'unpack, 'install'

and 'install-licenses' need to be modified to drop the "/..." suffix,

if any? If this is done in go-build-system, then more packages could

benefit and perhaps some existing package definitions could be

simplified.

;; See https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns

;; for how wildcard paths work.

(define (unwildcard-import-path import-path

(if (string-suffix? "/..." import-path)

(string-drop-right import-path 4)

import-path))

;; in 'unpack', change

;; (when (string-null? unpack-path)

;; (set! unpack-path import-path))

;; to

;; (when (string-null? unpack-path)

;; (set! unpack-path (unwildcard-import-path unpack-path)))

;;

;; and likewise in 'install' and and 'install-license-files'.

Greetings,

Maxime.

Leo Nikkilä schreef op vr 25-03-2022 om 22:53 [+0000]:

TBC, this is a partial review that just looks at the package

definitions a bit. I did not look at the source code for bundling,

malware, license problems ...

Greetings,

Maxime.