mvisonneau / gitlab-ci-pipelines-exporter Goto Github PK

Hello Mvisonneau,

I would like to be able to fetch all refs within a project. We have only a handful of projects, but we have many branches / merge requests. We'd like to import all of these into prometheus, and do the filtering in grafana or in prometheus. Is this possible?

If not, I can also try and create a PR for this as well, if you'd point to me where I should start from

Hello,

is it possible to get the pipeline executions within a specific time period and to add this kind of informations in a panel in the grafana dashboard?

I see that defaults for pulling pipelines and discovering projects are pretty safe, yes our gitlab instance cannot handle the load. Is there currently some throttling mechanism that allows to prevent API query overflow , like no more that N number of simultaneous requests?

Thank you

@inge4pres, I was running some tests with a master build locally and ran into the following error:

panic: send on closed channel

goroutine 41 [running]:
github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProjectsWith.func1(0xc000192480, 0xc0000f61e0, 0xc00012fd00, 0xc000192780)
	/go/gitlab-ci-pipelines-exporter/cmd/gitlab.go:483 +0x1b2
created by github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProjectsWith
	/go/gitlab-ci-pipelines-exporter/cmd/gitlab.go:476 +0x120

I'll try to dig into it later on, just to let you know! :)

nb: probably related to #88

Recently experienced the following issue when a DNS lookup failed

{"level":"error","msg":"ListProjectPipelines: Get https://git.example.com/api/v4/projects/72/pipelines?ref=master: dial tcp: lookup git.sph.re on 10.4.64.10:53: read udp 10.5.1.25:56257-\u003e10.4.64.10:53: read: connection refused","time":"2019-10-01T14:29:54Z"}
panic: runtime error: index out of range [0] with length 0

goroutine 2555 [running]:
github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProjectRef(0xc0000e03e8, 0xc000582900, 0xc0003a2096, 0x6)
	/build/cmd/gitlab.go:275 +0xf30
created by github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProject
	/build/cmd/gitlab.go:244 +0x595

Need to implement correct handling of this error.

exporter version : 0.2.8

It would be very nice to be able to break the piplines down to the job level to be able to see how long each job takes, what job is currently running for a pipeline and what part of a pipeline did actually fail.

I'm not much of a go programmer, but could give it a go if it is something you think is a good idea

Hello there, long time without coming here! :)

Its possible to add a flag, to fetch (or not) tags from the discovered projects?

In my example here, we're using tags only for create releases, but not creating branches for each one of those, and when discover happens, all tags shows as branches on "healthy" dashboard section:

Hi!
It's be a good idea to monitor subchart in project like separete projects. My project user/foo counts 70 charts bar1 bar2 bar3 and etc, but triggering bar1 enable monitoring for all project user/foo.
It would be better if trigger user/foo/bar1 will have separete monitoring from user/foo/bar2

Hello there!

I did some "tunning" on the dashboard and want to share it with you to check if its useful to merge on code or not.

I've uploaded the .json at this gist and just need to change MYAWESOMEGITLABURL by your gitlab URL and check it.

Changes are (which I remember by now):

• Added a clock plugin (that its on our monitoring display and its kinda useful to remember time of daily standup)
• Added a column saying the latest id of each pipeline
• Separated running pipelines to a different table
• Added links to pipelines (and on healthy table, the link leads to latest pipeline)

Hi! How do you query prometeus for status panel?

Awesome collector, thanks a million for providing this tool!

Would it be possible to define a date in the config, from where the scraper should start running? Currently every restart of the docker project starts off with only the latest commit present.

Again, thank you so much!

Hi!

I have a test setup with the following config.yml

# URL and Token with sufficient permissions to access your GitLab's projects
# pipelines informations
gitlab:
  url: http://gitlab.cmaker.studio
  token: potato

# Waiting time between polls for each projects
polling_interval_seconds: 30

# The list of the projects you want to monitor
projects:
  - name: root/test1
    ref: master
  - name: root/test2
    ref: master
  - name: root/test3
    ref: master

The container starts correctly but it's crashing with the following error

root@docker01:/opt/gitlab-ci-exporter# docker run -d    --name gitlab-ci-pipelines-exporter    -v $(pwd)/config-test.yml:/etc/config.yml    -p 8080:8080    gitlabci-pipeline-cmaker:latest    -config /etc/config.yml
f82ac2c868cf2f16bc0f50e377d37a27c934b6fc75acae5958a37166b0e13149
root@docker01:/opt/gitlab-ci-exporter# docker logs f82ac2c868cf2f16bc0f50e377d37a27c934b6fc75acae5958a37166b0e13149
2019/07/19 04:38:03 -> Starting exporter
2019/07/19 04:38:03 -> Polling http://gitlab.cmaker.studio every 30s
2019/07/19 04:38:03 -> 3 project(s) configured
2019/07/19 04:38:03 --> Polling ID: 2 | root/test2:master
panic: runtime error: index out of range

goroutine 12 [running]:
main.main.func1(0xc0000a1400, 0xc00005a780, 0xc0000269f0, 0xa, 0xc000026a08, 0x6)
	/build/main.go:126 +0xbe7
created by main.main
	/build/main.go:108 +0x3af

If I switch to monitoring a single project. It works correctly and does not crash. 2 or more projects and the container crashes when launched.

I saw on old issue but you mentioned a fix being in the current release.

Any ideas?

Hello there,

First: Thanks for you code!

I'm getting a error while trying to deploy it using multiple projects (with 4 projects that work normally):

time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/ci-cd-test:dev (1039)"
time="2019-09-06T20:29:51Z" level=warning msg="Could not find any pipeline for myproject-saas/ci-cd-test:master"
time="2019-09-06T20:29:51Z" level=warning msg="Could not find any pipeline for myproject-saas/ci-cd-test:integ"
time="2019-09-06T20:29:51Z" level=warning msg="Could not find any pipeline for myproject-saas/ci-cd-test:dev"
time="2019-09-06T20:29:51Z" level=info msg="Found ref 'master' for project 'myproject-saas/back-end/complexity-api'"
time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/back-end/complexity-api:master (2617)"
time="2019-09-06T20:29:51Z" level=info msg="Found ref 'master' for project 'myproject-saas/nginx-mock-data'"
time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/nginx-mock-data:master (470)"
time="2019-09-06T20:29:51Z" level=info msg="Found ref 'master' for project 'myproject-saas/back-end/user-count-api'"
time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/back-end/user-count-api:master (1332)"
time="2019-09-06T20:29:51Z" level=info msg="Found ref 'master' for project 'myproject-saas/back-end/flow-api'"
time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/back-end/flow-api:master (1376)"
time="2019-09-06T20:29:51Z" level=info msg="Found ref 'master' for project 'myproject-saas/back-end/capa-plans-api'"
time="2019-09-06T20:29:51Z" level=info msg="Polling myproject-saas/back-end/capa-plans-api:master (1213)"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0x89f951]

goroutine 212 [running]:
main.(*client).pollProjectRef(0xc000108d30, 0xc00011ab40, 0xc0004113e8, 0x6)
	/build/exporter.go:350 +0x681
created by main.(*client).pollProject
	/build/exporter.go:310 +0x1e5

It would be nice to be able to let people try it out in an easy fashion. I believe we can assume that people will have a GitLab instance available, if not they can always use gitlab.com

We should look to automatically provision and configure

• prometheus
• grafana
• and of course the exporter

Originally posted by @mvisonneau in #4 (comment)

We've come up with another little feature enhancement. Currently the only time the exporter gets a list of projects is on startup. We've had an instance running for a few weeks, but as we're regularly creating new repos, they're not getting picked up.

The 2 options I can think of are:

1. After a certain amount of time, drop back from polling to getting the project list again.
2. After a certain amount of time, exit the app and let either systemd or the container schedule restart from scratch.

What do you reckon?

If the Gitlab get unavailable (e.g. caused by an update) while exporter is running the exporter will exit:

gitlab-ci-pipelines-exporter_1  | time="2019-10-19T04:07:09Z" level=error msg="ListProjectPipelines: Get https://gitlab/api/v4/projects/276/pipelines?ref=master: dial tcp 10.169.66.61:443: connect: no route to host"
gitlab-ci-pipelines-exporter_1  | panic: runtime error: index out of range [0] with length 0
gitlab-ci-pipelines-exporter_1  |
gitlab-ci-pipelines-exporter_1  | goroutine 556 [running]:
gitlab-ci-pipelines-exporter_1  | github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProjectRef(0xc00014a2e8, 0xc0003d58c0, 0xc0006d8910, 0x6)
gitlab-ci-pipelines-exporter_1  |       /build/cmd/gitlab.go:275 +0xf30
gitlab-ci-pipelines-exporter_1  | created by github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd.(*Client).pollProject
gitlab-ci-pipelines-exporter_1  |       /build/cmd/gitlab.go:244 +0x595

Is it possible to handle such unavailability inside the application?
Otherwise the current workaround is docker-compose.yml with restart: always which results in a lot of log output.

is there any mechanism to add links to view gitlab individual pipeline runs - either the ref or last run column ?

Hiya 👋 I've just started experimenting with gitlab-ci-pipelines-exporter and I'm also finding it really useful. Thanks @mvisonneau! ❤️

I think I have a similar use-case to @misaon and thought it might be helpful to share on this issue.

I'd like to instrument how long our main build/test/deploy pipeline takes on the master branch so we can optimize those durations. However, we have a couple of additional scheduled pipelines that also target the master branch (for example, scheduled nightly renovate jobs). These do something totally different to our build/test/deploy pipeline. Currently it's not possible to omit these from the gitlab_ci_pipeline_last_run_duration_seconds{ref="master"} metric.

GitLab does provide a CI_PIPELINE_SOURCE variable that can be used to identify whether the pipeline was triggered from a schedule. If we could add this variable as a label to the metric, we could filter out scheduled master pipeline durations, and focus on what we need to measure.

There are a lot of pipeline variables though, so I think it would make sense to whitelist which ones are sent as labels. Alternatively if a general purpose solution is hard to architect, maybe we could enumerate some common use-cases for this and just support those.

Originally posted by @nfm in #33 (comment)

Hey,

first of all: thank you for this awesome exporter. It's really useful and well done :)

I just noticed that the exporter uses a label job which is actually "reserved" by Prometheus. It looks like other exporters like the kube-state-metrics renamed their label job as well to mitigate the problem.

Therefore I propose to rename the label job to job_name. What do you think about it?

Currently it doesn't have a label for status of the pipeline, and it results in skewed reports. We have failed pipelines, and no way to filter them out from the reports.

Would be great to add status label to the time-specific metrics, then we could display HEALTHY & RUNNING based on true successful runs only.

Thank you for your consideration.

I cannot fetch project:

 gitlab-ci-pipelines-exporter_1  | time="2019-10-27T16:31:46Z" level=error msg="Unable to fetch project 'gitlab-org/gitlab-runner' from the GitLab API : Get https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-runner: dial tcp 35.231.145.151:443: connect: connection timed out"

but directly from docker (via curl) image it works:

{"id":11915984,"description":"","name":"auto-deploy-app","name_with_namespace":"GitLab.org / charts / auto-deploy-app","path":"auto-deploy-app","path_with_namespace":"gitlab-org/charts/auto-deploy-app","created_at":"2019-04-18T05:08:03.776Z","default_branch":"master","tag_list":[],"ssh_url_to_repo":"[email protected]:gitlab-org/charts/auto-deploy-app.git","http_url_to_repo":"https://gitlab.com/gitlab-org/charts/auto-deploy-app.git","web_url":"https://gitlab.com/gitlab-org/charts/auto-deploy-app","readme_url":"https://gitlab.com/gitlab-org/charts/auto-deploy-app/blob/master/README.md","avatar_url":null,"star_count":30,"forks_count":34,"last_activity_at":"2019-10-27T11:34:49.216Z","namespace":{"id":5032027,"name":"charts","path":"charts","kind":"group","full_path":"gitlab-org/charts","parent_id":9970,"avatar_url":null,"web_url":"https://gitlab.com/groups/gitlab-org/charts"}} ```

Does this support gitlab.com (gitlab cloud) ?

Is it possible to export job coverage data?

if I understand documentation well I should be able to configure fetch all with this config:

wildcards:

• refs: ".*"

But I'm getting:

"0 project(s) configured"
"Listing all projects using search pattern : '' with owner '' ()"
"Invalid owner kind '' must be either 'user' or 'group'"
"exporter started!"

I guess from other issues that 'fetch all' is a no go feature, for now, so I would suggest changes in documentation - so it doesn't give an impression it is possible.

Hi.

I've installed the exporter with a simple configuration.

In logs i see that he found projects, but i get no info about pipelines, and in the logs there is no information about the problem.

Teh token configuration must be right because i've used it with timoschwazer gitlab-monitor, and it worked.

Confgiuration:

# URL and Token with sufficient permissions to access your GitLab's projects
# pipelines informations
gitlab:
  url: [omitted]
  token: [omitted]
  health_url: https://gitlab.mag-news.it/-/health  # Alternative URL for determining health of GitLab API (readiness probe)
  skip_tls_verify: false                           # disable TLS verification

# Custom waiting time between polls for projects, their refs and pipelines (in seconds, optional)
projects_polling_interval_seconds: 1800 # only used for wildcards
refs_polling_interval_seconds: 300
pipelines_polling_interval_seconds: 60
pipelines_max_polling_interval_seconds: 1800 # when no pipeline exists for a given ref, the exporter will exponentially backoff up to this value

# Default regexp for parsing the refs (branches and tags) to monitor (optional, default to master)
# default_refs: "^master$"

# The list of the projects you want to monitor

# Dynamically fetch projects to monitor using a wildcard
wildcards:
  # Fetch projects belonging to a group and potentially its subgroups
  - owner:
      name: dms
      kind: group
      include_subgroups: true # optional (default: false)
    refs: "^master$"
    search: '' # optional (defaults to '')

Hey,

Been playing around with the exporter a bit and believes there's an error or inconsistency in the config.yaml file:

# Default regexp for parsing the refs (branches and tags) to monitor (optional, default to master)
# default_refs: "^master$"

# Search for projects globally
  - refs: ".*"
    search: 'baz' # optional (defaults to '')
    archived: true # optional (default: false)

Asssuming I want to monitor all projects but only the master branches, this config should work, as the first section defaults to master branches and the projects wildcard is set to find all projects, but instead it's exporting all branches of all projects.

Is there a configuration combination that does what I'm looking for?

Hello !
First of all, thank very much for your job ! I was able to get a CI status Dashboard for all the devs in my company.
All is working pretty smooth and correctly, but i still have an issue.

I can't retrieve Pipeline Status from merge request. I used wildcard to retrieve all groups / refs possible but merges requests pipelines won't come up.

From the image below, i'm able to get the release/1.1 pipeline status but not the "60" as it's a merge request.

Any ideas ? Did i miss something ?

Regards,

Since We can monitor git lab repos here. Can we also monitor status of any other sources like jenkins.

I have just added an example here on this link.
https://community.grafana.com/t/custom-dashboard-to-show-builds-progress/19871

In an organisation using self-installed GitLab within AWS, have deployed the exporter in a docker container on the same instance as GitLab, but port 8080 is reserved for GitLab login so I exposed on port 8081 instead, I can then view the gitlab-ci-pipelines-export metrics page but the GCPE doesnt show up as a Prometheus target and I can't use the GCPE metrics page as a Prometheus data source for my separate Grafana instance (hosted on Fargate) as I just get a "404 not found" error when trying to use http://gitlab.example.com:8081/metrics as my data source URL?

How does this software operate alongside the inbuilt Prometheus on Gitlab servers and how can we troubleshoot why that prometheus is not importing from GCPE? Is it then possible to simply use the data existing Prometheus data source to also send the GCPE stats?

Hi,

my gitlab instance does not have a valid cert (aka self-signed). Is there a way to disable the valid cert check?

I tried using the docker image and got the following error

Unable to fetch project xxx/xxx' from the GitLab API : Get https://git/api/v4/projects/xxx%2Fxxx: x509: certificate signed by unknown authority

Any help would be appreciated.

Thanks
Björn

Thanks for this project @mvisonneau!
We use GitLab and this project has a great value to monitor how we are performing.
This is more of a general design discussion rather than a real issue with the product.

While trying to refactor some part of the code to add a feature I noticed that the when the method Client.pollProject is called at gitlab.go#128](https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/blob/master/cmd/gitlab.go#L128) and gitlab.go#426 is never signalled for completion, so the goroutine will never return neither report the errors (only logging them), it will just loop for all projects with a time.Sleep at the end of it.
While this approach can work fine for a limited amount or projects, it does not scale well when the project number start to grow, especially with regards to the shared API client with rate limiting applied.

Another approach would be to build a processing pipeline that fans out to a limited number of processing goroutines doing the heavy-lifting of querying the GitLab API and fanning into the prometheus metrics collection.
I'll try to create a PR showing this in the next days so we can discuss.

Hello,
Is it possible to add a metric for commit count per repo since the last build?

Thanks

This is a great project.

I was wondering if there was a way to get all projects and ref? As the list of projects grows having everything monitored would be great. Instead of having to build a script to build the config file?

To handle limiting the scope we could make sure the token user only has access to the right projects.

Thank you for your work on this.

Would you be able to share the grafana dashboards that you build? They seem to be practical.

Thanks

Failed to install app gitlab-ci-pipelines-exporter. Error: YAML parse error on gitlab-ci-pipelines-exporter/templates/service.yaml: error converting YAML to JSON: yaml: line 11: could not find expected ':'

I saw change from Add service labels in helm template.

Great work on this tool, just started to use and it is dope! I hope GitLab integrates it as the official pipeline exporter in the future.

Would be possible to add pipeline id as a label? Then I could make some links to failed pipelines from the dashboard.

We need to ensure that the grafana.com dashboard is up to date with the one we maintain in the examples folder.

Originally posted by @cobra1978 in #34 (comment)

currently we can see status for: running, passed, failed - and different colouring for these statuses in the status Column

would it makes sense to add status for manual stages ?

it will return "status": "manual" in api/v4/projects/:id/pipelines. in UI would display as

currently we can see on dash:

I got an error, I can not understand what it is

2019/02/12 06:48:56 -> Starting exporter
2019/02/12 06:48:56 -> Polling *** every 30s
2019/02/12 06:48:56 -> 10 project(s) configured
2019/02/12 06:48:56 --> Polling ID: 286 | ***
...
2019/02/12 06:48:56 --> Polling ID: 261 | ***
panic: runtime error: index out of range

goroutine 16 [running]:
main.main.func1(0xc42009b200, 0xc4200165c0, 0xc420012e60, 0x11, 0xc420014cc0, 0x6)
/go/src/github.com/mvisonneau/gitlab-ci-pipelines-exporter/main.go:127 +0xafb
created by main.main
/go/src/github.com/mvisonneau/gitlab-ci-pipelines-exporter/main.go:108 +0x316

Originally posted by @sakojpa in #1 (comment)

Hello there,

A idea (don't know how hard its to implement it): Possibility to scrape all subprojects inside a project.

In my company we have a hosted gitlab, used by multiple teams, and each one has your own project, and inside this, we create our subprojects to work. But, to use the scrapper like it is (and like our structure is) is needed to add manually all subprojects since just can scan by user or group.

through this exporter we get metrics like ex:
gitlab_runner_counts(project="<groups_name>/<project_name>"
is there is any way to get group name and project name separately like this:
gitlab_runner_counts(group="<group_name>",project="<project_name>")

any solution for this??

Looks like one of the imports have broken checksum.

panic: Error processing import path "github.com/hashicorp/vault/sdk": Bad SHA256 for repo https://github.com/hashicorp/vault with rev v0.1.11

I forgot to handle the pagination which can lead to incorrect results for projects, branches and tags wildcards.

https://github.com/xanzy/go-gitlab/blob/d3666a4c977f472c24cf56bbdb9324e579502683/examples/pagination.go

we have with our own gitlab instance problems - the process starts and the only thing you can see is the following and then nothing more happend in the log but the container is running

we using gitlab-ci-pipelines-exporter in version 0.3.0 - any ideas what is going on under the hood ? we using gitlab version GitLab Community Edition 12.9.2

time="2020-04-30T12:24:50Z" level=info msg="polling metrics from projects refs" count=18
--
  | time="2020-04-30T12:24:50Z" level=debug msg="listing project pipelines" project-id=36054 project-ref=master
  | time="2020-04-30T12:24:50Z" level=debug msg="listing project pipelines" project-id=28373 project-ref=master
  | time="2020-04-30T12:24:50Z" level=debug msg="throttling GitLab requests" for=126.607654ms
  | time="2020-04-30T12:24:50Z" level=debug msg="throttling GitLab requests" for=326.552226ms
  | time="2020-04-30T12:24:50Z" level=debug msg="throttling GitLab requests" for=343.59615ms
  | time="2020-04-30T12:24:51Z" level=debug msg="throttling GitLab requests" for=318.693765ms

here is our config

gitlab:
  url: https://git.our.own.domain/
  token: $GIT_TOKEN

maximum_gitlab_api_requests_per_second: 5
maximum_projects_poller_workers: 2

#projects to monitor
projects:
  - name: spm/spm-aaa
  - name: spm/spm-zzz
  - name: spm/spm-yyy
  - name: spm/spm-xxx

Needs mechanism of asking old branch without pipelines with double delay ( based at previous asking)
If we see that "Could not find any pipeline for ***", we do 3 research and if it fails we do double-time between the asking for the branch.
Because now we have an extreme load of the server if we have many projects.

Thanks for your project)

Love the project and is just what I need for my current task, but all containers in our environment need to be clair scanned before they can be used.
Thing is, since the move to using busybox base images in 0.23 the containers can't be scanned, as Clair doesn't recognise busybox as a valid base OS.
Are there any post-0.22 changes that would break if I used alpine as a base image? I couldn't find any reasoning behind the base image change in the release notes.

Hi.

For us well be very useful to filter projects by tag, because we want to track with this dasboard only some types of project.

Unfortunately project names aren't consistent enaough t filter with a regexp.

Could you implement this type of filtering?

Thanks.

I couldn't find any source code for dashboard, so I'll report issue here.
I have applied this change locally in my JSON and my dashboard started working properly. As by default it wasn't.
all occurrences of gitlab_ci_pipeline_status suppose to be changed to gitlab_ci_pipeline_last_run_status in this file: https://grafana.com/grafana/dashboards/10620

Hi there and thanks so much for this, very useful.

I was wondering if there's a way to only retrieve refs starting from a certain period of time rather than ALL time. We are working with a TON of refs from years ago and it creates a lot of prometheus data we really don't need. Something like "only start getting refs from new jobs" or something like that. Am I overlooking a configuration option or is this not possible at this time?

Thanks!

Firstly, I would like to express my thanks for the excellent work on this exporter.
I would send PR but I have no experience with GO.

Idea:
Add variable information to metrics. See the pipelines API documentation (https://docs.gitlab.com/ee/api/pipelines.html#get-variables-of-a-pipeline).

Why?
Now, for example, in Grafana we are limited to working only with information "repository name" and "ref". I think it's a good idea to be able to add a column in Graphana to display your own information using variables from pipelines.

My scope:
I solve this problem because in my company we would like to use monitoring of export orders from our e-shops using Gitlab pipelines (via scheduler). Now we have a problem, because we have one ".gitlab-ci.yml" file, inside are defined "deploy" and "export of orders" stages. The "export of orders" stage is only started when I create a pipeline with the appropriate rules and the "deploy" starts when the merge request is created.
However, both "deploy" and "export orders" go to the metrics. It is mixed together and it is not possible to filter records.

Can you think of a solution to this problem?
Thanks a lot 👍