zoidyzoidzoid / gitlab-honeycomb-buildevents-webhooks-sink

GitLab Webhooks -> honeycomb.io buildevents <3

GitLab Honeycomb Buildevents Webhooks Sink

Server to create honeycomb.io buildevents from GitLab CI pipeline and job webhooks.

If you set this up, and add the public URL with the path of /api/message, to your GitLab project, and enable sending Pipeline and Job webhooks to it, then you will soon be able to see the events for them in your Honeycomb dataset.

We support the same environment variables as buildevents, most importantly BUILDEVENT_APIKEY. They are all documented here.

Check out the buildevents project that hugely influenced this project at https://github.com/honeycombio/buildevents . Another way to solve this would be doing something like the Circle CI's API usage with the buildevents watch command.

Overview

Basic usage

Advanced usage

We use the same logic as buildevents to generate trace IDs, so if you use the buildevents CLI to instrument steps and commands in your CI pipelines, those will show up in your pipeline traces in Honeycomb too!

Example

Work in Progress example of using buildevents cmd here: https://gitlab.com/zoidyzoidzoid/sample-gitlab-project/-/blob/master/.gitlab-ci.yml

Details

GET /healthz: healthcheck

POST /api/message: receive webhooks

GitLab Pipeline Webhooks Documentation GitLab Job Webhooks Documentation

https://github.com/honeycombio/buildevents/blob/06856ef24981b796af33bcf03e004b9cba4cb687/common.go#L68-L77

Similar Projects

• sluongng/buildkite-honeycomb-exporter - A honeycomb exporter that uses opentelemetry to export buildkite pipeline information to honeycomb or other observability platforms.

gitlab-honeycomb-buildevents-webhooks-sink's Issues

Which License is this tool published under?

Hi

Thanks for making this great tool can you please publish this under a license?

Thanks

Tidy up code

We could structure the repo a little bit differently.

Maybe following something like this https://github.com/golang-standards/project-layout

So doing something like

cmd/server/main.go
pkg/handler.go

Enable adding some Pipeline variables to the events

Job hooks don't have environment variables.

Pipeline hooks do.

It'd be useful to be able to map from environment variables to event keys where the value would be the value of the variable.

同学，您这个项目引入了164个开源组件，存在1个漏洞，辛苦升级一下

检测到 zoidbergwill/gitlab-honeycomb-buildevents-webhooks-sink 一共引入了164个开源组件，存在1个漏洞

漏洞标题：miekg Go DNS package 安全特征问题漏洞
缺陷组件：github.com/miekg/[email protected]
漏洞编号：CVE-2019-19794
漏洞描述：miekg Go DNS package是一款DNS服务器软件包。
miekg Go DNS package 1.1.25之前版本（用在CoreDNS 1.6.6之前版本和其他产品）中存在安全漏洞，该漏洞源于程序没有正确生成随机数。攻击者可利用该漏洞伪造响应。
国家漏洞库信息：https://www.cnvd.org.cn/flaw/show/CNVD-2019-45898
影响范围：(∞, 1.1.25)
最小修复版本：1.1.25
缺陷组件引入路径：github.com/zoidbergwill/gitlab-honeycomb-buildevents-webhooks-sink@->github.com/miekg/[email protected]

另外还有几个漏洞，详细报告：https://mofeisec.com/jr?p=ad3e6d

Make it possible to run a public instance

Summary

We could enable a public instance or something that folks could easily integrate with, and try out.

If we allowed people to specify the API key, dataset, and honeycomb host using query parameters, then we could create events for their dataset and things.

Explanation

From an implementation perspective, the main parameter I was thinking of was originally just the API key, but I guess we could support dataset + host as well, and standardise the command line options, environment variables, and query parameters that are supported:

env var	query parameter
BUILDEVENT_APIKEY	key
BUILDEVENT_DATASET	dataset
BUILDEVENT_APIHOST	host

Concerns

Not sure how interested folks would be in using something like this, cuz they'd obviously need to trust that we aren't using their API key for nefarious things, since it'd have write access.

Honeycomb's API keys can be given access to just write events but not do other things, so hopefully that'd make folks a little less concerned about this, but I'd still expect the primary use case being folks running their own copy.

Not sure if there is a way to somehow prove on Heroku or something that we are deploying and using the public open source codebase and not some dodgy patches that could be printing stuff like API keys that we have no interest in accessing, but ideally could prove.

Maybe the GitHub container image + cosign signing will help somehow a bit.

Otel example

Hey thanks for creating/opensourcing/maintaining this project.

It inspired me to create my own https://github.com/sluongng/buildkite-honeycomb-exporter/ recently.

However over there I used Open Telemetry SDK so it's a bit more portable and therefore not locked-in to honeycomb and users can swapped out their tracing backend to any other Otel compatible vendor/services.

Just wana give you a ping just in case you ever need Otel example :D

Add more pipeline and job info

Add more tests

Architecture diagram

Docs?

Add pipeline source

This is useful for people to find scheduled pipelines, etc.

Fetch more pipeline information from the GitLab API

This would make the code more complex and error prone, but if we took in the GitLab URL, API path, and a token, then we could query more information from the GitLab API for each pipeline and job, especially making it easier to connect child / generated pipelines, since GitLab doesn't have any fields in Pipeline or Job webhook events that'd help with this use case.

Integration tests?

It'd be cool if we had a GitHub actions workflow that pushed commits to GitLab to trigger GitLab CI pipelines and get GitLab CI webhooks, but it's a bunch of complex work.

Examples of adding instrumentation within a job using the honeycomb library

e.g.

from os import getenv
import libhoney

libhoney.init(
    writekey=getenv("BUILDEVENT_APIKEY"),
    dataset="buildevents",
)
parent_trace_id = getenv("CI_PIPELINE_ID")
libhoney.add(
    {
        "service_name": "custom_build_process",
        "trace.trace_id": parent_trace_id,
        "trace.parent_id": parent_trace_id,
        "repo": getenv("CI_PROJECT_URL"),
    }
)