coder / coder-logstream-kube Goto Github PK

View Code? Open in Web Editor NEW

10.0 10.0 3.0 266 KB

Stream Kubernetes Pod events to the Coder startup logs

License: GNU Affero General Public License v3.0

Go 88.92% Dockerfile 0.49% Shell 10.59%

coder-logstream-kube's Introduction

Self-Hosted Cloud Development Environments

Quickstart | Docs | Why Coder | Enterprise

Coder enables organizations to set up development environments in their public or private cloud infrastructure. Cloud development environments are defined with Terraform, connected through a secure high-speed Wireguard® tunnel, and automatically shut down when not used to save on costs. Coder gives engineering teams the flexibility to use the cloud for workloads most beneficial to them.

Define cloud development environments in Terraform
- EC2 VMs, Kubernetes Pods, Docker Containers, etc.
Automatically shutdown idle resources to save on costs
Onboard developers in seconds instead of days

Quickstart

The most convenient way to try Coder is to install it on your local machine and experiment with provisioning cloud development environments using Docker (works on Linux, macOS, and Windows).

# First, install Coder
curl -L https://coder.com/install.sh | sh

# Start the Coder server (caches data in ~/.cache/coder)
coder server

# Navigate to http://localhost:3000 to create your initial user,
# create a Docker template and provision a workspace

Install

The easiest way to install Coder is to use our install script for Linux and macOS. For Windows, use the latest ..._installer.exe file from GitHub Releases.

curl -L https://coder.com/install.sh | sh

You can run the install script with --dry-run to see the commands that will be used to install without executing them. Run the install script with --help for additional flags.

See install for additional methods.

Once installed, you can start a production deployment with a single command:

# Automatically sets up an external access URL on *.try.coder.app
coder server

# Requires a PostgreSQL instance (version 13 or higher) and external access URL
coder server --postgres-url <url> --access-url <url>

Use coder --help to get a list of flags and environment variables. Use our install guides for a complete walkthrough.

Documentation

Browse our docs here or visit a specific section below:

Templates: Templates are written in Terraform and describe the infrastructure for workspaces
Workspaces: Workspaces contain the IDEs, dependencies, and configuration information needed for software development
IDEs: Connect your existing editor to a workspace
Administration: Learn how to operate Coder
Enterprise: Learn about our paid features built for large teams

Support

Feel free to open an issue if you have questions, run into bugs, or have a feature request.

Join our Discord to provide feedback on in-progress features and chat with the community using Coder!

Integrations

We are always working on new integrations. Please feel free to open an issue and ask for an integration. Contributions are welcome in any official or community repositories.

Official

VS Code Extension: Open any Coder workspace in VS Code with a single click
JetBrains Gateway Extension: Open any Coder workspace in JetBrains Gateway with a single click
Dev Container Builder: Build development environments using devcontainer.json on Docker, Kubernetes, and OpenShift
Module Registry: Extend development environments with common use-cases
Kubernetes Log Stream: Stream Kubernetes Pod events to the Coder startup logs
Self-Hosted VS Code Extension Marketplace: A private extension marketplace that works in restricted or airgapped networks integrating with code-server.

Community

Provision Coder with Terraform: Provision Coder on Google GKE, Azure AKS, AWS EKS, DigitalOcean DOKS, IBMCloud K8s, OVHCloud K8s, and Scaleway K8s Kapsule with Terraform
Coder Template GitHub Action: A GitHub Action that updates Coder templates

Contributing

We are always happy to see new contributors to Coder. If you are new to the Coder codebase, we have a guide on how to get started. We'd love to see your contributions!

Hiring

Apply here if you're interested in joining our team.

coder-logstream-kube's People

Stargazers

Watchers

Forkers

toshikish

coder-logstream-kube's Issues

sample template

I was testing coder-logstream-kube last night so wrote a sample template

For some reason on the Coder installation I have on Kubernetes, the coder service account did not have enough permissions to create a workspace deployment (kubernetes_pod cannot be used yet/now)

The README.md gives kubectl commands to create a role and rolebinding to fix it. Again it may have been my Coder install which dates back to summer of 2022 when our helm chart was not mature yet. Who knows.

https://github.com/sharkymark/v2-templates/tree/main/logstream-kube

@kylecarbs I do have one question, as you can see in the UI logs, all it shows is the pod is created successfully on the node. You can see when the startup_script starts. Should I have seen more info, like container creating? The coder-logstream-kube README.md shows more info like container creation.

sending logs fails if workspace restart was triggered

Description

Restarting a workspace results in not sending logs to agent. Using stop/start everything is fine.

version: v0.0.5

POD dies after a short while...

pod restarts upon workspace restarts

i'm getting multiple restarts of the coder-logstream-kube pod when I restart my Coder workspace. i've tailed the logs and here's what's returned:

fatal error: concurrent map read and map write

goroutine 9668 [running]:
main.(*podEventLogger).sendLog(0xc000686120, {0xc00005dc60, 0x1b}, {0xc00073f6b0, 0x24}, {{0xc153aacb1a746786, 0xbfcbc6a11c, 0x358f460}, {0xc00031c660, 0x5d}, ...})
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:303 +0x8d
created by main.(*podEventLogger).sendLog
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:331 +0x772

goroutine 1 [select, 13 minutes]:
main.root.func1(0xc000004900, {0x21322c6?, 0x0?, 0x0?})
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/main.go:78 +0x7fc
github.com/spf13/cobra.(*Command).execute(0xc000004900, {0xc00011e210, 0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004900)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
main.main()
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/main.go:19 +0x1f

goroutine 33 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x6a
created by github.com/golang/glog.init.0
        /home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x1bf

goroutine 51 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0006afdb0, 0xc00054a678?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:802 +0x58
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 19 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0001b24f8, 0xa)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000715f80?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0001b24d0, 0xc0005440d0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:575 +0x256
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000524140)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:192 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x17e2ae5?, {0x24a8400, 0xc00057f6e0}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0005241b8?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000524140, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:163 +0x385
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004c6840, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:503 +0x542
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:150 +0x6b
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:148 +0x21b

goroutine 20 [sync.Cond.Wait, 13 minutes]:
sync.runtime_notifyListWait(0xc0001b2918, 0x2)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc00064efe0?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0001b28f0, 0xc000544480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:575 +0x256
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000524960)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:192 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x17e2ae5?, {0x24a8400, 0xc00057fef0}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0005249d8?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000524960, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:163 +0x385
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004c68f0, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:503 +0x542
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:150 +0x6b
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:148 +0x21b

goroutine 21 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0004c6a78, 0x10)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000468f20?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0004c6a50, 0xc0006aa5e0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:575 +0x256
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0007112c0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:192 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x17e2ae5?, {0x24a8400, 0xc000520360}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000711338?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc0007112c0, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:163 +0x385
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004c69a0, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:503 +0x542
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:150 +0x6b
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/informers/factory.go:148 +0x21b

goroutine 41 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00067f738?, {0x24a8400, 0xc000216000}, 0x1, 0xc000732000)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc00067f788?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002adb90)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 23 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0006afe50, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:802 +0x58
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 24 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:133 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:132 +0xc5

goroutine 25 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x358f460?}, {0x24b29a0, 0xc000134880}, {0x7fd822016c58, 0xc0004c6a50}, {0x24de330?, 0x20fb8e0}, 0x0, ...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:712 +0x185
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0004741e0, {0x0?, 0x0?}, 0xc000114240, 0xc0006596e0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:431 +0x54e
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0004741e0, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:356 +0x34e
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:289 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00014b4c0?, {0x24a83e0, 0xc0006afea0}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0004741e0, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:288 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 117 [select, 13 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0004741e0, 0xc000114240, 0xc000242120, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:368 +0x114
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:355 +0x32a

goroutine 52 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:133 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:132 +0xc5

goroutine 53 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x358f460?}, {0x24b29a0, 0xc000642dc0}, {0x7fd822016c58, 0xc0001b24d0}, {0x24de330?, 0x20fcea0}, 0x0, ...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:712 +0x185
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc00022e000, {0x0?, 0x0?}, 0xc000114240, 0xc000714040?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:431 +0x54e
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00022e000, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:356 +0x34e
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:289 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00046e140?, {0x24a83e0, 0xc000556050}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00022e000, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:288 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 31 [select, 13 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc00022e000, 0xc000114240, 0xc0001149c0, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:368 +0x114
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:355 +0x32a

goroutine 6733 [runnable]:
sync.runtime_SemacquireMutex(0x12?, 0xf0?, 0x477469?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:77 +0x26
sync.(*Mutex).lockSlow(0xc000127df8)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/mutex.go:90
github.com/coder/coder/codersdk/agentsdk.(*Client).QueueStartupLogs.func4()
        /home/runner/go/pkg/mod/github.com/coder/[email protected]/codersdk/agentsdk/agentsdk.go:679 +0x72
github.com/coder/coder/codersdk/agentsdk.closeFunc.Close(0x1e07260?)
        /home/runner/go/pkg/mod/github.com/coder/[email protected]/codersdk/agentsdk/agentsdk.go:722 +0x1a
main.(*podEventLogger).sendLog(0xc000686120, {0xc0006b5700, 0x1b}, {0xc00073f6b0, 0x24}, {{0xc153aacf131c00c1, 0xc37e1ada64, 0x358f460}, {0xc000516740, 0x34}, ...})
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:330 +0x619
created by main.(*podEventLogger).sendLog
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:331 +0x772

goroutine 58 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0006afe00, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:802 +0x58
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 59 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:133 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:132 +0xc5

goroutine 60 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x358f460?}, {0x24b29a0, 0xc000134440}, {0x7fd822016c58, 0xc0001b28f0}, {0x24de330?, 0x20f77a0}, 0x0, ...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:712 +0x185
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc00022e0f0, {0x0?, 0x0?}, 0xc000114240, 0xc0007142a0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:431 +0x54e
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00022e0f0, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:356 +0x34e
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:289 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00046e240?, {0x24a83e0, 0xc000556190}, 0x1, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00022e0f0, 0xc000114240)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:288 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 119 [select, 13 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc00022e0f0, 0xc000114240, 0xc0002426c0, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:368 +0x114
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:355 +0x32a

goroutine 42 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002adb90)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:938 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 65 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00072b738?, {0x24a8400, 0xc0003bc000}, 0x1, 0xc00031c000)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002adcb0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 66 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002adcb0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:938 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 81 [chan receive, 13 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001fb738?, {0x24a8400, 0xc0003d0060}, 0x1, 0xc00038c000)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002adc20)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 82 [select, 13 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002adc20)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:938 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 114 [IO wait]:
internal/poll.runtime_pollWait(0x7fd82236da18, 0x72)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00043c100?, 0xc00078e000?, 0x0)
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00043c100, {0xc00078e000, 0x5500, 0x5500})
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00043c100, {0xc00078e000?, 0xc00078e005?, 0xc3?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000130028, {0xc00078e000?, 0x41?, 0xc0004d6530?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00071eee8, {0xc00078e000?, 0xc00071eee8?, 0x0?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc0004d6610, {0x24a2f40, 0xc00071eee8})
        /opt/hostedtoolcache/go/1.20.7/x64/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0004d6380, {0x24a8d00?, 0xc000130028}, 0x5500?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0004d6380, 0x0)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc0004d6380, {0xc000268000, 0x1000, 0xc0002a4066?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc000266ae0, {0xc0005bc740, 0x9, 0xc000237d38?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x24a2d40, 0xc000266ae0}, {0xc0005bc740, 0x9, 0x9}, 0x9)
        /opt/hostedtoolcache/go/1.20.7/x64/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0005bc740?, 0x9?, 0xc000000000?}, {0x24a2d40?, 0xc000266ae0?})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005bc700)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000237f98)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2225 +0x12e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000244180)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2120 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:818 +0xc1f

goroutine 146 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00087ac48, 0x7)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000841a70?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc00087ac30, {0xc000722001, 0x5ff, 0x5ff})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000722001?, 0x2?, 0x213b90c?})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2508 +0x75
encoding/json.(*Decoder).refill(0xc000195b80)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000195b80)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000195b80, {0x1e390e0, 0xc00075cb88})
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0003bce70, {0xc00075ec00, 0x800, 0xc00})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0000ae4b0, 0xc000672480?, {0x24b27e8, 0xc000134d40})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000714380)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000134880)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x130

goroutine 162 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00087a948, 0x0)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000279bb0?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc00087a930, {0xc0004d2c00, 0x200, 0x200})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc0004d2c00?, 0x0?, 0x0?})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2508 +0x75
encoding/json.(*Decoder).refill(0xc000416780)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000416780)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000416780, {0x1e390e0, 0xc00075c0c0})
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0008407e0, {0xc00029f000, 0x400, 0x400})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000556320, 0xc000686660?, {0x24b27e8, 0xc000134480})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000468ae0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000134440)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x130

goroutine 145 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00087ac00, 0xc00037ea00)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1438 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0xfca8e6?, 0xc000642cc0?)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1300 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1229 +0x34a

goroutine 148 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00087a7c8, 0xb)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0006e0150?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc00087a7b0, {0xc000246001, 0x3dff, 0x3dff})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000246001?, 0x2?, 0x213b90c?})
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2508 +0x75
encoding/json.(*Decoder).refill(0xc000416140)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000416140)
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000416140, {0x1e390e0, 0xc00071ef18})
        /opt/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000840c00, {0xc000256000, 0x4000, 0x5000})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000556370, 0xc00071a6c0?, {0x24b27e8, 0xc0005bbf80})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000468c60)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000642dc0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x130

goroutine 151 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00087a900, 0xc00037e700)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1438 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1300 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1229 +0x34a

goroutine 9408 [runnable]:
main.(*podEventLogger).sendLog.func2()
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:331
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/asm_amd64.s:1598 +0x1
created by main.(*podEventLogger).sendLog
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:331 +0x772

goroutine 183 [IO wait]:
internal/poll.runtime_pollWait(0x7fd82236d838, 0x72)
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000756380?, 0xc000851300?, 0x0)
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000756380, {0xc000851300, 0x1300, 0x1300})
        /opt/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000756380, {0xc000851300?, 0xc0008513b6?, 0x1f?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000482238, {0xc000851300?, 0x3001cc000769878?, 0xc0001208b0?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00071f038, {0xc000851300?, 0xc00071f038?, 0x0?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc000120990, {0x24a2f40, 0xc00071f038})
        /opt/hostedtoolcache/go/1.20.7/x64/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000120700, {0x24a8d00?, 0xc000482238}, 0x124f?)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000120700, 0x0)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc000120700, {0xc000509000, 0x1000, 0x701d80?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc000713920, {0xc0005bcac0, 0x9, 0x71fa05?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x24a2d40, 0xc000713920}, {0xc0005bcac0, 0x9, 0x9}, 0x9)
        /opt/hostedtoolcache/go/1.20.7/x64/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /opt/hostedtoolcache/go/1.20.7/x64/src/io/io.go:351
net/http.http2readFrameHeader({0xc0005bcac0?, 0x9?, 0xc0006e1d70?}, {0x24a2d40?, 0xc000713920?})
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/http/h2_bundle.go:1567 +0x6e
net/http.(*http2Framer).ReadFrame(0xc0005bca80)
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/http/h2_bundle.go:1831 +0x95
net/http.(*http2clientConnReadLoop).run(0xc000769f98)
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/http/h2_bundle.go:9187 +0x12e
net/http.(*http2ClientConn).readLoop(0xc00087b380)
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/http/h2_bundle.go:9082 +0x6f
created by net/http.(*http2Transport).newClientConn
        /opt/hostedtoolcache/go/1.20.7/x64/src/net/http/h2_bundle.go:7779 +0xc3c

goroutine 147 [select, 3 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00087a780, 0xc00037e800)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1438 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0xfca8e6?, 0xc000642cc0?)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1300 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1229 +0x34a

goroutine 169 [runnable]:
github.com/coder/coder/codersdk/agentsdk.(*Client).QueueStartupLogs.func2()
        /home/runner/go/pkg/mod/github.com/coder/[email protected]/codersdk/agentsdk/agentsdk.go:635
created by github.com/coder/coder/codersdk/agentsdk.(*Client).QueueStartupLogs
        /home/runner/go/pkg/mod/github.com/coder/[email protected]/codersdk/agentsdk/agentsdk.go:635 +0x1ea

goroutine 9570 [runnable]:
runtime.goexit1()
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/proc.go:3634 +0x54
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/asm_amd64.s:1599 +0x6
created by main.(*podEventLogger).sendLog
        /home/runner/work/coder-logstream-kube/coder-logstream-kube/logger.go:331 +0x772

Add support for specifying `resources` on the pod

Right now the pod runs unconstrained, but it would be nice to set resources to help schedule the pod better since it needs few resources to run.

Stream ReplicaSet events (e.g. Kubernetes quota breached)

When I exceed a Kubernetes quota, the error is on the ReplicaSet, not the deployment or pod:

  Warning  FailedCreate  46s (x6 over 2m5s)  replicaset-controller  (combined from similar events): Error creating: pods "coder-benzo-my-workspace-6df987574-ffvcc" is forbidden: exceeded quota: my-quota, requested: limits.cpu=6,limits.memory=4Gi,requests.memory=512Mi, used: limits.cpu=8,limits.memory=8Gi,requests.memory=1792Mi, limited: limits.cpu=2,limits.memory=2Gi,requests.memory=2Gi

In the Coder UI, the workspace seems like it is hanging forever. No events are streamed

Add support for `affinity` selectors

nodeSelectors is a basic way to schedule where the pods will run but affinity is a much more flexible way. This is supported in the main coder helm chart, so it would be nice to use affinity here too.

Custom Certificates Through Helm Chart

Problem

Currently, to set a custom certificate for the Helm chart (required for Coder deployments with self-signed certificates) you must create a new Docker image that extends the current one. The extended image needs to add in your custom certificate and set the SSL_CERT_FILE or SSL_CERT_DIR environment variable. You must then reference this custom image in the Helm chart (example of both below).

Solution

The environment variables should be able to be set through values.yaml and the custom certificates should be able to be mounted as a volume or from a secret (like in the Coder helm chart for certs and tls).

Examples

Custom Dockerfile

FROM ghcr.io/coder/coder-logstream-kube:latest

ADD mycert.crt mycert.crt
ENV SSL_CERT_FILE="mycert.crt"

values.yaml

# url -- The URL of your Coder deployment. Must prefix with http or https
url: "https://coder.mydomain.local"

# namespace -- The namespace to searching for Pods within.
# If unspecified, this defaults to the Helm namespace.
namespace: ""

# image -- The image to use.
image:
  # image.repo -- The repository of the image.
  repo: "myCustomRepo/coder-logstream-kube"
  # image.tag -- The tag of the image, defaults to {{.Chart.AppVersion}}
  # if not set. If you're using the chart directly from git, the default
  # app version will not work and you'll need to set this value. The helm
  # chart helpfully fails quickly in this case.
  tag: "latest"
  # image.pullPolicy -- The pull policy to use for the image. See:
  # https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy
  pullPolicy: Always
  # image.pullSecrets -- The secrets used for pulling the Coder image from
  # a private registry.
  pullSecrets: []
  #  - name: "pull-secret"

serviceAccount:
  # serviceAccount.annotations -- The service account annotations.
  annotations: {}
  # serviceAccount.labels -- The service account labels.
  labels: {}
  # coder.serviceAccount.name -- The service account name
  name: coder-logstream-kube

include helm chart `.tgz` file in the next release

we have customers who cannot use the helm CLI to manually pull down chart files. they rely on repositories providing a Helm chart .tgz in the releases.

in the meantime, I am having customers manually clone the repo and extract the Helm chart files.

build for arm64

I run a little cluster on a Raspberry Pi, but it seems like we're only building for amd64

Streaming other events

We're using this and it works great!

We'd like to extend it to capture events from spot instance interruptions on underlying hosts. We already have another service that determines that.

Just wondering if you think this would be possible how I would need to format the event to coder to capture it and associate it with that users workspace.

CI credential file packaged into helm chart

there is a JSON file that is inadvertently included our helm chart. steps to repro:

helm pull coder-logstream-kube/coder-logstream-kube
tar -xzvf v0.0.8.tgz

this is a security finding on behalf of a customer.

Image should run as USER

Description

Currently it runs as unspecified user context. Aquasecurity complains about it. Please run logstream-kube as non root user in image

This logstream pod seems to restart periodically

On @sharkymark's cluster, the logstream pod seems to restart periodically. It still sends logs, but doesn't seem to run for a long time.

We were able to find some logs from GCP. The format isn't great but there are definitely errors which causes the container to exit and restart:

unsupported protocol scheme

setting the url value without http or https, the following error is returned in the pod logs:

upload startup logs failed  error="Patch \"/api/v2/workspaceagents/me/startup-logs\": unsupported protocol scheme \"\""  count=1

Don't limit CODER_NAMESPACE to a single namespace

From https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ :

Namespaces are a way to divide cluster resources between multiple users
via Resource Quotas : https://kubernetes.io/docs/concepts/policy/resource-quotas/
and RBAC : https://kubernetes.io/docs/reference/access-authn-authz/rbac/

There are benefits to deploying per-user namespaces:

Ability to give the user control over their own namespace via RBAC (deploying other objects / API Isolation)
Ability to persist expensive objects like cert-manager certs / let encrypt (some objects take a lot of time)
Ability to isolate traffic between multiple users / namespaces

We create a namespace per user, and do not destroy it when a workspace is torn down. This allows expensive objects (like cert-manager/letsencrypt certs/dns) to persist and be reused for multiple workspaces (from the same user) to access them.

Some resources we use per user/namespace:

Issuer (Cert-Manager w/ DNS01 for wildcard)
Certificate (this can take 40 seconds to provision from Lets Encrypt)
tls-secret (generated by TLS Cert from Certificate)
wildcard ingress (each user get's there own namespace AND *.username.coder.website [accessible without coder])
RoleBinding w/ admin over their own namespace (we allow them to create whatever other resources they want within their namespace) : RBAC
We use Resource Quotas to ensure one user doesn't take over all the resources on a node

server restarts when K8s pod workspaces are started

both @sharkymark and I have observed occasional restarts of the coder-logstream-kube deployment when it tries to stream startup events of workspace pods. this does not happen when a workspace is built as a deployment.

i've tailed the logs during a pod build, and grep-ed "error":

fatal error: stack overflow

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.