kubernetes-sigs / apiserver-runtime

Libraries for implementing aggregated apiservers

Kubebuilder: apiserver-runtime

Note: This project is Alpha with no commitments from its maintainers to provide any sort of support. If you intend to use this project in production, we recommend you become involved in maintaining it.

Experimental / Work in Progress

This project exists to explore the creation of new libraries for building Kubernetes API extension servers.

This project is experimental and may change drastically or be cancelled with little or no public notice.

If deletion of this repo would negatively impact you, then make sure this is documented:

• Comment on this issue

And make sure you will get any public notifications about this project.

• Join the [email protected] mailing list

Documenting your dependence on this project does not guarantee that it will not be cancelled if there was insufficient interest on the part of the maintainers to continue maintaining it.

Goals

apiserver-runtime provides libraries for building on top of the Kubernetes apiserver and apimachinery modules.

If you are simply looking to create new Kubernetes extension resource types backed by etcd storage, then you should use CRDs instead of this library. This library is only for the cases where you want to expose Kubernetes endpoints which are not backed by etcd storage.

apiserver-runtime is developed under the kubebuilder subproject.

See the docs: https://pkg.go.dev/sigs.k8s.io/apiserver-runtime/pkg/builder

See the example: sample/cmd/apiserver/main.go

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

• Slack
• Mailing List

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

apiserver-runtime's Issues

apiserver-runtime users

Add a comment to this issue if you depend upon apiserver-runtime, and join the kubebuilder mailing list.

Adjust to kubebuilder go/v4 layout

I noticed that the project does not support kubebuilder's go/v4 layout yet. This issue is to track changes towards it.
https://kubebuilder.io/migration/migration_guide_gov3_to_gov4

k8s.io/apiserver v0.27.2 version not supported

hi,kubernetes-sigs:
In this release, storage has increased the SingularQualifiedResource.

k8s.io/apiserver/pkg/registry/generic/registry/store.go

However, in the newStore method, no assignment is made to this field, resulting in an error when calling the store.CompleteWithOptions(options) method.

sigs.k8s.io/apiserver-runtime/pkg/builder/rest/rest.go

error message:

Decouple the builder from the sample

I observe that the pkg/builder uses packages in internal/sample-apiserver (example). In other words, the library seems hardcoded to the sample.

[Umbrella Issue] Tracking items for cutting the first release

here are a few things i can think of before we move to the first release:

• arbitrary subresource support:
  • providing extensibility for extending whatever new custom subresources
  • providing a new commonly-used non-standard subresource:
    • finalize subresource: controls writes upon .metadata.finalizers so that the finalizers can be protected by RBAC in finer granularity. e.g. namespaces/finalize
    • approve subresource: which is similar to a removed feature named initializer from kubernetes. the resource object is only supposed to work after the resource has been approved. e.g. certificatesigningrequests/approve
• a testing framework for aggregated apiserver. similar to how controller-runtime does, the apiserver-builder originally has a testing framework but now removed.

WithoutEtcd does not actually drop etcd flags

I would expect that if I build the command using WithoutEtcd() I will end up without any etcd related flags.
However this seems not to be the case. All etcd related cmd flags are still present.

A nasty workaround currently is to hide the flags before calling the Execute().

rootCmd.Flags().MarkHidden("etcd-cafile")
rootCmd.Flags().MarkHidden("etcd-certfile")
rootCmd.Flags().MarkHidden("etcd-compaction-interval")
rootCmd.Flags().MarkHidden("etcd-count-metric-poll-period")
rootCmd.Flags().MarkHidden("etcd-db-metric-poll-interval")
rootCmd.Flags().MarkHidden("etcd-healthcheck-timeout")
rootCmd.Flags().MarkHidden("etcd-keyfile")
rootCmd.Flags().MarkHidden("etcd-prefix")
rootCmd.Flags().MarkHidden("etcd-servers")
rootCmd.Flags().MarkHidden("etcd-servers-overrides")

Also since the entire runtime depends on the internal sample-apiserver it makes it impossible to tweak these arguments in the first place, see issue #58.

Stale k8s.io/apiserver dependency blocks downstream dependencies updates

Hi, I have been developing a custom k8s api server extension via API Aggregation Layer. My go.mod file has the following dependencies:

require (
  k8s.io/apimachinery v0.23.1
  k8s.io/apiserver v0.23.1
  k8s.io/klog/v2 v2.30.0
  sigs.k8s.io/apiserver-runtime v1.1.1
  mycustommodule vX.Y.Z
)

but mycustommodule depends on https://pkg.go.dev/go.opentelemetry.io/otel/[email protected]. As a result I have the following errors after go mod tidy(just extract):

k8s.io/apiserver/pkg/server imports
  k8s.io/apiserver/pkg/endpoints/filters imports
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp imports
  go.opentelemetry.io/otel/semconv: module go.opentelemetry.io/otel@latest found (v1.27.0),
  but does not contain package go.opentelemetry.io/otel/semconv
        
k8s.io/apiserver/pkg/server imports
  k8s.io/apiserver/pkg/endpoints/filters imports
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp tested by
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.test imports
  go.opentelemetry.io/otel/oteltest imports
  go.opentelemetry.io/otel/metric/number: module go.opentelemetry.io/otel/metric@latest found (v1.27.0),
  but does not contain package go.opentelemetry.io/otel/metric/number 

Solution is to update k8s.io/apiserver up to the latest version https://pkg.go.dev/k8s.io/[email protected], but at the same time it will make me to update sigs.k8s.io/apiserver-runtime as well up to the latest commit from master. My final go.mod will look like this:

require (
  k8s.io/apimachinery v0.30.1
  k8s.io/apiserver v0.30.1
  k8s.io/klog/v2 v2.120.1
  sigs.k8s.io/apiserver-runtime v1.1.2-0.20231213224704-8af026f5e12a
  mycustommodule vX.Y.Z
)

But at this case I will have a runtime error:

pkg/mod/sigs.k8s.io/[email protected]/internal/sample-apiserver/pkg/cmd/server/start.go:120:43: o.RecommendedOptions.Etcd.StorageConfig.Paging undefined (type storagebackend.Config has no field or method Paging) (exit status 1)
https://github.com/kubernetes-sigs/apiserver-runtime/blob/main/internal/sample-apiserver/pkg/cmd/server/start.go#L120

Could you please update sigs.k8s.io/apiserver-runtime's dependency up to k8s.io/[email protected]?

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

k8s.io/apiserver v0.25.0 version not supported

Hey kubernetes-sigs

I'm using in the project apiserver and apiserver-runtime but it's not working with apiserver starting from 0.25.0, that's because of new Destory method which was added in Storage interface: https://github.com/kubernetes/apiserver/blob/456c5a07f9118d0118129d0f80eb5b586b3d151f/pkg/registry/rest/rest.go#L63C2-L63C11

This method is not implemented in parentPlumbedStorageGetterUpdaterProvider and parentPlumbedStorageGetterProvider:

apiserver-runtime/pkg/builder/rest/provider.go

Line 52 in 8af026f

func (p ParentStaticHandlerProvider) Get(s *runtime.Scheme, g generic.RESTOptionsGetter) (rest.Storage, error) {

As I see the latest version 1.1.1 uses apiserver v0.23.1:

apiserver-runtime/go.mod

Line 17 in 7479e06

k8s.io/apiserver v0.23.1

Are there any plans/branch/thoughts to update the package to support the newest version of apiserver ? At least 0.25.0.

Thanks!

Anyway to avoid etcd dependency when it's not needed?

I'm building an apiserver with this project which doesn't use etcd as backend. However, the resulting binary still requires a --etcd-servers flag to get started normally, and it would keep connecting to that address until a real etcd service is connectd. And during this connecting period, the api endpoint would keep returning 503 error.

So, is there anyway I can avoid the etcd dep? If not, any suggestion to how to make it possible? I may compose a PR here.

Is there any method register subresource with 'WithResourceAndHandler' ?

I'm currently trying to use WithResourceAndHandler to link to an external database, but it doesn't seem to support subresource registration, is there any solution?

some struct missing Destroy method for rest.Storage interface

apiserver-runtime tag version v1.1.1

# sigs.k8s.io/apiserver-runtime/pkg/builder/rest
../../../pkg/mod/sigs.k8s.io/[email protected]/pkg/builder/rest/provider.go:60:10: cannot use parentPlumbedStorageGetterUpdaterProvider{…} (value of type parentPlumbedStorageGetterUpdaterProvider) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        parentPlumbedStorageGetterUpdaterProvider does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
../../../pkg/mod/sigs.k8s.io/[email protected]/pkg/builder/rest/provider.go:66:10: cannot use parentPlumbedStorageGetterProvider{…} (value of type parentPlumbedStorageGetterProvider) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        parentPlumbedStorageGetterProvider does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)

# sigs.k8s.io/apiserver-runtime/pkg/builder
../../../../github/kubes/apiserver-runtime/pkg/builder/storage_provider.go:83:10: cannot use &scaleSubResourceStorage{…} (value of type *scaleSubResourceStorage) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        *scaleSubResourceStorage does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
../../../../github/kubes/apiserver-runtime/pkg/builder/storage_provider.go:97:10: cannot use &connectorSubResourceStorage{…} (value of type *connectorSubResourceStorage) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        *connectorSubResourceStorage does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
../../../../github/kubes/apiserver-runtime/pkg/builder/storage_provider.go:109:11: cannot use &commonSubResourceStorage{…} (value of type *commonSubResourceStorage) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        *commonSubResourceStorage does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
../../../../github/kubes/apiserver-runtime/pkg/builder/storage_provider.go:131:9: cannot use &statusSubResourceStorage{…} (value of type *statusSubResourceStorage) as type "k8s.io/apiserver/pkg/registry/rest".Storage in return statement:
        *statusSubResourceStorage does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
../../../../github/kubes/apiserver-runtime/pkg/builder/storage_provider.go:228:30: cannot use &connectorSubResourceStorage{} (value of type *connectorSubResourceStorage) as type "k8s.io/apiserver/pkg/registry/rest".Storage in variable declaration:
        *connectorSubResourceStorage does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)
make: *** [Makefile:129: vet] Error 2

Please add a big "experimental" notice to the main readme

So that people know it's a WIP (it is, right?).

Can you get @deads2k and my attention when you're ready to remove this notice, so we know what is going on?

Missing WarningsOnCreate method

When trying to build it seems that WarningsOnCreate method is missing.

Seems there is already a commit that fixes this (a30ed8e) but need a new release.

❯ go build ./cmd/apiserver
# sigs.k8s.io/apiserver-runtime/pkg/builder/rest
../../../../go/pkg/mod/sigs.k8s.io/[email protected]/pkg/builder/rest/rest.go:42:31: cannot use s (type *DefaultStrategy) as type Strategy in argument to newStore:
	*DefaultStrategy does not implement Strategy (missing WarningsOnCreate method)
../../../../go/pkg/mod/sigs.k8s.io/[email protected]/pkg/builder/rest/rest.go:66:31: cannot use s (type *DefaultStrategy) as type Strategy in argument to newStore:
	*DefaultStrategy does not implement Strategy (missing WarningsOnCreate method)
../../../../go/pkg/mod/sigs.k8s.io/[email protected]/pkg/builder/rest/strategy.go:46:5: cannot use DefaultStrategy{} (type DefaultStrategy) as type Strategy in assignment:
	DefaultStrategy does not implement Strategy (missing WarningsOnCreate method)

Error: parent storageProvider for xxxxx must implement rest.StandardStorage

Hi,everyboy.
when i use the lastest version of the apiserver-runtime and controller-runtime. I can build the image successfully. But when i apply it in my cluster , it had a error : "Error: parent storageProvider for xxxxx must implement rest.StandardStorage". My project was right and can apply without mistake. But , now i want to update the lib of apiserver(newest include go.opentelemetry.io/otel v1.10.0 lib), it has the error. Can you help how to fix it ?

Thanks !
Jarvif !

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md