hiddeco / cronjobber Goto Github PK
View Code? Open in Web Editor NEWCronjobber is a cronjob controller for Kubernetes with support for time zones
License: Apache License 2.0
Cronjobber is a cronjob controller for Kubernetes with support for time zones
License: Apache License 2.0
As an admin I can easily deploy the examples you provide.
As another user (e.g. ci user) I get the following error:
tzcronjobs.cronjobber.hidde.co "my-dummy-service" is forbidden: User "ci-user" cannot get resource "tzcronjobs" in API group "cronjobber.hidde.co" in the namespace "mynamespace": RBAC: clusterrole.rbac.authorization.k8s.io "mynamespace-letsencrypt-role" not found
Do you have any ideas where to look? Do I have to change the rbac in some way?
The error:
Attempted to run a job with an invalid time zone: Europe/Berlin. open /usr/local/go/lib/time/zoneinfo.zip: no such file or directory
During maintenance purpose I want to disable all CronJobs, For default Kubernetes CronJob we can execute following command
kubectl patch cronjobs <job-name> -p '{"spec" : {"suspend" : true }}'
However I cannot see an option to do this in CronJobber . Is there a way to disable and enable Cronjob without playing with schedules.
Hi, I am trying to run cronjobber in an Openshift v4.6 cluster; Kubernetes v1.19. I was able to deploy cronjobber via Helm chart provided in the repo but when I schedule a TZCronJob, I am getting this error when the job starts:
Warning FailedCreate 78s (x91 over 16m ) cronjobber Error creating job: jobs.batch "hello-1630089060" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion batch/v1beta1 Kind TZCronJob: no matches for kind "TZCronJob" in version "batch/v1beta1"
Here is the TZCronJob yaml I am using:
apiVersion: cronjobber.hidde.co/v1alpha1
kind: TZCronJob
metadata:
name: hello
spec:
schedule: "30 15 * * *"
timezone: "America/Sao_Paulo"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo "Hello, World!"
restartPolicy: OnFailure
Hi,
Thank you for this controller, great work! Unfortunately, I had this odd issue recently and I was not able to replicate the issue in a test environment therefore I don't know the root cause. I think this is a critical bug, because it's about an intermittently occurring unexpected behaviour: A job for each TZCronjob get scheduled immediately, regardless of their schedules defined in TZCronJob, when cronjobber pod is recreated.
Platform: AWS EKS 1.14
Scenario
1- cronjobber 0.2.0 was installed with updatetz
2- A TZCronJob is created with schedule "45 14 * * 2" and timezone "Europe/London", no startingDeadlineSeconds set
3- A job is scheduled and completed successfully on 12 May 1:45 pm UTC, which is expected
NAME COMPLETIONS DURATION AGE
somejob-1589291100 1/1 7s 2d
4- Cronjobber pod gets recreated at "Wed, 13 May 2020 15:13:52 +0000":
cronjobber container logs
{"level":"info","ts":"2020-05-13T15:14:24.116Z","caller":"cronjobber/main.go:75","msg":"Starting Cronjobber version 0.2.0 revision f860bc912c395c58fa72741d8d34e8bf4b1a2c00"}
{"level":"info","ts":"2020-05-13T15:14:24.117Z","caller":"cronjobber/controller.go:106","msg":"Starting TZCronJob Manager"}
updatetz container logs
2020-05-13T15:14:29+0000 Local Time Zone database updated to version 2020a on /tmp/zoneinfo
5- Although its last schedule was for 12 May 1:45 pm UTC and it shouldn't be scheduled until next week, another job of this TZCronJob is started at "Wed, 13 May 2020 15:14:24 +0000", which is exactly the same to the second with the start time of cronjobber.
NAME COMPLETIONS DURATION AGE
somejob-1589291100 1/1 7s 2d
somejob-1589294700 1/1 17s 23h
Now, this happened for all of our TZCronJobs. They got rescheduled immediately after cronjobber pod is recreated 😰
Observation
1- updatetz updates the local tz database for the first time in the pod 5 seconds later than cronjobber controller's start time.
2- I've noticed that unix timestamps on Job specs can lead to something. 1589291100 (the expected behaviour) is 12 May 1:45 pm UTC, which is good.
It gets interesting for 1589294700: This is 12 May 2:45 pm (UTC) -> so it looks like cronjobber could not get the timezone data from the mounted volume when it started up. I guess this caused cronjobber to misinterpret existing schedules. Cronjobber synced the existing cronjobs with the misinterpreted tzdata, which led kube-scheduler to interpret that the last schedules were missed although they weren't missed and hence kube-scheduler rescheduled all jobs.
Potential workarounds/suggestions
Since I couldn't replicate the scenario, below would be only blind guesses;
1- We could set startingDeadlineSeconds, but that would not solve the issue permanently. There will be some possibility for the issue to reoccur if [misinterpreted t] + startingDeadlineSeconds > x, where t is the tzcronjob's last successful execution timestamp and x is the time when cronjobber pod is recreated.
2- I haven't written a single line of Go (yet) so this may be gibberish, but I think there may be some missing/wrong variable initialisation. For example, if there is an initial value of the current timestamp, that is used when cronjobber is not able to read tzdata, then this magic number should be of a max value, instead of a min value. In other words, it's better to do nothing (don't sync) than doing something wrong (rescheduling).
3- Cronjobber should not start until updatetz does its job at least once. Maybe docker-entrypoint in updatetz can touch a file on another volume and an init container within the same pod will check if this file exists in this volume. Since there will be an init container, the cronjobber container will not start until the init container completes successfully.
Hello,
I have a two CronJob the one set to 0 1 * * *
in Europe/Paris
which is running at 3 CEST, the other one scheduled at 0 3 * * *
in Europe/Paris
ran at 5 CEST...
Do you have any idea why it's not running at the time it was scheduled ?
Thanks,
Do you apply the same concurrent policy of k8scronjob?
Even if I apply 'forbid' and 'replace', it seems to be set to 'allow'.
Great Job! Thanks for sharing!.
I have a (probably) dummy question. How can I programmatically create a TZCronJob (in particular, I am using python). Is there and endpoint I need to call?
As reference, for regular CronJob, I am using pip package kubernetes==9.0.0 doing this:
cron_job.spec = V1beta1CronJobSpec(job_template=job_template, schedule=schedule)
client.BatchV1beta1Api(self._client).create_namespaced_cron_job(namespace_name, body=cron_job)
which in turn ends up calling
'/apis/batch/v1beta1/namespaces/{namespace}/cronjobs', 'POST'
Any advise very welcomed!
Alex
If the cronjobber must be a singleton, I think stateful set is probably more appropriate.
Nice job, thanks for releasing this!
Before starting to use this, I would like to know a little more about the behavior of cronjobber during edge cases:
There's been some changes to the Kubernetes CronJob controller since v1.13.3
It'd be good to track upstream and periodically sync CronJobber to the upstream CronJob version to avoid diverging too much form the defaults.
As you know, in cronjob, you can create jobs through --from.
However, tzcronjob seems impossible. Is there a way?
I am trying to get cronjobber running on EKS 1.23
The error:
Unable to update status for production/test-cronjob (rv = 8509036): tzcronjobs.cronjobber.hidde.co "test-cronjob" not found
Stacktrace:
github.com/hiddeco/cronjobber/pkg/controller/cronjobber.syncOne
/go/src/github.com/hiddeco/cronjobber/pkg/controller/cronjobber/controller.go:257
github.com/hiddeco/cronjobber/pkg/controller/cronjobber.(*TZCronJobController).syncAll
/go/src/github.com/hiddeco/cronjobber/pkg/controller/cronjobber/controller.go:146
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.Until
/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
Any ideas?
Deprecated in v1.16, planned for removal in v1.22. Use apiextensions.k8s.io/v1 CustomResourceDefinition instead.
CRDs v1 allows the TZCronJob to be registered as part of the K8s API and be detected on Helm using
.Capabilities.APIVersions.Has "cronjobber.hidde.co/v1alpha1"
At the moment I'm managing the detection manually
In the README it states:
- REFRESH_PERIOD (default: 7d) is how often it looks for updates
However, in docker-entrypoint.sh
, and in the init-updatetz
initContainer it is set to use REFRESH_INTERVAL:
sleep "${REFRESH_INTERVAL:=7d}"
in https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22 the CRD apiextensions.k8s.io/v1beta1
is deprecated and only apiextensions.k8s.io/v1
is supported going forward.
also some of the fields need to be updated:
CustomResourceDefinition in version "v1" cannot be handled as a CustomResourceDefinition: strict decoding error:
unknown field "spec.additionalPrinterColumns",
unknown field "spec.subresources",
unknown field "spec.version"
the deprecation guide also provides the alternative fields to use
First of all thanks for this awesome component. While official Kubernetes is expecting to add a Timezone soon but it is getting delayed a lot and we are using CronJobber which works great.
I am using EKS 1.21 and noticed an issue where CronJobber cronjobs will keep spinning up new pods if there are any error, or ImagePullBackoff , CrashLoopBack etc.. it is not honoring the fields concurrencyPolicy: Forbid . This works well with default CronJob API i.e. apiVersion: batch/v1 but not with CronJobber apiVersion: cronjobber.hidde.co/v1alpha1.
Is it possible to add a fix for this I am assuming this is happening because of the updates on CronJob in v 1.20
apiVersion: cronjobber.hidde.co/v1alpha1 kind: TZCronJob metadata: creationTimestamp: "2022-05-18T20:02:02Z" generation: 19 labels: app: cronjob-test-qa chart: test-cronjobs-68 release: cronjob-test-qa name: cronjob-test-qa namespace: qa resourceVersion: "600673922" uid: 667b2f60-02a9-4d19-992a-9b0dba7e8d50 spec: concurrencyPolicy: Forbid failedJobsHistoryLimit: 1 jobTemplate: metadata: labels: app: cronjob-test-qa spec: backoffLimit: 2
I cannot find a Helm chart version of it at https://hub.helm.sh
Do you have plans to create one or should everyone do it on their own manually?
Maybe it'd be a good idea to periodically (using CronJobs 😉?) run a script to update the timezone db instead of mounting the host one. That way CronJobber would not depend on the host having a timezone db at all.
deply/deploy.yaml
ok
vi test.yaml
~
timezone: "Asia/Seoul"
~
kubectl apply -f test.yaml
in 5 sec or 6 sed the pod was starting..
scheldule was in 5 min or 10min from now
ex)
date
Wed Sep 2 10:29:48 KST 2020
test.yaml
Schedule: "35 10 * * *"
There is breaking change in client-go v0.18.0 when is context required for REST client and is not possible to use generated clientset because context is missing (eg. in Get() function).
DoD:
With regular cronjobs you can create jobs from a cronjob when you want to run a task inmediatly:
kubectl create job --from=cronjob/<name of cronjob> <name of job>
With cronjobber this does not work:
$ kubectl create job --from=tzcronjob/update-campaign-metrics update-campaign-metrics-now
error: from must be an existing cronjob: no kind "TZCronJob" is registered for version "cronjobber.hidde.co/v1alpha1" in scheme "k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28"
$ kubectl create job --from=cronjob/update-campaign-metrics update-campaign-metrics-now
Error from server (NotFound): cronjobs.batch "update-campaign-metrics" not found
Would it be possible to fix this?
Trying to deploy cronjobber-updatetz
gives error about non existing image, it seems url is not working anymore.
https://quay.io/repository/hiddeco/cronjobber-updatetz
Warning Failed 8m48s (x4 over 10m) kubelet, ip-172-20-54-167.ec2.internal Failed to pull image "quay.io/hiddeco/cronjobber-updatetz:0.1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: EOF
We're using fluxcd and cronjobber, we would like to automatically update the cronjob image.
Simple testing seems to show that currently, fluxcd cannot understand the cronjobber resource.
The cronjob kind has this feature —from and wondering if we can add here? Ie, kubectl create job —from=cronjob/ .
CronJobber is deployed on cronjobber
NS and other regular CronJob
s are deployed on NS zzzzz
To test my TZCronJob
was deployed with schedule: */1 20 * * *
@ 20.02 then changed to schedule: 15 20 * * *
with the same result as below
There is a TODO just above that log statement. Am I suffering from a race condition of some sort? :(
{"level":"debug","ts":"2020-11-05T20:12:58.892Z","caller":"cronjobber/controller.go:136","msg":"Found 1 groups"}
{"level":"debug","ts":"2020-11-05T20:12:58.901Z","caller":"cronjobber/controller.go:281","msg":"No unmet start times for my-ns/my-job"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:126","msg":"Found 16 jobs"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:133","msg":"Found 1 cronjobs"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-1 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-1 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-2 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-2 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-3 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-3 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-4 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-4 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-5 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-5 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-6 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-6 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-7 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-7 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-8 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-8 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-9 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-9 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-10 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-10 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-11 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-12 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-12 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-14 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-15 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-16 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-17 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:136","msg":"Found 1 groups"}
{"level":"debug","ts":"2020-11-05T20:13:08.929Z","caller":"cronjobber/controller.go:281","msg":"No unmet start times for my-ns/my-job"}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.