Comments (42)
Update for all watching this issue:
Work on getting Curator to work with Elasticsearch 8 is currently underway, aggressively. You should know, however, that it will not be compatible with older configuration files, though action files should still work the same way. The reason for this is that Elasticsearch 8 has an updated python client which has been updated (majorly) to remove older methods, and to name functions, methods, classes, etc. to conform to Elasticsearch native patterns. Additionally, Elasticsearch 8 has not only deprecated (which was done in Elasticsearch 7) but fully removed doc_type
as an argument from API calls. This requires extensive refactoring of the Curator code (though mostly in tests) in order to work. Other API calls are similarly changing.
The changes to the python Elasticsearch 8 client are so substantial that I'm completely removing the client connection code from Curator in favor of making that its own module, es_client
. Work on that is nearly complete, though I have some more tests to write and more docs to update and/or write. This means that api keys, headers, and several other things will now just work out of the box. This also means, however, that Curator ceases to be reverse compatible with older versions. What will this mean for support for older versions?
Major version parity will be by major version names.
Work on both Curator 6 and Curator 7 is effectively complete, and they are in the 6.0 and 7.x branches already, respectively. No releases have been made yet, however, as documentation needs to be updated. As with Curator 8, they will only work within the same major version, i.e. Curator 7.x will only work with Elasticsearch 7.x clusters, though Curator 6 retains reverse compatibility with both Elasticsearch 6 and Elasticsearch 5 clusters. As both Elasticsearch 6 and Elasticsearch 7 are effectively EOL (though Elasticsearch 7 continues to receive security patches and other updates for a time in the 7.17.x branch), I don't expect Curator 6 and 7 to change much other than to fix execution bugs and to update dependencies to newer versions as needed for security patches and such.
It should also be noted that I will no longer be compiling and packaging Curator releases into DEB, RPM, or Windows MSI or Zip packages. Curator will be released to PyPi and Docker Hub (my personal repository there), and instructions will be shared for those who wish to build their own DEB/RPM/MSI/ZIP packages. The overhead of build-testing and maintaining the package repositories is part of the reason for slow release pacing, and I'd rather offer faster, more consistent updates without those luxuries than keep up the effort to maintain them. The Dockerfile in the root directory will be the one used for publication, so if you want to build your own you can use that, or alter it for your own purposes.
As Travis CI is also something of a luxury for me, I will be testing releases against local Docker images, and including the scripts necessary to build them so that it is easy for anyone to do likewise. If I can keep it working with Travis CI, I will, but it, too is a bit slower on the draw, so I'd just as soon keep this small, knowable, and nimble.
Other changes may follow. I was disappointed to learn that Python 3.10 has altered Collections such that I cannot easily make Curator work with it, though I haven't given up. I need to get Curator functional on Python 3.9 before I worry about 3.10+ support, so please be patient. The worry is that once I get it to work on 3.10, it may not work on 3.9, so there is that to consider.
Thank you all for your patience as I have kept this project on hold for so long. A lot has been going on in everyone's lives since the last Curator release, and I am no exception. I'm now in a place where I have more time and autonomy so I can dedicate my personal resources to this, and I'm excited to move it forward.
from curator.
I just finished 2 weeks off work on vacation, followed immediately by my wife having surgery. I have fallen behind somewhat. No ETA yet. Tons of refactoring still to do.
from curator.
I'm still using Curator because of this limitation in ILM.
TBH my use case is pretty basic, (count indices with prefix, delete those over n) so if Curator isn't going to be supported in future a custom script shouldn't be hard.
It would be nice to just continue using Curator though!
edit: FWIW this is my 'fix' as none of the Elastic APIs i'm using have changed
sed -ri 's/return \(7, 99, 99\)/return \(8, 99, 99\)/g' /usr/local/lib/python3.6/site-packages/curator/defaults/settings.py
from curator.
This is resolved with the release of Curator v8.
from curator.
It would definitely be nice to have Curator be maintained to support Elasticsearch 8.x. Using ILM requires that an index template be created for each index, which is not always desirable or feasible. I manage 8 separate instances of Elasticsearch for various production, staging, and development workloads. To be able to simply delete (prune older) indices of certain types by editing a Curator yml file is significantly less cumbersome and more convenient than setting up index templates for each separate instance.
I hope you reconsider keeping this maintained, or consider providing a minor update to be able to delete indices on Elasticsearch 8.x.
Thank you.
from curator.
Please could u speed up with the solution? Due to the Elastic 8 being released a long time ago.
from curator.
I can tell you there is a huge push now to get Curator 8 out the door in rapid fashion, and I hope that it is fully available for the general public before the end of January now.
I can also tell you that refactoring all of the APIs (and I mean all of them) is tedious and time consuming. I have to change not only the API calls that are made, but how the tests are structured, too. These changes will not be able to be backported to Curator 7 or Curator 6 (which will also be officially released at the same time as Curator 8), nor will any future work on Curator be backported to earlier releases. The nice thing is that the work I'm doing now will make it easier to release a Curator 9 when Elasticsearch 9 is released.
What happened? Why did it take so much work to do this? In a nutshell, the elasticsearch python client was majorly refactored to more closely match the API of Elasticsearch itself. On the simple side, elasticsearch-py function calls which sent a JSON body (well, a Python dict
that was automatically converted to JSON for the call) were frequently using body=...
as a parameter. Guess what isn't used any more? And where settings were being passed as part of body=
elsewhere, now there's dedicated transient=
and permanent=
options, which means completely changing the structures and how they were passed.
As stated in previous communications, I am not a full-time developer anymore, so all of my work has been slower, and a labor of love. It was hard to find time, and it was hard to find motivation when these changes are so tedious to chase down and fix. Now, however, there is internal pressure to get this ready for the general public. I'm excited that I get a few weeks to use nearly all of my work time to get Curator 8 ready. It's still tedious, but I've made great progress on the lower-hanging fruit. I have a ton of work to do for Reindex
and Freeze
, to say nothing of Snapshot
and Restore
ahead of me.
There will also be a few new features with regards to ILM, possibly SLM, and definitely about moving indices/shards through different tiers. I will also need to make Curator work with Searchable Snapshots, which is entirely new.
I appreciate the community's work in modifying older versions of Curator to ignore the <8.0
limitation in the current Curator 5.8.5 release. This is a workaround that will not pass all tests, unfortunately, but at least allows you the opportunity to get what you need working until such time as I can get a fully Curator 8 release to work.
NOTE: Curator 8 will NOT release on RPM/DEB packages any more. It will be available via source code, pip
, and Docker image. I will try to build a Docker image for arm64 in addition to x86_64, but that will depend on some external testing. I will put the scripts to build Docker images in this Github repo so anyone can build their own.
from curator.
UPDATE:
I have received some timely feedback from @sethmlarson and now believe I have a clean way forward to make it work with Python 3.10. You'll see it in the branch if I do get it there.
from curator.
Also note that much of what I have had to refactor is the removal of the AWS-ES supporting code that resulted in using requests
, which resulted in the error:
AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'
A tremendous amount of code clean-up has happened. I should warn you all that your existing ~/.curator/curator.yml
files will need to be modified. As a teaser, and for preparation, I have created a side project to handle Elasticsearch client object creation. I liked the work that I did in Curator to make this work, and wanted it to be available as a separate module, so I created it at https://github.com/untergeek/es_client. This now technically supports every client connection option that the official Elasticsearch Python client offers, including API keys, bearer tokens, and also allows the use of cloud_id
s to connect to Elastic Cloud and/or ECE.
There is a simple example of how to build a CLI client which accepts a config file and will add and override that file with CLI-specified options here. Not all options are addressed, but the most common ones are, making this example usable out of the box for many applications.
from curator.
I'm still using Curator because of this limitation in ILM. TBH my use case is pretty basic, (count indices with prefix, delete those over n) so if Curator isn't going to be supported in future a custom script shouldn't be hard.
It would be nice to just continue using Curator though!
edit: FWIW this is my 'fix' as none of the Elastic APIs i'm using have changed
sed -ri 's/return \(7, 99, 99\)/return \(8, 99, 99\)/g' /usr/local/lib/python3.6/site-packages/curator/defaults/settings.py
Hey
Help please, I switched to ELC 8.1.2 and my Curator stopped working
i found the file /opt/elasticsearch-curator/lib/curator/defaults/settings.pyc
but it is binary
Tell me how you managed to make changes to the binary and run it. I will be grateful for help
from curator.
Progress Report
I have a ton of work to do for
Reindex
andFreeze
, to say nothing ofSnapshot
andRestore
ahead of me.
Freeze
is no longer supported in 8, so I don't have to worry about that.
Last night I finished initial work on Reindex
. I have also finished Snapshot
and Restore
, and all of the tests are working for all of the refactoring work I've completed. I've had to update the es_client
module a few times, so that is now at 8.6.0.post3
as a result. The two messiest and hardest ones to refactor so far were Reindex
and Shrink
, but they are working and passing tests now. Yay!
The Index Settings
action is still working with the older body
type calls. There are probably a few more calls like that which are deprecated but working still. I want to clean it all up, but can do some of that kind of clean-up later, so long as test are passing and the code is working.
What remains, if I'm this close? I have to finish es_repo_mgr
and refactoring all of the click
command-line work for the singletons. Most of that will be the same, but I do need to add new options for the client configuration portion of those scripts.
I will likely do a pre-release at that point. I have some Searchable Snapshot functionality to add, but that won't matter to Basic License users of Elasticsearch. I will likely also need to get data tiers working ASAP as well, though this functionality is still covered by the Shard Allocation Routing that is already in Curator, from what I can tell.
from curator.
It is not. ILM is time based, not size based.
from curator.
My Gist https://gist.github.com/anjia0532/d8e779c97333d6fc0d045675e18e1224
Chinese Blog 074-Elastic Curator 支持ES8.x
docker pull anjia0532/curator:5.8.5
OR
docker build . -t curator:5.8.5
FROM python:3.9.4-alpine3.13
# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
# pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
# pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple
RUN apk --no-cache add --virtual .build-deps git && \
git clone https://github.com/elastic/curator.git /tmp/curator && \
cd /tmp/curator && \
git checkout -b tags/v5.8.4 && \
wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
git apply --check /tmp/curator.patch && \
git apply /tmp/curator.patch && \
apk del .build-deps
RUN cd /tmp/curator && python setup.py install
# Thanks for @arslanbekov
RUN rm -rf /tmp/curator
from curator.
@untergeek May I kindly ask if there is any movement about this issue?
ILM does NOT allow the removal of old indexes based on the disk used space, so if the cluster is running out of space it is throwing out the fresh logs instead of the old ones, and this is the only feature that I require from the curator.
Ofc it is possible to use the hacks advice above, but it feels so ridiculously wrong.
If the company is not willing to maintain this repo, then, please, allow someone else to do this job, officially.
Thank you
from curator.
Your client YAML file raises a few questions. First, it doesn't look properly indented. Second, you have added port
which appears to be a hold-over from a much older version of Curator. The hosts
definition should now include the full URL + port, e.g. http://127.0.0.1:9200
. Third, you're adding http_auth
when it should be username
and password
underneath other_settings
. This appears to be another holdover from an older release of Curator.
These changes are all documented in the official documentation. These breaking changes are documented in the Changelog.
Your client definition should look like this, instead:
elasticsearch:
client:
hosts:
- http://127.0.0.1:9200
other_settings:
username: elastic
password: mypassword
logging:
loglevel: INFO
logfile: /var/log/curator
logformat: default
I should also note that you should not be logging to /var/log/curator
while using Docker. Given that you've mounted a path (even though you're keeping it in /etc
locally) you could set logfile: /.curator/curator.log
and then your log file will appear in /etc/elasticsearch/curator/curator.log
.
from curator.
I cannot promise any rapid release of a version of Curator that supports ES 8. There are multiple API calls that have changed, and those will have to be addressed.
What is in Curator that is not in ILM and SLM that you require Curator for? If you're using ES 8, you have both ILM and SLM to cover a great many things, and while I know the feature set isn't perfectly aligned between the two, it seems unusual to me that you would require Curator, so you have piqued my curiosity.
from curator.
I am using a curator for the daily clearing of indices. I am rising it via cron and doing the cleaning.
from curator.
Does ILM not work for you? ILM's time-based policies should be able to accomplish this for you in a way that does not require cron.
The bottom line is that Curator was created long before ILM existed, and is effectively deprecated for most index management use cases in favor of ILM. Curator remains for use cases that ILM cannot address, and for older clusters that do not support ILM.
from curator.
I will try, i will be back.
from curator.
Can anyone help, I downloaded the archive:
curator-5.8.4.tar.gz
inside it, take the file:
curator-5.8.4.tar.gz\curator-5.8.4.tar\curator-5.8.4\curator\defaults\settings.py
open with notepad and correct version (return (7, 99, 99) to return (8, 99, 99)
Next, make the binary with the command python3 -m py_compile ss.py
replace the new binary in /opt/elasticsearch-curator/lib/curator/defaults/settings.pyc
After that, try to run the Curator again
from curator.
@alex2013a90, I compiled the edited settings.py file and tested it on some Windows hosts running Elasticsearch 8.1.2 successfully. I don't have a Linux box to test against but feel free to use the attachment, which contains the recompiled settings.py file for Curator 5.8.4.
from curator.
We are also using Curator to snapshot indeces and delete old ones.
We just upgraded to ES version 8 and run into the same problem here...
Regards
from curator.
We rely on curator to avoid running out of space and were about to finally upgrade to ES v8. Thankfully, I found this ticket. How is curator still not compatible with v8+ yet?
from curator.
Just for info: We used snapshot and delete indeces with Curator. I contacted ES support and they recommended to use ILM and SLM, so we just ditched Curator and switched to ILM and SLM.
Regards
Richard
from curator.
We specifically used the curator instead of ILM because to my knowledge, ILM can't maintain an aggregate max index size.
So we're doing this:
actions:
1:
action: delete_indices
description: >-
Delete indices if total size of logs-bla-* is greater than 400GB.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: false
allow_ilm_indices: true
filters:
- filtertype: pattern
kind: prefix
value: logs-bla-
exclude:
- filtertype: space
disk_space: 400
If this is possible now with ILM or SLM, I'm all ears.
from curator.
Well, we don't have this use case. But check out the ILM docs, maybe this is doable with ILM.
Regards
Richard
from curator.
@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator
, and then run python setup.py
Correct script:
FROM python:3.9.4-alpine3.13
# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
# pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
# pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple
RUN apk --no-cache add --virtual .build-deps git && \
git clone https://github.com/elastic/curator.git /tmp/curator && \
cd /tmp/curator && \
git checkout -b tags/v5.8.4 && \
wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
git apply --check /tmp/curator.patch && \
git apply /tmp/curator.patch
WORKDIR /tmp/curator
RUN python setup.py install
from curator.
@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deleted
rm -rf /tmp/curator
, and then runpython setup.py
Correct script:
FROM python:3.9.4-alpine3.13 # Chinese mirror #RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \ # pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \ # pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple RUN apk --no-cache add --virtual .build-deps git && \ git clone https://github.com/elastic/curator.git /tmp/curator && \ cd /tmp/curator && \ git checkout -b tags/v5.8.4 && \ wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \ git apply --check /tmp/curator.patch && \ git apply /tmp/curator.patch WORKDIR /tmp/curator RUN python setup.py install
Thanks And Fixed it.
from curator.
I would also appreciate if Curator will support Elasticsearch 8. I have created simple library, which helps us automatize indices and mappings changes. Our indices use index and component templates, but are not time-based neither size-based (so afaik ILM is not an option for us), but we have multiple environments and making changes manually is very annoying.
from curator.
Hi there, can make ILM work for me. Would thank you dearly if you can speed up the support for vesion 8 !
Also,
Anyone found an acceptable workaround for version 8 ?
from curator.
@untergeek Hi there! Any updates in terms of new curator release?
from curator.
I just finished 2 weeks off work on vacation, followed immediately by my wife having surgery. I have fallen behind somewhat. No ETA yet. Tons of refactoring still to do.
Thank you for your effort, should we be expecting it by Xmas ?
from curator.
My backup plan is just to use a simple Python script to delete old indices.
/_cat/indices?format=json
for e in resp.json()
if 'myindexprefix' in e['index'] && older than ndays
requests.delete
It's kind of surprising that no one has created a dirt simple curator in github
from curator.
@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deletedrm -rf /tmp/curator
, and then runpython setup.py
Correct script:FROM python:3.9.4-alpine3.13 # Chinese mirror #RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \ # pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \ # pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple RUN apk --no-cache add --virtual .build-deps git && \ git clone https://github.com/elastic/curator.git /tmp/curator && \ cd /tmp/curator && \ git checkout -b tags/v5.8.4 && \ wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \ git apply --check /tmp/curator.patch && \ git apply /tmp/curator.patch WORKDIR /tmp/curator RUN python setup.py install
Thanks And Fixed it.
Does this patch give you a functional curator? On my side, for a delete indices action against ES8, I get this:
AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'
from curator.
@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deletedrm -rf /tmp/curator
, and then runpython setup.py
Correct script:FROM python:3.9.4-alpine3.13 # Chinese mirror #RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \ # pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \ # pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple RUN apk --no-cache add --virtual .build-deps git && \ git clone https://github.com/elastic/curator.git /tmp/curator && \ cd /tmp/curator && \ git checkout -b tags/v5.8.4 && \ wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \ git apply --check /tmp/curator.patch && \ git apply /tmp/curator.patch WORKDIR /tmp/curator RUN python setup.py install
Thanks And Fixed it.
Does this patch give you a functional curator? On my side, for a delete indices action against ES8, I get this:
AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'
@lsoica Test it by my docker image ?
docker pull anjia0532/curator:5.8.5
from curator.
@anjia0532 thank you.
I've solved things with your help! 🙂
curator-dn64c 0/1 Completed 0 86s
from curator.
More info at https://es-client.readthedocs.io/en/v8.6.0/defaults.html
from curator.
@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deletedrm -rf /tmp/curator
, and then runpython setup.py
Correct script:FROM python:3.9.4-alpine3.13 # Chinese mirror #RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \ # pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \ # pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple RUN apk --no-cache add --virtual .build-deps git && \ git clone https://github.com/elastic/curator.git /tmp/curator && \ cd /tmp/curator && \ git checkout -b tags/v5.8.4 && \ wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \ git apply --check /tmp/curator.patch && \ git apply /tmp/curator.patch WORKDIR /tmp/curator RUN python setup.py install
Thanks And Fixed it.
Hello @anjia0532 ,
I have used your docker image "anjia0532/curator:5.8.5" in the cronjob, but getting the below error, could you please assist
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 93s default-scheduler Successfully assigned efk-system-logging/curator-elasticsearch-curator-27903900--1-mlrc5 to bit-rat-tst-2-md-1-5b996849bf-hsrsl
Normal Pulled 93s kubelet Container image "registry.test.com/test/curator:5.8.5" already present on machine
Normal Created 93s kubelet Created container elasticsearch-curator
Warning Failed 93s kubelet Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/curator/curator": permission denied: unknown
root@test:~$
My cronjob yaml:
apiVersion: batch/v1
kind: CronJob
metadata:
annotations:
meta.helm.sh/release-name: curator
meta.helm.sh/release-namespace: efk-system-logging
labels:
app: elasticsearch-curator
app.kubernetes.io/managed-by: Helm
chart: elasticsearch-curator-2.2.3
heritage: Helm
release: curator
name: curator-elasticsearch-curator
namespace: efk-system-logging
spec:
jobTemplate:
metadata:
labels:
app: elasticsearch-curator
release: curator
spec:
template:
metadata:
creationTimestamp: null
labels:
app: elasticsearch-curator
release: curator
spec:
containers:
containers:
- args:
- --config
- /etc/es-curator/config.yml
- /etc/es-curator/action_file.yml
command:
- /curator/curator
image: registry.test.com/test/curator:5.8.5
imagePullPolicy: IfNotPresent
name: elasticsearch-curator
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/es-curator
name: config-volume
dnsPolicy: ClusterFirst
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
runAsUser: 16
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: curator-elasticsearch-curator-config
name: config-volume
schedule: '*/5 * * * *'
successfulJobsHistoryLimit: 3
suspend: false
from curator.
@anjia0532谢谢!但它不起作用。首先,在脚本中,删除文件夹,然后运行正确的脚本:
rm -rf /tmp/curator``python setup.py
FROM python:3.9.4-alpine3.13 # Chinese mirror #RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \ # pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \ # pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple RUN apk --no-cache add --virtual .build-deps git && \ git clone https://github.com/elastic/curator.git /tmp/curator && \ cd /tmp/curator && \ git checkout -b tags/v5.8.4 && \ wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \ git apply --check /tmp/curator.patch && \ git apply /tmp/curator.patch WORKDIR /tmp/curator RUN python setup.py install
谢谢并修复了它。
你好@anjia0532,
我在 cronjob 中使用了您的 docker 映像“anjia0532/curator:5.8.5”,但收到以下错误,请您协助
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 93s default-scheduler Successfully assigned efk-system-logging/curator-elasticsearch-curator-27903900--1-mlrc5 to bit-rat-tst-2-md-1-5b996849bf-hsrsl Normal Pulled 93s kubelet Container image "registry.test.com/test/curator:5.8.5" already present on machine Normal Created 93s kubelet Created container elasticsearch-curator Warning Failed 93s kubelet Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/curator/curator": permission denied: unknown root@test:~$
我的 cronjob yaml:
apiVersion: batch/v1 kind: CronJob metadata: annotations: meta.helm.sh/release-name: curator meta.helm.sh/release-namespace: efk-system-logging labels: app: elasticsearch-curator app.kubernetes.io/managed-by: Helm chart: elasticsearch-curator-2.2.3 heritage: Helm release: curator name: curator-elasticsearch-curator namespace: efk-system-logging spec: jobTemplate: metadata: labels: app: elasticsearch-curator release: curator spec: template: metadata: creationTimestamp: null labels: app: elasticsearch-curator release: curator spec: containers: containers: - args: - --config - /etc/es-curator/config.yml - /etc/es-curator/action_file.yml command: - /curator/curator image: registry.test.com/test/curator:5.8.5 imagePullPolicy: IfNotPresent name: elasticsearch-curator resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/es-curator name: config-volume dnsPolicy: ClusterFirst restartPolicy: Never schedulerName: default-scheduler securityContext: runAsUser: 16 terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: curator-elasticsearch-curator-config name: config-volume schedule: '*/5 * * * *' successfulJobsHistoryLimit: 3 suspend: false
remove command , change args to 'curator --config /path/to/config_file.yml /path/to/action_file.yml'
from curator.
This might be relevant for the docker image of curator in docker hub: docker/hub-feedback#2314
from curator.
Thank you for the heads-up. As Curator is hosted on my personal, individual account, I do not believe this will affect me. I will eventually get around to having Curator's docker images hosted with Elastic, but for now, this seems safe.
from curator.
i'm trying to delete indices on es 8.3.3.
container shows exit status 0 , but indices still exists
docker run --network host --rm --name es-curator -v /etc/elasticsearch/curator:/.curator untergeek/curator:8.0.8 --config /.curator/client.yml /.curator/action.yml
actions:
1:
action: delete_indices
description: Delete indices older than 4 days (based on index name)
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: k8s-production-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 4
elasticsearch:
client:
hosts:
- 127.0.0.1
port: 9200
http_auth: elastic:mypassword
logging:
loglevel: INFO
logfile: /var/log/curator
logformat: default
from curator.
Related Issues (20)
- Docker build fail to create runable image HOT 12
- Unable to skip some indices while restoring
- [OLD docker] untergeek/curator `missing signature key` on server HOT 10
- max_primary_shard_size option for shrink & rollover ACTION HOT 1
- curator cronjob error HOT 2
- Edge case in metadata population at IndexList instantiation HOT 3
- Trying to snapshot all indices fails as Curator accesses system indices
- pip install elasticsearch-curator==7.0.0 failed due to pyyaml 6.0.0
- Manual ILM advance based on disk usage (per tier)
- docker pull throws `missing signature key` HOT 2
- Curator running failed in docker when upgrade to 8.0.8 as permssion error HOT 17
- Error: No such option: --config HOT 1
- Curator Snapshot/delete action is failing which points to snapshot_missing_exception. HOT 10
- Curator_cli requires the use of the --config parameter (verified for 8.0.8) HOT 1
- bug: Support PEP-517 in 6.x
- curator_cli forcemerge exception
- new curator version breaks the TLS verify functionality HOT 6
- Unexpected 'NoneType' error when running Curator HOT 5
- please release 8.0.16 HOT 1
- Dependency Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from curator.