Comments (23)
The backups look great, but could you also implement a configurable backup retention. Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier. So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access. Also you will definitely wont have complains for a full hard disk with a sensible default...
Absolutely right as well.
Keep within retention period(in days):
find /palworld/backups -name 'saved-*.tar.gz' -type f -mtime +$RETENTION_DAYS -exec rm {} \;
Or it might be better to just keep the latest n files in case you shut down the server for an extended period of time and set it back up later, all backups would get wiped. But people should be moving these backups offsite (to S3 or something) routinely anyway, so it's not a big deal.
Keep last N files:
cd /palworld/backups
ls -1t saved-*.tar.gz | tail -n +$(($KEEP_LAST_N_SAVES + 1)) | xargs -d '\n' rm -f --
from docker-palworld-dedicated-server.
The backups look great, but could you also implement a configurable backup retention. Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier. So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access. Also you will definitely wont have complains for a full hard disk with a sensible default...
@abdonkov - Im open for suggestions/ideas/code examples, everything im coming up has up-sides and drawbacks. I thought at least deliver the function to test first 😭 - Keeping only days, will kill inactive server with valid saves on a bug, therefore no backups anymore. Keeping the last 1?3?10? How to decide? How to do?
Looks like you nailed it. The only things I would add in future, (and maybe I will create a PR for this since you already have your hands full) is to still call /Save explicitly using the RCON commands before backing up. Some games broadcast a warning before saving because a save can interrupt gameplay, but I don't think Palworld has that issue.
@CameronJGrant - I think i have even seen something like this coded, dont remember right now where and what the context was 😭 but i agree send /save and /broadcast "Server-Save in progress ..." would be cool. -> https://tech.palworldgame.com/server-commands
I think we are on very different time zones.
Germany!
You don't need all that. You just need cron running in the container. I would have just installed cron via apt-get but @jammsen came up with a far more elegant solution with supercronic.
😊 ❤️
Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄
from docker-palworld-dedicated-server.
@abdonkov - Im open for suggestions/ideas/code examples, everything im coming up has up-sides and drawbacks. I thought at least deliver the function to test first 😭 - Keeping only days, will kill inactive server with valid saves on a bug, therefore no backups anymore. Keeping the last 1?3?10? How to decide? How to do?
Once we figure out how to use RCON in script's more easily we should use the /ShowPlayers command to check if anyone's in game before saving so that we aren't backing up unless someone is actually in there doing something. Then add a grace period after the server empties that backups can be created in.
I suggest we do that and allow users to choose their retention policy.
Possible environment variables:
Variable | Description | Default Value | Allowed Value |
---|---|---|---|
BACKUP_ENABLED | Backup function, creates backups in your game directory | true | false/true |
BACKUP_CRON_EXPRESSION | Needs a Cron-Expression - See https://github.com/aptible/supercronic#crontab-format or https://crontab-generator.org/ | 0 * * * * (meaning every hour) | Cron-Expression |
BACKUP_MAX_COUNT | The maximum number of backup files to keep | 5 | Integer value |
BACKUP_MAX_AGE | The maximum age for a backup file before it is deleted (in days) | 30 | Integer value (days) |
BACKUP_IF_IDLE | Perform backup only if the server is idle | false | false/true |
BACKUP_GRACE_PERIOD | Time to wait before starting the backup after the scheduled time (in minutes) | 10 | Integer value (minutes) |
BACKUP_ZIP | Whether to compress the backup files into ZIP format | true | false/true |
Weird side note. It saves backups of local data to the client's PC. I'm not sure if you could fix a server instance with that though.
https://docs.google.com/document/u/0/d/e/2PACX-1vR-lb9R2fh-IVK870PL82U071i-9-C_Y7YyroUN8EoD_34rbpKQzjsCejCHRb4XMGncY9HkI0OxQv1F/pub?pli=1#h.xl4nmt5nrqb9
@CameronJGrant - I think i have even seen something like this coded, dont remember right now where and what the context was 😭 but i agree send /save and /broadcast "Server-Save in progress ..." would be cool. -> https://tech.palworldgame.com/server-commands
Maybe it was here?
https://gist.github.com/mbasaglia/62d7fef83304f4b3ca90
Germany!
Cool!
Might I also suggest opening a develop branch?
from docker-palworld-dedicated-server.
@doodoori2 Read this #27 (comment) and realize you are already here ;)
from docker-palworld-dedicated-server.
Hey @CameronJGrant - I like the idea, would look into this tomorrow.
Few question though:
- Dont you mean a "tar.gz" of Saved?
- How would you run the /Save command inside the container? How would you inject that in the process? The RCON tool is outside of the container
- Are we now mixing internal and external work?
- Could you give me more details on your thoughts so i get a better understanding of the workflow you imagine?
- I will look into https://github.com/lloesche/valheim-server-docker tomorrow, its no 01:19 in the morning 🥱
from docker-palworld-dedicated-server.
You'd need a whole process manager to also have cron
run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)
I'd just add an example to the readme how to create backups with cron yourself.
Like backup.sh:
#!/bin/bash
# dir should be absolute for cron or you might end up anywhere:
cd /your/palworld/directory
mkdir -p backups
tar -czf backups/${something-with-date}.tar.gz game/Pal/Saved
And just call that via crontab.
from docker-palworld-dedicated-server.
You'd need a whole process manager to also have
cron
run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)I'd just add an example to the readme how to create backups with cron yourself.
Like backup.sh:
#!/bin/bash # dir should be absolute for cron or you might end up anywhere: cd /your/palworld/directory mkdir -p backups tar -czf backups/${something-with-date}.tar.gz game/Pal/SavedAnd just call that via crontab.
Im already done implementing a solution, finishing the last tests and then i will push the changes.
from docker-palworld-dedicated-server.
Valheim used something like that, but they had years of development ;)
I know and i would rather stay away from unnecessary bloating the image with an additional process manager like supervisord, im very happy i got it out again in my Sons Of The Forest and The Forest Dedicated.
Im using now a solution thats more container oriented and dont need all that stuff like cron does.
from docker-palworld-dedicated-server.
Hey @uncaught @CameronJGrant -
I have implemented a backup-solution that does hourly backups in your game
directory. If you want to change settings, you can enable or disable this and you can change the cron-expression, meaning the interval it happens. Therest is hardcoded on purpose and i prefer it that way. Too much open config can result in recursive backups and an early full harddisk and people beeing mad at me. I dont want that.
Please go ahead and try it out .
from docker-palworld-dedicated-server.
The backups look great, but could you also implement a configurable backup retention.
Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier.
So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access.
Also you will definitely wont have complains for a full hard disk with a sensible default...
from docker-palworld-dedicated-server.
Excellent work @jammsen!
Hey @CameronJGrant - I like the idea, would look into this tomorrow. Few question though:
- Dont you mean a "tar.gz" of Saved?
palworld/Pal/Saved would be better. I only had a few minutes to dig around the container yesterday and I wasn't sure if palworld/Pal/Saved/Config had anything that needed saving. I agree with your solution.
- How would you run the /Save command inside the container? How would you inject that in the process? The RCON tool is outside of the container
You can run the RCON commands using something like netcat directly from a shell script within the container. We can include anything you like in your "backupmanager.sh".
- Are we now mixing internal and external work?
Everything should be contained in one image. It's all internal.
- Could you give me more details on your thoughts so i get a better understanding of the workflow you imagine?
Looks like you nailed it. The only things I would add in future, (and maybe I will create a PR for this since you already have your hands full) is to still call /Save explicitly using the RCON commands before backing up. Some games broadcast a warning before saving because a save can interrupt gameplay, but I don't think Palworld has that issue.
- I will look into https://github.com/lloesche/valheim-server-docker tomorrow, its no 01:19 in the morning 🥱
I think we are on very different time zones.
You'd need a whole process manager to also have
cron
run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)
You don't need all that. You just need cron running in the container. I would have just installed cron via apt-get but @jammsen came up with a far more elegant solution with supercronic.
from docker-palworld-dedicated-server.
Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄
I mean if you don't want to decide how much backups to retain you can always set it to 0, meaning the retention by count won't be active. This will also require a retention days parameter as a fallback, like 2, 3 days. Something small so with the default options, the backups won't use too much space and you wont get complains 😄
If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.
from docker-palworld-dedicated-server.
Once we figure out how to use RCON in script's more easily we should use the /ShowPlayers command to check if anyone's in game before saving so that we aren't backing up unless someone is actually in there doing something. Then add a grace period after the server empties that backups can be created in.
I suggest we do that and allow users to choose their retention policy.
#Overkill 🤣
Maybe it was here? https://gist.github.com/mbasaglia/62d7fef83304f4b3ca90
No i could be here https://github.com/jammsen/docker-conanexiles/blob/master/src/usr/bin/conanexiles_controller#L103 it wasnt from the perspective of my build a thing, it was more of the experience of using person, like "im the admin of a dedicated server and host this for my friends and i find the feature ive not seen like this before very cool". Sadly i didnt fork it, because i didnt knew i would need it for something.
Might I also suggest opening a develop branch?
Yes, hopefully tomorrow.
from docker-palworld-dedicated-server.
If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.
Just to be on the safe side, you mean this one right? https://github.com/lloesche/valheim-server-docker
from docker-palworld-dedicated-server.
Hi guys, where is the "Rentention-Policy" you mentioned? is there any discord or ideas page? how can I read the thread?
from docker-palworld-dedicated-server.
Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄
I mean if you don't want to decide how much backups to retain you can always set it to 0, meaning the retention by count won't be active. This will also require a retention days parameter as a fallback, like 2, 3 days. Something small so with the default options, the backups won't use too much space and you wont get complains 😄
If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.
I think i will just go with a keep N files strategy, no need to overcomplicate everything.
Keep last N files:
cd /palworld/backups ls -1t saved-*.tar.gz | tail -n +$(($KEEP_LAST_N_SAVES + 1)) | xargs -d '\n' rm -f --
Actually that looks already nice to get a better understanding and get started
from docker-palworld-dedicated-server.
Just to be on the safe side, you mean this one right? https://github.com/lloesche/valheim-server-docker
Yes, that one.
from docker-palworld-dedicated-server.
Hey @CameronJGrant - Sorry for the delay i finaly got around to finish this. But Bugs > Features, sorry.
Ive added now a switch to turn it on or off and a number to keep with even you shell-code that i really liked. Works like a charm. Thanks for your contribution!
from docker-palworld-dedicated-server.
Im closing this issue as resolved, feel free to try it out and reopen again if needed.
If you like this project, please consider giving this repo and the docker-hub-repo a Star.
from docker-palworld-dedicated-server.
For sure I'll give you a star! Thanks for the hard work! Ironically I just had my server wipe lol. Luckily I had a few manual backups so it wasn't a big deal.
from docker-palworld-dedicated-server.
Hey @jammsen. There isn't a reopen issue button so maybe it's turned off?
Anyway, small thing. I found out the hard way that if disk space fills in your server Palworld will delete your server file. It looks like, when it saves your game, it deletes your old file first, then saves the new one.
That said, I notice the defaults we have set are:
So I notice your defaults are:
Variable | Default |
---|---|
BACKUP_ENABLED | true |
BACKUP_CRON_EXPRESSION | 0 * * * * |
BACKUP_RETENTION_POLICY | false |
BACKUP_RETENTION_AMOUNT_TO_KEEP | 30 |
This means, by default, we are backing up every hour and not removing any old backups. In a year people will have almost 9000 backups. It only comes to about 18GB but a lot of people run small servers. We might be sitting on a ticking time bomb as is.
I suggest having the retention policy on by default and set the time to once per day. Then 30 days are covered. (Since 30 at hourly is only one day).
from docker-palworld-dedicated-server.
For sure I'll give you a star! Thanks for the hard work! Ironically I just had my server wipe lol. Luckily I had a few manual backups so it wasn't a big deal.
That sounds horrible and good at the same time 🤣
Hey @jammsen. There isn't a reopen issue button so maybe it's turned off?
OH DANG SORRY - https://stackoverflow.com/a/21333938
I suggest having the retention policy on by default and set the time to once per day. Then 30 days are covered. (Since 30 at hourly is only one day).
I get your point and i would have done that when i introduced BOTH features at the same time, it wasnt the case for this, it was incremental. Thats why i had to really hard think about it and didnt wanted people to have to loose savegames they didnt backup before, thats why its on false by default. But i get your point, thats why i rewrote the Setup-6-Steps another time after splitting files to make sure people CONFIG it to their liking you know?
from docker-palworld-dedicated-server.
Thanks for the hard work!
No! Thank YOU for your contribution Keanu-Wave! ❤️
from docker-palworld-dedicated-server.
Related Issues (20)
- [Bug Report] Player join/left issue with character creation HOT 1
- [Bug Report] Player join/leave cannot have commas in name HOT 1
- [Bug Report] Player join/leave doesn't work with players of the same name HOT 1
- [Bug Report] Crash Loop (LowLevelFatalError) HOT 4
- [Feature Request] Automatic Restart should not wait 15 minutes when no player is on the server
- [Bug Report] RCON Port setting change in 1.50 HOT 6
- [Bug Report] Error when auto restart HOT 3
- docker部署后无法正常使用RCON,cli: execute: auth: rcon: dial tcp: lookup tcp/###RCON_PORT###:unknown port HOT 6
- [Feature Request] Make gosu-binary building more transparent
- [Bug Report] Log file consumes all remaining HDD space associated with Docker system HOT 3
- [Feature Request] Implement server OoM crash detection and restart HOT 2
- [Bug Report] RCON Log gets cut off
- [Bug Report] New alias for rconcli with timestamp
- [Bug Report] After docker container update Server failed to start with permission denied HOT 11
- [Bug Report] Container backups also backup the Palworld built-in backups HOT 1
- [Bug Report] Automatic Restart stopped working HOT 2
- [Question] 您好,我在游戏中,会接受到容器显示的备份 通知,我想询问下这个备份通知能否不在游戏内显示,可以在容器日志中 HOT 3
- [Feature Request] use semantic versioning HOT 7
- [Bug Report] No config found, generating one HOT 3
- [Bug Report] Your descriptive title here! HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-palworld-dedicated-server.