Comments (6)
quick followup after looking through the code a bit, it seems like the filesystem monitoring defaults to polling only instead of using an OS native implementation when available? If the observer.polling was replaced with the standard observer: https://pythonhosted.org/watchdog/api.html#module-watchdog.observers it should reduce the amount of processing overhead that's used. It'll still fail over to the polling implementation if the other ones aren't available.
from maestral.
I've noticed the indexing mode also chews up a ton of cpu time, I haven't checked yet but if all files are being hashed using the approach here: https://pythonhosted.org/watchdog/api.html#module-watchdog.observers it may be a good idea to swap that out for files below a specific threshold for a function that reads the whole file in one go then hashes once it's read into memory, instead of reading 1024 bytes at a time as this should allow you to reduce the number of filesystem actions taken.
from maestral.
I'm not very familiar with the docker setup, this was an external contribution. If run natively, Maestral will either use FSEvents on macOS or Inotify on Linux, and the idle CPU usage is close to 0%. It will only fall back to polling if the Platform is neither of those:
maestral/src/maestral/fsevents/__init__.py
Lines 25 to 30 in 5f0cf08
Regarding hashing, this is done in chunks of 65536 bytes, which has been a good tradeoff between CPU and memory usage. Note that multiple files may be hashed in parallel to better distribute load across CPU cores.
Finally, the config file has settings for max bandwidth and CPU usage, and Maestral will throttle its work and transfer speeds to stay below both.
Could you check if you can replicate this behavior outside of a Docker image?
from maestral.
Ah i hadn't noticed that in the init area and had found the polling observer elsewhere. My bad! I had been trying to avoid setting up an environment locally but will replicate my docker config to test and see if I can figure out what's going squirrely with the docker one if things behave more normally natively.
from maestral.
ok I've got maestral configured as a local install in a venv now and it is working, unfortunately even with max cpu time set to 5% the indexing step is now running at 1100% cpu usage. I'm going to try using a cgroup to manage the max cpu and see if that works.
from maestral.
Ok, so I wrote a new systemd user file and forced it to restrain cpu usage via cgroup and my laptop is no longer trying to burn holes in my desk!
[Unit]
Description="Maestral Service"
After=network.target
[Service]
Type=forking
ExecStart=/home/<user>/maestral-venv/bin/maestral start
ExecStop=/home/<user>/maestral-venv/bin/maestral stop
CPUQuota=5%
[Install]
WantedBy=default.target
I may consider bumping up the total amount of cpu it can use but for now it's working in the background and chugging along. I'll dig into the docker container in a bit, there's ways to do similar there and a compose file would be a more robust way of handling things long term.
edit: quick note for anybody using this, this is a user space systemd file it would go into ~/.local/share/systemd/user/maestral.service
and be started with systemctl --user daemon-reload
(to reparse the folder) and systemctl --user start maestral
to start the service, you can use enable to make it a user space service as well.
second note: moved to 20% and the sync and update finished in about an hour and is just idling around not really hurting at all. working solid.
from maestral.
Related Issues (20)
- Hanged system for external dropbox location HOT 2
- Selective Sync is not sorted correctly HOT 1
- App crashing constantly, AGAIN. Message says "contact the developer" HOT 1
- Docs on Raspbian Python version are outdated HOT 1
- Hight memory usage on macOS after a sync HOT 1
- Rename the System Service app to Maestral from Sam Schott HOT 2
- Missing dependency on importlib-metadata HOT 1
- 1.63 Won't Continue Sync HOT 1
- Maestral GUI windows not opening on Mac HOT 3
- Unable to add to excluded list, version 1.9.2 HOT 1
- Crashes when choosing files and folders for selective sync HOT 3
- OSError: [Errno 95] Operation not supported: HOT 5
- App won't start as of Sonoma 14.4.1 HOT 2
- Could not sync file or folder Insufficient read or write permissions.
- Add S3 backend support
- Resyncing folder when sharing is disabled HOT 1
- Insufficient read or write permissions for .Spotlight-V100 folder HOT 2
- Trying to sync system/spotlight folders
- Autostart not working: Cannot load keyring keyring.backends.SecretService.Keyring HOT 1
- Maestral on Multiple Macs - Authentication Failed Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maestral.