Comments (16)
Thanks for letting me know and I can confirm I am getting the same result.
It is working, and there are no errors in the log; it's just hungry :-)
I will wait for the next release and re-test
Thanks for all your great work.
from immich.
Below are some logs from the microservices; perhaps they might provide some insight:
[Nest] 7 - 05/13/2024, 11:34:38 AM LOG [EventRepository] WebSocket server initialized.
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Error: The input file contains an unsupported image format
at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14)
at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29)
at async /usr/src/app/dist/services/job.service.js:149:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Affected Job ID:
{
"id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8"
}
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
at ChildProcess.emit (node:events:518:28)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Affected Job ID:
{
"id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e"
}
Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient.
Here's the relevant part of my Docker Compose configuration:
immich-microservices:
container_name: immich_microservices
image: altran1502/immich-server:${IMMICH_VERSION:-release}
command: ['start.sh', 'microservices']
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro"
- /etc/localtime:/etc/localtime:ro
devices:
- /dev/dri:/dev/dri
env_file:
- .env
depends_on:
- redis
- database
restart: always
deploy:
resources:
limits:
cpus: '2'
memory: 4G
[Nest] 7 - 05/13/2024, 11:35:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:33 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:45 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:58 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:11 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:53 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:06 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:34 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:47 AM LOG [EventRepository] Initialized websocket server
I'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know.
On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts.
Thank you very much!
from immich.
I can also confirm this issue is affecting the installation on my NAS. Limiting memory causes frequent restarts of the microservices container. I also tried setting the Generate Thumbnails Concurrency to 1, but it didn't help. I found the NAS started overutalising the drives as they were at 100% utilisation. I imagine it was paging out memory to disk; CPU utilization was around 40%. I have increased the RAM in the NAS to 32GB, which has solved the problem for me, and the microservices container now runs constantly, with no restarting at all. The microservices container stabilised at around 28GB, the NAS memory resource monitor shows almost all of that usage is cache memory, the container limit is set to 4GB.
After restarting the container, it takes about 15-20min to stabilise at this level. Disk usage is now around 40% and CPU is around 80%, generating thumbnails at around 30/min. This is with no other jobs running.
I have around 230k assets in the external library, 50%jpg, 50%RAW. I also tried this in docker on a server (TrueNAS scale) and experienced the same issue and outcome.
Overall, I love the way Immich is going. It is going to be a great product. Well done to everyone involved, and thanks for the great work.
from immich.
How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each.
Do you have /tmp configured to be in-memory? No /tmp configured.
Is the increase completely linear, or are there spikes? Mostly linear overlayed with a small saw tooth shape.
I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth.
Any errors in the logs? No Errors, running smoothly.
from immich.
I made a test image for microservices with a possible fix: ghcr.io/immich-app/immich-server:pr-9665
. Would you be able to change your image to that and see if it affects RAM usage?
from immich.
Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-).
from immich.
So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that.
You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.)
from immich.
How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory.
from immich.
Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective.
from immich.
Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory.
from immich.
Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error?
As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage.
from immich.
I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs -
from immich.
Thanks for the detailed info!
- How many of those assets are RAW, if any?
- Do you have
/tmp
configured to be in-memory? - Is the increase completely linear, or are there spikes?
- Any errors in the logs?
from immich.
Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container.
from immich.
Not good news I am afraid. The immich_microservices fails.
immich_microservices
date stream content
2024/05/23 01:09:52 stderr Microservices worker exited with code 1
2024/05/23 01:09:52 stderr }
2024/05/23 01:09:52 stderr routine: 'parserOpenTable'
2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,
2024/05/23 01:09:52 stderr },
2024/05/23 01:09:52 stderr routine: 'parserOpenTable'
2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,
from immich.
Hmm, that error is about connecting to Postgres, not related to thumbnail generation.
from immich.
Related Issues (20)
- Upload via Domain -> Cloudflare is crashing HOT 1
- [BUG] Timeline endpoint returns the wrong video duration HOT 3
- IOS App - Downloading Image from Immich that is marked as Favourite, does not reflect in Apple Photos
- Issue with navbar in Android HOT 1
- I would like to try translating immich's web application into Chinese HOT 5
- Photos from external library do not show in iOS app HOT 5
- Auto Backup Logic bug
- Search by camera looks bad in Android
- Android app freezing after launch HOT 7
- Moving a picture in the trash doesn't change the photo album's or location album's thumbnail. HOT 1
- iOS app stops being able to connect to server on one device but not the other (and version discrepancy?) HOT 5
- 【BUG】The video is completely gray after transcoding HOT 3
- [BUG] Interrupted mobile uploads leave corrupt files HOT 9
- External Library - Permissions Denied on NFS shares, and mkdir on existing folders
- Transcoding gets blocked on certain videos creating multiple instances of ffmpeg running HOT 10
- iOS app showing what looks like an XML file on "advanced info" instead of nicely formatted information HOT 2
- Slideshow has problems loading images after running for a while HOT 1
- Some photos are treated as motion photos, even though they don't appear to be such
- Not everything is uploaded from the Android App HOT 2
- App Android and IOS: Deleted pictures are still shown in full view after deletion swiping left HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from immich.