iiab / calibre-web Goto Github PK

Describe the bug/problem

Calibre-Web currently has apscheduler for managing various tasks, including format conversions, uploads, and metadata-related activities. While the present job queuing system functions adequately, there's a need to further optimize and adapt this mechanism specifically for download tasks.

The existing job queuing mechanism, for its general functionality and reliability, can be retained and optimized to better accommodate and efficiently manage download-related tasks within the Calibre-Web environment per feature #7.

The following changes are proposed:

1. Fine-tuning for Downloads: Optimize the current job queue to better handle and prioritize download-related tasks.
2. Load Distribution: Explore methods to evenly distribute and manage the load when handling concurrent or multiple downloads.
3. Error Handling and Monitoring: Enhance error handling and monitoring within the job queuing system to ensure the smooth execution of download tasks.

To Reproduce

There is no specific bug to reproduce; this issue pertains to code consistency and best practices within the Calibre-Web project.

Expected behavior

By optimizing the current job queuing mechanism specifically for download tasks, the goal is to ensure smoother and more efficient download operations while maintaining the reliability and stability of the existing system.

Environment (please complete the following information):

• OS: Ubuntu 23.10
• Python version: 3.11.6
• Calibre-Web version: 0.6.21 - Romesa (customized per #7 )
• Docker container: No
• Special Hardware: No
• Browser: Firefox Nightly 121.0a1 (2023-10-31) (64-bit)

Additional Context

The intention is to build upon the strengths of the current job queuing system, ensuring it's well-suited for download-related tasks without requiring a complete overhaul. This approach aims to maintain compatibility and stability while improving the efficiency of download task management within Calibre-Web.

Problem Statement:
Calibre-Web is a robust platform designed for managing e-books efficiently. However, users often find the need to organize their personal multimedia content (videos, photos, audios) alongside their digital book collection. Presently, there's no seamless provision to integrate and manage these diverse media types within Calibre-Web.

Proposed Solution:
This expansion to support multimedia content is crucial. By incorporating support for various media formats (videos, photos, audios), users can effortlessly upload, organize, and manage their diverse digital media collections. Ensuring metadata persistence in the database will allow for easy categorization and searching. Additionally, implementing a feature to download videos from popular platforms directly into Calibre-Web would significantly enhance its utility.

Detailed Feature Specifications:

1. Multimedia Upload Support: Enable the uploading of various media formats supported by major browsers (e.g., MP4, WEBM for videos; JPEG, PNG for images; MP3, OGG and WAV for audios).
2. Metadata Handling: Implement a system to persist metadata associated with each uploaded media file (e.g., title, description, author, date, etc.).
3. Integration with Online Platforms: Provide an option to download videos directly from popular platforms (such as YouTube, Vimeo) for seamless integration and management.
   

Alternatives Considered:

• Custom Plugin Development: While a custom plugin could address this need, maintaining it separately might introduce complexities. Integrating these features directly within Calibre-Web would provide a more seamless and integrated solution for users.

Describe the bug/problem

The current implementation in editbooks.py utilizes direct imports from the subprocess module, which diverges from the consistent practice within the Calibre-Web codebase. To ensure uniformity and maintain code standardization, the file should use the subproc_wrapper.py as an interface for interacting with subprocesses, for #6.

To Reproduce

There is no specific bug to reproduce; this issue pertains to code consistency and best practices within the Calibre-Web project.

Expected behavior

Refactoring the code in editbooks.py to utilize the subproc_wrapper.py should lead to seamless subprocess operations in line with the established code standards.

Environment (please complete the following information):

• OS: Ubuntu 23.10
• Python version: 3.11.6
• Calibre-Web version: 0.6.21 - Romesa (customized per #7 )
• Docker container: No
• Special Hardware: No
• Browser: Firefox Nightly 121.0a1 (2023-10-31) (64-bit)

Additional context

It's recommended to review the code changes thoroughly, ensuring that the substitution from direct subprocess imports to the subproc_wrapper.py maintains or improves the functionality without introducing any regressions. The modification should promote uniformity in the codebase and simplify future maintenance tasks related to subprocess interactions.

@deldesir FYI attempting https://www.youtube.com/shorts/Mt187r0Ec9A returned "Invalid YouTube URL"

So I tried bare URL "youtu.be/Mt187r0Ec9A" (which works in a browser).

FWIW it failed with tail -f /var/log/calibre-log showing the following:

[2023-10-12 20:54:34,980] ERROR {cps.editbooks:338} Subprocess error (return code 1): WARNING: [generic] The url doesn't specify the protocol, trying with http
Importing playlist-less media youtu.be/Mt187r0Ec9A
, --------YB LOG--------\n\n
2023-10-12 20:54:30 - Moving /var/tmp/download.db aside if it already exists.
--------YB LOG--------\n\n
2023-10-12 20:54:30 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A

--------YB LOG--------\n\n
2023-10-12 20:54:34 - An error occurred while running the command. Please check /var/log/yb.log  for more details.

[2023-10-12 20:54:34,983]  INFO {cps.editbooks:364} Subprocess result: (False, "Subprocess error (return code 1): WARNING: [generic] The url doesn't specify the protocol, trying with http\nImporting playlist-less media youtu.be/Mt187r0Ec9A\n, --------YB LOG--------\\n\\n\n2023-10-12 20:54:30 - Moving /var/tmp/download.db aside if it already exists.\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:30 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A\n\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - An error occurred while running the command. Please check /var/log/yb.log  for more details.\n", '--------YB LOG--------\\n\\n\n2023-10-12 20:54:30 - Moving /var/tmp/download.db aside if it already exists.\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:30 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A\n\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - An error occurred while running the command. Please check /var/log/yb.log  for more details.\n', "WARNING: [generic] The url doesn't specify the protocol, trying with http\nImporting playlist-less media youtu.be/Mt187r0Ec9A\n")
[2023-10-12 20:54:37,994] ERROR {cps.editbooks:338} Subprocess error (return code 1): WARNING: [generic] The url doesn't specify the protocol, trying with http
Importing playlist-less media youtu.be/Mt187r0Ec9A
, --------YB LOG--------\n\n
2023-10-12 20:54:34 - Moving /var/tmp/download.db aside if it already exists.
--------YB LOG--------\n\n
2023-10-12 20:54:34 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A

--------YB LOG--------\n\n
2023-10-12 20:54:37 - An error occurred while running the command. Please check /var/log/yb.log  for more details.

[2023-10-12 20:54:37,994]  INFO {cps.editbooks:364} Subprocess result: (False, "Subprocess error (return code 1): WARNING: [generic] The url doesn't specify the protocol, trying with http\nImporting playlist-less media youtu.be/Mt187r0Ec9A\n, --------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - Moving /var/tmp/download.db aside if it already exists.\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A\n\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:37 - An error occurred while running the command. Please check /var/log/yb.log  for more details.\n", '--------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - Moving /var/tmp/download.db aside if it already exists.\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:34 - Running command: lb tubeadd /var/tmp/download.db youtu.be/Mt187r0Ec9A && lb dl /var/tmp/download.db --video youtu.be/Mt187r0Ec9A\n\n--------YB LOG--------\\n\\n\n2023-10-12 20:54:37 - An error occurred while running the command. Please check /var/log/yb.log  for more details.\n', "WARNING: [generic] The url doesn't specify the protocol, trying with http\nImporting playlist-less media youtu.be/Mt187r0Ec9A\n")

# cat /var/log/yb.log
No media found
No media found

# lb --version
2.2.112

Describe the Problem

Currently, in Calibre-Web, the creation and population of bookshelves are manual processes, dependent on user decisions after each upload/download. This manual approach can become time-consuming and less user-friendly, especially when multiple downloads occur.

The proposal is to automate the process of creating and populating bookshelves upon the occurrence of multiple downloads. This automation should streamline the organization of downloaded content and enhance user experience by reducing manual efforts.

Proposed Changes

1. Automated Bookshelf Creation: Implement a feature to automatically create bookshelves based on specified criteria (e.g., playlist title, custom name) for downloaded content.
2. Populating Bookshelves: Upon download completion, the system will automatically organize and populate the appropriate bookshelves based on predefined rules or user settings.
3. Customizable Automation: Provide users with the flexibility to customize the automation settings, allowing for personalized organization rules and preferences. This should be defaulted to available collection title. The current frontend code for the download modal dialog is misleading. It must be removed or optimized for custom naming/renaming of bookshelves.
4. User Notification: Notify users about the automated creation and population of bookshelves, ensuring transparency and awareness of the organization process.

Expected Outcomes

By automating the creation and population of bookshelves upon multiple downloads, the goal is to significantly reduce manual efforts for users, improving the organization and accessibility of downloaded content within Calibre-Web.

Additional Context

The aim of this proposed change is to enhance the user experience by automating the organization process without compromising user control. This enhancement seeks to minimize the burden on users in organizing downloaded content while allowing for customization and user-defined parameters for automation.