evan-buss / openbooks

Search and Download eBooks

openbooks

NOTE: Going forward only the latest release will be supported. If you encounter any issues, be sure you are using the latest version.

Openbooks allows you to download ebooks from irc.irchighway.net quickly and easily.

Getting Started

Binary

1. Download the latest release for your platform from the releases page.
2. Run the binary
   • Linux users may have to run chmod +x [binary name] to make it executable
3. ./openbooks --help
   • This will display all possible configuration values and introduce the two modes; CLI or Server.

Docker

• Basic config
  • docker run -p 8080:80 evanbuss/openbooks
• Config to persist all eBook files to disk
  • docker run -p 8080:80 -v /home/evan/Downloads/openbooks:/books evanbuss/openbooks --persist

Setting the Base Path

OpenBooks server doesn't have to be hosted at the root of your webserver. The basepath value allows you to host it behind a reverse proxy. The base path value must have opening and closing forward slashes (default "/").

• Docker
  • docker run -p 8080:80 -e BASE_PATH=/openbooks/ evanbuss/openbooks
• Binary
  • ./openbooks server --basepath /openbooks/

Usage

For a complete list of features use the --help flags on all subcommands. For example openbooks cli --help or openbooks cli download --help. There are two modes; Server or CLI. In CLI mode you interact and download books through a terminal interface. In server mode the application runs as a web application that you can visit in your browser.

Double clicking the executable will open the UI in your browser. In the future it may use webviews to provide a "native-like" desktop application.

Development

Install the dependencies

• go get
• cd server/app && npm install
• cd ../..
• go run main.go

Build the React SPA and compile binaries for multiple platforms.

• Run ./build.sh
• This will install npm packages, build the React app, and compile the executable.

Build the go binary (if you haven't changed the frontend)

• go build

Mock Development Server

• The mock server allows you to debug responses and requests to simplified IRC / DCC servers that mimic the responses received from IRC Highway.

• cd cmd/mock_server
  go run .
  # Another Terminal
  cd cmd/openbooks
  go run . server --server localhost --log

Desktop App

Compile OpenBooks with experimental webview support:

cd cmd/openbooks
go build -tags webview

Why / How

• I wrote this as an easier way to search and download books from irchighway.net. It handles all the extraction and data processing for you. You just have to click the book you want. Hopefully you find it much easier than the IRC interface.
• It was also interesting to learn how the IRC and DCC protocols work and write custom implementations.

Technology

• Backend
  • Golang
  • Chi
  • gorilla/websocket
  • Archiver (extract files from various archive formats)
• Frontend
  • React.js
  • TypeScript
  • Redux / Redux Toolkit
  • Mantine UI / @emotion/react
  • Framer Motion

openbooks's Issues

Option to disable downloading to the browser

Ive got a directory set up that books get downloaded into for ingestion into calibre, but when I download a book it downloads to both my browser and to my device. One potential solution could be that 'if a folder is defined, don't download to device' or an ENV variable able to be set to 'disable direct downloads'.

Language implementation

Hello,
I like your application, but I would like to be able to read in other languages than English (French in my case)
Would it be possible to implement such a feature?

Change it so it still passes lines with no ::INFO:: but show the file size as "Unknown" or similar

Missing LICENSE

I see you have no LICENSE file for this project. The default is copyright.

I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

Integrate with GoodReads Lists

While Readarr team mentioned integration with IRC servers is probably never going to happen, I believe this tool has the potential to become a Readarr replacement.

It is currently missing the ability to track pre-configured Goodreads lists for missing books.

Adding a local database similar to Readarr and the ability to track the various lists Readarr does would be an amazing feature set.

Progress bar?

Can you add a progress bar for when the server is downloading the book on the web app?

FWServer input needs parsing

See author column for example

Persist books with user (not root) permissions

Hey, I'm running the latest docker install of Openbooks. I absolutely love it, thanks for your work. One issue I run into is, when I download books, they are stored in the persist directory with root permissions. This stops Calibre from running Auto-import till I change the permission of the downloaded books, into my user permissions. I do not run the docker container as root, but instead as my specified docker user.

Is there any way I can define the owner / permissions of the books being persisted?

Thanks!

Obfuscated?

Closed

Fix for permanent web app

I've been using this as a webapp hosted on https://books.gabba.ga/

The changes I make were the following:
Auto-selected wss or ws based on html protocol and was window.location.host to find address
IRC seems to time out after a while so I implemented a timer that closes irc connection if no client was connected for 10 minutes.

I'll make a pull request if I figure out how to...
I'm trying currently to recreate this in node js, like you I don't want to use an irc library but rather make my own.
The advantage of making this solely in n-js would be that I can have many concurrent people using the app and the irc connection would be "hosted" locally on their computers

Option to change @search bot

Sometimes the @search bot is offline, but alternative search bots are present in the channel.

Please add an option to manually change the "search" bot.

Optional browser UI to control OpenBooks

Enable a flag to switch between terminal and browser mode.

• Write a front-end SPA that allows users to search and download files
  • This would provide a cross platform UI that allows users to search through search results without opening the file manually

Request - Option to download files to a watch folder.

Is it possible to download the files to a watch folder for automatic import

any plans on making a similiar app and/or adding other media ability ie movies etc

Add CLI option to customise Whois IRC string

Integration with Calibre Server API

Integrate with Calibre Server API so downloading a book will add it to a pre-configured Calibre server.

donate?

How to donate to this project?

arm support

Would it be possible to add support for arm,
I want to host this on my raspberry pi

Using CLI results in SIGSEGV

Unable to download

We can do searching but we are unable to download the file . Before MrAtoz one was very useful to download now none of the link is working . please fix this issue

Here is the screenshot where the script is running from 30 min still it not downloaded.

"Server connection closed"

When opening the page for openbooks, I get a "Welcome, connection established".
Sometimes I manage to use it for 20 seconds, make one search, or even successfully download a file.
However, the service soon stops working. The little bell for notifications turns from green to gray, and I get the following error when trying to do anything:
"Server connection closed. Reload page".
If I reload the page, I get to enjoy it for 20 more seconds or so.
Maybe the IRC server, or some bots in the channels, are recognizing openbooks and automatically kicking me out?
I have no visibility on what happens in the background, but that's the feeling I get.
Any idea how to avoid this behavior?

Thanks a lot for your great software

Issues when building

1. In the build.sh you do "cd server/app" and this breaks the whole script as you named it "app_old"
2. packr2 is not up anymore in the path you direc the user to - tell them to instead download the binary from https://github.com/gobuffalo/packr/releases/ and place in $GOBIN
3. tsconfig.app.json config overrides tsconfig.json and results in an error to fix I remove "types": [] from .app.json
4. To access webserver over wan one needs to change ws address in app/src/app.jsx

THIS IS AM AMAZING PROJECT BTW - WELL DONE!

panic: runtime error: invalid memory address or nil pointer dereference

2022/05/22 00:08:00 CLIENT (extreme_direction_1): invalid dcc send string
2022/05/22 00:59:11 CLIENT (extreme_direction_1): Connection Closed: websocket: close 1001 (going away)
2022/05/22 22:02:42 SERVER: Client connected from 127.0.0.1:34978
2022/05/22 22:02:42 CLIENT (extreme_direction_1): New client created.
2022/05/22 22:02:42 CLIENT (extreme_direction_1): Connection Closed: websocket: close 1001 (going away)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x7d58ec]

goroutine 164 [running]:
github.com/evan-buss/openbooks/irc.(*Conn).Disconnect(0xc00031c280)
	/home/runner/work/openbooks/openbooks/irc/irc.go:46 +0x4c
github.com/evan-buss/openbooks/server.(*server).readPump.func1()
	/home/runner/work/openbooks/openbooks/server/client.go:63 +0x2f
github.com/evan-buss/openbooks/server.(*server).readPump(0xc00006eb40, 0xc00031c300)
	/home/runner/work/openbooks/openbooks/server/client.go:80 +0x28c
created by github.com/evan-buss/openbooks/server.(*server).serveWs.func1
	/home/runner/work/openbooks/openbooks/server/routes.go:90 +0x8c5

$ ./openbooks_linux --version
openbooks version 4.3.0

Update "Publish Release Binaries" action.

• The current action uses "actions/create-release@v1" and "actions/upload-release-asset@v1" which are now unmaintained.
• There are more elegant alternatives like https://github.com/softprops/action-gh-release which allow creating the release and uploading release files in a single action. They also allow specify glob patterns so we don't need to list every single file in yml.

Web UI not working

Hi, from version 3.3.1 and onwards when I run the server through the cmd the browser only shows a blank page. On the contrary when I try to run any version before 3.3.1 the browser shows the expected web ui. Am i doing something wrong?

Blank main panel

Following an alert from diun, I updated my openbooks docker just now.

I now have an empty main pain where the search should be (in dev tools it is empty too...).
If I minimise the side bar, I then have a totally blank screen...

Tested in both Firefox and Chrome.

'Version' inquiries from IRC servers/client receive silence

Issue #52 was closed before my reply was read apparently. The 'version' reply doesn't have to have a version number but it needs to reply with -something- other than silence. If it looks like a bot and acts like a bot it will be treated as a bot.

Reset table filters on new search

The table filters for each column should be reset when the user enters a new search query. Otherwise they will see an empty table and not know why.

Base URL

Is there a way to have it listen on http://ip:port/baseurl instead of http://ip:port for putting behind a reverse proxy?

Empty files and Previous downloads is always Book persistence disabled

I am using the last section docker compose from https://github.com/evan-buss/openbooks/blob/master/docker.md

It will search, it will download a book correctly to the persistent location

but...

the browser download, results in an empty file with .txt attach to it. Not the file downloaded to the persistent location
secondly the Previous downloads always shows Book persistence disabled.

How to fix this, is there a hidden option, argument?

ARM docker image

Any chance of a docker arm image too? right now the docker is amd64 only. Thanks

Donation Opportunity?

@evan-buss I'd like to donate a few bucks to this project on behalf of a friend who appreciates it and had a recent loss. Could I paypal your gmail account or something?

Unable to parse authors

Unable to parse authors.

Example:

2021-12-29 13:28:52 | 2021/12/29 12:28:52 CLIENT (total-pressure-1): Error: unable to parse author. Line: !MrAtoz Barry -- Witchcraft and Demonology in South-West England, 1640–1789 [2012 #retail] «45E2E70B».pdf.
-- | --
  |   | 2021-12-29 13:28:52 | 2021/12/29 12:28:52 CLIENT (total-pressure-1): Error: unable to parse author. Line: !MrAtoz Barry -- Raising Spirits- How a Conjuror’s Tale Was Transmitted Across the Enlightenment [2013 #retail] «EA499429».pdf.

Frontend shows:

can't read books titles

ipad

mac

suggestion: hamburger menu and smaller author space

Please contact fruitloops at #ebooks

Please contact fruitloops in #ebooks regarding your script and its banning and how it can be coded in such a way that it is acceptable for us and users.

Expected a JavaScript module script but the server responded with a MIME type of "text/plain".

The command I use it just normal openbooks server

This is what the browser devtools console error came out with a blank white screen.

Nothing happens after searching

I get the connected to server message and I do get a nick which appears at the bottom left.

However on searching anything, I get a "search request sent" and then nothing happens. The search term appears on the left side of the screen with a spinner and nothing else happens. It just stays like that forever. No errors or results.

I am using the latest docker image of openbooks in a docker compose file like the example given in docker hub with the --persist command. The following is all I see in the logs:

openbooks  | 2022/02/13 21:15:43 SERVER: Base Path: /
openbooks  | 2022/02/13 21:15:43 SERVER: OpenBooks is listening on port 80
openbooks  | 2022/02/13 21:16:41 SERVER: Client connected from 172.18.0.2:37748
openbooks  | 2022/02/13 21:16:41 CLIENT (official_paul_1): New client created.
openbooks  | 2022/02/13 21:16:41 CLIENT (official_paul_1): CONNECT Message Received
openbooks  | 2022/02/13 21:16:53 CLIENT (official_paul_1): SEARCH Message Received

Please help

Add port customization feature

Allow user to specify a port to connect via the browser interface

• Just realized I can launch the browser from Go and then in React get the address with window.location.href. With this the websocket won't have to be hardcoded.
• This should also allow a single computer on the network to run openbooks and it should be accessible from other PC's since the websocket will be connecting to the proper IP instead of localhost:8080/ws

Calibre Integration?

Would integration into an existing Calibre Content Server be possible as it is with Readarr?

Still reporting 4.3.0 on IRC Whois

IRC Whois is still reported as 4.3.0, although the software version changed.

Add options that allow commands and searches to be directly passed into the CLI

Example:

openbooks cli --search "Dan Brown Inferno"
openbooks cli --get "!horla Inferno - Dan Brown.epub"

Implement local fuzzy search

One of the main ways of abusing the IRCHighway is by sending too many search requests. And although each user is limited to 10+ seconds between search requests, an increase in the usage of this tool will eventually again overload the search bot, because that is a single point of failure.

The architecture of IRCHighway with a single point of failure should not be a limiting factor for this amazing application.

Hence, caching locally the lists of running servers and implementing a local fuzzy search will distribute the load while significantly improving the ability of this application to serve requests faster.

Learn You a Haskell fails to parse

It appears the entry for the search results contains either unicode characters and also doesn't seem to contain author info. Perhaps rather than erroring out, a line could be displayed to retrieve the entry anyways?

Possible to set a save destination

Would be nice to point this to a directory — looking to have calibre monitor that directory and automatically import.

Web UI issues

Sometimes due to long name structure, the title gets hidden and so we cant identify which title the book is. Can we also put horizontal scroll along with vertical so that this problem can be removed?

Scroll Fixes

• After searching for a book the results grid makes the entire page scroll. The sidebar has a fixed height and doesn't extend to the bottom of the screen.
• Make the sidebar and the center content scroll independently.

Listen On A Port Other Than 80

This may be an edge case, but I run containers through a VPN container. To do that I have to add the port to the VPN container. I believe however the VPN container is already using port 80 internally.

example

services:
  gluetun:
    image: qmcgaw/gluetun
    restart: unless-stopped
    container_name: gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - 8888:8888/tcp # HTTP proxy
      - 8388:8388/tcp # Shadowsocks
      - 8388:8388/udp # Shadowsocks
      # Port Mapping
      - 10036:80 # Openbooks
    volumes:
      - gluetun:/gluetun
    environment:
      # See https://github.com/qdm12/gluetun/wiki
      VPN_SERVICE_PROVIDER: redacted
      VPN_TYPE: openvpn
      OPENVPN_USER: redacted
      OPENVPN_PASSWORD: redacted
      SERVER_CITIES: redacted
      DOT: "off"
      OPENVPN_CIPHER: aes-128-cbc
      TZ: redacted
    dns:
      - redacted
     
  openbooks:
    container_name: openbooks
    image: evanbuss/openbooks:latest
    # environment:
      #  BASE_PATH: /openbooks/
    volumes:
      - books:/books/books
    # ports:
    #   - '10036:80'
    command: --persist
    restart: unless-stopped
    network_mode: "service:gluetun"
    depends_on:
      - gluetun

Don't `mkdir books`

Is it possible not to run mkdir books? I could be completely off base, but generally when I mount a volume to a path I'm expecting the data to be in that path. Making another books directory in /books seems redundant. This is not a huge deal though, just a thought.

My Compose:

version: "3.9"

volumes:
  books:
    driver_opts:
      type: "nfs"
      o: "addr=${NAS_IP},nolock,soft,rw"
      device: ":${NAS_MEDIA_PATH}/Literature/Books"

services:
  openbooks:
    container_name: openBooks
    image: evanbuss/openbooks:latest
    environment:
      - BASE_PATH=/openbooks/
    volumes:
      - 'books:/books'
    ports:
      - '17012:80'
    restart: unless-stopped
    command: --persist

[FR] Ability to Download to servers HDD instead of devices

It would be an amazing addition if instead of downloading the result to the device you are viewing the webpage on (a phone for instance), if there was the ability to save it to the attached volume ie the 'books' directory which could be elsewhere on a server.

The Docker compose example shows the ability to map a volume to the container as 'books' for example, but as far as I can see it remains unused?

services:
    openbooks:
        ports:
            - '8080:80'
        volumes:
            - 'booksVolume:/books'
        restart: unless-stopped
        container_name: OpenBooks
        command: --persist
        environment:
          - BASE_PATH=/openbooks/
        image: evanbuss/openbooks:latest

volumes:
    booksVolume:

Could you port these features please? :)

I made a tool using your backend which allows for fully automated downloading needing only a search term and it does the rest of the decisions for you. Openbooks-Downloader

I want to port this to a user friendly gui like you have done with this program but I am not good at making any programs with a gui, so I was wondering if you could make a take on my program(in your preferred language) with a nice gui like you have done for openbooks?

If you are willing to give this a shot im thinking that it could look like the query response for when you search for a book in google books except this includes a download button.