h2owater425 / node-hitomi

Hitomi.la api for Node.js

H2Owater425 💻🌊

👋 Hello, stranger!

Let me introduce myself, shall I?

I'm H2Owater425, also known as KKM which is my initial name.

Currently, I'm a sophomore in the department of web programming at Korea Digital Media High School. 👨‍🎓👨‍💻

But for you guys who have visited, I have both bad and good news. 😮‍💨

The bad news is that I'm very suck poor at programming, and the good news is the fact that I'll keep growing my skills!

Contacts 📫

• Github Repository Issue
• Discord: 물워터#7826
• Twitter: @H2Owater425
• Email: [email protected], [email protected]

node-hitomi's Issues

Add order by popularity feature to getGalleryIdList

Apply popularized code style

Adding the type of the object at the name of the function or the property does not seem to be a good idea for now, so I would like to suggest changing the name of them (to future me)

Examples:

• getIdList => getIds,
• getTagList => getTags,
• Gallery['artistList'] => Gallery['artists'],
• Gallery['groupList'] => Gallery['groups']

getImageUrl is throwing 403

Any chance of getting this fixed again or is not worth the time? :(

Stop people from downloading source folder

HitomiError [REQEUST_REJECTED]: Request to 'https://ltn.hitomi.la/gg.js' was rejected

Description of bug

Error was thrown, although I quite don't know which function caused this.
Using proper catch didn't worked.

Expected behavior

The error can be catch on .catch
And, should be obvious, but this error should never be thrown, whatever the cause is.

Environment

• OS: Windows 10
• Version
  • NodeJS: 16.17.0
  • Package: 7.0.5

Additional context

Error:

HitomiError [REQEUST_REJECTED]: Request to 'https://ltn.hitomi.la/gg.js' was rejected
    at ClientRequest.<anonymous> (<omitted>\node_modules\node-hitomi\library\index.js:1:2451)
    at ClientRequest.emit (node:events:527:28)
    at TLSSocket.socketErrorListener (node:_http_client:454:9)
    at TLSSocket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'REQEUST_REJECTED'
}

Code (although I'm not sure which function caused this, these are two functions that I've used with):

// ...
var getImageUrl = new hitomi.ImageUrlResolver();
// ...

const downloadHitomi: DownloadMethod = function downloadHitomi(file, path) {
    return new Promise<string>(async (resolve, reject) => {
        try {
            const URL = <string>await getImageUrl(<hitomiType.Image>file.hitomiImage, 'webp', {}).catch(err => { return reject(err) });
            // ...
        }
        catch (err) { reject(err) }
    })
};

const getHitomi: GetMethod = function getHitomi(url) {
    return new Promise(async (resolve, reject) => {
        // ...
        const metadata: hitomiType.Gallery = await hitomi.getGallery(parseInt(id!), { includeFiles: true }).catch(err => { return reject(err) });
        // ...
    })
}

hitomi.ImageUriResolver is not defined @8.0.0

const hitomi = require('node-hitomi').default;

PS C:\Download\hitomi> node index.js
[class (anonymous)] // console.log(hitomi.ImageUrlResolver); @7.0.6

PS C:\Download\hitomi> node index.js
undefined // console.log(hitomi.ImageUrlResolver); @8.0.0

ImageUriResolver subdomain mismatch

Description of bug

ImageUriResolver.getImageUri subdomain mismatch

Expected behavior

Results right subdomain

Screenshots

Environment

• OS: Windows 10
• Version
  • NodeJS: 20.9.0
  • Package: 8.0.1

Additional context

Merge getIds and getQuriedIds

They do similar things, why don't we combine their features into one function

getImageUrl sub-domain mismatch

I ran another test again using this module. This time I tried using getImageUrl which should return the correct url for the file / image.

I don't know the logic behind this function, nor I know the logic behind hitomi's sub domain system. So I doubt I'll be able to give out details for this issue. But here's what I've tried.

import { getGalleryData, getImageUrl } from 'node-hitomi'
import axios from 'axios'

async function getImage(link: string): Promise<Buffer> {
  return new Promise(async (res, rej) => {
    await axios.get(link, {
      headers: { referer: "https://hitomi.la" },
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      responseType: "arraybuffer"
    })
      .then((r) => { res(r.data) })
      .catch(err => rej(err))
  })
}

getGalleryData(1954101)
  .then(async (gallery) => {
    for (let i = 0; i < gallery.files.length; i++) {
      var url = getImageUrl(gallery.files[i], 'jpg'), valid = false;
      await getImage(url).then(buffer => { /* ignore buffer for now */ valid = true }).catch(err => { valid = false })
      console.log(`page ${i}: ${url} (valid: ${valid})`);
    }
  })

Here's the result

Notice that valid with value as true means the url is correct and getImage function were able to fetch image, while false means the url were incorrect and site returning 404 not found, thus axios rejecting the promise.

This happen on all gallery I've tried. So I get curious and directly check the image from browser, and found out it's the sub-domain mismatch

module version used: v 2.0.0

getTags with artist(and group, series, character), return blank array

Description of bug

getTags with

• artist
• group
• series
• character

return blank array.
other types (ex. female, language...) are OK.

const hitomi = require("node-hitomi").default;

hitomi.getTags("female", { startWith: "z" }).then((value) => {
  console.log(value); // OK, 2 tags are returned.
});

// getTags with artist(and group, series, character), return blank array.
hitomi.getTags("artist", { startWith: "z" }).then((value) => {
  console.log(value); // NG, blank array.

  // getTagUrl return valid URL: https://ltn.hitomi.la/allartists-z.html
  console.log(hitomi.getTagUrl("artist", { startWith: "z" }));
});

getTagUrl return valid URL, so I guess reason of this result is this line,

node-hitomi/source/index.ts

Line 684 in cdaff69

const matchedNames: string[] = buffer.toString('utf8').match(RegExp(type === 'language' ? '(?<=")(?!all)[a-z]+(?=":)' : '(?<=\/tag\/' + (type === 'male' || type === 'female' ? type + '%3A' : '') + ')[a-z0-9%]+(?=-all\\.html)', 'g')) || [];

but I can't read these regular expression correctly...

Expected behavior

return valid tags array.

Screenshots

Environment

• OS: macOS Monterey 12.3.1
• Version
  • NodeJS: 16.14.2
  • Package: 7.0.3

Add range option to getGalleryData

Use reject function of promise to avoid UnhandledPromiseRejection

When use 'getIds' or 'getQueriedIds', throw REQUEST_REJECTED

Environment

• OS: Windows 10 21H1 build 19043.1466
• Nodejs: v16.13.2
• Npm: 8.3.1

Code

const page = parseInt(req.query.page) || 1;
if(page == 1) {
    const result = await hitomi.getIds({
        startIndex: 0,
        endIndex: 9
    })
    res.json(result);
}else{
    const result = await hitomi.getIds({
        startIndex: (page - 1) * 10,
        endIndex: (page - 1) * 10 + 9
    })
    res.json(result);
}

const result = await hitomi.getQueriedIds([{"type": "female","name": "sole_female"}]);
res.json(result);

Error message

// hitomi.getIds(...)
HitomiError [REQEUST_REJECTED]: Request to 'https://ltn.hitomi.la/index-all.nozomi' was rejected
    at ClientRequest.<anonymous> ([project_dir]\node_modules\node-hitomi\library\index.js:1:5326)

// hitomi.getQueriedIds(...)
HitomiError [REQEUST_REJECTED]: Request to 'https://ltn.hitomi.la/n/tag/female:sole female-all.nozomi' was rejected
    at ClientRequest.<anonymous> ([project_dir]\node_modules\node-hitomi\library\index.js:1:5326)

I try to connect above url to web browser and it show '403 Forbidden'.
And i try to connect same url with Referer header, it's working.
So i think requesting to 'ltn.hitomi.la/*.nozomi' required referer header set as 'https://hitomi.la'.

(Sorry to my dirt English. I'm student in Korea, so i not good at English)

Use tester to catch error before releasing new release

Hitomi only supports extension of webp, avif, jxl and mp4 only from now on

They have been gradually changing their image url system from the thumbnail, and eventually they dropped support for the extensions except for webp, avif, and mp4.

Add Support for "cg" type of galllery

An error occured when trying to download cg gallery, ex url: https://hitomi.la/cg/...

(node:6408) UnhandledPromiseRejectionWarning: HitomiError [REQEUST_REJECTED]: Request to 'https://hitomi.la/artistcg/redacted-xxx.html' was rejected
    at ClientRequest.<anonymous> (/Users/stillmen/Documents/PROJEKT/rej-hitomi/node_modules/node-hitomi/library/index.js:1:1472)
    at Object.onceWrapper (events.js:422:26)
    at ClientRequest.emit (events.js:315:20)
    at HTTPParser.parserOnIncomingClient (_http_client.js:641:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
    at TLSSocket.socketOnData (_http_client.js:509:22)
    at TLSSocket.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
(Use node --trace-warnings ... to show where the warning was created)
(node:6408) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:6408) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

As you can see, the request url changed from cg to artistcg
Thank you.

Edited
My code:

const hitomi = require('node-hitomi').default;

hitomi.getGallery(2006297)
.then(function (value) {
    console.log(value['title']['display'], value['id']);
});

Make document

Make wiki or something

getGalleryData TypeError: Cannot read property 'push' of undefined

I ran a simple test with this module with several id to try.

Some ids ran fine with getGalleryData, but 1316592 did not end well.

import { getGalleryData } from 'node-hitomi'
getGalleryData(1316592).then(console.log);

Error result:

(node:16524) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'push' of undefined
    at <anonymous>\node_modules\node-hitomi\library\index.js:1:2287
    at Array.forEach (<anonymous>)
    at <anonymous>\node_modules\node-hitomi\library\index.js:1:2241
    at Array.forEach (<anonymous>)
    at <anonymous>\node_modules\node-hitomi\library\index.js:1:2127
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:16524) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:16524) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

403 Error

Description of bug

When I request the image link it sends it to me but when I click on it I get the Nginx 403 error. When I open the doujin before the image it works but suddenly I can't download the image.
It gives me a 403 error.

Code

doujin.js

const hitomi = require('node-hitomi').default;
const { downloadImage } = require('../../utils/download');
const ids = await hitomi.getIds({
    tags: [
        {
            type: "type",
            name: "doujinshi"
        }, {
            type: "female",
            name: "handjob"
        }
    ] 
})
const gallery = await hitomi.getGallery(ids[0], { includeFiles: true });
const IUR = new hitomi.ImageUrlResolver();
await IUR.synchronize()
const url = IUR.getImageUrl(gallery.files[1], 'avif')
console.log(url)
await downloadImage(url, `${__dirname}/../../assets/`)

download.js

const download = require('image-downloader');
const { SimpleConsole } = require('./errors');

async function downloadImage(url, filepath) {
    const options = {
        url: url,
        dest: filepath,
    };
    await download.image(options)
        .then(({ filename }) => {
            SimpleConsole(`saved to ${filename}`)
        })
        .catch((err) => console.error(err));
}

module.exports = { downloadImage };

And even if I add this code in doujin.js it does not work

await open("https://hitomi.la/reader/2510773.html#3")  //doujin link of the request

Environment

• Version
  • NodeJS: 18.12.1
  • Package: 7.0.6

UnhandledPromiseRejectionWarning: Error: incorrect header check

Had a version jump, so I don't actually know from which version this error appears (from v3.0.1 to v3.1.0)

(node:30432) UnhandledPromiseRejectionWarning: Error: incorrect header check
    at Zlib.zlibOnError [as onerror] (zlib.js:181:17)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:30432) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which 
was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:30432) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

code:

import hitomi from 'node-hitomi'
hitomi.getGalleryData(2008107)
  .then(async (gallery) => {
    console.log(gallery);
})

just a guess, but probably ever from dependency change from node-fetch to https

403 for image url created using getImageUrl

Description of bug

Passing Image to the ImageUrlResolver.getImageUrl() generates URL which returns 403

Expected behavior

Return working URL

Screenshots

none needed

Environment

• OS: Windows 11
• Version
  • NodeJS: v17.7.1
  • Package: 7.0.3

Additional context

const galleryId = 2187810
const resolver = await (new hitomi.ImageUrlResolver()).synchronize()
const gallery = await hitomi.getGallery(galleryId)
const imageUrl = resolver.getImageUrl(gallery.files[0], 'webp', {})

Generated url: https://ba.hitomi.la/webp/1649354401/701/333e4e57f16697902987a77abde2d7e65693538330bff5866890cb9e7619dbd2.webp

getTags can't find artists include dot character

Description of bug

getTags can't find artists include dot character.
Other types (group, tag, series...) too.

const main = async () => {
  const resp = await hitomi.getTags("artist", { startWith: "0-9" });
  // missing `.357-inch` and `.s`
  console.log(resp.slice(0, 5));
  // =>
  // [
  //   { type: 'artist', name: '-1semuy' },
  //   { type: 'artist', name: '0-g' },
  //   { type: 'artist', name: '0-jirow' },
  //   { type: 'artist', name: '007' },
  //   { type: 'artist', name: '013' }
  // ]
};

Environment

• OS: macOS Monterey 12.3.1
• Version
  • NodeJS: 16.14.2
  • Package: 7.0.4

HitomiError [REQEUST_REJECTED]: Request to 'https://ltn.hitomi.la/index-all.nozomi' was rejected

It seems to 'https://ltn.hitomi.la/index-all.nozomi' sends HTTP status code 206 Partial Content, add status code exception to the fetchBuffer function

Add text search to getIds function

Implement e-/exhentai api

Add gallery related data to Gallery

Recently, Hitomi made some differences on their gallery api, and one of them is addition of gallery related datas.
It would be nice to have this feature.

Improve performance of array related function

Using the array functions such as map, filter performs worse than using the for loop

Implement more data fetcher from hitomi (Ex: supported language, etc...)

Handle long gallery title

If the length of the gallery title is overs 200 bytes, the getGallery function doesn't return the gallery object that includes full data.

Add extension support for other extensions supported on hitomi

Extension of interface Image doesn't contain gif.

Missing npm banner image

Subdomain mismatch on the url of the image

Environment

• OS: MacOS Catalina 10.15.7
• Nodejs: v16.13.2
• Npm: 8.1.2

Code

const hitomi = require('node-hitomi').default;

const gallery hitomi.getGallery(1434389).then(image => {
    const isThumb = true; // same to false
    const ext = (isThumb ? (image.hasAvif ? 'avif' : image.extension) : (((image.hasAvif) ? 'avif' : ((image.hasWebp) ? 'webp' : image.extension))));
    const url = hitomi.getImageUrl(image, ext, {isThumbnail: isThumb})
    console.log(url); // it return wrong url
})

I try getImageUrl to gallery No 1434389, and it return wrong url.
I also try to connect url with Referer header, it same.
So i think Hitomi uses different image routing rules according to the presence of avif.

Remove dependency from node-fetch

I believe dependency from node-fetch can be easily replaced with default module of nodejs, something like https.
Removing dependencies from everything would make this package standalone, and also more light.

Make getQueriedIds more efficient

You can use key-value object like dictionary to make the performance better.
Instead of using set, put id on key so that you can get access speed of almost zero.

Default export error: The expression of an export assignment must be an identifier or qualified name in an ambient context.

Something's wrong with the default export.

The error occurs only with just importing / calling the module.

Apply change of function name to README.md

Make method to get image url flexible

Currently, the getImageUrl function uses hard-coded image resolving data to resolve an image URL and this produces the wrong URL when Hitomi changes its image resolving data.
You can solve this problem just by fetching the data whenever the function is called, although it costs a lot.
So, I suggest making a class that can obtain and store the data out of necessity.
In that way, users can get proper URLs at a low cost even if Hitomi changes their method to get the URLs.

Add image download function with image typed objects

Make getTags work properly

Make function and parameter name more intuitive and coherent

Improve performance of queryTag

queryTag is run synchronously and as we all know, it spends more time than when it run asynchronously.