yimura / scraper

A Youtube Scraper with zero dependencies.

YouTube Scraper

Table of Contents

• Why use this package
  • Timings
• Options
  • Example Options
• Example Code
  • Output
• Return Object Structure

Why use this package?

This is a YouTube scraper with zero dependencies. Everything has been coded to have a minimal footprint creating a small package that's aimed at being as fast as possible.

Timings

These are the timings I would get on average over 20 tests, ofcourse the Fetch time depends on how good your connection is to YouTube and how loaded YouTube is at that point.

Type	Fetch Time	Processing Time
video	585.632055 ms	3.117175 ms
channel	494.026065 ms	not tested
playlist	569.424545 ms	not tested

Check here

Options

Property	Default	Description
language	en	Set the language that you would like for results to be returned in. A list of supported language types can be found here.
searchType	video	Which type to search for on YouTube, supported types are any, channel, live, movie, playlist and video

"Sort by" has not been implemented as of now.
All data is sorted in the default order that YouTube returns these in.

Example Options

You can set the global language which YouTube should return results in or set the return language per search/request:

import youtube from '@yimura/scraper'

// This will set the language to French from France globally
const yt = new youtube.default('fr-FR');

// Sets the language communicated to YouTube to Dutch from Belgium for this search
const results = yt.search('Never gonna give you up', {
    language: 'nl-BE',
    searchType: 'video' // video is the default search type
});

Example Code

CommonJS:

const Scraper = require('@yimura/scraper').default;

const youtube = new Scraper();

youtube.search('Never gonna give you up').then(results => {
    console.log(results.videos[0]);
});

ESModule:

import youtube from '@yimura/scraper'

const yt = new youtube.default();
yt.search('Never gonna give you up').then(results => {
    console.log(results.videos[0]);
});

Output

{
    channel: {
        name: 'Official Rick Astley',
        link: 'https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw',
        verified: true
    },
    description: "Rick Astley's official music video for “Never Gonna Give You Up” Listen to Rick Astley: https://RickAstley.lnk.to/_listenYD Subscribe ...",
    duration: 213,
    duration_raw: "3:33",
    id: 'dQw4w9WgXcQ',
    link: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
    thumbnail: 'https://i.ytimg.com/vi/dQw4w9WgXcQhqdefault.jpg',
    shareLink: 'https://youtu.be/dQw4w9WgXcQ',
    title: 'Rick Astley - Never Gonna Give You Up (Video)',
    uploaded: '11 years ago',
    views: 788551856
}

Return Object Structure

{
    channels: [
        {
            channelId: String,
            description: String,
            link: String,
            thumbnails: [
                {
                    url: String,
                    width: Number,
                    height: Number
                }
            ],
            subscribed: Boolean,
            uploadedVideos: Number,
            verified: Boolean
        }
    ],
    playlists: [
        {
            preview: [
                {
                    duration: Number,
                    duration_raw: String,
                    views: Number,
                    id: String,
                    link: String,
                    thumbnail: String,
                    title: String,
                    shareLink: String
                }
            ],
            id: String,
            link: String,
            thumbnail: String,
            title: String,
            videoCount: Number
        }
    ],
    streams: [
        {
            watching: Number,
            channel: {
                name: String,
                link: String,
                verified: Boolean
            },
            id: String,
            link: String,
            thumbnail: String,
            title: String,
            shareLink: String
        }
    ],
    videos: [
        {
            description: String,
            duration: Number,
            duration_raw: String,
            uploaded: String,
            views: Number,
            channel: {
                name: String,
                link: String,
                verified: Boolean
            },
            id: String,
            link: String,
            thumbnail: String,
            title: String,
            shareLink: String
        }
    ]
}

scraper's Issues

TypeError The "listener" argument must be of type function. Received an instance of Object

Describe the bug

This issue comes when I am using Electron With React TypeScript Webpack and Electron. It works like charm in Separate Node.js Application.

To Reproduce
Steps to reproduce the behavior:

1. Clone [this repo.](https://github.com/electron-react-boilerplate/electron-react-boilerplate
2. yarn add @yimura/scraper
3. Type the following:

import youtube from '@yimura/scraper'

const yt = new youtube.default();
yt.search('Never gonna give you up').then(results => {
   console.log(results[0]);
});

4. yarn start and you should see the error in Electron.

Expected behavior
Should Print the JSON data for the YT query.

Screenshots
If applicable, add screenshots to help explain your problem.

Thumbnail picture is incorrectly formatted

The thumbnail is incorrectly returned, there is a missing / before hqdefault.jpg:

Scraper/src/Util.js

Line 31 in 3a27585

return 'https://i.ytimg.com/vi/'+ id +'hqdefault.jpg';

Set Accept-Language header

Is your feature request related to a problem? Please describe.
Currently YouTube will decide the language based of off the location (from your IP).

Describe the solution you'd like
Add an option with each search to set the preferred language or set the preferred language when initiating the Scraper class.

Describe alternatives you've considered
none

Additional context
This might be possible to communicate to Google/YouTube through the Accept-Language header.

"TypeError: The "listener" argument must be of type Function. Received type object" when using with NextJS

After a clean installation of NextJS and calling the scraper in index,js following the documentation, it will throw the error "TypeError: The "listener" argument must be of type Function. Received type object".

Running the scraper from a file in the terminal runs as expected.

Node Version: 14.19.1
NPM Version: 6.14.16

Steps to reproduce the behavior:

1. Install NextJS
2. Install Scraper
3. Call scraper in Index.js
4. See error

Doing logging the result in index,js apparently works, but the frontend will be broken.

Unexpected token '.', but normal fix isn’t working

Describe the bug
The picture below shows the error and the code is your template, there is no other module other then this one, it’s a new project and only has one js file which has your template.

Environment
Node Version: 14.17.6
NPM Version: 6.14.15

Expected behavior
I expected to get the template output which is :

{
channel: {
name: 'Official Rick Astley',
link: 'https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw',
verified: true
},
description: "Rick Astley's official music video for “Never Gonna Give You Up” Listen to Rick Astley: https://RickAstley.lnk.to/_listenYD Subscribe ...",
duration: 213,
duration_raw: "3:33",
id: 'dQw4w9WgXcQ',
link: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
thumbnail: 'https://i.ytimg.com/vi/dQw4w9WgXcQhqdefault.jpg',
shareLink: 'https://youtu.be/dQw4w9WgXcQ',
title: 'Rick Astley - Never Gonna Give You Up (Video)',
uploaded: '11 years ago',
views: 788551856
}

Allow providing video url directly

Is your feature request related to a problem? Please describe.
According to #10 searching by providing a url is not always consistent. Custom search solutions could also benefit from scraper's speed.

Describe the solution you'd like
A _fetch param that will use the query directly as the request URL and a function to wrap it up (because search is misleading).

Describe alternatives you've considered
Calling the functions directly, however it feels redundant as it will be exactly the same as search but will require manually GETting page.

Additional context
While search kinda works when you provide a url, it might not be on-par depending on language, video age rating, location etc.

Unexpected token '.'

I'm having this issue rn, do you know how to fix?

Thumbnail scrape issues

Scraper is working really good but there is an issue with thumbnail sizes.
Let me show the issue:

Normal thumbnail from YouTube Search:

Oversized thumbnail from Yimura Scraper:

Did you see the black bars? It is the issue.

get `videoInfo` from url link

Is your feature request related to a problem? Please describe.
This package is fast in searching youtube videos, and should state in the documentation the limit it search. As I wanted to get video info from the link, sometime the title return undefined, thus forcing me to make it Unknown Track.

Describe the solution you'd like
Get video info at least title from the provided link parameter

Describe alternatives you've considered
No idea

Additional context

image