yggverse / yggo

YGGo! Distributed Web Search Engine

Project archived. Please, visit Yo! - the next generation of YGGo project based on Manticore search.

YGGo! - Distributed Web Search Engine

Written by inspiration to explore Yggdrasil ecosystem. Engine could be useful for crawling regular websites, small business resources, local networks.

The project goal - simple interface, clear architecture and lightweight server requirement.

Overview

https://github.com/YGGverse/YGGo/tree/main/media

Online instances

• http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo/

Database snaps

• 17-09-2023 http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggtracker/en/torrent/15

Requirements

php8^
php-dom
php-xml
php-pdo
php-curl
php-gd
php-mbstring
php-zip
php-mysql
php-memcached
memcached
sphinxsearch

Installation

• git clone https://github.com/YGGverse/YGGo.git
• cd YGGo
• composer install

Setup

• Server configuration /example/environment
• The web root dir is /src/public
• Deploy the database using MySQL Workbench project presented in the /database folder
• Install Sphinx Search Server
• Configuration examples presented at /config folder
• Make sure /src/storage/cache, /src/storage/tmp, /src/storage/snap folders are writable
• Set up the /src/crontab by following example
• To start crawler, add at least one initial URL using search form or CLI

JSON API

Build third party applications / index distribution.

Could be enabled or disabled by API_ENABLED option

Address

/api.php

Search

Returns search results.

Could be enabled or disabled by API_SEARCH_ENABLED option

Request attributes

GET action=search  - required
GET query={string} - optional, search request, empty if not provided
GET type={string}  - optional, filter mime type of available or empty
GET page={int}     - optional, search results page, 1 if not provided
GET mode=SphinxQL  - optional, enable extended SphinxQL syntax

Hosts distribution

Returns hosts collected with fields provided in API_HOSTS_FIELDS option.

Could be enabled or disabled by API_HOSTS_ENABLED option

Request attributes

GET action=hosts - required

Application manifest

Returns node information for other nodes that have same CRAWL_MANIFEST_API_VERSION and DEFAULT_HOST_URL_REGEXP conditions.

Could be enabled or disabled by API_MANIFEST_ENABLED option

Request attributes

GET action=manifest - required

Search textual filtering

Default constructions

word prefix:

yg*

operator OR:

hello | world

operator MAYBE:

hello MAYBE world

operator NOT:

hello -world

strict order operator (aka operator "before"):

aaa << bbb << ccc

exact form modifier:

raining =cats and =dogs

field-start and field-end modifier:

^hello world$

keyword IDF boost modifier:

boosted^1.234 boostedfieldend$^1.234

Extended syntax

https://sphinxsearch.com/docs/current.html#extended-syntax

Could be enabled with following attributes

GET m=SphinxQL

Roadmap

Basic features

• Web pages full text ranking search
  • Sphinx
• Unlimited content MIME crawling
• Flexible settings compatible with IPv4/IPv6 networks
• Extended search syntax support
• Compressed page history snaps with multi-provider storage sync
  • Local (unlimited locations)
  • Remote FTP (unlimited mirrors)
  • Privacy-oriented downloads counting, traffic controls

UI

• CSS only, JS-less interface
• Unique host ident icons
• Content MIME tabs (#1)
• Page index explorer
  • Meta
  • Snaps history
  • Referrers
• Top hosts page
• Safe media preview
• Results with found matches highlight
• The time machine feature by content snaps history

API

• Index API
  • Manifest
  • Search
  • Hosts
  • Snaps
• Context advertising API

Crawler

• Auto crawl links by regular expression rules
  • Pages
  • Manifests
• Robots.txt / robots meta tags support (#2)
• Specific rules configuration for every host
• Auto stop crawling on disk quota reached
• Transactions support to prevent data loss on queue failures
• Distributed index crawling between YGGo nodes trough manifest API
• MIME Content-type settings
• Ban non-condition links to prevent extra requests
• Debug log
• Index homepages and shorter URI with higher priority
• Collect target location links on page redirect available
• Collect referrer pages (redirects including)
• URL aliasing support on PR calculation
• Host page DOM elements collecting by CSS selectors
  • Custom settings for each host
• XML Feeds support
  • Sitemap
  • RSS
  • Atom
• Palette image index / filter
• Crawl queue balancer, that depends of CPU available
• Networks integration
  • yggdrasil
    • YGGstate (unlimited nodes)
      • DB
      • API

Cleaner

• Banned pages reset by timeout
• DB tables optimization

CLI

*CLI interface still under construction, use it for your own risk!

• help
• db
  • optimize [x] crontab
  • crawl
  • clean
• hostSetting
  • get
  • set
  • list
  • delete
  • flush
• hostPage
  • add
  • rank
    • reindex
• hostPageSnap
  • repair
    • db
    • fs
  • reindex
  • truncate

Other

• Administrative panel for useful index moderation
• Deployment tools
• Testing
• Documentation

Contributions

Please make a new branch of main|sqliteway tree for each patch in your fork before create PR

git checkout main
git checkout -b my-pr-branch-name

See also: SQLite tree

Donate to contributors

• @d47081: BTC | LTC | XMR | ZEPH | Support our server by order Linux VPS

License

• Engine sources MIT License
• Home page animation by alvarotrigo
• CLI logo by patorjk.com
• Transliteration by php-translit
• Identicons by jdenticon

Feedback

Feel free to share your ideas and bug reports!

Community

• Mastodon
• [matrix]

See also

• Yo! Micro Web Crawler in PHP & Manticore
• YGGwave ~ The Radio Catalog
• YGGstate - Yggdrasil Network Analytics
• YGGtracker - BitTorrent Tracker and Catalog for Yggdrasil

yggo's Issues

White list / black list websites, robots.txt pre-sets

So, trackers with external seeders is shit inside the network

Nice start..

I mean this subject for the websites we need to crawl and some maybe a mirrors we need to block or limit by the crawlPageLimit/CRAWL_HOST_DEFAULT_PAGES_LIMIT

Ideas here, just few relevant relations
#1 (comment)

And I would to ask, do we need to enable the GitHub Discussions page, or do Issues to resolve, not talk.

Uncaught PDOException: SQLSTATE[42000]: Syntax error or access violation

We will get an error message if we try to enter such a search query: http://[202:5855:991b:6e6b:e306:53f2:b058:2186] or even just /.

UTF-8 normalization

Some pages have unsupported character set that causes DB error in crawler

it have been temporarily fixed but requires proper solution (without db character set changing),

so I found useful library that could solve this problem
https://github.com/neitanod/forceutf8

Please add support for the robots.txt file

Please add reading the robots.txt file and crawling based on its content in the crawler functions.

Implement MySQL + Sphinx data driving model

Just tried to make search request on 2.5M rows on SQLite / FTS5 and seems that we starting to have performance issue.

According to following conversation we need to rewrite current DB driver model. Suppose MySQL is the nice accessible alternative.

I have experience with Sphinx engine, it stores compiled data in RAM and able to process at least 8M rows in milliseconds with same server resources, comparing to the current result.

If some one have better ideas - you are welcome here.

Add missed keywords index recording

https://github.com/YGGverse/YGGo/blob/sqliteway/library/sqlite.php#L90
https://github.com/YGGverse/YGGo/blob/sqliteway/library/sqlite.php#L99

Search in translit feature

Implemented with library
https://github.com/ashtokalo/php-translit

At this moment, it supports lot of locales, but not reverse transliteration
ashtokalo/php-translit#8

Core upgrade

Thoughts to change DB structure or/and current search model implementation.

As the project have no releases yet, current repository could be separated to the sphinxway branch.

By this way, following changes could be wanted:

• Rewrite crawler
  • Make it more flexible for changes (currently it's monolith)
  • Improve multimedia index / caching
  • Improve content semantics by internal content parsing to the ranked keywords
• Framework for web UI
• Make installation simpler for distributed usage
• Add more features related with snaps exploring

imho, as the basement, perspective replacement for sphinx engine is
https://github.com/manticoresoftware/manticoresearch

Main goal for the future changes - make search useful by collected experience.

Reindex page feature

Ability to make page re-index request manually, on explorer page or so

Check 404 pages not banned in search index

According to Yggdrasil core v 0.4 - 0.5 migration

Engine has snap history feature, where pages with 404 must be available in search index.

I think the search results should be divided into categories somehow...

I think it would make sense to add something like this panel:

If I want to find sites, then such search results don't make much sense to me: