nelsonjchen / github-wiki-see-rs

🔎 Did you know most GitHub Wikis can't index on search engines? Search Engine Enablement for GitHub Wikis service. 400,000+ GitHub Wikis, now indexable by your favorite search engine.

GitHub Wiki SEE

As seen at https://github-wiki-see.page by search engines and archivers!

GitHub Wiki Search Engine Enablement is a service to allow GitHub Wikis to be indexed by search engines.

This is a terribly and hastily built service. However, it is usable and MVP!

This was made in response to community/community#4992, an issue that has been present for 10+ years.

Design

This is designed as a Rust web proxy application. It is not very Rusty and has lots of hackiness. Cleanup, major overhauls, and straightening are much appreciated!

It is designed to run as a simple Docker application on a service such as fly.io and/or Google Cloud Run. Uptime and latency are important so that the service appears on search engines and gets a high ranking.

301/302/307/308s are intentionally not used as to not give search engines the impression that the page is a redirect and to ignore the content. Humans should see the "content" as a redirect; the robots should not.

All links rendered in the tool going outside of GitHub are tagged with rel="nofollow ugc" to prevent ranking manipulation which is probably one the reason wiki content was excluded from indexing.

A Cloudflare Worker is placed in front to additionally protect against the service accidentally mirroring indexable content on GitHub. The worker also enriches a "last modified" header date on the proxied content if possible from the original content if the original content isn't indexable to better hint to search engines the freshness of content and better utilize their crawler budget.

Decommissioning

Please see:

https://github-wiki-see.page/#decommissioning

But basically if GitHub lets it be indexed, this service will 308 redirect it.

github-wiki-see-rs's Issues

Remove per-page Sitemaps in favor of per-wiki sitemaps with `wiki_index` route

Search engines seem to be wasting their crawl budget on dead pages in the old per-page sitemap. With the per-page sitemap still there, they'll keep trying and thus wasting resources.

Time for another scan of GitHub for indexed Wikis (April 2022)

rubs hands

Let's see what they permit indexing of and what they aren't

Anchor links not working

Fixing of #345 resulted in anchor links showing up in index. Maybe good, maybe bad. Either way, the anchor links don't work for some reason.

Redirect to wiki page?

It would be cool if clicking on the see.page link automatically redirected you to GitHub

Allow cloudflare workers to rewrite a last updated space

Should help with the last modified situation and help crawling budget. Maybe also make it modify the header too with that.

Websites Hosting

Resolve the problem

Additionally use Cloudflare Workers to check if GitHub permits indexing before loading the proxied page

Preliminary tests seem to indicate this can handle the load whereas the "origin" server cannot implement this.

Please get rid of the flashing yellow background

The blinking is so 2010 and it's extremely annoying, especially with Dark Reader. Please just make it big bold red text or whatever but get rid of the flashing background.

Monitor https://github-wiki-see.page/m/aiegoo/tmux/wiki/startify-vim as it is detected as a redirect when it should not be

https://github-wiki-see.page/m/aiegoo/tmux/wiki/startify-vim

• Not decommissioned
• Contentful
• Yet detected as a redirect?

Will check every few days.

https://search.google.com/search-console/inspect?resource_id=sc-domain%3Agithub-wiki-see.page&id=UCy4w0VNemi8RuuiwHLGCw&hl=en

Make a defined stable "API" for extensions/scripts/etc. to click to go to the original page

Example: #136 (comment)

Monitor and Blacklist Shellcheck Wiki

Waiting on reasonable indexing/listing of shellcheck.net's mirror of the wiki entries.

koalaman/shellcheck#2487

Will add to the redirect blacklist.

Monitor https://github.com/keirf/flashfloppy/wiki and see why it's out of survey.

https://github.com/keirf/flashfloppy/wiki

It's over 800 stars.

It's unpublically editable

It still has noindex.

Add Mediawiki Link Support

https://docs.github.com/en/communities/documenting-your-project-with-wikis/editing-wiki-content

Handle non-markdown backing

Probably will just try to look for non-markdown after markdown and display the raw source.

Remove anchor links from sitemap and wiki_index path

These are needlessly duplicating.

Detect Renamed/Moved Repositories and act accordingly

https://github.com/node-config/node-config

-->

https://github.com/lorenwest/node-config

Update Touched Sitemap List

Missing stuff like https://github.com/alygin/vscode-tlaplus/wiki/Settings somehow. Doesn't show in touched wiki list for wiki_index.

Add Last Modified to the HTML from the redirect checker

Header response wasn't enough. Or maybe it was. But doesn't show up on summary. Hmm.

Upgrade to a tokio 1.0+ stack

The stack is using old tokio 0.2 stuff. The Rust world went towards newer tokio versions.

Wikis that are unindexable and uneditable may not expose a raw URL

Check for handling of this case and fallback to GitHub.

https://gist.github.com/nnja/9136152c163091614e70defcf3753d06#gistcomment-4469816

"View Page On GitHub" button is too big and flashy, or looks too much like a logo, triggering banner blindness

Speaking from personal experience here...whenever I find myself on one of these pages, my mind completely skips over the big blue button, and instead I hunt for the tiny link below it with the "URL" label. It would be much better, I think, to put it a bit lower so it doesn't look like a logo.

Fallback breaks for SoapySDR wiki home

https://github-wiki-see.page/m/pothosware/SoapySDR/wiki

Gradually Decommission known indexable GitHub Wikis

• Use BigQuery to locate all wikis
• Scan/Check all Wikis's Home to see which are are not blocked by the header tag
• Burn in list of wikis that have indexing enabled. The list is small enough to fit in the binary.
• Redirect users/search engines automatically to GitHub where indexing is enabled.

Thank you

I just want to say "Thank you!" for this project! :-)

My wiki pages now appear in search engines. First visitors come from https://github-wiki-see.page/ , this was how I got attention of your project.

Broken 520 for some reason

https://github-wiki-see.page/m/Dlloydev/sTune/wiki/Examples_MAX31856_PTC_SSR

🚨🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 🚨 issacs/github#1683 viewers! That issue is closed. See this GitHub Discussion on Wiki Indexing on GitHub's feedback discussion forum.

Viewers of isaacs/github#1683 (comment)

See community/community#4992

This issue is just a stub to redirect.

You are first result in Google

hey, thanks for push forward on indexing wikis! I wonder how you got ranked so quickly, without having any backlinks to your site?

wiki index and debug sitemaps are broken

https://github-wiki-see.page/debug_sitemaps/pyenv/pyenv/sitemap.xml

https://github-wiki-see.page/m/pyenv/pyenv/wiki_index

Return 404 when page is non-existent, not blindly following redirect.

Just wanted to drop by and give my thanks for this tool

This helped me find an answer today - clever idea and neat execution. I appreciate it!

Add a social graph image that says "Explanation on Front Page"

https://twitter.com/pocmatos/status/1579784657314643970

Apparently people are not visiting the front page. The existing social graph tags try to clear it up but even with the preview text on this one i guess it didn't clear it up. Let's add a graphic.

Page isn't rendered at all (https://github.com/wlsdn2316/1-tetris-/wiki/Functions)

https://github.com/wlsdn2316/1-tetris-/wiki/Functions

It's just one line. Hmm.

Change 308 permanent redirects to 301

Searching site:github-wiki-see.page in Google seems to return a lot of pages that redirect automatically, but aren't stricken from the https://github-wiki-see.page domain.

Maybe changing the responses to the older 301 redirect might help get the site entries off the index for these pages.

Rescan Wikis in February

These weren't indexable before in the last survey:

• https://github.com/NuGet/Home/wiki/Repository-Signatures-and-Countersignatures-Technical-Specification

• https://github.com/ajayyy/SponsorBlock/wiki/Edge

and probably more

They are now.

These were indexed:

• https://github.com/imchillin/Anamnesis/wiki

but they are now noindexed.

Add missing image CSS

Create synthetic wiki index version of sitemaps for wiki depth debugging purposes.

As a search console user, I would like to know what pages underneath one particular wiki are indexed.