markjaquith / wordpress-plugin-directory-slurper

Slurps down every stable version of every plugin in the WordPress plugin directory.

WordPress Plugin Directory Slurper

A command line PHP script that downloads and updates a copy of the latest stable version of every plugin in the WordPress.org plugin repository.

Really handy for doing local searches across all WordPress plugins.

Requirements

• Unix system (tested on Mac OS X and Linux)
• PHP 5.2 or higher
• wget and svn command-line executables installed

A Windows/Unix compatible version of this project that has comparable performance (if you can use cURL and PHP's pThreads) is available here.

Instructions

1. cd WordPress-Plugin-Directory-Slurper
2. ./update

The plugins/ directory will contain all the plugins when the script is done.

Run ./update help for more options.

Scanning the repo

You can use ack or ag to scan the plugins repository. See also the plugins directory maintenance documentation page.

This repository also includes a script to show a summary of a scan. For example:

$ ag --php --skip-vcs-ignores rest_get_date_with_gmt | tee scans/rest_get_date_with_gmt.txt
$ ./summarize-scan.php scans/rest_get_date_with_gmt.txt

5 matching plugins
Matches  Plugin                             Active installs
=======  ======                             ===============
      4  rest-api                                   40,000+
      4  wptoandroid                                    30+
      5  custom-contact-forms                       60,000+
      2  appmaker-wp-mobile-app-manager                 50+
      4  appmaker-woocommerce-mobile-app-manager       200+

FAQ

Why download the zip files? Why not use SVN?

An SVN checkout of the entire repository is a BEAST of a thing. You don't want it, trust me. Updates and cleanups can take hours or even days to complete.

Why not just do an SVN export of each plugin's trunk?

There is no guarantee that the plugin's trunk is the latest stable version. The repository supports doing development in trunk, and designating a branch or tag as the stable version. Using the zip file gets around this, as it figures it all out and gives you the latest stable version.

How long will it take?

Your first update will take a while (at least a couple of hours, and potentially overnight, depending on your connection and disk speeds).

But subsequent updates are smarter. The script tracks the SVN revision number of your latest update and then asks the plugins SVN repository for a list of plugins that have changed since. Only those changed plugins are updated after the initial sync.

How much disk space do I need?

As of December 2017, the plugin repository contains over 70,000 plugins. The script will download around 20 GB of zip files which, when unpacked, will use around 45 GB of disk space.

Something went wrong, how do I do a partial update?

The last successful update revision number is stored in plugins/.last-revision. You can just overwrite that and the next update will start after that revision.

What is this thing actually doing to my computer?

Once downloads have started, you can use a command like this to monitor the tasks being executed by this tool:

watch -n .5 "pstree -pa `pgrep -f '^xargs -n 1 -P .+ ./download'`"

Copyright & License

Copyright (C) 2011 Mark Jaquith

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

wordpress-plugin-directory-slurper's Issues

Delete plugins that have been removed from the repository

As @Ipstenu pointed out to me today, this script does not delete plugins that you've downloaded but that have subsequently been removed from the repository.

As of 68d6912 the summarize-scan.php script will detect this condition and print REMOVED next to any such plugins.

The script should create the zips directory

Currently, if the zips directory doesn't exist, the program proceeds through the whole repository with errors like:
Updating 03talk-community-conferencezips/03talk-community-conference.zip: No such file or directory
... download failed.

Also some spaces or periods at the end of the updating line would make that more readable.

Question about multi-threaded capabilities for this slurper

I noticed https://github.com/krakjoe/pthreads exists

Does it seem feasible to perhaps adjust that php library to slurp all the plugins more quickly?

Slurp themes too

It shouldn't be too much trouble to make this script work with both plugins and themes. Here's how to change it over to just themes: Ipstenu/WordPress-Theme-Directory-Slurper@4ae9d79

count() warning on first run

I setup a fresh copy of the repo and got this warning when running update for the first time:

Warning: count(): Parameter must be an array or an object that implements Countable in wordpress-plugin-directory-slurper/common.php on line 49

Everything seems to still be working, though, so it's a minor issue.

Sometimes I get a prompt to 'Replace' files

Updating auto-seo-tagsreplace plugins/auto-seo-tags/readme.txt.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename:

Generally I've been hitting A on these, but it means I have to monitor my download.

You could edit unzip -d plugins to unzip -d -o plugins and that would force overwrite. (or -n to never overwrite)

I'm not sure which is best.

WP.org now requires User Agents

Submitted Pull Request #34 to correct this

tl;dr thanks to spammers, WP.org needs an UA. I grabbed a Firefox one I'd been using. Not great but gets the job done.

Also patched an API call for removed plugins that return null in some cases (IIRC it's for 'permanent closures' only -- plugins that are very unlikely to ever be reopened)

PHP Warning, Not Updating

Hey Mark,

Thanks for this code!

My initial slurp went just fine, but I just tried to run an update several times and am not sure what's wrong. I'm using the latest code from the git repo.

Here's what I get when updating:

[eric@ea-fedora WordPress-Plugin-Directory-Slurper]$ sudo ./update
Determining most recent SVN revision...
Most recent SVN revision: 725464
Last synced revision: 678503
PHP Warning: file_get_contents(http://plugins.trac.wordpress.org/log/?
verbose=on&mode=follow_copy&format=changelog&rev=725464&limit=46961): failed to > open stream: HTTP request failed! in /home/eric/Dropbox/dev/vhost/WordPress-Plugin-
Directory-Slurper/update on line 57
[CLEANUP] Updated plugins/.last-revision to 725464
[SUCCESS] Done updating plugins!
It took 1 minutes and 23 seconds to update 0 plugins
[DONE]

It's probably been about two months since the original slurp, which went flawlessly and I have never yet updated (or tried).

Organise plugins into directories based on the first letter of their slug

I'm sure this process has a name but I can't think of it.

I've been working on a change to the slurper which means it places plugins into subdirectories based on the first letter of the plugin slug. The reason for this is to avoid the huge number of subdirectories inside the plugins directory that you otherwise get.

This results in a directory structure that looks something like:

- plugins
  - a
    - aa-disable-click
    - aajako-rashifal
    - aa-result
    - ...
  - b
    - b2-logos
    - babble
    - babelbee-translation-request
    - ...
  - c
    - c4d-post-views
    - c7-history-developer
    - cached-twitter-followers
    - ...

PR coming at some point soon.