gatographql / gatographql

Interact with all your data in WordPress using GraphQL

Gato GraphQL

Gato GraphQL is a tool for interacting with data in your WordPress site. You can think of it as a Swiss Army knife for dealing with data, as it allows to retrieve, manipulate and store again any piece of data, in any desired way, using the GraphQL language.

With Gato GraphQL, you can:

• Query data to create headless sites
• Expose public and private APIs
• Map JS components to Gutenberg blocks
• Synchronize content across sites
• Automate tasks
• Complement WP-CLI to execute admin tasks
• Search/replace content for site migrations
• Send notifications when something happens (new post published, new comment added, etc)
• Interact with cloud services
• Convert the data from a 3rd-party API into the required format
• Translate content in the site
• Update thousands of posts with a single action
• Insert or remove Gutenberg blocks in bulk
• Validate that a new post contains a mandatory block
• And much more...

Check out the Tutorial section in gatographql.com which demonstrates how to implement these use cases using the plugin.

View screenshots

GraphiQL client to execute queries in the wp-admin:

Interactively browse the GraphQL schema, exploring all connections among entities:

The GraphiQL client for the single endpoint is exposed to the Internet:

Interactively browse the GraphQL schema exposed for the single endpoint:

Persisted queries are pre-defined and stored in the server:

Requesting a persisted query URL will retrieve its pre-defined GraphQL response:

We can create multiple custom endpoints, each for a different target:

Endpoints are configured via Schema Configurations:

We can create many Schema Configurations, customizing them for different users or applications:

Custom endpoints and Persisted queries can be public, private and password-protected:

Manage custom endpoints and persisted queries by adding categories to them:

We can configure exactly what custom post types, options and meta keys can be queried:

Configure every aspect from the plugin via the Settings page:

Modules with different functionalities and schema extensions can be enabled and disabled:

Augment the plugin functionality and GraphQL schema via extensions:

The Tutorial section explains how to achieve many objectives, exploring all the elements from the GraphQL schema:

Development Requirements

• PHP 8.1
• Lando
• Composer

Recommended to use

• XDebug (integrated out of the box when using VSCode and the PHP Debug addon for VSCode)

Install for Development

Follow these steps:

Clone repo locally

Clone the repo in your local drive:

git clone https://github.com/GatoGraphQL/GatoGraphQL

Install Composer dependencies

Run:

$ cd {project folder}
$ cd submodules/GatoGraphQL && composer install && cd ../.. && composer install

Run the webserver for DEV

A Lando webserver for development is already set-up:

• Runs on PHP 8.1
• It directly uses the source code on the repo
• XDebug is enabled

Follow these steps:

Build the Lando webserver for DEV

Run (only the first time):

composer build-server

After a few minutes, the website will be available under https://gatographql.lndo.site.

To print the server information, including the port to connect to the MySql database (so you can visualize and edit the data in the DB using an external client, such as MySQLWorkbench), run:

composer server-info

What plugins are installed in the webserver? 🤔

• Gato GraphQL
• Gato GraphQL - Testing
• Gato GraphQL - Testing Schema

(The last two are utilities to run integration tests for Gato GraphQL. Among others, they provide CPT "dummy-cpt" and custom taxonomies "dummy-category" and "dummy-tag").

Log-in to the wp-admin

Credentials for https://gatographql.lndo.site/wp-admin/:

• Username: admin
• Password: admin

Open the Gato GraphQL plugin in the wp-admin

Click on the Gato GraphQL link on the menu to open the GraphiQL client, and execute the following GraphQL query:

{
  posts {
    id
    title
  }
}

If the installation of the webserver was successful, you will receive a response:

Run Integration Tests

Run:

composer integration-test

Start the Lando webserver for DEV

Building the webserver (above) is needed only the first time.

From then on, run:

composer init-server

Run the webserver for PROD

A Lando webserver is set-up and configured to test the released plugins, generated by GitHub Actions.

• Runs on PHP 7.2
• You must download the generated plugins for PROD (from GitHub Actions) and install them on the webserver
• XDebug is not enabled

Follow these steps:

Build the Lando webserver for PROD

Run (only the first time):

composer build-server-prod

After a few minutes, the website will be available under https://gatographql-for-prod.lndo.site.

(The URL is the same one as for DEV above, plus appending -for-prod to the domain name.)

To print the server information, run:

composer server-info-prod

The wp-admin login credentials are the same ones as for DEV.

Start the Lando webserver for PROD

Building the webserver (above) is needed only the first time.

From then on, run:

composer init-server-prod

Manage the Lando webservers

Re-install the WordPress site (DEV and PROD)

You can at any moment re-install the WordPress site (and import the initial dataset).

On the DEV webserver:

composer reset-db

On PROD:

composer reset-db-prod

This is useful when:

• The installation when doing build-server was halted midway (or failed for some reason)
• Running the integration tests was not completed (modifying the DB data to a different state, so that running the tests again will fail)

Re-build the Lando webserver for DEV

When a plugin or package folder has been renamed, you need to update the path in the overrides section of the .lando.upstream.yml Lando config file, and then rebuild the Lando webserver to reflect these changes.

Run:

composer rebuild-server

Regenerate the Composer autoload files for DEV

When a new plugin is added to the monorepo, it must have its Composer autoload file generated, and the plugin must be symlinked to the Lando webserver.

Run:

composer rebuild-app-and-server

Releasing the plugins for PROD

(This is done by the admin of the repo.)

The monorepo includes scripts that completely automate the process of releasing the plugins in the monorepo.

Follow these steps:

Tag the monorepo as patch, minor or major

Choose which version you will be releasing. The same version will be applied to all plugins in the monorepo.

(Given that the current version is 0.0.0...)

To release version 0.0.1, run:

composer release-patch

To release version 0.1.0, run:

composer release-minor

To release version 1.0.0, run:

composer release-major

What do these commands do? 🤔

Executing these commands will first prepare the repo for PROD:

• Update the version (in the plugin file's header, readme.txt's Stable tag, others) for all the extension plugins in the monorepo
• Update the documentation image URLs to point to that tag, under raw.githubusercontent.com
• Commit and push
• Git tag with the version, and push tag to GitHub

And then, it will prepare the repo for DEV again:

• Update the version to the next DEV version (next semver + -dev)
• Commit and push

To preview running the command without actually executing it, append -- --dry-run:

composer release-patch -- --dry-run

Create release from tag in GitHub

After tagging the repo on the step above, we must create a release from the tag to generate the extension plugins for production.

To create the release, we must head over to the tags page in the GitHub repo, and click on the new tag (eg: 0.1.0).

Then, on the tag page, click on Create release from tag.

This will trigger the generate_plugins.yml workflow, which will generate the extension plugins and attach them as assets to the tag page.

For instance, after tagging Gato GraphQL with 1.0.9, the tag page GatoGraphQL/GatoGraphQL/releases/tag/1.0.9 had the following assets attached to it:

• gatographql-1.0.9.zip
• gatographql-testing-1.0.9.zip
• gatographql-testing-schema-1.0.9.zip

Install the extension in the PROD webserver

Once the extension plugin has been generated, install it on the PROD webserver to test it, whether manually or using WP-CLI.

Using WP-CLI, to test the released version 0.1.0, run:

$ cd webservers/gatographql-for-prod
$ lando wp plugin install https://github.com/GatoGraphQL/GatoGraphQL/releases/latest/download/gatographql-0.1.0.zip --force --activate
$ cd ../..

Query the extension in the wp-admin

Once you've installed the release on the Lando webserver for PROD, log-in to the wp-admin, access the GraphiQL client, and execute the following GraphQL query:

{
  posts {
    id
    title
  }
}

If the installation of the webserver was successful, you will receive a response:

Run Integration Tests for PROD

Run:

composer integration-test-prod

Development Process

Purging the cache

When developing the plugin and testing it in the DEV webserver, whenever we create a new PHP service or modify the signature of an existing one (such as the PHP classname), we need to purge the container cache.

Run:

composer purge-cache

How does Gato GraphQL use a service container? 🤔

The Gato GraphQL plugin uses a service container (via the Symfony DependencyInjection library), to manage all services in PHP.

Services are PHP classes, and must be defined in configuration files services.yaml and schema-services.yaml to be injected into the container.

The first time the application is invoked, the container gathers all injected services and compiles them, generating a single PHP file that is loaded in the application.

Generating this file can take several seconds. To avoid waiting for this time on each request, the Gato GraphQL plugin caches this file after it has been generated the first time.

The container needs to be purged whenever a service is created, or an existing one updated or removed.

(In production, the Gato GraphQL plugin purges the cache whenever a new extension plugin is activated, or when the plugin Settings are updated. During development, it can in addition be triggered manually, by running composer purge-cache.)

This applies to resolvers (type resolvers, field resolvers, directive resolvers, and any other resolver that gives shape to the GraphQL schema), as these are PHP services. Whenever a resolver is added or removed, or is updated in such a way that modifies the GraphQL schema, the cached container must be purged.

Some example resolvers are:

• String type: StringScalarTypeResolver
• Field User.name (and others): UserObjectTypeFieldResolver
• @skip directive: SkipFieldDirectiveResolver

Regenerating the monorepo configuration

After adding a plugin or package to the monorepo, the configuration (containing all the packages) must be regenerated.

Run:

composer update-monorepo-config

What does command update-monorepo-config do? 🤔

The update-monorepo-config command will:

• Regenerate the root composer.json, adding the new packages
• Regenerate the root phpstan.neon, adding the new packages

Development Tools

Error logs

Access the error logs from the Lando webserver:

composer log-server-errors

For PROD:

composer log-server-errors-prod

SSH

SSH into the Lando webserver:

composer ssh-server

For PROD:

composer ssh-server-prod

Debugging

XDebug is already integrated when using VSCode.

Add a breakpoint in the source code and then, in the Run and Debug tab, press on Start Debugging with the corresponding configuration (defined in .vscode/launch.json):

• [Lando webserver] Listen for Xdebug: For debugging the source code while running the Lando webserver for DEV
• [PHPUnit] Listen for Xdebug: For debugging PHPUnit tests

XDebug is enabled but inactive; it must be activated when requesting the webpage (see below).

Debugging in the Lando webserver for DEV

Activate XDebug for a request by appending parameter ?XDEBUG_TRIGGER=1 to the URL (for any page on the Gato GraphQL plugin, including any page in the wp-admin, the GraphiQL or Interactive Schema public clients, or other).

For instance:

• https://gatographql.lndo.site/wp-admin/edit.php?page=gatographql&action=execute_query&XDEBUG_TRIGGER=1
• https://gatographql.lndo.site/graphiql/?XDEBUG_TRIGGER=1

Debugging PHPUnit tests

Activate XDebug by prepending XDEBUG_TRIGGER=1 before the phpunit command to run the unit tests.

For instance:

XDEBUG_TRIGGER=1 vendor/bin/phpunit layers/GatoGraphQLForWP/phpunit-packages/gatographql/tests/Unit/Faker/WPFakerFixtureQueryExecutionGraphQLServerTest.php

Monorepo Commands

Retrieve the list of all Composer commands available in the monorepo:

composer list

View all monorepo commands

composer command	Description
analyse	Run PHPStan static analysis of the code
build-js	Build all JS packages, blocks and editor scripts for all plugins in the Gato GraphQL - Extension Starter repo
build-server	Initialize the Lando webserver with the 'Gato GraphQL' demo site, for development. To be executed only the first time
build-server-prod	Initialize the Lando webserver with the 'Gato GraphQL' demo site, for production. To be executed only the first time
check-style	Validate PSR-12 coding standards (via phpcs)
debug	Run and debug PHPUnit tests
delete-settings	Delete the plugin settings from the DB
deoptimize-autoloader	Removes the optimization of the Composer autoloaders for all the plugins installed in the webserver
destroy-server	Destroy the Lando webserver with the 'Gato GraphQL' demo site
destroy-server-prod	Destroy the Lando webserver with the 'Gato GraphQL' demo site for PROD
disable-caching	Disable caching for the 'Gato GraphQL' in DEV
disable-restrictive-defaults	Do not use restrictive default values for the Settings
enable-caching	Enable caching for the 'Gato GraphQL' in DEV
enable-restrictive-defaults	Use restrictive default values for the Settings
fix-style	Fix PSR-12 coding standards (via phpcbf)
import-data	Imports pre-defined data into the DB (posts, users, CPTs, etc)
improve-code-quality	Improve code quality (via Rector)
init-server	Alias of 'start-server
init-server-prod	Runs the init-server-prod script as defined in composer.json
install-deps-build-js	Install all dependencies from npm to build the JS packages, blocks and editor scripts for all plugins in the Gato GraphQL - Extension Starter repo
install-site	Installs the WordPress site
install-site-prod	Installs the WordPress site in the PROD server
integration-test	Execute integration tests (PHPUnit)
integration-test-prod	Execute integration tests (PHPUnit) against the PROD webserver
log-server-errors	Show (on real time) the errors from the Lando webserver
log-server-errors-prod	Show (on real time) the errors from the Lando webserver for PROD
log-server-warnings	Show (on real time) the warnings from the Lando webserver
log-server-warnings-prod	Show (on real time) the warnings from the Lando webserver for PROD
merge-monorepo	Create the monorepo's composer.json file, containing all dependencies from all packages
merge-phpstan	Generate a single PHPStan config for the monorepo, invoking the config for the PHPStan config for all packages
preview-code-downgrade	Run Rector in 'dry-run' mode to preview how the all code (i.e. src/ + vendor/ folders) will be downgraded to PHP 7.2
preview-src-downgrade	Run Rector in 'dry-run' mode to preview how the src/ folder will be downgraded to PHP 7.2
preview-vendor-downgrade	Run Rector in 'dry-run' mode to preview how the vendor/ folder will be downgraded to PHP 7.2
propagate-monorepo	Propagate versions from the monorepo's composer.json file to all packages
purge-cache	Purge the cache for the 'Gato GraphQL' in DEV
rebuild-app-and-server	Update the App dependencies (from Composer) and rebuild the Lando webserver
rebuild-server	Rebuild the Lando webserver
rebuild-server-prod	Rebuild the Lando webserver for PROD
release-major	Release a new 'major' version (MAJOR.xx.xx) (bump version, commit, push, tag, revert to 'dev-master', commit, push)
release-minor	Release a new 'minor' version (xx.MINOR.xx) (bump version, commit, push, tag, revert to 'dev-master', commit, push)
release-patch	Release a new 'patch' version (xx.xx.PATCH) (bump version, commit, push, tag, revert to 'dev-master', commit, push)
remove-unused-imports	Remove unused use imports
reset-db	Resets the WordPress database
reset-db-prod	Resets the WordPress database in the PROD server
server-info	Retrieve information from the Lando webserver
server-info-prod	Retrieve information from the Lando webserver for PROD
ssh-server	SSH into the Lando webserver with the 'Gato GraphQL' demo site
ssh-server-prod	SSH into the Lando webserver for PROD with the 'Gato GraphQL' demo site
start-server	Start the Lando webserver with the 'Gato GraphQL' demo site, for development
start-server-prod	Start the Lando webserver for PROD with the 'Gato GraphQL' demo site, for development
stop-server	Stop the Lando webserver
stop-server-prod	Stop the Lando webserver for PROD
stopping-integration-test	Runs the stopping-integration-test script as defined in composer.json
stopping-integration-test-prod	Execute integration tests (PHPUnit) against the PROD webserver, stopping as soon as there's an error or failure
stopping-unit-test	Runs the stopping-unit-test script as defined in composer.json
unit-test	Execute unit tests (PHPUnit)
update-deps	Update the Composer dependencies for the 'Gato GraphQL' plugins
update-monorepo-config	Update the monorepo's composer.json and phpstan.neon files, with data from all packages
use-default-restrictive-defaults	Remove the set value, use the default one
validate-monorepo	Validate that version constraints for dependencies are the same for all packages

Extending the GraphQL schema

Create an extension for Gato GraphQL using GatoGraphQL/ExtensionStarter.

Development

Supported PHP features

Check the list of Supported PHP features.

Gutenberg JS builds

Compiled JavaScript code (such as all files under a block's build/ folder) is added to the repo, but only as compiled for production, i.e. after running npm run build.

Code compiled for development, i.e. after running npm start, cannot be commited/pushed to the repo.

Architectural resources

PHP Architecture

Articles explaining how the plugin is "downgraded", using PHP 8.1 for development but deployable to PHP 7.2 for production:

1. Transpiling PHP code from 8.0 to 7.x via Rector
2. Coding in PHP 7.4 and deploying to 7.1 via Rector and GitHub Actions
3. Tips for transpiling code from PHP 8.0 down to 7.1
4. Including both PHP 7.1 and 8.0 code in the same plugin … or not?

Service container implementation:

• Building extensible PHP apps with Symfony DI

Explanation of how the codebase is split into granular packages, to enable CMS-agnosticism:

1. Abstracting WordPress Code To Reuse With Other CMSs: Concepts (Part 1)
2. Abstracting WordPress Code To Reuse With Other CMSs: Implementation (Part 2)

Description of how the plugin is scoped:

• Gato GraphQL is now scoped, thanks to PHP-Scoper!

GraphQL by PoP documentation

Gato GraphQL is powered by the CMS-agnostic GraphQL server GraphQL by PoP.

Technical information on how the GraphQL server works:

• GraphQL by PoP documentation.

Description of how a GraphQL server using server-side components works:

• Implementing a GraphQL server with components in PHP

These articles explain the concepts, design and implementation of GraphQL by PoP:

1. Designing a GraphQL server for optimal performance
2. Simplifying the GraphQL data model
3. Schema-first vs code-first development in GraphQL
4. Speeding-up changes to the GraphQL schema
5. Versioning fields in GraphQL
6. GraphQL directives are underrated
7. Treating GraphQL directives as middleware
8. Creating an @export GraphQL directive
9. Adding directives to the schema in code-first GraphQL servers
10. Coding a GraphQL server in JavaScript vs. WordPress
11. Supporting opt-in nested mutations in GraphQL
12. HTTP caching in GraphQL

Gutenberg

These articles explain the integration with Gutenberg (the WordPress editor).

1. Adding a Custom Welcome Guide to the WordPress Block Editor
2. Using Markdown and Localization in the WordPress Block Editor

Monorepo documentation

GatoGraphQL/GatoGraphQL is a monorepo containing the several layers required for Gato GraphQL. Check Monorepo_README.md for documentation of the different projects.

Standards

PSR-1, PSR-4 and PSR-12.

To check the coding standards via PHP CodeSniffer, run:

composer check-style

To automatically fix issues, run:

composer fix-style

Release notes

• 2.2 (current)
• 2.1
• 2.0
• 1.5
• 1.4
• 1.3
• 1.2
• 1.1
• 1.0
• 0.10
• 0.9
• 0.8
• 0.7
• 0.6

Change log

Please see CHANGELOG for more information on what has changed recently.

Testing

To execute PHPUnit, run:

composer test

Static Analysis

To execute PHPStan, run:

composer analyse

Downgrading code

To visualize how Rector will downgrade the code to PHP 7.2:

composer preview-code-downgrade

Report issues

To report a bug or request a new feature please do it on the GatoGraphQL monorepo issue tracker.

Contributing

We welcome contributions for this package on the GatoGraphQL monorepo (where the source code for this package is hosted).

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

Security

If you discover any security related issues, please email [email protected] instead of using the issue tracker.

Credits

• Leonardo Losoviz

License

GPLv2 or later. Please see License File for more information.

gatographql's Issues

🐢

Rename methods to follow camelCase for PSR-1

To add full support for PSR-1, all functions must be renamed to their camelCase equivalent. Eg: get_modules => getModules.

Service Workers => Offline first

Once the Service Workers are implemented #32, give the website an Offline first functionality:

• Load website from SW Cache
• Cache visited pages/tabs, offer to open them again to the user

Enable the Media Manager for cross-domain access

There is an issue doing a POST operation in the Media Manager for external domains: somehow plupload does not work, it never sends the cookies to the other domain, so then the PHP session is lost. As a consequence, for the time being, it is not allowed in PoP to upload media when creating a post or comment into another domain.

In theory (as explained here), plupload should work, by passing option "send_browser_cookies" under required_features in object _wpPluploadSettings.defaults (defined in file wp-includes/media.php), however, after trying multiple times, it never worked.

The .htaccess will need the following code:

<IfModule mod_headers.c>
<FilesMatch "(async-upload)\.php$">
    Header set Access-Control-Allow-Origin "*"
    Header add Access-Control-Allow-Headers Content-Type
    Header add Access-Control-Expose-Headers Content-Type
</FilesMatch>
</IfModule>

For GET operations it did work, even though we need to introduce a hack in file wp-includes/js/media-models.min.js: replace c.data.query=d,wp.media.ajax(c) with c.data.query=d,/* Hack PoP WP: Allow for cross-domain fetching of data */c.xhrFields={withCredentials:true},c.crossDomain=true,/* End PoP Hack WP */wp.media.ajax(c). In addition, we need to add the full (external) domain to ajaxurl.

We can also wait to see the progress of the project Gutenberg which will replace the TinyMCE in WordPress and, possibly too, the Media Manager.

Create alternate code for Gravity Forms

PoP relies on Gravity Forms (only in the back-end, front-end is all PoP), for the following forms:

• Contact us
• Contact user
• Volunteer
• Flag post as inappropriate
• Share by email
• Newsletter

If the developer doesn't have a Gravity Forms' license, then he/she'll be unable to use these functionalities. To fix this:

• Create a simple functionality to send emails for each of these forms
• If the developer does have GF, then the current GF functionality will take precedence

Create an app for PoP

Because all PoP websites understand the same JSON code, a single mobile app could serve pretty much all of them. Even more, once PoP becomes a standalone plug-in (#1), any existing Wordpress website, just by installing this plug-in, will be able to export its data to feed the mobile app, so a single mobile app could serve all Wordpress websites!

The app could have pre-defined channels or domains from which to read the data, and allow the user to input a new one.

Refactor global variables

The variables below must be migrated to not using global anymore:

• $pop_cssconverter_conversiongenerator
• $pop_cssconverter_conversionmanager
• $PoPFrontend_Initialization
• $pop_module_runtimecontentmanager
• $pop_serverside_kernelhelpers
• $pop_module_processordynamicdatadecorator_manager
• $counter
• $pop_resourcemoduledecoratorprocessor_manager
• $pop_jsresourceloaderprocessor_manager
• $popfrontend_resourceloader_scriptsandstyles_registration
• $popfrontend_resourceloader_initialization
• $pop_resourceloader_acrossthememodes_cssbundlefilegenerator
• $pop_resourceloader_acrossthememodes_cssbundlegroupfilegenerator
• $pop_resourceloader_acrossthememodes_jsbundlefilegenerator
• $pop_resourceloader_acrossthememodes_jsbundlegroupfilegenerator
• $pop_resourceloaderprocessor_manager
• $pop_resourceloader_mirrorcode_renderer
• $pop_resourceloader_singlethememode_cssbundlefilegenerator
• $pop_resourceloader_singlethememode_cssbundlegroupfilegenerator
• $pop_resourceloader_singlethememode_jsbundlefilegenerator
• $pop_resourceloader_singlethememode_jsbundlegroupfilegenerator
• $pop_resourceloader_allroutes_filegenerator_bundlefiles
• $pop_resourceloader_currentroute_filegenerator_bundlefiles
• $pop_resourceloader_generatedfilesmanager
• $pop_resourceloader_storagegenerator
• $pop_frontend_resourceloader_mappingparser
• $pop_frontend_resourceloader_mappinggenerator
• $pop_frontend_resourceloader_mappingmanager
• $pop_cssresourceloaderprocessor_manager
• $pop_resourceloader_mappingstoragemanager
• $pop_resourceloaderprocessor_state
• $pop_templateresourceloaderprocessor_manager
• $popfrontend_resourceloader_scriptsandstyles_manager
• $pop_resourceloader_generatedfilesfilelocation
• $pop_resourceloader_bundlemappingstoragefilelocation
• $pop_resourceloader_bundlegroupmappingstoragefilelocation
• $pop_resourceloader_keymappingstoragefilelocation
• $pop_resourceloader_bundleversionstoragefilelocation
• $pop_resourceloader_bundlegroupversionstoragefilelocation
• $pop_sparesourceloader_mappingstoragemanager
• $pop_sparesourceloader_hierarchyformatcombinationresources_configfile_renderer
• $pop_sparesourceloader_hierarchyformatcombinationresources_configfile_generator
• $pop_sparesourceloader_initialresources_configfile_renderer
• $pop_sparesourceloader_initialresources_configfile_generator
• $pop_sparesourceloader_resources_configfile_renderer
• $pop_sparesourceloader_resources_configfile_generator
• $pop_sparesourceloader_configfile_renderer
• $pop_sparesourceloader_configfile_generator
• $pop_sparesourceloader_resources_configfile_generator/*
• $pop_sparesourceloader_initialresources_configfile_generator*/
• $pop_sparesourceloaderprocessor_manager
• $pop_automatedemails_manager
• $pop_cdn_configfile_generator
• $pop_cdn_filerenderer
• $pop_cdn_thumbprint_manager
• $pop_multidomain_initdomainscripts_generator
• $pop_multidomain_initdomainscripts_filerenderer
• $pop_multidomain_resourceloader_filerenderer
• $pop_multidomain_resourceloader_configfile_generator
• $pop_resourceloader_initialresources_configfile_generator*/
• $pop_serviceworkers_manager
• $pop_serviceworkers_job_manager
• $pop_serviceworkers_frontend_resourceloader_scriptsandstyles_registration
• $pop_serverside_arrayshelpers
• $pop_serverside_comparehelpers
• $pop_serverside_datehelpers
• $pop_serverside_dbobjecthelpers
• $pop_serverside_feedbackmessagehelpers
• $pop_serverside_labelshelpers
• $pop_serverside_latestcounthelpers
• $pop_serverside_modhelpers
• $pop_serverside_multilayouthelpers
• $pop_serverside_operatorshelpers
• $pop_serverside_replacehelpers
• $pop_serverside_showmorehelpers
• $pop_serverside_statushelpers
• $pop_serverside_urlparamhelpers
• $pop_serverside_formatvaluehelpers
• $pop_serverside_formcomponentshelpers
• $popcore_userloggedinstyles_filegenerator
• $popcore_userloggedinstyles_filerenderer
• $popthemewassup_backgroundimage_filegenerator
• $popthemewassup_feedthumb_filegenerator
• $popthemewassup_backgroundimage_filerenderer
• $popthemewassup_feedthumb_filerenderer
• $popthemewassup_multidomainstyles_filerenderer
• $gd_theme_mesym

Faster initial load: clean .css, compress .js, bundle WP-loaded files together

PoP allows to create Single-Page Application websites, which have a drawback: plenty of files must be loaded initially, before the first page is even rendered.

PoP allows lazy-loading, so not everything needed is loaded initially. However, it does load 3 big files:

• 1 bundled&minified .css file
• 1 bundled&minified .js file with all javascript functions
• 1 bundled&minified .tmpl.js file with all Handlebars templates

Some ways to make it load faster:

• Clean all .css files
• Compress .js files thoroughly, eg: shortening internal variable names

Additionally, WP-loaded files have not been taken care of, and these are plenty of requests. These could, similarly, be all bundled together into 1 single .css and 1 single .js files

Don't cache Custom-Query API requests (with WP Super Cache, etc)?

The custom-query API allows to specify what data-fields to retrieve from the queried object through URL param fields. Certain data-fields, such as "edit-url", involve user state (in this case, the value of the added _nonce param depends on the logged-in user). Hence, if a page with data-field "edit-url" is requested, then this should not be cached, or it will expose sensitive information to all other users. However, this is an issue only for data-fields involving user state, which are a minority, and not the others. Hence, should these requests never be cached, or only when they include user-state data-fields, and if so how?

Allow for decentralized POST operations

Decentralized POST operations are currently not working if the plugins activated in the aggregator and aggregated websites are different. This is because of the definition of processors, coupled with the settings_id mangling for the processors, which creates a miscommunication between them.

Send email to network on user's activity

The user's network is composed by:

• Users following the user
• Users belonging to the same Organizations the user belongs to

All these users should receive a notification email for the following cases:

• When users in their network creates a new post

Add runtime version for function get_js_setting

Some settings are runtime, eg: the typeahead component's thumbprint, yet it is being cached in pop-cache

Consequence of #71 + #72: Make PoP a server-side implementation of blocks for the open web

After resolving #71, PoP will be able to act as the server-side implementation of Gutenberg blocks in WordPress. Because Gutenberg has also been adopted by Drupal, if we manage to resolve #72, then PoP can also act as a server-side implementation of blocks for Drupal! And through the separation of plugins into 2 entities, a generic one containing the bulk of the code, and a specific one for the CMS, most of the code will be reusable across platforms.

If Gutenberg is poised to be the editor for the open web, then PoP can become the component provider for the open web... exciting!

CMS-agnostic interfaces: decouple from WordPress function signatures?

Making functionality become CMS-agnostic is accomplished by splitting plugins into 2 entities: a generic one containing definitions of interfaces for all the required functionalities, and the implementation of the interfaces for the specific CMS.

For instance, plugin PoP CMS Model is a generic implementation, declaring all the functionalities it requires through interface PoP_CMSModel_CMS_FunctionAPI. This interface is implemented by plugin PoP CMS Model for WordPress through class PoP_CMSModelWP_FunctionAPI.

Then, all throughout the application, we must access the functionality through the interface and never directly through the CMS library. For instance, function get_posts, which is defined in the interface mentioned above, must be invoked like this:

$cmsapi = PoP_CMS_FunctionAPI_Factory::get_instance();
$query = array(
	// Query parameters here...
);
return $cmsapi->get_posts($query);

Currently, all these interfaces have the same function signatures as in WordPress. Not just the name of the function but, more importantly, also the parameters. For instance, function get_posts receives a $args parameter, which is an array with the following properties: 'numberposts', 'category', 'include', 'exclude', etc. PoP is currently expecting these same parameters! Should this be abstracted too?

Remove plug-in Aryo Activity Log, copy functionality from within

The notifications functionality is based on plug-in Aryo Activity Log, however this plug-in was not thought for notifications but for logging the user activity, so it had to be hacked.

It makes more sense to fork this plug-in instead, and create a new plug-in to be used for notifications and messaging.

Then, fix all the hacks:

• Fix all checking "VERSION" instead of "INITIALIZED" constants in all load.php files. (Otherwise, when disabling Aryo Activity Log plug-in, it generates a Server Error)

  if(defined('AAL_POP_VERSION')) => if(defined('AAL_POP_INITIALIZED'))
  if (defined('POP_USERAVATAR_VERSION')) => if (defined('POP_USERAVATAR_INITIALIZED'))
  if (defined('AAL_POPPROCESSORS_VERSION')) => if (defined('AAL_POPPROCESSORS_INITIALIZED'))

Create tutorials

For, at least, the following use cases:

• Lesson 1 - Hello World!
• Lesson 2 - Configuration
• Lesson 3 - Loading Database Data
• Lesson 4 - Loading Relational Objects
• Lesson 5 - Filters
• Lesson 6 - Executing Operations
• Lesson 7 - Forms
• Lesson 8 - Posting Data
• Lesson 9 - Validations and User State
• Lesson 10 - Preloading Views
• Lesson 11 - Multidomain
• Lesson 12 - Changing the Structure of the Response
• Lesson 13 - Filtering Modules
• Lesson 14 - Extending the ModuleProcessor with Decorators
• Lesson 15 - Loading Resources
• Lesson 16 - Bundling Resources
• Lesson 17 - Sending Transactional Emails
• Lesson 18 - A/B Testing

Clicking on "Log-in" before the loaders' request is back, breaks everything

When I clicked on link "Invite Users", before the initial-frames had loaded, its configuration overrid the configuration for initial-frames for modals target! So then, this page should not override? how does it work?

For sure, it overrid configuration classes => bodies, with value 'modal-body' for modal Invite new members, and it also produced a .js error.

I think it's enough if the configuration is still hosted under the other key, and not under its own URL

🐠

Add Virtual DOM library for creating highly-responsive elements using javascript?

Currently, we are able to produce HTML output from merging Handlebars templates in the frontend, using javascript. However, this may not be fast enough to support highly-responsive elements (eg: realtime preview of a formatted article, as it is being written in a text editor).

We could implement a diffing algorithm for DOM elements, which coupled with PoP's html creation could accomplish this functionality. In theory, it should work using the diffDOM library

Integrate the GraphQL syntax into GraphQL API for PoP

graphql-php parses the GraphQL query and renders the output. Either GraphQL API for PoP uses this library, or makes its own implementation to parse the GraphQL syntax.

Add namespaces all over

The PHP code is a mess, because there are more than one hundred plugins (not uploaded yet, but soon will), and their class names and interface names keep getting longer and longer. So we must implement namespacing to keep the names shorter and cleaner.

🐌

Make PoP CMS-agnostic

Because it was originally conceived for WordPress, PoP's current implementation is in PHP, which can be perfectly used for other PHP-based CMSs (such as Joomla or Drupal). To attain this goal, we have transformed to codebase to support multiple CMSs by splitting plugins into 2 entities: a generic implementation that should work for every CMS (eg: pop-engine) and a specific one for WordPress (eg: pop-engine-wp), so that only the latter one should be re-implemented for other CMSs.

Then, in the generic plugin we define what functions from a CMS are required, such as interface PoP_CMS_FunctionAPI, which is resolved by the corresponding WordPress implementation class PoP_CMSWP_FunctionAPI.

So PoP's architecture has already support for multiple CMSs. However, a deep assessment of this feature being feasible or not must still be carried out. For instance, plenty of code has been implemented following the WordPress architecture, such as basing the object model on posts, pages and custom post types, and we must make sure if these can also work in Drupal/Joomla/etc or if another level of abstraction must be added.

Implement autoloading

Follow https://www.php-fig.org/psr/psr-4/ and https://phpbestpractices.org/#auto-loading to add autoloading and remove all load.php files. It will require renaming folders and filenames following namespaces and class names, hence namespaces must be well thought of first, and renamed accordingly (#76)

Release standalone plug-ins for several functionalities

The PoP engine can run on any WordPress website, so it makes sense to provide standalone functionalities, independently of the website using a PoP theme or not. Eg:

• Comments
• Front-end editor
• Search/typeahead input
• Media player
• User notifications
• Messaging

Extract do_action and apply_filters for other CMSs

Copy the code for function do_action and apply_filters to PoP, so that the hooks functionality is available to PoP whichever the implementing CMS. These functions must be added like this, so they don't conflict with WordPress:

if (!function_exists('do_action')) :
    // Copy/paste code for function
endif;

These functions must be placed on a new plugin which must be loaded first, before any other plugin, since all plugins call function do_action when being initialized. How to guarantee a plugin to be the first loaded? Can the plugin loading order be defined in WordPress?

🐙

Implement Code Splitting

The idea is to initially load assets as small as they can be, containing only what is needed for that page (JS, CSS, Handlebars templates) and nothing more. This way, loading the first page will get a significant performance boost.

The challenge is how to make it work with the "previous session openTabs()" functionality, which allows the user to immediately open many more pages to the current one, possibly from pages cached using Service Workers. This implies that, by the time these pages from the Service Workers cache are loaded (which is pretty much immediately), their assets must also have been pre-loaded.

Can that be achieved using code-splitting? If so, will the code be manageable? (eg: importing all the Handlebars templates one per file is not pretty... it's normally nicer to import them as a bundle.) Does it make sense to implement it?

As a reference, WebPack is doing code-splitting

Create "User Preferences" page

Among others, a page "User Preferences" should allow users to configure what notification emails to receive (from users who belong to the user's network, eg: when they create a post, add a comment, etc).

Implement after #18

Decentralization: different plug-ins installed in different systems make them incompatible to use each other's data

When different plug-ins are installed, the names produced by function PoP_ServerUtils::get_template_definition will produce different results. Then, 2 different system will not be able to talk to each other.

Solution:

1. if get_dataloadsource_domain() array is not empty, output dictionary: block name => compressed name
2. add param ?referrer=external when doing fetchBlock with module=data (no need for clicking on pages, which will then load its own data)
3. seeing param referrer=external, output dictionary: block name => compressed name
4. from 2nd dictionary, from compressed name get block name, go to 1st dictionary, from block name (which is the same!) get compressed name

Implement Service Workers

Using Service Workers, it is possible to have the website pre-loaded on an empty page, and then retrieve the data for the requested URL, making the whole loading muuuuch faster

Make JSON requests immutable, access them through a CDN

After loading the initial app-shell (which loads all .js and .css files, and outputs some configuration), the website solely operates by doing JSON requests. This is done by simply adding parameter output=json to the URL to load, eg:

https://getpop.org/en/?output=json

This JSON output is mutable, since it contains dynamic content, such as posts, users' data, etc. However, it can be made immutable, or static, simply by adding a "thumbprint", i.e. a number to express the last time any value was inserted/updated/deleted (eg: a post was created, a comment was added, a user changed his/her name, etc), like this:

https://getpop.org/en/?output=json&thumbprint=3452223554222

The addition of the thumbprint allows the content to become static, because if content has been updated, a new "latest thumbprint" will be generated, and this new value will be added as a parameter to all JSON requests. In order to always get the latest thumbprint, a process in background must always get this value from the server, either every x amount of time (such as is currently done to fetch the notifications), or through WebSockets.

The thumbprint is hidden to the user: it is added in runtime, after the user has clicked on a link (this happens in functions fetchPageSection and executeFetchBlock). After the thumbprint has been added to the requested URL, the request can be routed through a CDN instead of going straight to the server. This way, dynamic content will have become static and delivered to the client from a location near the user:

1. The user clicks on https://getpop.org/en/
2. PoP modifies the URL to be requested, adding the thumbprint (and output=json), and changing the domain pointing to the CDN: https://cdn.getpop.org/en/?output=json&thumbprint=3452223554222
3. The CDN, if it doesn't have this entry, will fetch it from https://getpop.org/en/?output=json&thumbprint=3452223554222
4. The server gives the response
5. The CDN caches the response, and returns it to the client
6. Next time a user clicks on https://getpop.org/en/, if the thumbprint has not changed, then this content will be fetched from the CDN

Add documentation

For the following items, in whatever project they belong to:

• Choosing the Entry Module
• API Custom-Querying Capabilities
• Code Splitting
• Content CDN
• Single-Page Application
• Multiple URIs
• JavaScript as Progressing Enhancement
• Client-Side Rendering through Handlebars
• Executing JavaScript functions
• Isomorphic Server-Side Rendering
• Client-Side Layout Prefetch

Integrate with ZeroNet?

Read discussion below.

Note: previously this issue was about integration with React. But since the conversation on ZeroNet took over, I just renamed this issue.

Create Static Google Map before JS loads

Show a static Google Maps image before the javascript kicks in and draws the proper map.

Example static map:
https://maps.googleapis.com/maps/api/staticmap?center=Brooklyn+Bridge,New+York,NY&zoom=13&size=600x300&maptype=roadmap&markers=color:blue%7Clabel:S%7C40.702147,-74.015794&markers=color:green%7Clabel:G%7C40.711614,-74.012318&markers=color:red%7Clabel:C%7C40.718217,-73.998284&key=AIzaSyA9pdYHZ9jZdcdiqM9ttPdWqqnQrxHBWkM

Reference: https://developers.google.com/maps/documentation/static-maps/intro

Integrate with React?

The front-end has so-far been implemented with javascript and jQuery, however it can also be perfectly integrated with advanced JS frameworks. React, in particular, is designed to be only the View part of an MVC architecture, so it could fit perfectly within the PoP architecture.

It could be used for:

• Give bigger autonomy to modules. Eg: right now, only blocks are able to fetch data from the webserver, and modules are not independent from the block; using React, modules could communicate with the webserver under a different URL from that of the block, eg: "Recommend" button count

Multiple data sources for components

The decentralization feature will be complete only once any component is able to fetch its data from several sources concurrently. Eg: the events calendar should be able to fetch events, in real time, from many different websites.

Server-side configuration cache: Allow to inject Redis/Memcache instead of saving files to disk

Configuration can be cached on the server. Currently this is implemented by saving a file to disk for each configuration. It would be awesome to be able to inject a different implementation of the FileStorage, to save it under some memory-caching system such as Redis or Memcache instead of saving to disk.

🕷

Improve automation through scripts

PoP currently does not have scripts for automating:

• The installation process (eg: download dependencies)
• Creation of needed pages/categories (addressed in #38)
• Compilation of Handlebars .tmpl files
• Mangling/minifying .css/.js files

Help from anyone with experience with the right tools (Gulp? Grunt? Composer? WP CLI?) will be appreciated!

Email digests

Create the functionality to send daily email digests in the website, following these preferences added in #35:

• GD_METAKEY_PROFILE_EMAILDIGESTS_DAILYCONTENT
• GD_METAKEY_PROFILE_EMAILDIGESTS_BIWEEKLYUPCOMINGEVENTS
• GD_METAKEY_PROFILE_EMAILDIGESTS_DAILYNETWORKACTIVITY
• GD_METAKEY_PROFILE_EMAILDIGESTS_DAILYSUBSCRIBEDTOPICSACTIVITY

Make all code access functionality through the CMS-agnostic interfaces

All throughout the application, we must access the functionality through the interface and never directly through the CMS library. For instance, function get_posts, which is defined in interface PoP_CMSModel_CMS_FunctionAPI, must be invoked like this:

$cmsapi = PoP_CMS_FunctionAPI_Factory::get_instance();
$query = array(
	// Query parameters here...
);
return $cmsapi->get_posts($query);

So, we must find all instances of function get_posts invoked directly, and replace it with the code above. And similarly, for all other functions too.

Make Composer the Dependency Manager, and make WordPress simply a dependency?

Currently, PoP uses WordPress as its underlying CMS (it's the only implementation so far). WordPress itself doesn't use Composer by default, because of a philosophical question to which there is no agreement yet: Is WordPress the site, or simple a site dependency? For instance, the author of this blog post makes the case that WordPress is a dependency of the site, to which I agree.

Then, if WordPress can be considered a dependency, and similarly we consider PoP's CMS be a dependency whichever the CMS is (which makes a lot of sense), then we can justify making Composer the pre-established dependency manager for PoP, and we make it work with WordPress through composer.json file, which would add order to the project.

Replace all the Factories with Symfony's Dependency Injection Container

Replace all the factories, such as FunctionAPI_Factory, with the DependencyInjection component from Symfony

🐡

Bring forward the first meaningful paint

Make it quicker for users to see the website upon the initial load, as suggested in this article: https://developers.google.com/web/updates/2015/11/app-shell

Integration with bbPress for forums

Integrating with bbPress will allow to add user forums in PoP websites.

Make $query an interface Query

Deducing the hierarchy from the URL is done through interface's function query_is_hierarchy($query, $hierarchy). The signature of this function receives a $query object, which is based on the WordPress WP_Query class, and which is in charge of calculating the hierarchy from either the current URL or from the properties in $vars. Hence, an implementation of the $query object must be implemented for the other CMSs.

As such, it makes sense to create an interface Query to represent the instance $query, abstracting WP_Query in the WordPress implementation, and ready to be implemented by the other CMSs.

Create script to create pages/categories/etc needed for configuration

Most plug-ins need some configuration to be done, which involves creating pages (eg: POPTHEME_WASSUP_PAGE_ADDCONTENT) / categories (eg: POPTHEME_WASSUP_CAT_WEBPOSTS) / etc in the system, and then saving their IDs in a configuration file, such as environment-constants.php.

The creation of these is currently a manual process. It would be awesome to automate it, adding scripts to create the objects in the DB upon installation of the plug-in, and then automatically saving the IDs on the configuration file.

Add server-side html rendering

Implement .php templates, similar to the .tmpl templates, to execute the same logic and produce the same html.

The produced html can be used for:

• Generating fallback html for no-JS devices
• Generating email html to send in the email digests
• Generating email html for new functionality "Send post by email"