semanticmediawiki / semanticscribunto

Provides service functions to support the Scribunto extension

Semantic Scribunto

Semantic Scribunto (a.k.a. SSC) is a Semantic Mediawiki extension to provide native support for the Scribunto extension.

Requirements

• PHP 7.3 or later
• MediaWiki 1.31 or later
• Semantic MediaWiki 3.0 or later

Installation

The recommended way to install Semantic Scribunto is using Composer with MediaWiki's built-in support for Composer.

Note that the required extensions Semantic MediaWiki and Scribunto must be installed first according to the installation instructions provided.

Step 1

Change to the base directory of your MediaWiki installation. If you do not have a "composer.local.json" file yet, create one and add the following content to it:

{
	"require": {
		"mediawiki/semantic-scribunto": "~2.2"
	}
}

If you already have a "composer.local.json" file add the following line to the end of the "require" section in your file:

"mediawiki/semantic-scribunto": "~2.2"

Remember to add a comma to the end of the preceding line in this section.

Step 2

Run the following command in your shell:

php composer.phar update --no-dev

Note if you have Git installed on your system add the --prefer-source flag to the above command.

Step 3

Add the following line to the end of your "LocalSettings.php" file:

wfLoadExtension( 'SemanticScribunto' );

Usage

A description of the mw.smw library functions can be found here.

Contribution and support

If you want to contribute work to the project please subscribe to the developers mailing list and have a look at the contribution guideline.

• File an issue
• Submit a pull request
• Ask a question on the mailing list

Tests

This extension provides unit and integration tests that are run by a continues integration platform but can also be executed using the composer phpunit command from the extension base directory that will run all tests. In order to run only a specific test suit, the following commands are provided for convenience:

• composer unit to run all unit tests
• composer integration to run all integration tests (which requires an active MediaWiki, DB connection)

License

GNU General Public License, version 2 or later.

semanticscribunto's Issues

mw.smw.ask returns always false on printouts for category membership

Setup and configuration

• SMW version: 2.5.5
• SSC version: master
• MW version: 1.27
• DB (MySQL etc.): mysql

Issue

for a page PAGE in category member the query

local q = '[[PAGE]]|?Category|?Category:Member|?Category:INVALID'
mw.logObject(mw.smw.ask(q))

returns

table#1 {
 table#2 {
  "[[:PAGE]]",
  ["Category"] = "[[:Category:Member|Member]]",
  ["Member"] = false,
  ["INVALID"] = false,
 },
}

Make 2.0.0 release

Tracking issue for the 2.0.0 release.

• Release notes
• Changes since 1.2.0

• Finalize RELEASE NOTES
• Update "README.md"
• Update "/docs/"
• Update "composer.json"
• Update version number "extension.json"
• Create tag
• Add release to Wikidata
• Update homepage
• Announce via mail
• Do a release tweet on Twitter
• Run composer update on smw.o
• Update "LocalSettings.php"
• Start development of new version
  • Update "composer.json" (update dev alias)
  • Update "RELEASE NOTES" (start new section)
  • Update "extension.json" (new alpha version)

SemanticScribunto abort mediawiki installation

Setup and configuration

• SMW version: 2.4.6
• SSC version: 1.0.0
• MW version: 1.28
• DB (MySQL etc.): MySQL 5.6

Issue

Scribunto extension must be registered in LocalSettings.php, thus it is not available during installation process. Meanwhile during installation all composer managed extensions are loaded.

SemanticScribunto then complaints:

Error: Semantic Scribunto requires Scribunto, please enable or install the extension first

I do know this is a very minor issue, but simply adding a installer check can help us who may want to reinstall mw or those who work on multiple wikis.

Thanks!

Bubble "error" in mw.smw.info is broken

Setup and configuration

• SMW version: 3.2.2
• SSC version: 2.2.0
• MW version: 1.35.1
• DB (MySQL etc. and version): doesn't matter here

Issue

When mw.smw.info is called with error as its second parameter, instead of a red icon specified in the docs, a default info type icon appears.

That should be easy to fix, I'll submit a patch soon.

Tests are failing in travis

getQueryResult doesn't return entire table

Setup and configuration

• SMW version: 2.5.0.alpha, 2.4.0-rc1
• SemanticScribunto version: 1.0.0-alpha
• MW version: 1.26.3, 1.27.1
• DB (MySQL etc.): MySQL, MariaDB

Issue

getQueryResult returns only the first element of the result table rather than the entire table.

See http://sandbox.semantic-mediawiki.org/wiki/Utilisateur:Mathompson

put documentation on https://doc.semantic-mediawiki.org/

does this make any sense?

Make 2.2.0 release

• Finalize release notes
• Update readme
• Update "composer.json"
• Update "extension.json"
• Create tag
• Add release to Wikidata
• Update homepage
• Announce via mail
• Do a release tweet on Twitter
• Run composer update on smw.o
• Update "LocalSettings.php"
• Start development of new version
  • Update "composer.json" (update dev alias)
  • Update "extension.json" (new alpha version)

"mw.smw.ask" does not work for "format=count" queries

Setup and configuration

• SMW version: 3.1.6
• SSC version: 2.1.0
• MW version: 1.34.2
• DB version: MySQL 5.7.31-0ubuntu0.18.04.1

Issue

format=count" is a pseudo result format that requires special handling. Without this patch, "ask" returns nil. Performing a "getQueryResult" and looking at the row count is a possible workaround but it's incredibly inefficient for larger datasets.

Steps to reproduce the observation (recommendation is to use the sandbox):

Perform "mw.smw.ask" query with "format=count". It will alway return "nil".

Error using language filter with renamed column

Setup and configuration

• SMW version: 3.2.3
• SSC version: 2.1.0
• MW version: 1.35.5
• DB (MySQL etc. and version): 10.6.5 MariaDB

Issue

Using a language filter on a renamed column results in the language filter being appended to the renamed name.

Examples:

mw.smw.ask( { '[[-Has subobject::Example]][[Typ::+]]', '?Column|+lang=de' } )

Produces correct results, a table of tables containing only the german text for Column.

mw.smw.ask( { '[[-Has subobject::Example]][[Typ::+]]', '?Example=renamed_de|+lang=de'} )

Produces incorrect results:

  table#2 {
    "<span class=\"smw-subobject-entity\">[[...]]</span>",
    ["renamed_de|+lang=de"] = table#3 {
      "Entry (de)",
      "Entry (en)",
    },
  },

Am I calling mw.smw.ask wrong?

tests not up to date with changes in smw

some tests (unit and integration) are failing due to changes in smw

ask and getQueryResult fail on format=debug

Setup and configuration

• SMW version: 2.4.2
• SSC version: 1.0.0-alpha
• MW version: 1.27.1
• DB (MySQL etc.): mysql

Issue

Since \SMW\Scribunto\LibraryFactory::newQueryResultFrom does not return an object of class \SMWQueryResult on format=debug, function ask fails in \SMW\Scribunto\LuaAskResultProcessor::__construct

Stacktrace is:

1) LuaSandbox: SMW\Scribunto\Tests\ScribuntoLuaLibraryAskTest[5]: ask (format debug)
Argument 1 passed to SMW\Scribunto\LuaAskResultProcessor::__construct() must be an instance of SMWQueryResult, string given, called in /var/www/luasmw/mediawiki-1.27.1/extensions/SemanticScribunto.github/src/LibraryFactory.php on line 71 and defined

/var/www/luasmw/mediawiki-1.27.1/extensions/SemanticScribunto.github/src/LuaAskResultProcessor.php:48
/var/www/luasmw/mediawiki-1.27.1/extensions/SemanticScribunto.github/src/LibraryFactory.php:71
/var/www/luasmw/mediawiki-1.27.1/extensions/SemanticScribunto.github/src/ScribuntoLuaLibrary.php:66
/var/www/luasmw/mediawiki-1.27.1/extensions/Scribunto/engines/LuaSandbox/Engine.php:403
/var/www/luasmw/mediawiki-1.27.1/extensions/Scribunto/engines/LuaSandbox/Engine.php:316
/var/www/luasmw/mediawiki-1.27.1/extensions/Scribunto/tests/engines/LuaCommon/LuaDataProvider.php:49
/var/www/luasmw/mediawiki-1.27.1/extensions/Scribunto/tests/engines/LuaCommon/LuaEngineTestBase.php:244
/var/www/luasmw/mediawiki-1.27.1/tests/phpunit/MediaWikiTestCase.php:156

and getQueryResult fails in \SMW\Scribunto\ScribuntoLuaLibrary::getQueryResult on operation

  $result = $queryResult->toArray();

with

PHP Fatal error:  Call to a member function toArray() on string in /var/www/luasmw/mediawiki-1.27.1/extensions/SemanticScribunto.github/src/ScribuntoLuaLibrary.php on line 119

Reproduction

Steps to reproduce the observation (recommendation is to use the sandbox):

I added the tests

{ name = 'ask (format debug)', func = mw.smw.ask,
	args = { { '[[Modification date::+]]', '?Modification date', limit = 2, mainlabel = 'main', format = 'debug' } },
	expect = { nil }
},
{ name = 'getQueryResult (format debug)', func = mw.smw.getQueryResult,
	args = { { '[[Modification date::+]]', '?Modification date', limit = 2, mainlabel = 'main', format = 'debug' } },
	expect = { nil }
}

note

• Neither \SMW\Store::getQueryResult nor \SMW\Scribunto\LibraryFactory::newQueryResultFrom documents this possible behavior correctly. :(
• since the online test system has no data, the tests mentioned above cannot be reproduced there. When setting limit to 0, ask produces a null result whereas getQueryResult returns a correct table.

Remove composer dependencies to Scribunto in "composer.json"

Setup and configuration

• SMW version: 2.4
• SSO version: 1.0-dev
• MW version: 1.27.1
• DB (MySQL etc.): mysql

Issue

I ran the install as described in the installation manual. Note that with 1.27.1 composer is used for mediawiki core and stable is the mininum stability required by default. Consequently the install fails with

Your requirements could not be resolved to an installable set of packages.
Problem 1
  - The requested package mediawiki/semantic-scribunto ~1.0 is satisfiable by mediawiki/semantic-scribunto[1.x-dev] but these conflict with your requirements or minimum-stability.

Which hopefully will be solved when sso has its first stable release.

Running composer require mediawiki/semantic-scribunto:~1.0@dev let us get one step further but still fails with

Your requirements could not be resolved to an installable set of packages.
  Problem 1
    - Installation request for mediawiki/semantic-scribunto ~1.0@dev -> satisfiable by mediawiki/semantic-scribunto[1.x-dev].
    - mediawiki/semantic-scribunto 1.x-dev requires mediawiki/scribunto * -> satisfiable by mediawiki/scribunto[ ... ]  but these conflict with your requirements or minimum-stability.

Adding both

        "mediawiki/scribunto": "master@dev",
        "mediawiki/semantic-scribunto": "^1.0@dev"

to composer.local.json let me finally install sso. Which brings me to the core of this issue.

It is not so difficult for people to install scribunto and sso both manually and little gained in putting scribunto in sso's require list. In my case as described above I had to install it manually, even though I already downloaded and used it (only not with composer). Also, there might be more users like me trying to keep composer installed extensions at a mininum, so until now I installed Scribunto directly from its repository. [0]

Furthermore my line of thinking: SSO will be installed by users who want to bridge scribunto and smw making it very likely they have both extensions installed already. All functionaliy of SSO on the other hand is tied to the hook ScribuntoExternalLibraries. So in the unlikely case someone does not install Scribunto, SSO simply does nothing and should not produce any exceptions or errors.

Concluding: Could you please either remove the scribunto dependency or put in into the "suggest" section (maybe in require-dev?)?

Regards,
Tobi

[0]: long story being, that I have my mw software in a repository and installed all extensions via submodule. Excluded from that are the smw composer extensions which unfortunately always complicate merges, pulls, upgrades, etc...

Write test case to cover "... return results in correct order"

See #6

Integration tests with test data and lua scripts

It would be good to have online test data to run some more plausible tests with travis for mw.smw.ask and mw.smw.getQueryResult. Unfortunately, my knowledge and experience is verly limited in that regard, as I am just beginning to work with unittests (who would have guessed ;) ).

My approach would be to have install-semantic-scribunto.sh import some xml files and run the maintenance scripts once, but I'm guessing there is a better way.

Make 1.2.0 release

• Finalize release notes
• Update readme
• Update composer.json (remove SMW 3.0)
• Update version number (1.2.0)
  • SemanticScribunto.php
• Create tag
• Announce via mail
• Do a release tweet on Twitter

1.0.0 release

• Finalize release notes
• Update readme - looks good as is since we do the initial release
• Update version number (1.0.0)
  • SemanticScribunto.php
• Create tag
• Announce via mail
• Do a release tweet on Twitter
• Shortcut should be SSC rather than SSO
• Extension author update
• Examples on sandbox - done according to #11 (comment)
• Add extension to code documentation generation (s-mw.o - issue 8)

Get i18n rolling

• add repo to user group with push rights
• review en.json and qqq.json
• ping twn to add the repo
• affirmation by twn that support was added

Object of class LuaSandboxFunction could not be converted to string when passing iterators into SMW objects

Setup and configuration

• SMW version: 3.1.0
• SSC version: 2.1.0
• MW version: 1.31.5
• DB (MySQL etc. and version): 10.4.10-MariaDB-1:10.4.10+maria~buster
• 7.3.11-1+0~20191026.48+debian9~1.gbpf71ca0 (fpm-fcgi)

Issue

When people create modules that save bad values into Semantic Objects, the calling page errors out as HTTP error 500. In the example someone tried to save an iterator into a property, which obviously does not work, getting an error 500 instead of an error message was not very obvious to them though. Returning an error message instead of a hard error 500 would probably be advisable.

Produces a stack trace and outputs:

PHP Recoverable fatal error:  Object of class LuaSandboxFunction could not be converted to string in /extensions/SemanticScribunto/src/ScribuntoLuaLibrary.php on line 363

Steps to reproduce the observation (recommendation is to use the sandbox):

Example Module:

local p = {}
function p.t()
	local dataStore = {}
	dataStore['some property'] = mw.text.gsplit('some','strings')
	mw.smw.set(dataStore)
	-- mw.smw.subobject(dataStore) -- this produces the same result
end
return p

and the module would simply be called as {{#invoke:Module name|t}}

Given it is easy to work around this by not passing broken things in, I do not think this is super urgent, but it might be a nice to have to catch such errors.

API query does not work for wikis that require login to read

Setup and configuration

• SMW version: 2.4.1
• SSO version: 1.0
• MW version: 1.26
• DB (MySQL etc.): MySQL

Issue

SemanticScribunto uses the ask API to make the query. This does not work when the wiki requires login to read. SemanticScribunto could either go through the login flow to make the API request (probably more trouble than it is worth) or make the query directly in PHP without using the API.

[0d9efd17] [no req] UsageException from line 1368 of .../includes/api/ApiBase.php: You need read permission to use this module

Backtrace

#0 .../includes/api/ApiBase.php(1956): ApiBase->dieUsage(string, string)
#1 .../includes/api/ApiMain.php(1180): ApiBase->dieUsageMsg(string)
#2 .../includes/api/ApiMain.php(1251): ApiMain->checkExecutePermissions(SMW\MediaWiki\Api\Ask)
#3 .../includes/api/ApiMain.php(436): ApiMain->executeAction()
#4 .../extensions/SemanticScribunto/src/ScribuntoLuaLibrary.php(57): ApiMain->execute()
#5 [internal function]: SMW\Scribunto\ScribuntoLuaLibrary->getQueryResult(string)

Steps to reproduce the observation (recommendation is to use the sandbox):

Set the following in the wiki configuration file:

$wgGroupPermissions['*']['read'] = false;

Attempt to use SemanticScribunto.

ask tests failing due to changes in mw master

ref #49

getQueryResult returns values in wrong order

Setup and configuration

• SMW version: 2.5.0.alpha, 2.4.0-rc1
• SemanticScribunto version: 1.0.0-alpha
• MW version: 1.26.3, 1.27.1
• DB (MySQL etc.): MySQL, MariaDB

Issue

When a sort parameter is given to getQueryResult, it returns the results out-of-order.

See http://sandbox.semantic-mediawiki.org/wiki/Utilisateur:Mathompson

Decimal number formats should be LUA compatible

Setup and configuration

• SMW version: 4.1.1
• SSC version: 2.1
• MW version: 1.39.3

Issue

Queries using the #-n format result in string type numbers in LUA.

Example:
Value with unit: 1,509 Minuten
Value returned with #-n: 1,509
Expected value: 1.509

While the returned value is "correct" with regards to the content language, trying to use this value directly in LUA results in Arithmetic on String errors, as the , changes the type to string.

Ideally the SSC should query SMW with en as its content language.

Use IncomingSemanticDataValidator instead of `assertInProperties`

SemanticScribuntoJsonTestCaseScriptRunnerTest to replace assertInProperties with IncomingSemanticDataValidator.

refs #29, SemanticMediaWiki/SemanticMediaWiki#2161

mw.smw.set should recognize multiple values for one property given in a sub-table

Setup and configuration

• SMW version: 2.5.6
• SSC version: 1.1.1
• MW version: 1.30
• DB (MySQL etc.): 10.0.34-MariaDB

Issue

mw.smw.set() allows setting multiple values for the same property by having arguments like

table#1 { 
  "prop=value", 
  "prop=value2", 
}

It would be great if it could handle arguments of the form

table#1 { 
  ["prop"] = table#2 { 
    "value",
    "value2", 
  }, 
}

(This is also the format returned by mw.smw.ask() when the result has multiple values for one property.)

Subobject parameter @category cannot be set

Setup and configuration

• SMW version: 3.1.6
• SSC version: 2.1.0
• MW version: 1.34.1
• DB (MySQL etc. and version): MySQL

Issue

The category of subobjects cannot be set using the Scribunto function mw.smw.set.
It should be supported in the same way it is possible to define categories with the subobject template.

{{#subobject: |Has page=Example/0301/1 |@category=ABC }}

mw.set drops property names with numbers in key-value mode

Setup and configuration

• SMW version: 2.2.5
• SSC version: 1.0.0
• MW version: 1.30.0
• DB (MySQL etc.): MySQL

Issue

When setting properties by using key-value form, if the "key" part contains numbers, then the property will not be accepted (more like simply get ignored).

Steps to reproduce the observation (recommendation is to use the sandbox):
Using following table to setup properties. (source)

local settings = {
	['hello'] = 'world',
	['hell0'] = 'world',
	'HELLO=WORLD',
	'HELL0=WORLD',
}

Only ['hell0'] = 'world' doesn't show up on browse page.

Make 1.1.0 release

Isn't it time to switch from 1.1.0-alpha to 1.1.1?

Make 2.1.0 release

• Finalize release notes
• Update readme
• Update "composer.json"
• Update "extension.json"
• Create tag
• Add release to Wikidata
• Update homepage
• Announce via mail
• Do a release tweet on Twitter
• Run composer update on smw.o
• Update "LocalSettings.php"
• Start development of new version
  • Update "composer.json" (update dev alias)
  • Update "extension.json" (new alpha version)

Replace deprecated Hooks::register / Hooks::isRegistered / Hooks::clear

Issue

The functions Hooks::register / Hooks::isRegistered / Hooks::clear are soft-deprecated and going to emit warnings, see
https://phabricator.wikimedia.org/T335536

Error: 1137 Can't reopen table: 'p' ()

Setup and configuration

• SMW version: all
• SSC version: all
• MW version: all
• DB (MySQL etc. and version): MySQL+

Issue

Query: SELECT p.smw_title AS prop, o_id AS id0, o0.smw_title AS v0, o0.smw_namespace AS v1, o0.smw_iw AS v2, o0.smw_sortkey AS v3, o0.smw_subobject AS v4, o0.smw_sort AS v5 FROM `unittest_smw_di_wikipage` INNER JOIN `unittest_smw_object_ids` AS p ON p_id=p.smw_id INNER JOIN `unittest_smw_object_ids` AS o0 ON o_id=o0.smw_id WHERE (s_id='29') AND (p.smw_iw!=':smw') AND (p.smw_iw!=':smw-delete')
Function: SMW\SQLStore\EntityStore\SemanticDataLookup::fetchSemanticDataFromTable
Error: 1137 Can't reopen table: 'p' ()

When running tests, MW's default behaviour (we have to rely on Scribunto_LuaEngineTestBase for that matter) is to clone tables as TEMPORARY TABLE which isn't an issue with MariaDB but when used with MySQL as in case of Travis-CI, tests run into Error: 1137 which translates into "You cannot refer to a TEMPORARY table more than once in the same query" [0].

The reason is that a table like unittest_smw_object_ids is created as TEMPORARY TABLE with p being another TEMP reference to a TEMPORARY TABLE which isn't allowed in MySQL.

[0] https://dev.mysql.com/doc/refman/8.0/en/temporary-table-problems.html