phpmyadmin / motranslator

Translation API for PHP using Gettext MO files

motranslator

Translation API for PHP using Gettext MO files.

Features

• All strings are stored in memory for fast lookup
• Fast loading of MO files
• Low level API for reading MO files
• Emulation of Gettext API
• No use of eval() for plural equation

Limitations

• Default InMemoryCache not suitable for huge MO files which you don't want to store in memory
• Input and output encoding has to match (preferably UTF-8)

Installation

Please use Composer to install:

composer require phpmyadmin/motranslator

Documentation

The API documentation is available at https://develdocs.phpmyadmin.net/motranslator/.

Object API usage

// Create loader object
$loader = new PhpMyAdmin\MoTranslator\Loader();

// Set locale
$loader->setlocale('cs');

// Set default text domain
$loader->textdomain('domain');

// Set path where to look for a domain
$loader->bindtextdomain('domain', __DIR__ . '/data/locale/');

// Get translator
$translator = $loader->getTranslator();

// Now you can use Translator API (see below)

Low level API usage

// Directly load the mo file
// You can use null to not load a file and the use a setter to set the translations
$cache = new PhpMyAdmin\MoTranslator\Cache\InMemoryCache(new PhpMyAdmin\MoTranslator\MoParser('./path/to/file.mo'));
$translator = new PhpMyAdmin\MoTranslator\Translator($cache);

// Now you can use Translator API (see below)

Translator API usage

// Translate string
echo $translator->gettext('String');

// Translate plural string
echo $translator->ngettext('String', 'Plural string', $count);

// Translate string with context
echo $translator->pgettext('Context', 'String');

// Translate plural string with context
echo $translator->npgettext('Context', 'String', 'Plural string', $count);

// Get the translations
echo $translator->getTranslations();

// All getters and setters below are more to be used if you are using a manual loading mode
// Example: $translator = new PhpMyAdmin\MoTranslator\Translator(null);

// Set a translation
echo $translator->setTranslation('Test', 'Translation for "Test" key');

// Set translations
echo $translator->setTranslations([
  'Test' => 'Translation for "Test" key',
  'Test 2' => 'Translation for "Test 2" key',
]);

// Use the translation
echo $translator->gettext('Test 2'); // -> Translation for "Test 2" key

Gettext compatibility usage

// Load compatibility layer
PhpMyAdmin\MoTranslator\Loader::loadFunctions();

// Configure
_setlocale(LC_MESSAGES, 'cs');
_textdomain('phpmyadmin');
_bindtextdomain('phpmyadmin', __DIR__ . '/data/locale/');
_bind_textdomain_codeset('phpmyadmin', 'UTF-8');

// Use functions
echo _gettext('Type');
echo __('Type');

// It also support other Gettext functions
_dnpgettext($domain, $msgctxt, $msgid, $msgidPlural, $number);
_dngettext($domain, $msgid, $msgidPlural, $number);
_npgettext($msgctxt, $msgid, $msgidPlural, $number);
_ngettext($msgid, $msgidPlural, $number);
_dpgettext($domain, $msgctxt, $msgid);
_dgettext($domain, $msgid);
_pgettext($msgctxt, $msgid);

Using APCu-backed cache

If you have the APCu extension installed you can use it for storing the translation cache. The .mo file will then only be loaded once and all processes will share the same cache, reducing memory usage and resulting in performance comparable to the native gettext extension.

If you are using Loader, pass it an ApcuCacheFactory before getting the translator instance:

PhpMyAdmin\MoTranslator\Loader::setCacheFactory(
    new PhpMyAdmin\MoTranslator\Cache\AcpuCacheFactory()
);
$loader = new PhpMyAdmin\MoTranslator\Loader();

// Proceed as before 

If you are using the low level API, instantiate the ApcuCache directly:

$cache = new PhpMyAdmin\MoTranslator\Cache\ApcuCache(
    new PhpMyAdmin\MoTranslator\MoParser('./path/to/file.mo'),
    'de_DE',     // the locale
    'phpmyadmin' // the domain
);
$translator = new PhpMyAdmin\MoTranslator\Translator($cache);

// Proceed as before

By default, APCu will cache the translations until next server restart and prefix the cache entries with mo_ to avoid clashes with other cache entries. You can control this behaviour by passing $ttl and $prefix arguments, either to the ApcuCacheFactory or when instantiating ApcuCache:

PhpMyAdmin\MoTranslator\Loader::setCacheFactory(
    new PhpMyAdmin\MoTranslator\Cache\AcpuCacheFactory(
        3600,     // cache for 1 hour
        true,     // reload on cache miss
        'custom_' // custom prefix for cache entries
    )
);
$loader = new PhpMyAdmin\MoTranslator\Loader();

// or...

$cache = new PhpMyAdmin\MoTranslator\Cache\ApcuCache(
    new PhpMyAdmin\MoTranslator\MoParser('./path/to/file.mo'),
    'de_DE',
    'phpmyadmin',
    3600,     // cache for 1 hour
    true,     // reload on cache miss
    'custom_' // custom prefix for cache entries
);
$translator = new PhpMyAdmin\MoTranslator\Translator($cache);

If you receive updated translation files you can load them without restarting the server using the low-level API:

$parser = new PhpMyAdmin\MoTranslator\MoParser('./path/to/file.mo');
$cache = new PhpMyAdmin\MoTranslator\Cache\ApcuCache($parser, 'de_DE', 'phpmyadmin');
$parser->parseIntoCache($cache);

You should ensure APCu has enough memory to store all your translations, along with any other entries you use it for. If an entry is evicted from cache, the .mo file will be re-parsed, impacting performance. See the apc.shm_size and apc.shm_segments documentation and monitor cache usage when first rolling out.

If your .mo files are missing lots of translations, the first time a missing entry is requested the .mo file will be re-parsed. Again, this will impact performance until all the missing entries are hit once. You can turn off this behaviour by setting the $reloadOnMiss argument to false. If you do this it is critical that APCu has enough memory, or users will see untranslated text when entries are evicted.

History

This library is based on php-gettext. It adds some performance improvements and ability to install using Composer.

Motivation

Motivation for this library includes:

• The php-gettext library is not maintained anymore
• It doesn't work with recent PHP version (phpMyAdmin has patched version)
• It relies on eval() function for plural equations what can have severe security implications, see CVE-2016-6175
• It's not possible to install it using Composer
• There was place for performance improvements in the library

Why not to use native gettext in PHP?

We've tried that, but it's not a viable solution:

• You can not use locales not known to system, what is something you can not control from web application. This gets even more tricky with minimalist virtualisation containers.
• Changing the MO file usually leads to PHP segmentation fault. It (or rather Gettext library) caches headers of MO file and if it's content is changed (for example new version is uploaded to server) it tries to access new data with old references. This is bug known for ages: https://bugs.php.net/bug.php?id=45943

Why use Gettext and not JSON, YAML or whatever?

We want translators to be able to use their favorite tools and we want us to be able to use wide range of tools available with Gettext as well such as web based translation using Weblate. Using custom format usually adds another barrier for translators and we want to make it easy for them to contribute.

motranslator's Issues

Include twig support

Can you add the following files (from phpMyAdmin) to provide twig support:

• Twig\I18nExtension
• Twig\I18n\TokenParserTrans
• Twig\I18n\NodeTrans

In my local test this works perfectly (using my namespace)

$twig->addExtension(new PhpMyAdmin\Twig\I18nExtension());

Number format may be changed inadvertently

At \PhpMyAdmin\MoTranslator\Loader::setlocale() we can find this:

        $this->locale = $locale;
        // Set system locales as well
        if (function_exists('setlocale')) {
            setlocale(0, $locale);
        }

In my Windows box with the official PHP/5.6.21 binaries (php-5.6.21-Win32-VC11-x86) the LC_... constant that has zero as value is LC_ALL. This basically breaks all code that assumes that numbers gets serialised in locale-independent way, e.g.:

/* @var SimpleXMLElement $xml */
/* @var float $amount */
$xml->addChild('amount', $amount); // Implicit cast to string produces: <amount>3,14</amount>

setlocale() should be called with constants rather than plain integers.

Helper script and sample output:

<?php
echo php_uname(), PHP_EOL;
echo PHP_VERSION, PHP_EOL;
$constants = get_defined_constants(true);
asort($constants['standard']);
foreach ($constants['standard'] as $name => $value) {
	if (mb_substr($name, 0, 3) === 'LC_') {
		printf('% -11s = %s', $name, $value); echo PHP_EOL;
	}
}

Windows NT SOFT2121B 6.2 build 9200 (Unknow Windows version Business Edition) i586
5.3.28
LC_ALL      = 0
LC_COLLATE  = 1
LC_CTYPE    = 2
LC_MONETARY = 3
LC_NUMERIC  = 4
LC_TIME     = 5

Linux SOFT2121b 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64
5.5.9-1ubuntu4.21
LC_CTYPE    = 0
LC_NUMERIC  = 1
LC_TIME     = 2
LC_COLLATE  = 3
LC_MONETARY = 4
LC_MESSAGES = 5
LC_ALL      = 6

Remove use of eval()

For gettext we need something what is able to evaluate it's plural expressions and not use eval() (see phpmyadmin/phpmyadmin#6363), unfortunately I was not able to find single arithmetic expression evaluation library for PHP which supports ternary operator, but they tend to support dozen of other stuff we don't need like functions. Therefore I think it will be better to implement own code for this particular use case. The final goal is to remove eval usage.

Example expression we need to evaluate for any arbitrary value of n: n==0 ? 0 : n==1 ? 1 : n==2 ? 2 : n%100>=3 && n%100<=10 ? 3 : n%100>=11 ? 4 : 5

The syntax is quite limited:

• n variable
• ternary operator ? :
• comparisons == < > <= => !=
• division and modulo / %
• numbers

PS: More plural examples can be found in the Localization Guide.

PHP Notice in extractPluralCount function

Run:

include 'vendor/autoload.php';
$loader = new PhpMyAdmin\MoTranslator\Translator('test.mo');
$loader->extractPluralCount('nplurals');

result:

PHP Notice:  Undefined offset: 1 in .../vendor/phpmyadmin/motranslator/src/Translator.php on line 200

Tests fail on i386

https://salsa.debian.org/phpmyadmin-team/motranslator/-/jobs/1168052#L1509

1) PhpMyAdmin\MoTranslator\Tests\MoFilesTest::testEmptyMoFile with data set #4 ('./tests/data/error/fpdle.mo')
TypeError: Argument 3 passed to PhpMyAdmin\MoTranslator\StringReader::readintarray() must be of the type int,
 float given, called in src/Translator.php on line 140

motranslator/src/Translator.php

Line 140 in 392db58

$tableOriginals = $stream->readintarray($unpack, $originals, $total * 2);

motranslator/src/StringReader.php

Line 101 in 392db58

return unpack($unpack . $count, $this->read($pos, 4 * $count));

int $number parameter marked as string

The selectString function receive $n as int

param int $n count of objects

while the $number parameter in ngettext & npgettext functions are marked as string

param string $number Number of objects

while it's passed to selectString function.

fix:
change description of $number to int.

Bad usage of global functions.

Cannot redeclare __() (previously declared in /var/www/site/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php:967

This package does not work with Laravel, and possible others because of the usage of global functions.

It would be great if you would wrap all the global functions in an if (!function_exists()) call to ensure this does not happen.

PHP Notice in ngettext function

in https://github.com/phpmyadmin/motranslator/blob/master/src/Translator.php#L290:

        return $list[$select];

Run:

<?php
include 'vendor/autoload.php';
$t=new PhpMyAdmin\MoTranslator\Translator('2.mo');
print_r($t);
print_r($t->ngettext(chr(90).chr(4).chr(88),chr(67).chr(0).chr(68),8));
?>

Result:

PHP Notice:  Undefined offset: 16 in /vendor/phpmyadmin/motranslator/src/Translator.php on line 290

2.mo file was generated by nodejs script:

var gettextParser = require("gettext-parser");

d = {
  "headers": {
    "plural-forms": "nplurals=20; plural=(n*2);"
  },
 
  "translations": {
    "Z": {
      "X": {
        "msgid_plural": "C\x00D",
      }
    }
  }
}

var output = gettextParser.mo.compile(d);
var fs = require('fs');
fs.writeFile("2.mo", output, function(err) {
    if(err) {
        return console.log(err);
    }

    console.log("The file was saved!");
});

Get rid of eval() call

Hey,
Kindly help me, let me know how to get started ?

use single quotes instead of double quotes when possible

in preg_match call at:
https://github.com/phpmyadmin/motranslator/blob/master/src/Loader.php#L105
double quotes can be replaced by single quotes.

in other instances (array_push calls , $filename variable) it might be replaced (which improve performance) but will reduce legibility, more information:
https://stackoverflow.com/questions/3316060/single-quotes-or-double-quotes-for-variable-concatenation#3316126

PHP type error - Translator::selectString() must be of the type integer, boolean returned

 PhpMyAdmin\Tests\LanguageTest::testGettext with data set #3 ('en_GB')
TypeError: Return value of PhpMyAdmin\MoTranslator\Translator::selectString() must be of the type integer, boolean returned

https://travis-ci.org/github/phpmyadmin/phpmyadmin/jobs/744078077#L690

Possible incorrect dependencies: PHP/5.3.0 does not seem to be supported

I suspect that motranslator requirements don't consider dependencies properly:

"require": {
    "php": ">=5.3.0",
    "symfony/expression-language": "^4.0 || ^3.2 || ^2.8"
},

… but oldest supported expression-language requirements say:

"require": {
    "php": ">=5.3.9"
},

So there's no way to install with Composer under PHP versions earlier than 5.3.9.

(expression-language bumped PHP version from 5.3.3 to 5.3.9 in v2.7.0-BETA1, not idea why.)

trim usage when ltrim/rtrim can be used instead

1. extractPluralCount function in:

if (strtolower(trim($nplurals[0])) != 'nplurals') {

trim can be replaced with rtrim (because previous call to trim on $parts[0] get rid of spaces from beginning of the string)

2. sanitizePluralExpression function in

$expr = trim(substr($expr, 6));
$expr = trim(substr($expr, 1));

trim can be replaced by ltrim (because previous call to trim on strtolower($expr) get rid of spaces from end of the string)

add stripping of disallowed chars in sanitizePluralExpression

consider adding:

$expr = preg_replace('@[^n0-9:\(\)\?=!<>/%&|]@', '', $expr); 

as was happened before: a3ceb85#diff-cd35f60a79ff75867f0d5cbd2e9fbcf2L175
to restrict which chars can be passed to symfony/expression-language which allow usage of constant function to retrieve constant data, and regex usage via matches (both may be used together to leak constant data)
https://symfony.com/doc/current/components/expression_language/syntax.html

fpd on 32bit via negative result of unpack

unpack function on PHP that run in 32bit mode can return negative result as noted by the documentation: https://secure.php.net/manual/en/function.unpack.php#refsect1-function.unpack-notes

Note that PHP internally stores integral values as signed. If you unpack a large unsigned long and it is of the same size as PHP internally stored values the result will be a negative number even though unsigned unpacking was specified.

thus can result in PHP Warning.
example: (run under PHP 32bit)

<?php
$negative32 = pack('CCCC', 0xFF, 0xFF, 0xFF, 0xFF);
file_put_contents('fpdle.mo',"\xde\x12\x04\x95".str_repeat($negative32,6));

include 'vendor/autoload.php';
$t=new PhpMyAdmin\MoTranslator\Translator('fpdle.mo');
print_r($t);
?>

result:

PHP Warning:  unpack(): Type V: not enough input, need 4, have 0 in /vendor/phpmyadmin/motranslator/src/StringReader.php on line 88
PHP Warning:  unpack(): Type V: not enough input, need 4, have 0 in /vendor/phpmyadmin/motranslator/src/StringReader.php on line 88

Switching locale during one process run

Hi there,

I have and issue where I need to be able to switch locale for the domain during the same process run. Reading through the code, I found that Loader instance is memoizing Translator instances into domains variable under only domain key. So when I switch to another locale, it does not instantiate new Translator instance for the same domain.

Does it make sense to add locale to the keys for Translators. Either as sub-array or change the keys to have both domain and locale as one key (e.g messages_en_US) ?

>= usage instead of > in read function

in read function:

if ($pos + $bytes >= $this->_len) {
            throw new ReaderException('Not enough bytes!');
        }

>= should be replaced by > (there will be enough bytes if $pos + $bytes equal to $this->_len)

Replace apigen with Sami in CI

This has been already done on our server (see phpmyadmin/scripts#11), so it doesn't make sense to test documentation build with apigen.

If Sami supports some kind of lint mode, it should be enabled in the CI.

on the readme.md Gettext compatability usage add these

I might be a good thing to add these under the Gettext compatibility usage so people know they have been replicated.

// Other Functions
_dnpgettext($domain, $msgctxt, $msgid, $msgidPlural, $number)
_dngettext($domain, $msgid, $msgidPlural, $number)
_npgettext($msgctxt, $msgid, $msgidPlural, $number)
_ngettext($msgid, $msgidPlural, $number)
_dpgettext($domain, $msgctxt, $msgid)
_dgettext($domain, $msgid)
_pgettext($msgctxt, $msgid)

PHP Notice in getPluralForms function

Run:

<?php
$negative32 = pack('CCCC', 0xFF, 0xFF, 0xFF, 0xFF);
$total = pack('V',8);
$dummy = pack('V',2);
$originals = pack('V',1); // total
$translations = pack('V',4);
file_put_contents('fpd1.mo',"\xde\x12\x04\x95".$total.$originals.$translations.$translations.'dddddddddd');

include 'vendor/autoload.php';
$t=new PhpMyAdmin\MoTranslator\Translator('fpd1.mo');
print_r($t);
print_r($t->ngettext(chr(18).chr(4).chr(149).chr(8),chr(0).chr(0).chr(1),1));
?>

Result:

PHP Notice:  Undefined index:  in /vendor/phpmyadmin/motranslator/src/Translator.php on line 235

Change $cache_translations to protected or add method setCachedTranslations(array)

Hi!

I am trying to implement in-memory cache for motranslator, but I noticed I have to loop the whole array and call setTranslation() for each msgid. Maybe you could simplify it by changing cache_translations to protected or by adding another method for setting all translations at once?

class CachedTranslator extends Translator
{
	/**
	 * @param string $filename
	 */
	public function __construct($filename)
	{
		// TODO: read from cache, set $cached_translations and return or let it continue with reading the file on cache miss.
		parent::__construct($filename);
	}
}

Thank you!

Dummy _bind_textdomain_codeset implementation

The implementation of _bind_textdomain_codeset is:

/**
 * Dummy compatibility function, MoTranslator assumes
 * everything is UTF-8
 *
 * @return void
 */
function _bind_textdomain_codeset($domain, $codeset) {
    return;
}

in documentation (under Limitations):

Input and output encoding has to match (preferably UTF-8)

fix:
1. change the documentation (if non UTF-8 isn't supported) or change comment before _bind_textdomain_codeset function (assume input & output encoding are the same)
2. throw exception (NotImplemented) in _bind_textdomain_codeset function if:

		strtolower($codeset) !== 'utf-8'

as the code will do nothing instead of changing codeset.

PHP Fatal error: Uncaught Symfony\Component\ExpressionLanguage\SyntaxError

There missing catching of exception caused by incorrect input to Symfony evaluate method, example:
Run:

<?php
include 'vendor/autoload.php';
$t=new PhpMyAdmin\MoTranslator\Translator('2.mo');
print_r($t);
print_r($t->ngettext(chr(90).chr(4).chr(88),chr(67).chr(0).chr(68),8));

Result:

PHP Fatal error:  Uncaught Symfony\Component\ExpressionLanguage\SyntaxError: Unclosed "(" around position 0 for expression `(n*2`. in /vendor/symfony/expression-language/Lexer.php:98

2.mo file was generated by nodejs script:

var gettextParser = require("gettext-parser");
d = {
  "headers": {
    "plural-forms": "nplurals=20; plural=(n*2;"
  },
 
  "translations": {
    "Z": {
      "X": {
        "msgid_plural": "C\x00D",
      }
    }
  }
}

var output = gettextParser.mo.compile(d);
var fs = require('fs');
fs.writeFile("2.mo", output, function(err) {
    if(err) {
        return console.log(err);
    }

    console.log("The file was saved!");
});