php-kafka / php-avro-schema-generator Goto Github PK

Adapt the token parser to support the ReflectionUnionType that has been introduced in PHP:8.0
Might be obsolete after #26

Right now the export also happens in the schema merger
Having dedicated exporters will give more flexibility in the future.

Right now only string values are supported, add support for:

• empty string
• numbers
• null

Now most interesting for me to define optional (nullable fields), so, instead of just:

"type": "string",

Allow to set something like:

"type" : [ "null", "string" ],
"default": null

But there are also more interesting cases when e.g. instead of just int I would like to use logicalType date.

Please allow providing some functional interface to override fixed mechanism by providing an array of mappers, or just callable.

It was discovered (@bajdzun ), that the SR does not include namespaces of subtypes when they are in the same space.
To cover this a new option --optimizeSubSchemaNamespaces was added (#5, #10) to make this possible without breaking the current behaviour.
Upon further investigation it seems this is part of the Avro specification and should be default behaviour, see discussion in (#12).
This will be properly addressed and will result in a major release.

Thanks @bajdzun, @healerz & co for the help and uncovering this issue 🙏

Reported in #33 by @Hubbitus
Make sure, that the default is the first type if we have a union type

Said I have class:

<?php
...
namespace Pimcore\Model\Document\Editable\Loader;

use Pimcore\Loader\ImplementationLoader\PrefixLoader as BasePrefixLoader;

/**
 * @internal
 */
final class PrefixLoader extends BasePrefixLoader
{

Parser failed on it with:

In ClassParser.php line 205:
                                                                                                        
  Parent class [BasePrefixLoader] for [Pimcore\Model\Document\Editable\Loader\PrefixLoader] not found! 

(Error message include fix in #38)

Upgrade phpstan and infection, code / tests will need to be adjusted

The readme is a bit short and leaves out some of the details that are needed to understand the purpose of this project better.
Create a few wiki pages with more elaborate explanation

TokenParser was pieced together from other projects.
Evaluate if it would be feasible to replace it with something that is better maintained like PHP-Parser

In #34 capability for union types and annotation types was added. Cover these cases in integration tests as well

        $registry = (new ClassRegistry())
            ->addClassDirectory($this->projectDir . '/var/classes/DataObject')
            ->load();

The directory contains files from PIMcore, and among others scanned class Pimcore\Model\DataObject\AbstractObject with property with annotation:

    /**
     * @internal
     *
     * @var self|null
     */
    protected $o_parent;

Should be there self keyword resolved automatically into Pimcore\Model\DataObject\AbstractObject class?

Since PHP 7.4 type declarations were introduced.

But php-avro-schema-generator does not look in it.

Steps to reproduce:

// in ../DAO/DemoNotification.php
class DemoNotification {
    private string $content;

    public function __construct(string $content) {
        $this->content = $content;
    }

    public function getContent(): string {
        return $this->content;
    }
}

$data = new DemoNotification("Hello");
$registry = (new ClassRegistry())
    ->addClassDirectory(__DIR__ . '/../DAO')
    ->load();
$generator = new SchemaGenerator($registry, '');
$schemas = $generator->generate();
echo(current($schemas)); // Out: {"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":""}]}

Actual behavior

Despite explicit type declaration:

private string $content;

Script outputs:

{"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":""}]}

Expected behavior

{"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":"string"}]}

To leverage all the information we have about a template, we should probably chance the interface from

OptimizerInterface::optimize(string $definition): string

to

OptimizerInterface::optimize(SchemaTemplateInterface $schemaTemplate): SchemaTemplateInterface

This way the optimizers can leverage all the information from a SchemaTemplateInterface to optimize the schema.
This will be a breaking change

When embedding a root schema in another schema, the field schema_level is not being removed during schema merge

$registry = (new ClassRegistry($converter))
            ->addClassDirectory($this->projectDir . '/vendor/pimcore/pimcore/lib/Loader/')
                ->load();

And I have got error:

In ClassParser.php line 197:                                                  
  Class "AbstractClassNameLoader" does not exist  

Problem happened on class declaration starting from:

<?php

declare(strict_types = 1);

namespace Pimcore\Loader\ImplementationLoader;

class ClassMapLoader extends AbstractClassNameLoader
{
...

Class AbstractClassNameLoader indeed did not listed in the use statements, but that present in the same namespace and directory.

Said run your example of generation we got output schema PhpKafka.PhpAvroSchemaGenerator.Example.SomeTestClass.avsc (part):

    {
      "name": "someOtherTestClass",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.SomeOtherTestClass"
    },
    {
      "name": "someOtherTestClasses",
      "type": {
        "type": "array",
        "items": "PhpKafka.PhpAvroSchemaGenerator.Example.SomeOtherTestClass"
      }
    },
    {
      "name": "blaaaaaaaa",
      "type": [
        "int",
        "string"
      ]
    },
    {
      "name": "unicorn",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland"
    }

Which is incorrect!
AVRO does not have types like PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland!

In your example it is not very representative, because no any inner classes have fields, but said if Wonderland will have single field like:

**                                                                                                                                                                                                                                                                           
* Country of miracles
**/
class Wonderland                                                                                                                                                                                                                                                              
{                                                                                                                                                                                                                                                                             
    private string $land;                                                                                                                                                                                                                                                     
}

That generated part instead of

    {
      "name": "unicorn",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland"
    }

should look like:

{
  "name": "unicorn",
  "type": "record",
  "doc": "Country of miracles",
  "fields": [
    {
      "name": "land",
      "type": "string"
    }
  ]
}

As pointed out in #52 some examples are missing.
Add examples for

• generate & merge
• cli generate
• cli merge
• cli generate & merge

Add a small readme to example folder as well with a short introduction about these examples