php-kafka / php-avro-schema-generator

PHP avro subschema merger and experimental PHP Class avro schema generator

PHP Kafka extension (php-simple-kafka-client)

Documentation

Please read the documentation here

Libraries

• php-kafka/php-simple-kafka-lib a small wrapper library to make usage easier

Support

Join the Slack Workspace or Gitter

Credits

This extension relies on librdkafka
This extension is based on php-rdkafka
Many thanks to all contributors ❤️

php-avro-schema-generator's Issues

Add more examples

As pointed out in #52 some examples are missing.
Add examples for

• generate & merge
• cli generate
• cli merge
• cli generate & merge

Add a small readme to example folder as well with a short introduction about these examples

Upgrade phpstan and infection

Upgrade phpstan and infection, code / tests will need to be adjusted

Support union types for SchemaGenerator

Adapt the token parser to support the ReflectionUnionType that has been introduced in PHP:8.0
Might be obsolete after #26

Improvement: Check default is first type

Reported in #33 by @Hubbitus
Make sure, that the default is the first type if we have a union type

The @var annotation contains a non existent class "self"

        $registry = (new ClassRegistry())
            ->addClassDirectory($this->projectDir . '/var/classes/DataObject')
            ->load();

The directory contains files from PIMcore, and among others scanned class Pimcore\Model\DataObject\AbstractObject with property with annotation:

    /**
     * @internal
     *
     * @var self|null
     */
    protected $o_parent;

Should be there self keyword resolved automatically into Pimcore\Model\DataObject\AbstractObject class?

Refactor export of merged schema

Right now the export also happens in the schema merger
Having dedicated exporters will give more flexibility in the future.

Consider replacing current TokenParser

TokenParser was pieced together from other projects.
Evaluate if it would be feasible to replace it with something that is better maintained like PHP-Parser

Add Wiki

The readme is a bit short and leaves out some of the details that are needed to understand the purpose of this project better.
Create a few wiki pages with more elaborate explanation

Improve @avro-default annotation

Right now only string values are supported, add support for:

• empty string
• numbers
• null

Improve optimizer interface

To leverage all the information we have about a template, we should probably chance the interface from

OptimizerInterface::optimize(string $definition): string

to

OptimizerInterface::optimize(SchemaTemplateInterface $schemaTemplate): SchemaTemplateInterface

This way the optimizers can leverage all the information from a SchemaTemplateInterface to optimize the schema.
This will be a breaking change

Generator: nested record types generated incorrectly

Said run your example of generation we got output schema PhpKafka.PhpAvroSchemaGenerator.Example.SomeTestClass.avsc (part):

    {
      "name": "someOtherTestClass",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.SomeOtherTestClass"
    },
    {
      "name": "someOtherTestClasses",
      "type": {
        "type": "array",
        "items": "PhpKafka.PhpAvroSchemaGenerator.Example.SomeOtherTestClass"
      }
    },
    {
      "name": "blaaaaaaaa",
      "type": [
        "int",
        "string"
      ]
    },
    {
      "name": "unicorn",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland"
    }

Which is incorrect!
AVRO does not have types like PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland!

In your example it is not very representative, because no any inner classes have fields, but said if Wonderland will have single field like:

**                                                                                                                                                                                                                                                                           
* Country of miracles
**/
class Wonderland                                                                                                                                                                                                                                                              
{                                                                                                                                                                                                                                                                             
    private string $land;                                                                                                                                                                                                                                                     
}

That generated part instead of

    {
      "name": "unicorn",
      "type": "PhpKafka.PhpAvroSchemaGenerator.Example.Wonderland"
    }

should look like:

{
  "name": "unicorn",
  "type": "record",
  "doc": "Country of miracles",
  "fields": [
    {
      "name": "land",
      "type": "string"
    }
  ]
}

Parser do not account use alias for parent classes

Said I have class:

<?php
...
namespace Pimcore\Model\Document\Editable\Loader;

use Pimcore\Loader\ImplementationLoader\PrefixLoader as BasePrefixLoader;

/**
 * @internal
 */
final class PrefixLoader extends BasePrefixLoader
{

Parser failed on it with:

In ClassParser.php line 205:
                                                                                                        
  Parent class [BasePrefixLoader] for [Pimcore\Model\Document\Editable\Loader\PrefixLoader] not found! 

(Error message include fix in #38)

ParserClass do not resolve parent classes without use statements

$registry = (new ClassRegistry($converter))
            ->addClassDirectory($this->projectDir . '/vendor/pimcore/pimcore/lib/Loader/')
                ->load();

And I have got error:

In ClassParser.php line 197:                                                  
  Class "AbstractClassNameLoader" does not exist  

Problem happened on class declaration starting from:

<?php

declare(strict_types = 1);

namespace Pimcore\Loader\ImplementationLoader;

class ClassMapLoader extends AbstractClassNameLoader
{
...

Class AbstractClassNameLoader indeed did not listed in the use statements, but that present in the same namespace and directory.

Handling namespaces of embedded types correctly

It was discovered (@bajdzun ), that the SR does not include namespaces of subtypes when they are in the same space.
To cover this a new option --optimizeSubSchemaNamespaces was added (#5, #10) to make this possible without breaking the current behaviour.
Upon further investigation it seems this is part of the Avro specification and should be default behaviour, see discussion in (#12).
This will be properly addressed and will result in a major release.

Thanks @bajdzun, @healerz & co for the help and uncovering this issue 🙏

Explicit type declarations is not handled!

Since PHP 7.4 type declarations were introduced.

But php-avro-schema-generator does not look in it.

Steps to reproduce:

// in ../DAO/DemoNotification.php
class DemoNotification {
    private string $content;

    public function __construct(string $content) {
        $this->content = $content;
    }

    public function getContent(): string {
        return $this->content;
    }
}

$data = new DemoNotification("Hello");
$registry = (new ClassRegistry())
    ->addClassDirectory(__DIR__ . '/../DAO')
    ->load();
$generator = new SchemaGenerator($registry, '');
$schemas = $generator->generate();
echo(current($schemas)); // Out: {"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":""}]}

Actual behavior

Despite explicit type declaration:

private string $content;

Script outputs:

{"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":""}]}

Expected behavior

{"type":"record","name":"DemoNotification","namespace":"App.DAO","fields":[{"name":"content","type":"string"}]}

Add more test cases for SchemaGenerator

In #34 capability for union types and annotation types was added. Cover these cases in integration tests as well

Please allow types re-definition (custom mapping)

Now most interesting for me to define optional (nullable fields), so, instead of just:

"type": "string",

Allow to set something like:

"type" : [ "null", "string" ],
"default": null

But there are also more interesting cases when e.g. instead of just int I would like to use logicalType date.

Please allow providing some functional interface to override fixed mechanism by providing an array of mappers, or just callable.

schema_level not removed if root schema is embedded

When embedding a root schema in another schema, the field schema_level is not being removed during schema merge