yahoo / yql-plus

The YQL+ parser, execution engine, and source SDK.

YQL+ Engine

This is the YQL+ parser, execution engine, and source SDK.

Modules

• yqlplus_language - YQL+ grammar and parser

  An ANTLRv4-based parser for the YQL+ grammar (see docs/language.md) which parses programs into "OperatorNodes" -- an AST representation of a YQL+ program consisting of OperatorNodes of different types of operators:

  • StatementOperator - The top-level PROGRAM as well as argument declarations, view statements, queries, etc.
  • SequenceOperators - Represents queries. Logically produces a sequence of records, includes operators like PROJECT, FILTER, SCAN
  • ExpressionOperator - Represents an expression -- used to represent projection expressions, filters, sorts, etc.
  • SortOperator - Represents ASC and DESC operators on sort expressions.
  • ProjectOperator - Operations in a projection -- FIELD, FLATTEN (SELECT a.* ...)

• yqlplus_engine -- YQL+ execution engine

  This is the engine implementation. It has APIs for turning a stream or file into a CompiledProgram which is the interface for executing a program and getting results.

• yqlplus_source_api -- Source annotations and interfaces

This module defines the API for writing sources, transforms (pipes), and UDFs. It contains interfaces for tagging an implementation of each as well as annotations for communicating binding information to the engine.

Guice Multibindings are used to publish these implementations to the engine with names.

Usage

• YQL+ language parser

ProgramParser parser = new ProgramParser();
parser.parse("query", 
             "PROGRAM (@uuid string,
                       @logo_type string=""); 
              SELECT p.id,
                     p.provider_id, 
                     p.provider_name,
                     p.provider_alias,  
                    {"name" : plogo.name, "image" :  plogo.image} logo
              FROM provider({}) AS p 
              LEFT JOIN provider_logo(@logo_type,{}) AS plogo 
              ON p.id = plogo.provider_id  
              WHERE p.id=@uuid
              OUTPUT AS provider;");

• Query a Data Source

  (1) Create a Data Source

  public class InnerSource implements Source {
      @Query
      @TimeoutBudget(minimumMilliseconds = 5, maximumMilliseconds = 100)
      public List<Person> scan(@TimeoutMilliseconds long timeoutMs) {
        Assert.assertTrue(timeoutMs <= 100, "timeoutBudget <= 100");
        // checking minimum is dodgy and leads to failures
        return ImmutableList.of(new Person("1", "joe", 1));
      }

      @Query
      public Person lookup(@Key("id") String id) {
        if ("1".equals(id)) {
            return new Person("1", "joe", 1);
        } else if ("3".equals(id)) {
            return new Person("3", "smith", 1);
        } else {
            return null;
        }
      }

      @Query
      public Person lookup(@Key("iid") Integer id) {
        return lookup(String.valueOf(id));
      }
  }

(2) Bind Source instance

public class JavaTestModule extends AbstractModule {
   @Override
    protected void configure() {
        install(new JavaEngineModule());
        MapBinder<String, Source> sourceBindings = MapBinder.newMapBinder(binder(), String.class, Source.class);
        sourceBindings.addBinding("innersource").to(InnerSource.class);
    }
}

(3) Create Program to query the Data Source

Injector injector = Guice.createInjector(new JavaTestModule());
YQLPlusCompiler compiler = injector.getInstance(YQLPlusCompiler.class);
CompiledProgram program = compiler.compile("SELECT * FROM innersource WHERE id = '1' OUTPUT AS b1;");
ProgramResult myResult = program.run(ImmutableMap.<String, Object>of(), true);
YQLResultSet b1 = myResult.getResult("b1").get();
List<Person> b1r = b1.getResult();
 

• Import Java function to YQL+ Program

  (1) Create an Export class

  public class DateTime implements Exports {

      @Export
      public Instant from_epoch_second(long epochSecond) {
          return Instant.ofEpochSecond(epochSecond);
      }
  }

(2) Bind Export class

public class StandardLibraryModule extends AbstractModule {

    @Override
    protected void configure() {
        MapBinder<String, Exports> exportsBindings = MapBinder.newMapBinder(binder(), String.class, Exports.class);
        exportsBindings.addBinding("datetime").to(DateTime.class);
    }
}

(3) Use the Export function in Program

YQLPlusCompiler compiler = injector.getInstance(YQLPlusCompiler.class);
CompiledProgram program = compiler.compile("SELECT datetime.from_epoch_second(1378489457) date OUTPUT AS d1;");
ProgramResult result = program.run(ImmutableMap.<String, Object>of(), false);
List<Record> f2 = result.getResult("d1").get().getResult();

Maven

   
   <dependency>
      <groupId>com.yahoo.yqlplus</groupId>
      <artifactId>yqlplus_engine</artifactId>
      <version>1.0.1</version>
   </dependency>
   ...
   <dependency>   
      <groupId>com.yahoo.yqlplus</groupId>
      <artifactId>yqlplus_source_api</artifactId>
      <version>$1.0.1</version>
   </dependency>
   ...
   <dependency>   
      <groupId>com.yahoo.yqlplus</groupId>
      <artifactId>yqlplus_language</artifactId>
      <version>$1.0.1</version>
   </dependency> 
   ...
   <dependency>
      <groupId>com.yahoo.yqlplus</groupId>
      <artifactId>yqlplus_stdlib</artifactId>
      <version>1.0.1</version>
   </dependency>
   ...
   <repositories>
      <repository>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
        <id>bintray-yahoo-maven</id>
        <name>bintray</name>
        <url>http://yahoo.bintray.com/maven</url>
      </repository>
   </repositories>

LICENSE

Copyright 2016 Yahoo Inc.

Licensed under the terms of the Apache version 2.0 license. See LICENSE file for terms.

yql-plus's Issues

Non-constant Map expression can't have key which is not a valid Java variable name

when compiling program:

PROGRAM();
SELECT * FROM mapsource({'a.b.c':'abc'}) WHERE id = 2 OUTPUT AS out;

got

java.lang.IllegalArgumentException: Invalid name (must be a valid unqualified name): a.b.c
at org.objectweb.asm.util.CheckMethodAdapter.checkUnqualifiedName(Unknown Source)
at org.objectweb.asm.util.CheckMethodAdapter.visitFieldInsn(Unknown Source)

Commons-cli dependency should be removed from engine

Instead of bundling YQLPlusRun.java in the engine jar, create a tools or shell module that builds a ypshell (Naming is hard). Investigate if jshell api's can be used for building a usable YQLPlus REPL.

Upgrade guava to 24.1.1

“NoSuchMethodErrors” due to multiple versions of com.fasterxml.jackson.core:jackson-core:jar

Issue description

Hi, there are multiple versions of com.fasterxml.jackson.core:jackson-core in yql-plus-master(language module). As shown in the following dependency tree, library com.fasterxml.jackson.core:jackson-core:2.9.10 is transitively introduced by com.fasterxml.jackson.core:jackson-databind:2.9.10.3, but has been managed to be version 2.7.9.

However, several methods defined in shadowed version com.fasterxml.jackson.core:jackson-core:2.9.10 are referenced by client project via com.fasterxml.jackson.core:jackson-databind:2.9.10.3, but missing in the actually loaded version com.fasterxml.jackson.core:jackson-core:2.7.9.

For instance, the following missing methods(defined in com.fasterxml.jackson.core:jackson-core:2.9.10) are actually referenced by yql-plus-master(language module), which will introduce a runtime error(i.e., "NoSuchMethodErrors") into yql-plus-master(language module).

1. com.fasterxml.jackson.core.JsonGenerator: void writeFieldId(long) is invoked by yql-plus-master(language module) via the following path:

paths------
<com.yahoo.yqlplus.language.operator.JsonOperatorDump$ProgramDumpModule$1: void serialize(com.yahoo.yqlplus.language.operator.OperatorNode,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> yql-plus-master\yqlplus_language\target\classes
fasterxml.jackson.databind.SerializerProvider: void defaultSerializeValue(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.StdKeySerializers$Default: void serialize(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.JsonGenerator: void writeFieldId(long)>

2. com.fasterxml.jackson.core.JsonGenerator: com.fasterxml.jackson.core.type.WritableTypeId writeTypeSuffix(com.fasterxml.jackson.core.type.WritableTypeId) is invoked by yql-plus-master(language module) via the following path:

paths------
<com.yahoo.yqlplus.language.operator.JsonOperatorDump$ProgramDumpModule$1: void serialize(com.yahoo.yqlplus.language.operator.OperatorNode,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> yql-plus-master\yqlplus_language\target\classes
<com.fasterxml.jackson.databind.SerializerProvider: void defaultSerializeValue(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer: void serialize(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.impl.UnknownSerializer: void serializeWithType(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider,com.fasterxml.jackson.databind.jsontype.TypeSerializer)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.jsontype.impl.TypeSerializerBase: com.fasterxml.jackson.core.type.WritableTypeId writeTypeSuffix(com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.core.type.WritableTypeId)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.JsonGenerator: com.fasterxml.jackson.core.type.WritableTypeId writeTypeSuffix(com.fasterxml.jackson.core.type.WritableTypeId)>

3. com.fasterxml.jackson.core.JsonGenerator: void writeEmbeddedObject(java.lang.Object) is invoked by yql-plus-master(language module) via the following path:

paths------
<com.yahoo.yqlplus.language.operator.JsonOperatorDump$ProgramDumpModule$1: void serialize(com.yahoo.yqlplus.language.operator.OperatorNode,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> yql-plus-master\yqlplus_language\target\classes
<com.fasterxml.jackson.databind.SerializerProvider: void defaultSerializeValue(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.TokenBufferSerializer: void serialize(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.TokenBufferSerializer: void serialize(com.fasterxml.jackson.databind.util.TokenBuffer,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.util.TokenBuffer: void serialize(com.fasterxml.jackson.core.JsonGenerator)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.JsonGenerator: void writeEmbeddedObject(java.lang.Object)>

4. com.fasterxml.jackson.core.JsonGenerator: com.fasterxml.jackson.core.type.WritableTypeId writeTypePrefix(com.fasterxml.jackson.core.type.WritableTypeId) is invoked by yql-plus-master(language module) via the following path:

paths------
<com.yahoo.yqlplus.language.operator.JsonOperatorDump$ProgramDumpModule$1: void serialize(com.yahoo.yqlplus.language.operator.OperatorNode,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> yql-plus-master\yqlplus_language\target\classes
<com.fasterxml.jackson.databind.SerializerProvider: void defaultSerializeValue(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer: void serialize(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.impl.UnknownSerializer: void serializeWithType(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider,com.fasterxml.jackson.databind.jsontype.TypeSerializer)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.jsontype.impl.TypeSerializerBase: com.fasterxml.jackson.core.type.WritableTypeId writeTypePrefix(com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.core.type.WritableTypeId)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.JsonGenerator: com.fasterxml.jackson.core.type.WritableTypeId writeTypePrefix(com.fasterxml.jackson.core.type.WritableTypeId)>

5. com.fasterxml.jackson.core.JsonpCharacterEscapes: com.fasterxml.jackson.core.JsonpCharacterEscapes instance() is invoked by yql-plus-master(language module) via the following path:

paths------
<com.yahoo.yqlplus.language.operator.JsonOperatorDump: java.lang.String dump(com.yahoo.yqlplus.language.operator.OperatorNode)> yql-plus-master\yqlplus_language\target\classes
<com.fasterxml.jackson.databind.ObjectWriter: java.lang.String writeValueAsString(java.lang.Object)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ObjectWriter: void _configAndWriteValue(com.fasterxml.jackson.core.JsonGenerator,java.lang.Object)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ObjectWriter$Prefetch: void serialize(com.fasterxml.jackson.core.JsonGenerator,java.lang.Object,com.fasterxml.jackson.databind.ser.DefaultSerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.DefaultSerializerProvider: void serializePolymorphic(com.fasterxml.jackson.core.JsonGenerator,java.lang.Object,com.fasterxml.jackson.databind.JavaType,com.fasterxml.jackson.databind.JsonSerializer,com.fasterxml.jackson.databind.jsontype.TypeSerializer)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.JsonValueSerializer: void serializeWithType(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider,com.fasterxml.jackson.databind.jsontype.TypeSerializer)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.SerializableSerializer: void serialize(java.lang.Object,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.ser.std.SerializableSerializer: void serialize(com.fasterxml.jackson.databind.JsonSerializable,com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.databind.util.JSONPObject: void serialize(com.fasterxml.jackson.core.JsonGenerator,com.fasterxml.jackson.databind.SerializerProvider)> Repositories\com\fasterxml\jackson\core\jackson-databind\2.9.10.3\jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.JsonpCharacterEscapes: com.fasterxml.jackson.core.JsonpCharacterEscapes instance()>

Suggested fixing solutions

Declare a direct dependency com.fasterxml.jackson.core:jackson-core:2.9.10 in the pom file of yql-plus-master(language module), to override com.fasterxml.jackson.core:jackson-core's managed version.

Thank you very much for your attention.
Best regards,

Dependency tree----

[INFO] com.yahoo.yqlplus:yqlplus_language:jar:1.0.16
[INFO] +- org.testng:testng:jar:6.8.5:test
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:test
[INFO] |  +- com.beust:jcommander:jar:1.27:test
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:test
[INFO] +- org.antlr:antlr4-runtime:jar:4.5:compile
[INFO] |  \- org.abego.treelayout:org.abego.treelayout.core:jar:1.0.1:compile
[INFO] +- com.google.guava:guava:jar:24.1.1-jre:compile
[INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for conflict with 2.0.1)
[INFO] |  +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:2.0.1:provided (scope not updated to compile)
[INFO] +- com.google.inject:guice:jar:4.0:compile
[INFO] |  +- javax.inject:javax.inject:jar:1:compile
[INFO] |  +- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  \- (com.google.guava:guava:jar:24.1.1-jre:compile - version managed from 16.0.1; omitted for duplicate)
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.9:compile
[INFO] \- com.fasterxml.jackson.core:jackson-databind:jar:2.9.10.3:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.10:compile
[INFO]    \- (com.fasterxml.jackson.core:jackson-core:jar:2.7.9:compile - version managed from 2.9.10; omitted for duplicate)

JOIN based on UDF output doesn't work

Compile following program

SELECT * FROM left JOIN right ON left.id = right.id WHERE false AND left.category = 1 AND right.woeid = 2 AND true

Giving error

java.lang.IllegalArgumentException: Operator FILTER argument count mismatch: expected: 2 got 1
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
	at com.yahoo.yqlplus.language.logical.ArgumentsTypeChecker.check(ArgumentsTypeChecker.java:28)
	at com.yahoo.yqlplus.language.logical.SequenceOperator.checkArguments(SequenceOperator.java:74)
	at com.yahoo.yqlplus.language.operator.OperatorNode.create(OperatorNode.java:41)
	at com.yahoo.yqlplus.engine.rules.JoinFilterPushDown.visitSequenceOperator(JoinFilterPushDown.java:122)
	at com.yahoo.yqlplus.language.logical.LogicalOperatorTransform.apply(LogicalOperatorTransform.java:32)
	at com.yahoo.yqlplus.language.operator.OperatorNode.transform(OperatorNode.java:151)
	at com.yahoo.yqlplus.language.logical.LogicalOperatorTransform.visitSequenceOperator(LogicalOperatorTransform.java:23)
	at com.yahoo.yqlplus.engine.rules.JoinFilterPushDown.visitSequenceOperator(JoinFilterPushDown.java:73)

Dependency conflicts on com.fasterxml.jackson.core:jackson-core, leading to inconsistent program behaviors

Hi, in yql-plus/yqlplus_language, there are mulptiple versions of library com.fasterxml.jackson.core:jackson-core. However, according to Maven's dependency management strategy: "first declaration wins", only com.fasterxml.jackson.core:jackson-core:2.7.9 can be loaded, and com.fasterxml.jackson.core:jackson-core:2.9.10 will be shadowed.

In total, there are 29 conflicting API pairs between these two library version.

As shown in the following figure, your project expects to invoke method <com.fasterxml.jackson.core.json.UTF8StreamJsonParser: nextToken()Lcom/fasterxml/jackson/core/JsonToken;> in library com.fasterxml.jackson.core:jackson-core:2.9.10 (along the original dependency path). As it has been shadowed, this method defined in com.fasterxml.jackson.core:jackson-core:2.7.9 is actually forced to be referenced via the following invocation path (along the actual dependency path):

<com.yahoo.yqlplus.language.operator.OperatorNode: toString(Ljava/lang/StringBuilder;)V> /home/wwww/sensor/unzip/yql-plus-1.0.16/yqlplus_language/target/classes
<com.fasterxml.jackson.databind.util.TokenBuffer: toString()Ljava/lang/String;> /home/wwww/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.9.10.3/jackson-databind-2.9.10.3.jar
<com.fasterxml.jackson.core.json.UTF8StreamJsonParser: nextToken()Lcom/fasterxml/jackson/core/JsonToken;>

Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----

Code snippet of <com.fasterxml.jackson.core.json.UTF8StreamJsonParser: nextToken()Lcom/fasterxml/jackson/core/JsonToken;> in com.fasterxml.jackson.core:jackson-core:2.9.10 (shadowed but expected to invoke method):

detailed method body

@Override
    public JsonToken nextToken() throws IOException
    {
        /* First: field names are special -- we will always tokenize
         * (part of) value along with field name to simplify
         * state handling. If so, can and need to use secondary token:
         */
        if (_currToken == JsonToken.FIELD_NAME) {
            return _nextAfterName();
        }
        // But if we didn't already have a name, and (partially?) decode number,
        // need to ensure no numeric information is leaked
        _numTypesValid = NR_UNKNOWN;
        if (_tokenIncomplete) {
            _skipString(); // only strings can be partial
        }
        int i = _skipWSOrEnd();
        if (i < 0) { // end-of-input
            // Close/release things like input source, symbol table and recyclable buffers
            close();
            return (_currToken = null);
        }
        // clear any data retained so far
        _binaryValue = null;

        // Closing scope?
        if (i == INT_RBRACKET) {
            _closeArrayScope();
            return (_currToken = JsonToken.END_ARRAY);
        }
        if (i == INT_RCURLY) {
            _closeObjectScope();
            return (_currToken = JsonToken.END_OBJECT);
        }

        // Nope: do we then expect a comma?
        if (_parsingContext.expectComma()) {
            if (i != INT_COMMA) {
                _reportUnexpectedChar(i, "was expecting comma to separate "+_parsingContext.typeDesc()+" entries");
            }
            i = _skipWS();
            // Was that a trailing comma?
            if ((_features & FEAT_MASK_TRAILING_COMMA) != 0) {
                if ((i == INT_RBRACKET) || (i == INT_RCURLY)) {
                    return _closeScope(i);
                }
            }
        }

        /* And should we now have a name? Always true for Object contexts
         * since the intermediate 'expect-value' state is never retained.
         */
        if (!_parsingContext.inObject()) {
            _updateLocation();
            return _nextTokenNotInObject(i);
        }
        // So first parse the field name itself:
        _updateNameLocation();
        String n = _parseName(i);
        _parsingContext.setCurrentName(n);
        _currToken = JsonToken.FIELD_NAME;

        i = _skipColon();
        _updateLocation();

        // Ok: we must have a value... what is it? Strings are very common, check first:
        if (i == INT_QUOTE) {
            _tokenIncomplete = true;
            _nextToken = JsonToken.VALUE_STRING;
            return _currToken;
        }        
        JsonToken t;

        switch (i) {
        case '-':
            t = _parseNegNumber();
            break;

            // Should we have separate handling for plus? Although it is not allowed per se,
            // it may be erroneously used, and could be indicate by a more specific error message.
        case '0':
        case '1':
        case '2':
        case '3':
        case '4':
        case '5':
        case '6':
        case '7':
        case '8':
        case '9':
            t = _parsePosNumber(i);
            break;
        case 'f':
            _matchFalse();
             t = JsonToken.VALUE_FALSE;
            break;
        case 'n':
            _matchNull();
            t = JsonToken.VALUE_NULL;
            break;
        case 't':
            _matchTrue();
            t = JsonToken.VALUE_TRUE;
            break;
        case '[':
            t = JsonToken.START_ARRAY;
            break;
        case '{':
            t = JsonToken.START_OBJECT;
            break;

        default:
            t = _handleUnexpectedValue(i);
        }
        _nextToken = t;
        return _currToken;
    }

Code snippet of <com.fasterxml.jackson.core.json.UTF8StreamJsonParser: nextToken()Lcom/fasterxml/jackson/core/JsonToken;> in com.fasterxml.jackson.core:jackson-core:2.7.9 (loaded version):

detailed method body

@Override
    public JsonToken nextToken() throws IOException
    {
        /* First: field names are special -- we will always tokenize
         * (part of) value along with field name to simplify
         * state handling. If so, can and need to use secondary token:
         */
        if (_currToken == JsonToken.FIELD_NAME) {
            return _nextAfterName();
        }
        // But if we didn't already have a name, and (partially?) decode number,
        // need to ensure no numeric information is leaked
        _numTypesValid = NR_UNKNOWN;
        if (_tokenIncomplete) {
            _skipString(); // only strings can be partial
        }
        int i = _skipWSOrEnd();
        if (i < 0) { // end-of-input
            // Close/release things like input source, symbol table and recyclable buffers
            close();
            return (_currToken = null);
        }
        // clear any data retained so far
        _binaryValue = null;

        // Closing scope?
        if (i == INT_RBRACKET) {
            _updateLocation();
            if (!_parsingContext.inArray()) {
                _reportMismatchedEndMarker(i, '}');
            }
            _parsingContext = _parsingContext.clearAndGetParent();
            return (_currToken = JsonToken.END_ARRAY);
        }
        if (i == INT_RCURLY) {
            _updateLocation();
            if (!_parsingContext.inObject()) {
                _reportMismatchedEndMarker(i, ']');
            }
            _parsingContext = _parsingContext.clearAndGetParent();
            return (_currToken = JsonToken.END_OBJECT);
        }

        // Nope: do we then expect a comma?
        if (_parsingContext.expectComma()) {
            if (i != INT_COMMA) {
                _reportUnexpectedChar(i, "was expecting comma to separate "+_parsingContext.getTypeDesc()+" entries");
            }
            i = _skipWS();
        }

        /* And should we now have a name? Always true for
         * Object contexts, since the intermediate 'expect-value'
         * state is never retained.
         */
        if (!_parsingContext.inObject()) {
            _updateLocation();
            return _nextTokenNotInObject(i);
        }
        // So first parse the field name itself:
        _updateNameLocation();
        String n = _parseName(i);
        _parsingContext.setCurrentName(n);
        _currToken = JsonToken.FIELD_NAME;

        i = _skipColon();
        _updateLocation();

        // Ok: we must have a value... what is it? Strings are very common, check first:
        if (i == INT_QUOTE) {
            _tokenIncomplete = true;
            _nextToken = JsonToken.VALUE_STRING;
            return _currToken;
        }        
        JsonToken t;

        switch (i) {
        case '-':
            t = _parseNegNumber();
            break;

            /* Should we have separate handling for plus? Although
             * it is not allowed per se, it may be erroneously used,
             * and could be indicate by a more specific error message.
             */
        case '0':
        case '1':
        case '2':
        case '3':
        case '4':
        case '5':
        case '6':
        case '7':
        case '8':
        case '9':
            t = _parsePosNumber(i);
            break;
        case 'f':
            _matchToken("false", 1);
             t = JsonToken.VALUE_FALSE;
            break;
        case 'n':
            _matchToken("null", 1);
            t = JsonToken.VALUE_NULL;
            break;
        case 't':
            _matchToken("true", 1);
            t = JsonToken.VALUE_TRUE;
            break;
        case '[':
            t = JsonToken.START_ARRAY;
            break;
        case '{':
            t = JsonToken.START_OBJECT;
            break;

        default:
            t = _handleUnexpectedValue(i);
        }
        _nextToken = t;
        return _currToken;
    }

The detailed informantion of the remaining 28 conflicting API pairs can be found in the following attachment.
29 conflicting API pairs in project yqlplus_language.txt

Dependency tree--

[INFO] com.yahoo.yqlplus:yqlplus_language:jar:1.0.16
[INFO] +- org.testng:testng:jar:6.8.5:test
[INFO] | +- org.beanshell:bsh:jar:2.0b4:test
[INFO] | +- com.beust:jcommander:jar:1.27:test
[INFO] | - org.yaml:snakeyaml:jar:1.6:test
[INFO] +- org.antlr:antlr4-runtime:jar:4.5:compile
[INFO] | - org.abego.treelayout:org.abego.treelayout.core:jar:1.0.1:compile
[INFO] +- com.google.guava:guava:jar:24.1.1-jre:compile
[INFO] | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for conflict with 2.0.1)
[INFO] | +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
[INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | - org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:2.0.1:provided (scope not updated to compile)
[INFO] +- com.google.inject:guice:jar:4.0:compile
[INFO] | +- javax.inject:javax.inject:jar:1:compile
[INFO] | +- aopalliance:aopalliance:jar:1.0:compile
[INFO] | - (com.google.guava:guava:jar:24.1.1-jre:compile - version managed from 16.0.1; omitted for duplicate)
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.9:compile
[INFO] - com.fasterxml.jackson.core:jackson-databind:jar:2.9.10.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.10:compile
[INFO] - (com.fasterxml.jackson.core:jackson-core:jar:2.7.9:compile - version managed from 2.9.10; omitted for duplicate)

Suggested solutions:

Solution: Declare version com.fasterxml.jackson.core:jackson-core:2.9.10 as a direct dependency, to override the version 2.7.9 (based on Maven's nearest wins loading strategy).

Thanks.
Best regards,
Coco

Add support for WildcardType

Mocking framework

Hi,

hope find you well with this cold call.

I am an author of mocking framework for Kotlin

I see you are using mockito-kotlin.

I just want you to be aware that there is solution that fully supports Kotlin and ask to try it in your new/current projects.

I can help you if you answer to this issue.

Thanks and please star it

ProgramParser doesn't consider the case of select statement without source

When parsing following statement

SELECT  1 WHERE @next = "";

java.lang.NullPointerException: Argument 0 of FILTER must not be null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:251)
at com.yahoo.yqlplus.language.logical.NodeTypeChecker.check(NodeTypeChecker.java:31)
at com.yahoo.yqlplus.language.logical.ArgumentsTypeChecker.check(ArgumentsTypeChecker.java:31)
at com.yahoo.yqlplus.language.logical.SequenceOperator.checkArguments(SequenceOperator.java:74)
at com.yahoo.yqlplus.language.operator.OperatorNode.create(OperatorNode.java:31)

Merged tables should run parallel, not sequential

CREATE TEMP TABLE sample1 AS (SELECT * FROM sample('id1'));
CREATE TEMP TABLE sample2 AS (SELECT * FROM sample('id2'));
SELECT * FROM sample1
MERGE
SELECT * FROM sample2
OUTPUT AS samples;

sample1 and sample2 should run in parallel.

Export class loader is not registered

If only UDFs are used in a program the program compilation will fail for the error no class found.

ArrayIndexAdapter missed emitting code of count and idx write

Missed explicit cast for generic return type

FAILED: testGenericResult
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/yahoo/yqlplus/generated/gen1/invocable_unit11.invoke(Lcom/yahoo/yqlplus/generated/gen1/obj_unit2;Lcom/yahoo/yqlplus/engine/TaskContext;Lcom/yahoo/yqlplus/engine/sources/SampleResultSource$Result;)Lcom/yahoo/yqlplus/generated/gen1/struct_unit12; @21: getfield
Reason:
Type 'java/lang/Object' (current frame, stack[0]) is not assignable to 'com/yahoo/yqlplus/engine/sources/SampleResultSource$Sample'
Current Frame:
bci: @21
flags: { }
locals: { 'com/yahoo/yqlplus/generated/gen1/obj_unit2', 'com/yahoo/yqlplus/engine/TaskContext', 'com/yahoo/yqlplus/engine/sources/SampleResultSource$Result', 'com/yahoo/yqlplus/generated/gen1/struct_unit12', top, 'java/lang/Object' }
stack: { 'java/lang/Object' }
Bytecode:
0x0000000: bb00 0e59 b700 0f4e 2cb6 0015 3a05 1905
0x0000010: c600 0b19 05b4 001b a700 0401 3a04 1904
0x0000020: c600 092d 1904 b500 1e2d b0
Stackmap Table:
append_frame(@27,Object[#14],Top,Object[#4])
same_locals_1_stack_item_frame(@28,Object[#35])
full_frame(@41,{Object[#37],Object[#39],Object[#17],Object[#14],Object[#35],Object[#4]},{})

at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getField0(Class.java:2975)
at java.lang.Class.getField(Class.java:1701)
at com.yahoo.yqlplus.engine.internal.bytecode.UnitGenerator.prepare(UnitGenerator.java:314)
at com.yahoo.yqlplus.engine.internal.bytecode.ASMClassSource.build(ASMClassSource.java:191)
at com.yahoo.yqlplus.engine.internal.compiler.ProgramEnvironment.compile(ProgramEnvironment.java:67)