os72 / protobuf-dynamic Goto Github PK

First of all, thank you for your contribution
The problem is
I want to build protobuf schema from .proto file
I use your DynamicSchema to build schema then use the schema to parse a byte message which is generated by precompiled protobuf java class
but I get the following exception:
While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.

this is my code : https://github.com/cecol/protobuf-to-avro/blob/dev/src/main/java/wilber/com/transform/Main.java
is any misunderstand of your DynamicSchema or any mistake in my code?

both mFileDescSetBuilder and mFileDescSetBuilder is useful to be accessible too make use of it in other methods. I already have use for mFileDescProtoBuilder.
would it be possible to make it public ?

Will this be enough to compose gRPC definitions in Java? And without need to generate code? (I guess no)

I found it could not support create bellow schema dynamic:

import "google/protobuf/wrappers.proto";
message TestWrapperEntity{
  google.protobuf.DoubleValue field01 = 1 [json_name="field01"];
}

Do you have some suggest how to solve this problem? Thanks

I get a NoSuchMethod exception at line 203 in Dynamic Schema:

List dependencyList = fdProto.getDependencyList();

But if I change it for:

ProtocolStringList dependencyList = fdProto.getDependencyList();

Everything works correctly. Is this normal or just my issue? I don't know if this matters but I am using Java 8.

I can do a pull request if you like with the necessary changes to get it working with protobuf-java-2.6.1 .

Protobuf 3.1 was released in September 2016

does the protobuf-dynamic support the map field type

Whether proto3 is supported ?

Hi,
I had written producer, consumer application, here producer thread produces messages, serializes it with protobuff and publishes it with zeroMQ publisher.
At consumer side thread will subscriber on zeroMQ port and deserializes the messsages with protobuff. I am getting below issues randomly at my consumer side while decoding protobuff byte array:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.

ProtobuffIssues12HrsTest.txt

These issues i am getting is random issue it's not that i am getting it for each and every message, out of 47,00,000 messages in 12hrs i got this issue for 10 messages, and i am testing this on a single machine.

Below is my producer proto file:-
ProtoFileProducer1.txt

Below is my producer class:-
Producer.txt

Below is my consumer proto file:-
ConsumerProtoFile.txt

Below is my consumer class:
Consumer.txt

I think add way to inspect/iterator current msgTypeNames lives in DynamicSchema would be helpful.

well, I think if it ever appear, it will come with your participation:

Java IDL for protobuf (Protocl Buffers in Java)

That is instead of

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

I would like to have

public interface Greeter {
	@Grpc
	HelloReply sayHello (HelloRequest req);

}

@GrpcMessage()
public class HelloReply{
	@GrpcField(1)
	string name;
} 

That is annotation like Hibernate/JPA over my POJO, instead of heaps of generated code.

http://stackoverflow.com/questions/40425211/java-idl-for-protobuf-protocl-buffers-in-java

I have the following method:

public static Descriptors.Descriptor parseProto (String protoPath, String messageType) throws IOException, Descriptors.DescriptorValidationException {
        FileInputStream schemaFile = new FileInputStream(protoPath);
        DynamicSchema schema = DynamicSchema.parseFrom(schemaFile);
        return schema.getMessageDescriptor(messageType);
    }

But when executed, the program is caught in an infinite loop in DynamicSchema.init() on line 219:

while (fileDescMap.size() < fileDescSet.getFileCount()) {

afaik, the condition is always true since fileDescMap is cleared just before and nothing is ever added into it, and fileDescSet has elements (1 in my case) which are never removed.

EDIT: I just tried with your sample schema (Schema2.desc), it is working well, but I guess it is because there is no dependency. I am personally using the example proto from Google: https://github.com/google/protobuf/blob/master/examples/addressbook.proto

EDIT 2: I finally found the problem: it came from my .proto file. If an import statement is not available (external statement), it will keep looping trying to add it to the set. It is a problem in my case as I am not aware of the proto file that will be parsed, I don't want the parser to freeze my app. Maybe throwing an exception in case of missing dependency?

Is there any method to generate proto file from DynamicSchema?
The output of toString method is not ptoto format.

can we provide getmFileDescSet() ? currently its private so its not accessible.

Hi,

There is a comment about the ProtoBuf API change in DynamicSchema.java.

// getDependencyList() signature was changed and broke compatibility in 2.6.1; workaround with reflection
//List<String> dependencyList = fdProto.getDependencyList();

link: https://github.com/os72/protobuf-dynamic/blob/master/src/main/java/com/github/os72/protobuf/dynamic/DynamicSchema.java#L223-L224

I just checked this method signature in ProtoBuf 2.4.1 and 2.6.1

In 2.4.1

public java.util.List<String> getDependencyList() {
      return dependency_;
}

in 2.6.1

public com.google.protobuf.ProtocolStringList getDependencyList() {
      return dependency_;
}

but ProtocolStringList is actually a List<String>

public interface ProtocolStringList extends List<String> {

  /** Returns a view of the data as a list of ByteStrings. */
  List<ByteString> asByteStringList();

}

So what does the broken compatibility here mean?

Thank you,
Justin

I'm trying to create a DynamicSchema that includes an existing proto message type that has been previously defined, compiled and included in my classpath. Could you show how to do that?