Comments (6)
Does it not work with 1.7.0? I would assume the savepoint/checkpoint format hasn't changed much
from bravo.
It might be that I'm doing something wrong, but I now get this error:
Caused by: java.lang.IllegalStateException: Cannot read state of a stateless operator.
at com.king.bravo.reader.OperatorStateReader.lambda$readKeyedStates$0(OperatorStateReader.java:101)
at java.util.Optional.orElseThrow(Optional.java:290)
at com.king.bravo.reader.OperatorStateReader.readKeyedStates(OperatorStateReader.java:101)
And when I try doing something super simple like
val savepoint = StateMetadataUtils.loadSavepoint(savepointPath)
val env = ExecutionEnvironment.getExecutionEnvironment
val reader = new OperatorStateReader(env, savepoint, stateOperatorUid)
reader.getAllKeyedStateRows.first(3).print()
I get
Caused by: java.lang.NoSuchMethodError: org.apache.flink.runtime.state.KeyedBackendSerializationProxy.<init>(Ljava/lang/ClassLoader;Z)V
at com.king.bravo.utils.StateMetadataUtils.getKeyedBackendSerializationProxy(StateMetadataUtils.java:177)
at com.king.bravo.reader.inputformat.RocksDBSavepointIterator.openIfNeeded(RocksDBSavepointIterator.java:136)
at com.king.bravo.reader.inputformat.RocksDBSavepointIterator.nextRecord(RocksDBSavepointIterator.java:102)
at com.king.bravo.reader.inputformat.RocksDBSavepointIterator.next(RocksDBSavepointIterator.java:87)
at com.king.bravo.reader.inputformat.RocksDBSavepointIterator.next(RocksDBSavepointIterator.java:54)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at com.king.bravo.reader.inputformat.RocksDBKeyedStateInputFormat.reachedEnd(RocksDBKeyedStateInputFormat.java:104)
from bravo.
To be clear, I'm compiling with flink 1.7.0 and bravo 0.3 in the same code base. It's possible if I used Flink 1.6.2 and bravo 0.3 together to read flink 1.7 savepoints, it would work, but that's much more challenging to operate in this code base.
from bravo.
Alright, thanks for checking. I will try to fix this and release 0.4 tomorrow
from bravo.
I have released the 0.4 to maven central that should work for Flink 1.7.0.
Please try it when you have some time to make sure everything still works as expected.
from bravo.
Works now, thanks!
from bravo.
Related Issues (16)
- Add support for operations on Map states
- Add support for operations on List states
- Support reading and modifying states with Namespaces
- Support reading/writing RocksDB incremental checkpoints and make formats explicit HOT 1
- Add support for managing timers
- Utilities for handling window operator states
- Invalid deserialized value for MapState HOT 19
- License HOT 1
- Publish artifacts to Maven Central HOT 3
- Build failures: unknown property 'nexusUsername' & no configured signatory HOT 3
- StateMetadataUtils hides actual errors HOT 1
- Support blink 1.5.1 HOT 3
- add support to scala 2.12 and flink 1.8.x HOT 1
- King
- For me
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bravo.