Comments (9)
Currently there's no way to do this automatically. I'm assuming your stream is exposed as a Haskell System.IO.Handle
? You could implement it manually with the following steps:
- Read a varint from the steam
- Read a
ByteString
of that length from the stream (e.g., withData.ByteString.hGet
) - Call
Data.ProtoLens.decodeMessage
on the returnedByteString
.
I don't know if there's any standalone code for reading a varint (there probably ought to be). You could try copying proto-lens's internal routine, for example adapting its inner loop to read bytes from a Handle
instead of calling attoparsec's anyWord
parser:
https://github.com/google/proto-lens/blob/master/proto-lens/src/Data/ProtoLens/Encoding/Bytes.hs#L38
from proto-lens.
ah ok, so there is not some obvious function i somehow missed. I hoped this would be an easy, straightforward project to do some practical haskell stuff ;) It still looks doable, i just might try solving it. If i get it to work and am confident in the code, would an PR be welcome, or is this too much of an edge case? The PR would probably need some serious reviewing, since i am not yet well versed in best practices/code style etc.
from proto-lens.
Hm...too be honest, such a function might be better as a separate package, at least until proto-lens has a better story around streaming in general. For example, we could provide
parseMessage :: Message a => Parser a
parseDelimitedMessage :: Message a => Parser a
and you could use parseDelimitedMessage
along with a library like attoparsec-conduit
to process a stream of inputs. But it's not clear that we'll stick with attoparsec
in the long term (#62), so I'm hesitant to expose such an API from the core of proto-lens
at this time.
from proto-lens.
ok, that makes sense! thank you.
from proto-lens.
This was done by #268.
from proto-lens.
well, this was a long time ago! Thanks!
from proto-lens.
@judah I think, this is not quite done yet. The new parseMessageDelimited
can help to get a message (via runParser
) only if you already have a ByteString
. In contrast, #61 concerns a Handle
(or a socket, I guess): how do I know exactly how many bytes to take from my Handle
to build the bytestring? I still seem to have to parse a varint
from my Handle
manually, right? Something like a loop reading one byte at a time and checking the MSB. Only after turning this varint
into an Int
can I read the exact amount of bytes needed to get the correct ByteString
, which, in turn, I can feed to runParser
.
I'm facing the problem right now and curious if you agree that's an issue. And if so, what would be the plan to tackle it.
from proto-lens.
Hm, I think you're right @ulysses4ever. Previously we were using a streaming parser based on attoparsec, which would have made this possible. But in proto-lens-0.5, we switched to a strict parser that needs a strict ByteString
. Unfortunately, I don't think it's feasible at this time to make Parser
non-strict without significantly affecting its performance.
I think it's reasonable to add more functionality to proto-lens to handle this use case. One idea I can think of is an API directly using Handles, which would copy the implementation of Data.ProtoLens.Encoding.Bytes.getVarInt
:
decodeDelimitedMessageH :: Message m => Handle -> IO (Either String m)
(I'm not wild about duplicating the code, but don't see another obvious way forward; maybe having some tests to verify the new code would be sufficient.)
Would you be interested in sending a PR?
from proto-lens.
@judah thanks for your reply! Yes, I'm interested in making a PR, given that I've already mostly implemented it in my project. I am gonna test it first, and then make a PR.
from proto-lens.
Related Issues (20)
- Generic runtime representation HOT 2
- QuickCheck and bytestring upper bounds are outdated
- Find a path forward for CI HOT 2
- Installing protoc doc
- Text format parser doesn't accept all valid boolean values
- Too strict bounds on profunctors on hackage HOT 2
- Release for Stackage LTS-18.0 HOT 5
- Support GHC 9
- `readMessage` from `TextFormat` chokes on unicode in strings
- Re-add blackgnezdo HOT 2
- Increment upper bounds for some dependencies HOT 2
- New Hackage release HOT 3
- Support lazy construction for byte fields HOT 2
- Not compatible with mtl-2.3 and transformers-0.6 HOT 3
- google/protobuf/struct.proto not included in protobuf types HOT 2
- Show instance not working for first/default enum values
- Show instance not working for first/default enum values
- Support GHC 9.4 HOT 10
- Compilation hangs after some `default-extensions` added. HOT 1
- Replace deprecated `set-output` command with environment file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proto-lens.