Add support for partial input about bytestring-lexing

[BebeSparkelSparkel]

@winterland1989 Can you provide a link?

[winterland1989]

This is 7years ago issue, I can't really find it any more lol.

[winterland1989]

The main idea is in the type, feel free to close it if it's not useful.

[wrengr]

The only case I can see for introducing partial results (in a lexer) is if the token spans the boundaries between input buffers. For integral lexing, it's quite easy (albeit annoying) to patch things up after the fact; whereas for non-integrals, although it can be patched up after the fact, it's a lot more involved (since you'd need to copy stuff over into a new buffer and try again from the beginning).

Since the behavior only differs at end-of-buffer, it should be doable to add support for partial inputs without introducing any slowdown in the main loop. I'll take a look, though I'll also have to figure out some decent benchmarks in order to ensure that the different return type doesn't introduce some unexpected overhead