API footgun: `infer_next_token` still works after end of text about llm

[setzer22]

I'd say being able to infer beyond eot is a feature some might want, even if it's just to run some experiment to see what would happen. But I'm OK with making the API harder to misuse, as long as it's still possible to request inference for a new token after EOT.

Returning some sort of "EOT" error in infer_next_token sounds like it might be what we want?

I wonder if A[EOT]B infers differently to AB...

Yes, for sure. EOT is an important token and the transformer will interpret both strings completely differently 👍

[philpax]

Yeah, that's what I was thinking. It wouldn't allow your use-case, though: if you returned an EOT error, you wouldn't be able to get the inferred token back.

[philpax]

Discussed this on the Discord - it makes much more sense to return the EOT token itself as an error, and then let users continue inferring if they want.