Using cached sessions in REPL mode ? about llm

[philpax]

Hey, thanks for reporting this! Just noticed now that we don't wire up loaded sessions to the REPL - that should be easy enough to fix.

With switching out the cached sessions per prompt, I think that'd be out of scope for the CLI, but that should be easy enough to implement. You can see that session loading is done here:

https://github.com/rustformers/llama-rs/blob/main/llama-cli/src/main.rs#L224-L244

and the REPL just runs in a loop, feeding the user prompt to the session:

https://github.com/rustformers/llama-rs/blob/main/llama-cli/src/main.rs#L21-L63

So all you'd have to do is load the model once, and for each session: load an existing session if required, run inference, and then persist the session back to disk.

[nsarrazin]

Thanks for the quick update!

I'll try to implement the switching of the cache myself, never worked with Rust but I'll give it a go it should be fun. Thanks for pointing me in the right direction.