Dumb question: definitions.py model parameters about llama-api

[c0sogi]

RoPE is a technique that allows a model to operate in a higher context (max_total_tokens) than the context length in which it was trained (maybe 4096). That's what you see. By default, if you do not set the parameter, it is calculated automatically. In other words, just delete the line with the parameter starting with rope.

There's no obivous golden rule for rope_freq_base, but I recommend using this calculations:

    def calculate_rope_alpha(self) -> float:
        """Calculate the RoPE alpha based on the n_ctx.
        Assume that the trained token length is 4096."""
        # The following formula is obtained by fitting the data points
        # (comp, alpha): [(1.0, 1.0) (1.75, 2.0), (2.75, 4.0), (4.1, 8.0)]
        compress_ratio = self.calculate_rope_compress_ratio()
        return (
            -0.00285883 * compress_ratio**4
            + 0.03674126 * compress_ratio**3
            + 0.23873223 * compress_ratio**2
            + 0.49519964 * compress_ratio
            + 0.23218571
        )

    def calculate_rope_freq(self) -> float:
        """Calculate the RoPE frequency based on the n_ctx.
        Assume that the trained token length is 4096."""
        return 10000.0 * self.calculate_rope_alpha() ** (64 / 63)

    def calculate_rope_compress_ratio(self) -> float:
        """Calculate the RoPE embedding compression ratio based on the n_ctx.
        Assume that the trained token length is 4096."""
        return max(self.max_total_tokens / Config.trained_tokens, 1.0)

    def calculate_rope_scale(self) -> float:
        """Calculate the RoPE scaling factor based on the n_ctx.
        Assume that the trained token length is 4096."""
        return 1 / self.calculate_rope_compress_ratio()

Note that this auto calculation methods are present in dev branch now and will be merged soon.

You can refer other parameters in this path:
llama_api/schemas/models.py. I recommend you to use IDE such as VSCode, as it will show you some hints for hidden parameters.

[Dougie777]

Thank you so much!