What Lags you use when pre-training the lag-llama? about lag-llama

[SpeeeedLee]

Also, I do not fully understand how the model gets the prediction for multiple future time points autoregressively at inference time.

Say I want to make 3 future predictions, and 100 trajectories, does it work as follows?

for _ in range(100):

1. Get the t-distribution parameters (Parameter_1) for the future first day.
2. Sample one data (Data_1) from t-distribution with Parameter_1.
3. Get the Parameter_2, by including the Data_1
4. Sample one data (Data_3) from t-distribution with Parameter_2.
5. trajectories.append([Data_1, Data_2, Data_3])

Additionally, it would be great if you could give me a hint on where I can find the detail code for autoregressive prediction, thanks!

[sudongwanglijuan]

lag-llama/lag_llama/gluon/lightning_module.py

Lines 229 to 261 in 7454088

	for t in range(self.prediction_length):
	if self.time_feat:
	params, loc, scale = self.model(
	*args,
	past_time_feat=repeated_past_time_feat,
	future_time_feat=repeated_future_time_feat[..., : t + 1, :],
	past_target=repeated_past_target,
	past_observed_values=repeated_past_observed_values,
	use_kv_cache=self.use_kv_cache,
	)
	else:
	params, loc, scale = self.model(
	*args,
	past_time_feat=None, # repeated_past_time_feat,
	future_time_feat=None, # repeated_future_time_feat[..., : t + 1, :],
	past_target=repeated_past_target,
	past_observed_values=repeated_past_observed_values,
	use_kv_cache=self.use_kv_cache,
	)
	sliced_params = [
	p[:, -1:] for p in params
	] # Take the last timestep predicted. Each tensor is of shape (#bsz*#parallel_samples, 1)
	distr = self.model.distr_output.distribution(sliced_params, loc, scale)
	sample = distr.sample() # (#bsz*#parallel_samples, 1)
	if self.nonnegative_pred_samples:
	sample = F.relu(sample)
	future_samples.append(sample)
	repeated_past_target = torch.cat((repeated_past_target, sample), dim=1)
	repeated_past_observed_values = torch.cat(
	(repeated_past_observed_values, torch.ones_like(sample)), dim=1
	)

[ashok-arjun]

Hello, it's Arthur, Thank you for your great work. I would like to ask whether it is possible to get the specific lag indices you use during the pre-training or zero-shot phases.

In the Colab tutorial notebook, it is indicated that the context length is set to 32, and the maximum potential lag index could be 1092. However, the exact indices used to tokenize the 32 historical time points remain unclear to me. Do you employ all of the 1092 lags, or is there a specific subset that is used?

Thank you!

Hi @SpeeeedLee .

The 32 historical time series points are consecutive, and sampled before the timestep to be predicted.

The lags however are sampled possibly even beyond this 32-length context, but sparsely as denoted by the lag indices.
The figure below might be useful to clarify the difference.

As for the indices of the lags, in our experiments, we sample lags of certain frequencies, upto a certain length.

The frequencies are denoted here:

lag-llama/lag_llama/gluon/estimator.py

Line 137 in 35f62a9

lags_seq: list = ["Q", "M", "W", "D", "H", "T", "S"],

The corresponding code to sample lags is here. We use the get_lags_for_frequency function of GluonTS in our code:

lag-llama/lag_llama/gluon/estimator.py

Lines 158 to 161 in 35f62a9

	for freq in lags_seq:
	lag_indices.extend(
	get_lags_for_frequency(freq_str=freq, num_default_lags=1)
	)

To give an example, the lags of the "D" frequency (daily) frequency look like this:

[0, 7, 12, 13, 14, 19, 20, 21, 26, 27, 28, 29, 30, 55, 83, 362, 363, 364, 726, 727, 728, 1090, 1091, 1092]

As for the actual lag indices that come from all these frequencies:

[0, 7, 8, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 34, 35, 36, 46, 47, 48, 50, 51, 52, 55, 57, 58, 59, 60, 61, 70, 71, 72, 83, 94, 95, 96, 102, 103, 104, 117, 118, 119, 120, 121, 142, 143, 144, 154, 155, 156, 166, 167, 168, 177, 178, 179, 180, 181, 334, 335, 336, 362, 363, 364, 502, 503, 504, 670, 671, 672, 718, 719, 720, 726, 727, 728, 1090, 1091, 1092]

For the code GluonTS uses to generate these lag indices, you can refer to the source code of the get_lags_for_frequency function.

[ashok-arjun]

Feel free to follow up if you have clarifications or close the issue if it answers your questions. Thanks!