Hi, thank you guys for your outstanding work. And I would like to first reproduce the experiment listed in your paper and then finetune the model on my own dataset. Could you please help to provide the tutorial of how to reproduce the experiment results? It would be very helpful to us I think. Thanks again for your effort.

Dataset using

Dataset Link

Code I am using to load the df

import numpy as np
from gluonts.dataset.pandas import PandasDataset

url = (
    "/content/Historical Product Demand.csv"
)
df = pd.read_csv(url, index_col=0, parse_dates=True)
# remove_ind = np.random.choice(np.arange(df.shape[0]), size=100, replace=False)
# mask = [False if i in remove_ind else True for i in range(df.shape[0])]
# df_missing_val = df.loc[mask, :]  # dataframe with 100 rows removed from df
def change_to_int(val):
    try:
        return int(val.strip('()'))  # Assuming the value is like "(1234)"
    except Exception as e:
        return 0

# Convert date column to datetime
df['Date'] = pd.to_datetime(df['Date'])


# Apply the function to convert Order_Demand to int
df['Order_Demand'] = df['Order_Demand'].apply(change_to_int)
df['item_id'] = df.index
df['target']= df['Order_Demand']
df.index = df.Date
df = df[['item_id',"target"]]

#Drop duplicate entiries for date
df_missing_val = df[~df.index.duplicated(keep='first')]
df_missing_val=df_missing_val.loc[pd.notnull(df_missing_val.index)]
df
# Get the max end date
freq="1D"
max_end = max(df.groupby("item_id").apply(lambda _df: _df.index[-1]))
print(f"{max_end=}")
dfs_dict = {}
for item_id, gdf in df_missing_val.groupby("item_id"):
    # Get the full (regular) date range
    # print(f"{gdf.index[0]=}")
    new_index = pd.date_range(gdf.index[0], end=max_end, freq=freq)
    # Reindex the dataframe
    dfs_dict[item_id] = gdf.reindex(new_index).drop("item_id", axis=1)
    dfs_dict[item_id].target.fillna(100,inplace=True)
    # Conver the columns to float32 for lag-llama
    for col in dfs_dict[item_id]:
        # Check if column is not of string type
        if dfs_dict[item_id][col].dtype != 'object' and pd.api.types.is_string_dtype(dfs_dict[item_id][col]) == False:
            dfs_dict[item_id][col] = dfs_dict[item_id][col].astype('float32')

# Create a PandasDataset
ds = PandasDataset(dfs_dict, target="target")

But I am getting the below error when predicting

131     ) -> Iterator:
--> 132         for data_entry in data_it:
    133             try:
    134                 yield self.map_transform(data_entry.copy(), is_train)

[/usr/local/lib/python3.10/dist-packages/gluonts/transform/_base.py](https://localhost:8080/#) in __call__(self, data_it, is_train)
    130         self, data_it: Iterable[DataEntry], is_train: bool
    131     ) -> Iterator:
--> 132         for data_entry in data_it:
    133             try:
    134                 yield self.map_transform(data_entry.copy(), is_train)

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in __iter__(self)
    412 
    413     def __iter__(self):
--> 414         for input, _label in self.test_data:
    415             yield input
    416 

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in __iter__(self)
    384 
    385     def __iter__(self) -> Generator[Tuple[DataEntry, DataEntry], None, None]:
--> 386         yield from self.splitter.generate_test_pairs(
    387             dataset=self.dataset,
    388             prediction_length=self.prediction_length,

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in generate_test_pairs(self, dataset, prediction_length, windows, distance, max_history)
    249             for window in range(windows):
    250                 offset = window * distance
--> 251                 test = self.test_pair(
    252                     entry, prediction_length=prediction_length, offset=offset
    253                 )

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in test_pair(self, entry, prediction_length, offset)
    299             label_slice = slice(offset_, None)
    300         return (
--> 301             slice_data_entry(
    302                 entry, input_slice, prediction_length=prediction_length
    303             ),

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in slice_data_entry(entry, slice_, prediction_length)
    161     entry: DataEntry, slice_: slice, prediction_length: int = 0
    162 ) -> DataEntry:
--> 163     slice_ = to_positive_slice(
    164         to_integer_slice(slice_, entry[FieldName.START]),
    165         entry[FieldName.TARGET].shape[-1],

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in to_positive_slice(slice_, length)
    118     if stop is not None and stop < 0:
    119         stop += length
--> 120         assert stop >= 0
    121     return slice(start, stop, slice_.step)
    122 

AssertionError:```

The complete code on Colab:

https://ts.gluon.ai/stable/tutorials/data_manipulation/pandasdataframes.html#Use-case-1---Loading-data-from-a-long-dataframe

Error：

First, I just wanted to say this project is very exciting and I plan to follow along with as it begins to mature, so thank you!

My request would be to have a comparison of Lag-Llama and Meta's Prophet. I know these are different approaches to time series forecasting, but Prophet is pretty widely used in the industry now, so I think potential users will be attracted to Lag-Llama if they see quantifiable comparisons.

Ideally this comparison would include tests from both tools against different datasets/sizes to show accuracy and speed of each.

Is it normal to have negative training losses?

Hi, I wanted to fine tune the model on my own dataset however with my own custom loss. Could you give an example on how to do that?
Thanks

Hi,

I would like to know the exact location in the lag-llama source code of the last hidden state to capture embeddings.

Regards,
Surendra

I have a dataset with only 1 time series array, so item_id is always the same, which is equivalent to not to use it. On the Demo you mention:
The item_id is required only when your dataset has multiple series. However its not working when not specifying this parameter, similarly when using PandasDataset(dfs_dict, target="target"). Do you have any suggestions on how to adapt the data on this case?

Thanks a lot

File ~/miniconda3/envs/env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:580, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
573 ckpt_path = self._checkpoint_connector._select_ckpt_path(
574 self.state.fn,
575 ckpt_path,
...
(...)
137 * self.dummy_value
138 )

KeyError: 'observed_values'

I am trying to use a dataset with lots of timeseries data as specified in the:
https://ts.gluon.ai/stable/tutorials/data_manipulation/pandasdataframes.html Usecase number 3.

I am unsure on how to use this pandas dataset for this model.

Even when using a single timeseries data as mentioned in the another issue, I am unable to get the forecasts and it shows assertion error: (then its blank)

I am having bad accuracy (MSE of 2500+)

I am giving the model 10 days worth of data and using the default settings given in the updated colab with a wide dataframe.
Prediction length = 24

While i see improvement if i increase the prediction length i.e. inplace of predicting for 1 day i predict for 72 time stamps (3 days) that i can get a mse of 400. Which is still high and not as good.

This image is 3 days prediction:

Do let me know if i am making a large error when giving this data which is causing this low accuracy.

Hi guys, thanks for your effort in this work. I just open this issue to figure out how should I compute the CRPS metric, which is mentioned in your papers. I found that the weighted average quantile loss looks very similar to it. Are they same on values? If not , how could I compute the CRPS? Thanks again.

Hi,
Thank you for publishing the pretraining and finetuning scripts! They are really helpful.
For a university project, we are trying to reproduce the results from the paper. However, running the pretrain script, we observe very slow training speeds (~1 minute per epoch) on our hardware.
Running the pytorch profiler for 16 training batches, we see the following:

Apparently the data loader needs 5 seconds for each batch, which is 84% of the full time of the training step.
After some further investigation, we found that the train data loader does the following:

1. Apply the transformation to a full time series.
2. Sample a window from the transformed data (inside the InstanceSplitter).
3. Extract the window from the transformed data (InstanceSplitter).
4. Create the batches of data according to the batch size.

This means a full timeseries gets transformed and then most of the transformed data is not used. This is then done for each item in a batch. We observed ~10 ms for transforming a full timeseries and with a batch size of 512, we get the >5 seconds reported by the profiler.

The order of execution is partly given by the gluonts package. So I am not aware of an obvious solution without addressing it there.

Now my question. Did you face the same issue during your experiments? How can we solve the problem we observe?

Congrats on this foundation model.

Your publication doesn't mention anything about training time, hardware requirements, and costs. Could you elaborate on this?

Best regards

Thanks for your great work. I'm looking forward to training it on my custom data.
When will you release the finetuning code?

Good Morning, and thanks for the open source code. I am trying to use the code for a time series classification task. Do you have some hint on what do I have to change?
Thanks for the answer

ckpt = torch.load("lag-llama.ckpt", map_location=torch.device('mps'))
estimator_args = ckpt["hyper_parameters"]["model_kwargs"]

code above is ok but then run following code. occur a error:

estimator = LagLlamaEstimator(
    ckpt_path="lag-llama.ckpt",
    prediction_length=prediction_length,
    context_length=context_length,

    # estimator args
    input_size=estimator_args["input_size"],
    n_layer=estimator_args["n_layer"],
    n_embd_per_head=estimator_args["n_embd_per_head"],
    n_head=estimator_args["n_head"],
    scaling=estimator_args["scaling"],
    time_feat=estimator_args["time_feat"],
)

lightning_module = estimator.create_lightning_module()
transformation = estimator.create_transformation()
predictor = estimator.create_predictor(transformation, lightning_module)

`RuntimeError Traceback (most recent call last)
Cell In[31], line 15
1 estimator = LagLlamaEstimator(
2 ckpt_path="lag-llama.ckpt",
3 prediction_length=prediction_length,
(...)
12 time_feat=estimator_args["time_feat"],
13 )
---> 15 lightning_module = estimator.create_lightning_module()
16 transformation = estimator.create_transformation()
17 predictor = estimator.create_predictor(transformation, lightning_module)

File ~/private/time-series-analysis/lag-llama/lag_llama/gluon/estimator.py:280, in LagLlamaEstimator.create_lightning_module(self, use_kv_cache)
264 model_kwargs = {
265 "input_size": self.input_size,
266 "context_length": self.context_length,
(...)
277 "dropout": self.dropout,
278 }
279 if self.ckpt_path is not None:
--> 280 return LagLlamaLightningModule.load_from_checkpoint(
281 checkpoint_path=self.ckpt_path,
282 loss=self.loss,
283 lr=self.lr,
...
262 'to map your storages to the CPU.')
263 device_count = torch.cuda.device_count()
264 if device >= device_count:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.`

and i modify code still error

ckpt = torch.load("lag-llama.ckpt", map_location=torch.device('cpu'))
estimator_args = ckpt["hyper_parameters"]["model_kwargs"]

RuntimeError Traceback (most recent call last)
Cell In[33], line 15
1 estimator = LagLlamaEstimator(
2 ckpt_path="lag-llama.ckpt",
3 prediction_length=prediction_length,
(...)
12 time_feat=estimator_args["time_feat"],
13 )
---> 15 lightning_module = estimator.create_lightning_module()
16 transformation = estimator.create_transformation()
17 predictor = estimator.create_predictor(transformation, lightning_module)

File ~/private/time-series-analysis/lag-llama/lag_llama/gluon/estimator.py:280, in LagLlamaEstimator.create_lightning_module(self, use_kv_cache)
264 model_kwargs = {
265 "input_size": self.input_size,
266 "context_length": self.context_length,
(...)
277 "dropout": self.dropout,
278 }
279 if self.ckpt_path is not None:
--> 280 return LagLlamaLightningModule.load_from_checkpoint(
281 checkpoint_path=self.ckpt_path,
282 loss=self.loss,
283 lr=self.lr,
...
262 'to map your storages to the CPU.')
263 device_count = torch.cuda.device_count()
264 if device >= device_count:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Hi, thank you very much for your work.

May I ask if the following error occurs when loading pre-trained model parameters while running your code?
Is it due to the Torch version? My Torch version is 2.2.0.

Thanks again for your great work!
I applied your pre-trained model to my dataset, which has a frequency of 5 minutes (freq = 5T). Unfortunately, the prediction results are quite poor. Can you please confirm whether I need to fine-tune the model or if there might be issues with my implementation?

The loss calculation during training uses the probability density function loss of all targets in context and predict, rather than just predict. Why?

First of thank you for your article and for sharing results
I'am trying to inference model on CPU only device.
And I modified model load code like below
ckpt = torch.load("lag-llama.ckpt", map_location=torch.device('cpu'))
But when execute lightning module still getting Error
lightning_module = estimator.create_lightning_module()

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Summary:

Encountered an incompatibility issue with pandas==2.2.0 where using to_offset('Q') raises an exception for "invalid frequency". This functionality works as expected in pandas==2.1.4. Suggesting an update to the requirements to specify the pandas version to ensure compatibility.

Steps to Reproduce:

Set up a Docker container with dependencies specified in the requirements file, which defaults to using pandas==2.2.0.
Execute the following Python code:

from pandas.tseries.frequencies import to_offset
to_offset('Q')

Observe the raised exception.

Expected Behavior:
The to_offset('Q') call should correctly interpret 'Q' as a quarterly frequency without raising an exception.

Actual Behavior:

When running with pandas==2.2.0, the code raises the following exception:

raise Exception("invalid frequency")

I noticed that the dimensions of the output of the first term in the list of:

forecast_it, ts_it = make_evaluation_predictions( dataset=backtest_dataset, predictor=predictor, )

is always (100, prediction_length), at least with the example and when using my own dataset, independent of the length of the test data length. Am I right here? Is this a bug? If not, how should I interpret the 100?

This the link to the Jupyter Notebook used for my own dataset https://drive.google.com/file/d/1WThIccnVKq_V48bC2FzmdzvHLEquLK94/view?usp=sharing

Kind Regards,

Hugo

Can we use this model for forecasting on our custom data as of now?

Thanks for releasing this foundation model! I wonder if you have code that demonstrates how to finetune Lag-Llama with custom training data? Thanks!

I used best checkpoint after turning then used get_lag_llama_predictions function to predict from best checkpoint . However, the predicted result is much lower than when I turned and trained that checkpoint.

Can the Taiwan Lottery accurately predict the next draw based on the results of the past ten draws with a 95% accuracy rate?
In my experiment, I need to compare the difference and accuracy of specialized forecasting models and general time series forecasting models.

(tf2) li@lieMacBook-Pro-M1 lag-llama % python run.py
Traceback (most recent call last):
File "/Users/liulan/Downloads/常用脚本/lag-llama/run.py", line 30, in
from gluonts.evaluation import Evaluator, make_evaluation_predictions
ModuleNotFoundError: No module named 'gluonts'

Hello. How do I freeze the predictions? I have a different prediction every time.

Hello, it's Arthur,
Thank you for your great work.
I would like to ask whether it is possible to get the specific lag indices you use during the pre-training or zero-shot phases.

In the Colab tutorial notebook, it is indicated that the context length is set to 32, and the maximum potential lag index could be 1092. However, the exact indices used to tokenize the 32 historical time points remain unclear to me.
Do you employ all of the 1092 lags, or is there a specific subset that is used?

Thank you!

Hi,
Why you did not consider holidays in the preprocessing?
Best regards

Hi, thanks for the model and instructions. I could not get the idea of predicting the last part of the data. I tried zero-shot learning with my own data. Let's say I have 1 week of hourly data. When I want to use the "get_lag_llama_predictions" function, the "forecasts" object has the same timestamp as the last part of the given data. But I want to predict the future timestamps. I want to give 1 week of data to model and predict the next 2 days. How can I achieve that? Thanks.

Hello,

Thanks for sharing your model.
Could you please provide an example of how to use this outstanding model on a . CSV dataset containing time-series data?

Why repeat here? The results generated after repeat (i.e. the parameters of the distribution) should be the same, right? What is the significance of this?

First , thank you for the great job , still did not finish the paper but trying to see what's the embedding space of time series looking like .
It does not seem obvious how to get them out especially with the gluonTS wrapper , any thoughts here ?
Thank you !

I have tried the zero-shot prediction on my own dataset, which is related to Healthcare. However, I am not satisfied enough with the performance and perform fine-tuning.
Then things get weird as the performance drops a lot compared to zero-shot. Is this because of one(or both) of the following?

1. I only use data from a single univariate time series to fine-tune. (training data is around 200~300)
2. I finetune the whole Lag-Llama's parameters

For cases that there is only 1 time series available and needed to be predicted, should one only fine-tune the outer layers? Or maybe prepare more related domain time series(in my case Healthcare) to perform the fine-tuning?

Any suggestion would be helpful, thanks a lot!

When I run the fine-tuning code on my dataset I get this error:

TypeError: model must be a LightningModule or torch._dynamo.OptimizedModule, got LagLlamaLightningModule

But the zero-shot works perfectly.

Hi !

Thank you for this foundation model, I've been trying to finetune it on a dataset of 10K timeseries but each timeseries only has 100 values (1min frequency and values ranging from 100 to -100). The timeseries are not following each other, so I can't append them.

Using your finetuning notebook, results are not good at all currently. What I'm interested in is predicting the last 20 values for each timeseries I feed the model so the context size should always be 80 + 20.

If I'm not mistaken, currently the finetuning script takes a random context inside each timeseries of the dataset right ?
Do you think my dataset is something the model can be finetuned on or should I build a model from scratch for this task ?

Thank you very much for any insights :)

Hi, thanks for your great work.
I have a question about the suggested values for context_length, which seem to be powers of 2. Is there a specific reason for this choice, or could we use other values for context_length that are not powers of 2?

I'm trying to fine-tune the model using my own data. When using the fine-tuned model to make predictions, I find that lag-llama gives all zero value predictions, which does not reproduce the fine-tuning effect in the demo. I followed the following steps:

1. Call from_long_dataframe to convert the data from the dataframe format to the standard format.
   
   

2. The data contains 7000 second-level data points. I use the first 4000 points for fine-tuning training, and the remaining 3000 data points as model input. The task is to predict the last 60 data points.
   

3. The prediction result is shown in the following figure. The result provided by lag-llama is 60 zero values.
   

What is the cause of this, or Which step did I do wrong?

Hello. Thank you so much for the open-source implementation.
I have a custom dataset with cryptocurrency values for daily prediction which except from the target values,
includes also 3 columns of related features.
Those columns could be included in your implementation? Are features taken into account from the model or is just the target value?

Thanks for such a great open source work. I feel so excited. I can’t wait to ask how long it will take to update the code for fine-tuning and training，You can close this issue at any time, it's just a little complaint from those who are waiting.

Hi,
The current fine-tuning notebook on colab uses gluon-ts datasets.
Can you quickly demonstrate fine-tuning for any open-source dataset in a pandas dataframe?
That will be helpful for the larger audience.
Thanks.

Hello, thank you for sharing the initial code. I aim to replicate the experiment outlined in your paper before proceeding to fine-tune the model with additional datasets. Could you kindly assist me in accessing a tutorial on reproducing the experiment results? Currently, the model seems to lack convergence, so any guidance you could provide would be greatly appreciated.

Once again, thank you for your efforts.

best,
koublal

Unable to make future predictions for future timesteps beyond the validation window where there are no known target values. Is that something this model is capable of?

Hi all,

Thank you for open-sourcing this library.

I am working on the task of anomaly detection in multivariate time series data. I would like to know how to use Lag-Llama to accomplish this. Is it possible for Lag-Llama to achieve this through fine-tuning?

Thanks!

Enviroments

#!git clone https://github.com/time-series-foundation-models/lag-llama/
#cd lag-llama
#!pip install -r requirements.txt --quiet
#!huggingface-cli download time-series-foundation-models/Lag-Llama lag-llama.ckpt --local-dir /content/lag-llama

from gluonts.dataset.common import ListDataset
from itertools import islice
from matplotlib import pyplot as plt
import matplotlib.dates as mdates
import torch
from gluonts.evaluation import make_evaluation_predictions, Evaluator
from gluonts.dataset.repository.datasets import get_dataset
from lag_llama.gluon.estimator import LagLlamaEstimator
import json
import pandas as pd

Custom data preparation

url = '/content/last_data.csv'
data = pd.read_csv(url)
data['date'] = pd.to_datetime(data['date'], format='%Y-%m-%d %H:%M:%S')
data = data[['date', 'target']]
data.resample('H', on='date')['target'].sum().reset_index()
data = data.set_index('date').asfreq('1H')

context_length = 24
prediction_length = 24*7
freq = "1H"
time_series = data['target'].values
start = data.index[0]
train_ds = ListDataset(
                                                [{'target': time_series[:-prediction_length], 'start': start}],
                                                freq=freq
)

test_ds = ListDataset(
                                                [{'target': time_series, 'start': start}],
                                                freq=freq
)

Model training and evalution

ckpt = torch.load("lag-llama.ckpt", map_location=torch.device('cuda:0'))
estimator_args = ckpt["hyper_parameters"]["model_kwargs"]

estimator = LagLlamaEstimator(
                                ckpt_path="lag-llama.ckpt",
                                prediction_length=prediction_length,
                                context_length=context_length,
                                # estimator args
                                input_size=estimator_args["input_size"],
                                n_layer=estimator_args["n_layer"],
                                n_embd_per_head=estimator_args["n_embd_per_head"],
                                n_head=estimator_args["n_head"],
                                scaling=estimator_args["scaling"],
                                time_feat=estimator_args["time_feat"],
)

lightning_module = estimator.create_lightning_module()
transformation = estimator.create_transformation()
predictor = estimator.create_predictor(transformation, lightning_module)


forecast_it, ts_it = make_evaluation_predictions(
                                                                                    dataset=train_ds,
                                                                                    predictor=predictor,
                                                                                )
                                                                                
                                                                                
                                                                                
forecasts = list(forecast_it)
tss = list(ts_it)
evaluator = Evaluator()
agg_metrics, ts_metrics = evaluator(iter(tss), iter(forecasts))
print(json.dumps(agg_metrics, indent=4))
                                                                                

The results were so bizarre, I couldn't believe it. After 20-30 days on a project, I developed a forecaster model in just 1 minute that performed better than the last model I developed, which took 3 days with an RTX3090. I'm going to lose my mind, holly shit.

Old TCN model

Log-Llama model

Hello and thank you for the model. I'm experimenting with my dataset, and one kind of systematic error I found are charts like below, where the first prediction point is way off the latest known point from the context. I'm wondering what could be the reason for this and if the problem could be alleviated somehow? Many thanks.

Hi! and thankyou for your excellent contribution to the world of time series!

I am currently using lag llama for finetuning, and was wondering if you had any rules of thumb for fine tuning yet?
I have read that transformers generally require many epochs, and noticed your early stopping patience is fifty. Does this mean we should generally train for many epochs? Or was that early stopping patience set on a very small dataset?

For context, my dataset has about 3 years with of price/energy demand/air temp/solar output data at 5 minute intervals. I have set a long context length to try and capture seasonal effects. Wondering how many epochs I should train for. The base foundation model did not work very on my data