kmax12 / gridstatus

Extract data from ISOs and other energy grid sources

gridstatus is a Python library that provides a uniform API for accessing electricity supply, demand, and pricing data for the major Independent System Operators (ISOs) in the United States. It currently supports data from CAISO, SPP, ISONE, MISO, Ercot, NYISO, and PJM.

GridStatus.io and Hosted API

To preview some of the data this library provide access to, visit GridStatus.io.

If you are trying to use our hosted API, you might want to check out the gridstatusio library here.

To learn more about our hosted API visit: https://www.gridstatus.io/api.

Community

• Need Help? Post a GitHub issue
• Want to chat? Join our Slack
• Want to stay updated? Follow us on Twitter @grid_status
• Want to contribute? Read our Contributing Guide

Installation

gridstatus supports python 3.9+. Install with pip

python -m pip install gridstatus

Upgrade using the following command

python -m pip install --upgrade gridstatus

Environment Variables

• Some parsers require the use of environment variables
  • The EIA class
  • The ErcotAPI class
• Environment variables can be set in the following ways
  • In a .env file in the root of the project
  • In the environment where the code is run
• See .env.template for the required environment variables and how to register for them

Documentation and Examples

To learn more, visit the documentation and view example notebooks.

Get Help

We'd love to answer any usage or data access questions! Please let us know by posting a GitHub issue.

gridstatus's Issues

Add example notebook for `make_availability_df`

A users might want to programmatically determine method support for each iso. They can do that with make_availability_df, but it's not documented. We should add a Example notebook for how it can be used.

The example notebook should be in docs/Examples/misc

Add gridstatus to conda-forge channel

• The instructions here (https://conda-forge.org/docs/maintainer/adding_pkgs.html) can be followed for adding packages to the conda-forge channel and allowing conda users to install the library.

Add import data

Import data is a critical piece of understand where electricity is flowing in the grid

get_historical_lmp takes a market object, not the default string

The function for get_historical_lmp prescribes a string that represents the market (e.g."REAL_TIME_HOURLY") but the conditional in the function takes an object (e.g. self.REAL_TIME_HOURLY)

I got around this with the following:

date= "June 1, 2020"
rth=isodata.base.Markets.REAL_TIME_HOURLY
data=caiso.get_historical_lmp(date=date, market=rth, nodes=None)

I found this example for CAISO, but I'm not sure if it persists across the other ISOs.

Cannot localize tz-aware Timestamp, use tz_convert for conversions

Following error reported running locally on a mac, but not showing up

        date_str = "20220722"
        for m in markets:
            print(iso.iso_id, m)
            latest = iso.get_latest_lmp(m, nodes=nodes)
>           assert isinstance(latest, pd.DataFrame)

isodata/tests/test_isos.py:251:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
isodata/miso.py:102: in get_latest_lmp
    time = pd.to_datetime(time_str).tz_localize(self.default_timezone)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: Cannot localize tz-aware Timestamp, use tz_convert for conversions

pandas/_libs/tslibs/timestamps.pyx:1849: TypeError

Add Viz Module to Documentation API Reference

There is only one viz method right now, but hope to add more in the future

Make Plotly Graphs interactive in the documention

Currently, the plotly graphs in documentation that are part of ipynb files are showing as SVGs. it should be possible to make them interactive.

Interactive graphs are working in index.md where code is defined in a code cell, but not in the example ipynb files

Example svg in our documentation that isn't working: https://docs.gridstatus.io/en/latest/Examples/caiso/DAM%20Heat%20Map.html

Here is an example of it working the documentation template that we use: https://myst-nb.readthedocs.io/en/v0.9.0/examples/interactive.html#plotly

Improve doc strings

Now that the new docs are up, it makes sense to go through and clean up the doc strings for all the methods.

Notes:

• Can we document methods on the base class and have the doc strings show up for subclasses in sphinx?
• Can we add usage examples in doc string i.e doctest?
• How can we document methods on subclass that are not defined?

utils._handle_date creates timezone-less pd.Timestamp objects

Description

Given the current utils._handle_date code

def _handle_date(date, tz=None):
    if not isinstance(date, pd.Timestamp):
        date = pd.to_datetime(date)

    if tz and date.tzinfo is None:
        date = date.tz_localize(tz)
    return date

passing in today from a timezone that's on the other side of the date boundary from US timezones may be problematic.

>>> pd.to_datetime('today')
Timestamp('2022-12-16 00:00:00.000000')
>>> pd.to_datetime('today').tzinfo is None
True
# Checking the equivalent in US/Eastern
>>> pd.to_datetime('2022-12-16 00:00:00.00000+0000').tz_convert('US/Eastern')
Timestamp('2022-12-15 19:00:00-0500', tz='US/Eastern')

Solution

I think the solution involves:

• using pd.Timestamp(date, tz=tz)
• enforcing tz from callers (which should always be self.default_timezone from ISO classes), or falling back to tz=tzlocal()
• writing a boatload of tests

Create documentation website

Currently, just documenting this in the readme. Now that the library is getting more fleshed out it probably makes sense to add a documentation website.

Thinking about using sphinx and hosting on readthedocs

ISO.get_historical_lmp : add feature to query historical time period

Hey, would be nice to be able to pull large time spans instead of individual days at a time. Thanks!

Eliminate with_markets

We should be able to use @lmp_config to ensure test coverage of get_lmp/get_spp instead of the explicit test decorator @with_markets.

ISONE DAM Inconsistent Redirects

Documenting strange inconsistent redirects when fetching DAM CSV in ISONE (test_isone_dst_end):

Code

date = "Nov 6, 2022"
data = iso.get_lmp(date=date, market=Markets.DAY_AHEAD_HOURLY)

Success

$ curl -L -I https://www.iso-ne.com/static-transform/csv/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
HTTP/1.1 302 
Server: ISO New England
Content-Type: text/csv
Date: Wed, 28 Dec 2022 22:12:33 GMT
Location: http://www.iso-ne.com/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
Transfer-Encoding: chunked
Connection: Keep-Alive
X-FRAME-OPTIONS: ALLOW-FROM http://isone.csod.com

HTTP/1.1 200 OK
Server: ISO New England
Content-Type: text/csv
Date: Wed, 28 Dec 2022 22:12:33 GMT
Accept-Ranges: bytes
ETag: "180448-2750a7-5f07e8b2374b5"
Connection: Keep-Alive
Last-Modified: Fri, 23 Dec 2022 13:12:54 GMT
Content-Length: 2576551

Failure

$curl -L -I https://www.iso-ne.com/static-transform/csv/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
HTTP/1.1 302 
Server: ISO New England
Content-Type: text/csv
Date: Wed, 28 Dec 2022 22:12:34 GMT
Location: http://www.iso-ne.com/status.html?path=/static-transform/csv/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
Transfer-Encoding: chunked
Connection: Keep-Alive
X-FRAME-OPTIONS: ALLOW-FROM http://isone.csod.com

HTTP/1.1 302 Found
Server: ISO New England
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 28 Dec 2022 22:12:34 GMT
Location: https://www.iso-ne.com/status.html?path=/static-transform/csv/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Frame-Options: SAMEORIGIN

HTTP/1.1 307 Temporary Redirect
Server: ISO New England
Content-Type: text/html; charset=iso-8859-1
Strict-Transport-Security: max-age=31536000; includesubdomains
Date: Wed, 28 Dec 2022 22:12:34 GMT
Location: https://www.iso-ne.com/markets-operations/system-forecast-status/current-system-status?path=/static-transform/csv/histRpts/da-lmp/WW_DALMP_ISO_20221106.csv
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Frame-Options: SAMEORIGIN

HTTP/1.1 200 OK
Server: ISO New England
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/html;charset=UTF-8
Strict-Transport-Security: max-age=31536000; includesubdomains
Content-Security-Policy: connect-src 'self' https://bf44331iwz.bf.dynatrace.com/ https://qif46274.live.dynatrace.com/ https://*.dynatrace.com https://snap.licdn.com/ https://cdn.linkedin.oribi.io/; img-src 'self' https://*.iso-ne.com http://*.iso-ne.com https://ssl.google-analytics.com https://*.twimg.com https://6002825.global.siteimproveanalytics.io/ https://px.ads.linkedin.com/ data: ; font-src 'self' data: ;
Date: Wed, 28 Dec 2022 22:12:35 GMT
Expires: -1
X-XSS-Protection: 1 ;mode=block
Pragma: no-cache
Transfer-Encoding: chunked
Content-Language: en-US
X-Content-Type-Options: nosniff
Connection: Keep-Alive
Set-Cookie: alfresco-crafter_lb=ROUTE.alfresco-crafter0; path=/;
X-Frame-Options: SAMEORIGIN

CAISO historical load giving parser error

Hey, seeing a parser error when trying to pull some caiso load data...

gridstatus.CAISO().get_historical_load(start=pd.Timestamp("1/1/2018").normalize(), end=pd.Timestamp("1/1/2019").normalize())

ParserError: Error tokenizing data. C error: Expected 1 fields in line 21, saw 2

Add Interconnection Queues

Interconnection queues give a sense of what production is coming online and when. All ISO publish this info, so it'd be great have it as part of the library

Remove yeseterday methods

The get_*_yesterday methods are basically short hand for get_*_historical methods. They don't seem particularly useful and appear to bloat the API.

Let's remove them for now. We can always add them back in later if needed

Add Weather Data

Many of the ISO report weather data that can be added to isodata

requirements.txt is empty

Hey there,

Could you update requirements.txt so I can do a quick pip install of the required packages?

Add Recent Realtime LMP data to PJM

Currently, the PJM implementation uses "Data Miner 2" for real-time LMP data in the REAL_TIME_HOURLY and REAL_TIME_5_MIN markets. However, I've observed a delay of approximately 3 days in the update of this data.

One possible solution is to look into getting the data from the "Data Viewer" application (https://dataviewer.pjm.com/dataviewer/pages/public/lmp.jsf) when the users requests data not yet available in Data Miner 2.

SPP Day Ahead Market returning tomorrow's data for today

I am running the following code at 530pm CST on Feb 2. The expected behavior is to return the day ahead market for Feb 2, but it is actually giving data from Feb 3

import gridstatus
iso = gridstatus.SPP()
iso.get_lmp(
    date="today",
    market=gridstatus.Markets.DAY_AHEAD_HOURLY,
    verbose=True,
)

Here is the output

Add Generation Outage Information

Need to do work filling out sources

Data Source

MISO

ISONE

• Short Term Report
• Long Term Report

NYISO

• Available Datasets

SPP

PJM

CAISO

Ercot

• Source 1
• Source 2

API

TBD

Use Github Action Caching to Speed Up Tests

More info: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

Add historical Fuel Mix data for Ercot

Available here: https://www.ercot.com/gridinfo/generation

Change data source for Ercot load data for current day

There is a different endpoint we can use for Ercot load data that is 5 minute resolution that will work for the current day. The current implementation provides hourly data. Let's switch to the new endpoint

The url is: https://www.ercot.com/api/1/services/read/dashboards/todays-outlook.json. Downloading and parsing it is already implemented in the unused function Ercot._get_supply, so we should just need to update get_load

Add interregional flows

Add the flow of electricity between regions

Data Source

• NYISO
• MISO - Net Scheduled Interchange 1 and 5 minute resolutions available
• ISONE
• PJM
  ERCOT -
• SPP
• CAISO - can get net imports/exports from fuel mix

API

Common columns

• Time
• Interface Name
• Interface Id (optional)
• Flow (positive is import, negative is export)
• Import Limit (optional)
• Export Limit (optional)

iso = gridstatus.ISONE()
# TBD which is best method name
df = iso.get_interface_flows()
df = iso.get_interregional_flow()
df = iso.get_flows()
df = iso.get_interchange()

Refactor ISOBase.markets

After #135 is merged, we no longer need to explicitly define the supported markets in each of the ISO subclasses.

I propose we define a dynamic property of ISOBase that determines the list of support markets when accessed.

Related to removing redundant specifications that also needs to be done in #138

Document Historical Data Access Limits

isodata can only access historical data provided by the public APIs it uses.

This issue tracks documenting the historical data access dates for each of the ISO

NYISO - Nov 1, 2017
CAISO - ~last 39 months for data coming from OASIS.
ISONE - TBD
PJM - Jan 1, 2016

SPP `get_status` Failing

SPP.get_status scrapes the SPP current grid conditions page

When originally implemented that page looked like this

However, SPP grid conditions page currently looks like this, leading to parsing issue. Note the change in where the updated at time and status are displayed

Expected Behavior

It's not clear to me if this is just a temporary format change or not. Let's wait at least few days to see what happens with the page structure before revisiting the implementation.

Add Curtailment Data

Curtailment data tells us when generator capacity is reduced for planned or unplanned reasons.

Available for CAISO: http://www.caiso.com/market/Pages/OutageManagement/CurtailedandNonOperationalGenerators.aspx

Looking for sources for other ISOs

lmp_config supports location and/or location_type filtering

@lmp_config could be expanded to support location and location_type filtering and "compile"-time checking.

Real-time price data

While there seems to be functionality for grabbing day-ahead electricity prices, I'm not sure there is any functionality on collecting real-time pricing data yet. Adding this functionality could be desirable.

EDIT: Thank you for taking on this effort!

Remove `get_supply` methods

For every ISO besides Ercot, get_supply is just the sum of the fuel mix.

For ercot, it's useful because the fuel mix on break down solar and wind.

Therefore, I propose to remove the get supply method and then add an "Other" category to ercot that is the equal to supply - solar - wind

Clean Up Interconnection Queue Data

Each of the iso's now have a get_interconection_queue method that returns their respective queues. Some work was done to standardize columns names.

Each queue is useful on it's own, but combined there are a still small data difference

Status Column

One first area to look at is the Status field where there are many different but similar values

all_queues = gridstatus.get_interconnection_queues()
all_queues["Status"].value_counts()

Withdrawn                                    7405
Active                                       6431
WITHDRAWN                                    1481
In Service                                   1028
Completed                                     751
Done                                          746
ACTIVE                                        594
Engineering and Procurement                   301
Confirmed                                     223
COMPLETED                                     199
Retracted                                     131
Suspended                                      74
Under Construction                             58
Deactivated                                    54
Partially in Service - Under Construction      46
Annulled                                       24
Withdrawn - Pending Cure                        2
                                                1
Name: Status, dtype: int64

We should standardize these values. There can be an extra column preserved where the original status remains.

There is a InterconnectionQueueStatus enum in base.py that should contain all possible values

Interconnection Agreement Completed Date vs Project Completed Date

Should confirm that the completion dates are all accurately and consistently mapped

Tutorial on using `save_to` parameter when downloading historical data

Make a example notebook on using the save_to parameter to save intermediate results. The notebook should then also demonstrate using the load_folder method to load that data

Add LMP Data for ERCOT

Some potential sources:

https://www.ercot.com/mktinfo
https://www.ercot.com/api/1/services/read/dashboards/systemWidePrices.json
https://www.ercot.com/mp/data-products/markets/real-time-market?id=NP6-788-CD

Rename demand methods to load?

Some ISOs call it demand, some call it load. In general, seems like load is more descriptive phrasing. Should we change the method names to use load instead?

get_latest_demand --> get_latest_load
get_demand_today --> get_load_today
get_demand_yesterday --> get_load_yesterday
get_historical_demand --> get_historical_load

Refactor tests

Let's refactor test_isos.py. Below are some initial thoughts

Goal

1. Make it easy to run only tests for a specific ISO
2. Make it easy to identify which ISOs have failing tests
3. Keep shared testing code in a central location
4. Still allow iso specific tests to be written

Proposed approach

Outlined in this branch: https://github.com/kmax12/gridstatus/blob/5ad2795d397c400d20d0a4b71023eab01ef638ff/gridstatus/tests/test_subclass.py

Open questions

• How do we specify tests that are shared by some, but not all ISO. e.g test_get_latest_status

SPP Historical LMP Data

SPP historical LMP data is available in various compacted forms:

DAM LMP by Location

• Yearly: 2013.zip https://marketplace.spp.org/pages/da-lmp-by-location#%2F2013
• Monthly: DA-LMP-MONTHLY-SL-202112.csv https://marketplace.spp.org/pages/da-lmp-by-location#%2F2021%2F12
• Daily: DA-LMP-SL-202112010100.csv https://marketplace.spp.org/pages/da-lmp-by-location#%2F2021%2F12%2FBy_Day

RTM LMP by Location

• Monthly: RTBM-LMP-MONTHLY-SL-202201.csv https://marketplace.spp.org/pages/da-lmp-by-location#%2F2021%2F12
• Daily: RTBM-LMP-DAILY-SL-20221201.csv https://marketplace.spp.org/pages/rtbm-lmp-by-location#%2F2022%2F12%2FBy_Day
• By Interval: RTBM-LMP-SL-202212270015.csv https://marketplace.spp.org/pages/rtbm-lmp-by-location#%2F2022%2F12%2FBy_Interval%2F27

Daily compaction seems to lag by 5 days in both example data sets.

Based on the requested range of historical data, SPP::get_lmp will need to piece together the right list of files to fetch after fetching the candidate directories.

Optimize NYISO Date Range Queries over Historical Data

NYISO historical data is stored in monthly zip file. Currently, if you are accessing multiple days in one month, isodata downloads the zip file each time. We should update isodata to only download the monthly archives once.

Approach, update support_date_range to chunk queries into months. it may be possible to use frequency parameter of the existing call to pd.date_range. Alternatively, use update_dates parameter to @support_date_range decorator chunk dates into months.

update _download_nyiso_archive to accept date ranges that are contained within a single month

Add natural gas price data

NYISO Interconnection Queue bug in latest release

When I run:

import gridstatus
nyiso = gridstatus.NYISO()
nyiso_queue = nyiso.get_interconnection_queue()

using gridstatus 0.17.0 I get this error:

ValueError                                Traceback (most recent call last)
Cell In[310], line 3
      1 import gridstatus
      2 nyiso = gridstatus.NYISO()
----> 3 nyiso_queue = nyiso.get_interconnection_queue()

File ~/.local/lib/python3.9/site-packages/gridstatus/nyiso.py:279, in NYISO.get_interconnection_queue(self, verbose)
    277 completed.insert(17, "Proposed Initial-Sync", None)
    278 completed["Status"] = InterconnectionQueueStatus.COMPLETED.value
--> 279 completed.columns = active.columns
    281 # the spreadsheet doesnt have a date, so make it null
    282 completed["Proposed  In-Service"] = None

File ~/.local/lib/python3.9/site-packages/pandas/core/generic.py:5500, in NDFrame.__setattr__(self, name, value)
   5498 try:
   5499     object.__getattribute__(self, name)
-> 5500     return object.__setattr__(self, name, value)
   5501 except AttributeError:
   5502     pass

File ~/.local/lib/python3.9/site-packages/pandas/_libs/properties.pyx:70, in pandas._libs.properties.AxisProperty.__set__()

File ~/.local/lib/python3.9/site-packages/pandas/core/generic.py:766, in NDFrame._set_axis(self, axis, labels)
    764 def _set_axis(self, axis: int, labels: Index) -> None:
    765     labels = ensure_index(labels)
--> 766     self._mgr.set_axis(axis, labels)
    767     self._clear_item_cache()

File ~/.local/lib/python3.9/site-packages/pandas/core/internals/managers.py:216, in BaseBlockManager.set_axis(self, axis, new_labels)
    214 def set_axis(self, axis: int, new_labels: Index) -> None:
    215     # Caller is responsible for ensuring we have an Index object.
--> 216     self._validate_set_axis(axis, new_labels)
    217     self.axes[axis] = new_labels

File ~/.local/lib/python3.9/site-packages/pandas/core/internals/base.py:57, in DataManager._validate_set_axis(self, axis, new_labels)
     54     pass
     56 elif new_len != old_len:
---> 57     raise ValueError(
     58         f"Length mismatch: Expected axis has {old_len} elements, new "
     59         f"values have {new_len} elements"
     60     )

ValueError: Length mismatch: Expected axis has 20 elements, new values have 21 elements

However, when I install gridstatus from the main branch I don't receive this error. Is it possible to do a release soon so the patch to this bug is accessible in the pypi package? Thank you!

Refactor method naming

Current

Currently we implement methods for latest, today, and historical for each type of data.

nyiso = gridstatus.NYISO()
nyiso.get_latest_fuel_mix()
nyiso.get_fuel_mix_today()
nyiso.get_historical_fuel_mix("jan 1, 2022")

Proposed

now I am thinking it might be more usable to do

nyiso.get_fuel_mix("latest")
nyiso.get_fuel_mix("today")
nyiso.get_fuel_mix("jan 1, 2022")

by supporting the today value, we could also update the date range api to support something like this

nyiso.get_fuel_mix(start="jan 1, 2022", end="today")

One downside to this is that it becomes less clear how to easily document support for historical data.

For example, MISO and SPP currently only support getting the latest fuel mix. If you tried to call the today or historical methods, it would raise a NotImplemented. We can still support this, but need to manually check for it rather than rely on the subclassing.

Also would require updating the logic to generate the availability table in the documentation

Load Bus LBMP Availability?

Last week @kbaranko and I were chatting about the meaning of NYISO "Generator" LMBPs and thought I would summarize some things I learned from poking around in some NYISO documentation. Hoping that this might help get more granular LBMP data available in the tool at some point (I have no urgent need for it personally at the moment).

• Definition of LBMP (location based marginal price) - the cost to provide the next MW of load at a specific location in the grid.
• A gen bus LBMP is calculated where generator injects power (easily available)
• A zonal LBMP is the load weighted-average of all the load bus LBMPs in that load zone (easily available)
  • Unfortunately it doesn't seem like NYISO publishes the load bus LBMPs on either of the two main NYISO websites
  • I dont think its possible to back-calculate each load bus LBMP, even if you have the sub-zonal load accessible through the NYISO's Metering API. @kmax12 as a sidenote, its unclear to me how extensive NYISOs APIs are since Iv only ever explored the MIS website both gridstatus and NYISOToolkit are 'scraping' from.
  • FYI some load names with their zones can be found at the bottom of the MIS website

Some useful LBMP docs I came across...

• https://www.nyiso.com/documents/20142/31589128/4%20MST%2017.1%20-%20MATC%20Incremental%20Redline%20v3.pdf/5f89445d-fd44-3b88-aaed-0031cb0eb744
• https://www.nyiso.com/documents/20142/3037451/3-LMBP.pdf/f7682e03-e921-eaab-09bf-690524b5ade6

Standardized Fuel Names

Here are the fuels used by each of the 7 isos

New York ISO: ['Dual Fuel', 'Natural Gas', 'Hydro', 'Nuclear', 'Wind', 'Other Renewables', 'Other Fossil Fuels']
California ISO: ['Natural Gas', 'Imports', 'Large Hydro', 'Wind', 'Nuclear', 'Batteries', 'Solar', 'Geothermal', 'Biomass', 'Small hydro', 'Biogas', 'Coal', 'Other']
Electric Reliability Council of Texas: ['Wind', 'Solar']
ISO New England: ['Natural Gas', 'Nuclear', 'Net Imports', 'Hydro', 'Wind', 'Refuse', 'Wood', 'Landfill Gas', 'Oil', 'Other']
Midcontinent ISO: ['Coal', 'Natural Gas', 'Nuclear', 'Wind', 'Other', 'Solar']
Southwest Power Pool: ['Natural Gas', 'Coal', 'Wind', 'Nuclear', 'Hydro', 'Diesel Fuel Oil', 'Solar', 'Other', 'Waste Disposal Services', 'Waste Heat']
PJM: ['Gas', 'Nuclear', 'Coal', 'Hydro', 'Wind', 'Multiple Fuels', 'Other Renewables', 'Oil', 'Other', 'Solar', 'Storage']

Here is how many times each occurs

Wind                       7
Nuclear                    6
Other                      5
Natural Gas                5
Solar                      5
Hydro                      4
Coal                       4
Oil                        2
Other Renewables           2
Geothermal                 1
Small hydro                1
Biomass                    1
Other Fossil Fuels         1
Batteries                  1
Large Hydro                1
Imports                    1
Biogas                     1
Net Imports                1
Waste Heat                 1
Wood                       1
Multiple Fuels             1
Gas                        1
Refuse                     1
Diesel Fuel Oil            1
Waste Disposal Services    1
Landfill Gas               1
Dual Fuel                  1
Storage                    1

Looks like they are mostly standardized, but some are repeated

• "Imports" and "Net Imports"
• Gas in PJM must be what others call "Natural Gas"
• "Other" and "Other fossil Fuels"
• "Large Hydro", "Small Hydro", and "Hydro"

Code to generate above

import isodata

all_fuels = []
for i in isodata.all_isos:
    f = i().get_latest_fuel_mix()._mix_df.index.tolist()
    all_fuels.extend(f)
    print(i.name + ": " + str(f))

print(pd.Series(all_fuels).value_counts().sort_values(ascending=False))

Improve MISO LMP Data Sources

Here is a much better data source for MISO day ahead LMP market: https://www.misoenergy.org/markets-and-operations/real-time--market-data/market-reports/#nt=%2FMarketReportType%3ADay-Ahead%2FMarketReportName%3ADay-Ahead%20Pricing%20(xls)&t=10&p=0&s=MarketReportPublished&sd=desc

Here is the similar source but for the real time market: https://www.misoenergy.org/markets-and-operations/real-time--market-data/market-reports/#nt=%2FMarketReportType%3AReal-Time%2FMarketReportName%3AReal-Time%20Pricing%20Report%20(xls)&t=10&p=0&s=MarketReportPublished&sd=desc

These sources provide
historical data back to Jan 1, 2020. Only two downsides appears to be that it only have top line LMP and doesn't break down components and also only has data about the hubs. I think both of those tradeoffs are worth it

Add battery charging and discharging data to CAISO

Available are urls like this http://www.caiso.com/outlook/SP/History/20220801/storage.csv, which is similar to other CAISO data

Dashboard link

As far as I know, no other ISO publishes this data yet

Add building docs to automated tests that run on github

Would be nice to have the CI check that docs can build without error

Standard Logging Function

This issue tracks making a standardized logging function.

The codebase currently has a lot of blocks the following structure

if verbose:
    print(msg)

As a first pass, I am thinking something simple like

def log(msg: str, verbose: bool = False) -> None:
    if verbose:
        print(msg)

gridstatus.log("This is a message", verbose=True)  # prints "This is a message"
gridstatus.log("This is a message", verbose=False) # does nothing

Questions

• Print to stderr or stdout? Should this be a parameter
• Supporting logging to file
• Use python's builtin logging module

Add API for installed capacity

Data sources found so far:

• CAISO installed renewables - http://www.caiso.com/Style%20Library/caiso/csv/installedRenewables.csv
• Ercot - https://www.ercot.com/files/docs/2022/02/08/ERCOT_Fact_Sheet.pdf

Support querying across multiple days in historical queries

Currently, the historical API only supports querying by one day at a time.

Current API:

caiso.get_historical_demand("Jan 1, 2020")

Proposed API 1:

caiso.get_historical_demand(("Jan 1, 2020", "Jan 30, 2020"))

Proposed API 2:

caiso.get_historical_demand(start="Jan 1, 2020", end="Jan 30, 2020")

Implementation

Some ISOs have APIs where querying for multiple days at once is more efficient. In those cases, that API should be used. Otherwise, this API can be thought of as was a way to avoid multiple method calls.