sandialabs / pvops

A set of documented functions for supporting operations research of photovoltaic energy systems.

pvops contains a series of functions to facilitate fusion of text-based data with time series production data collected at photovoltaic sites. The package also contains example datasets and tutorials to help demonstrate how the functions can be used.

Installation

pvops can be installed using pip. See more information at readthedocs.

Tutorials

To get started with pvops we recommended working with the tutorials

Package Layout and Documentation

The package is delineated into the following directories.

├───docs                : Documentation directory
|
├───tutorials           : Contains tutorials of functionality
│   └─── example_data   : └─── Example data
|
└───pvops               : Source function library
    ├───tests           : ├─── Library stability tests
    ├───text            : ├─── Text processing functions
    ├───text2time       : ├─── Text2Timeseries functions
    ├───timeseries      : ├─── Timeseries functions
    └───iv              : └─── Current-voltage functions

More information about these modules is available at readthedocs.

Citing

If using this package, please cite our JOSS paper using the following:

Citation:

Bonney et al., (2023). pvOps: a Python package for empirical analysis of photovoltaic field data. 
Journal of Open Source Software, 8(91), 5755, https://doi.org/10.21105/joss.05755

BibTex:

@article{Bonney2023, 
  doi = {10.21105/joss.05755}, 
  url = {https://doi.org/10.21105/joss.05755}, 
  year = {2023}, 
  publisher = {The Open Journal}, 
  volume = {8}, 
  number = {91}, 
  pages = {5755}, 
  author = {Kirk L. Bonney and Thushara Gunda and Michael W. Hopwood and Hector Mendoza and Nicole D. Jackson}, 
  title = {pvOps: a Python package for empirical analysis of photovoltaic field data}, 
  journal = {Journal of Open Source Software} }

Contributing

The long-term success of pvops requires community support. Please see the Contributing page for more on how you can contribute.

Logo Credit: Daniel Rubinstein

Copyright and License

pvops is copyright through National Technology & Engineering Solutions of Sandia (NTESS). The software is distributed under the Revised BSD License. See the LICENSE file for more information.

pvops's Issues

[JOSS review] Remove travis.yml in MANIFEST.in

I think the line below can be removed.

pvOps/MANIFEST.in

Line 4 in fe6e657

exclude .travis.yml

[JOSS review] Merge Documentation and Developing pvOps

To simplify the left panel, it could be considered merging the Documentation and Developing pvOps pages.

This is merely a suggestion, which I do not feel strongly about - but perhaps worth considering.

timeseries model: provide option to pass in test-train dfs

[JOSS review assistance] Timeseries expected energy models

• The iec_calc model is described as being "based on IEC calculations" and using "IEC standards" in https://pvops.readthedocs.io/en/latest/pages/moduleguides/timeseries.html#module-overview and elsewhere. It looks like a simple performance ratio. A few suggestions:
  • mention which IEC standard/model is is based on
  • clarify in docs (e.g., https://pvops.readthedocs.io/en/latest/pages/apidoc/timeseries.html#module-pvops.timeseries.models.iec) that it needs plane of array irradiance as an input (assuming that is the case)
  • Make at least a brief reference to potential downsides/limitations of a simple model like this, e.g., when temperature is ignored.
• As far as I can tell, none of the models make use of temperature data (either back of module temp or the combo of ambient temp and wind speed) based on their own documentation. I would suggest adding this, or at least documenting why it isn't necessary. I would worry that someone like a performance engineer that is new to solar could start with this package and then assume that irradiance is all that really matters. I do see ambient temperature and wind speed passed in some of the examples, but it isn't clear to me if/how they are being used (for example, wind_speed_ms is commented out in the dynamic linear modeling example in tutorial_timeseries.ipynb).
• Similar to the last point, AC capacity might be useful. If the AIT or linear model tool can estimate AC capacity on their own, this should probably be mentioned in the docs.
• A comparison of the structure of these models with something like a pvwatts-style model, or PVUSA/ASTM regression, would be helpful.

[JOSS review] Broken image reference in tutorial

There's a broken image reference at the bottom of this tutorial: https://pvops.readthedocs.io/en/latest/pages/examples/tutorial_iv_diode_extractor.html

[JOSS review assistance] Tutorials - general feedback

Here are some of my initial thoughts about the tutorials in general:

• a markdown file summarizing each of the tutorials and/or suggesting an order in which a user should work through them would be helpful. I see that there is a bit more info in https://pvops.readthedocs.io/en/latest/pages/examples.html. If the readthedocs example page is the best place to start, that should be linked in the README.md when referencing the examples directory. E.g., "Refer to the examples directory, following the guide at https://pvops.readthedocs.io/en/latest/pages/examples.html, for a full run-through of the available functionality."
• https://pvops.readthedocs.io/en/latest/pages/examples.html calls these "examples" instead of "tutorials" - I would suggest consistent terminology.
• Incomplete abbreviations page in docs: I noticed that https://pvops.readthedocs.io/en/latest/pages/abbreviations.html is mostly incomplete. This came up when I was trying to find more context for the Timeseries AIT Tutorial.
• In general, more text in the tutorials to give context would be helpful. Perhaps this could be done by making setting up a fictional solar plant that all of the tutorials and example data would be based on. Give it a fake name, say where it is located, how big it is, what assets it has, etc. Talk about how it has a maintenance team that keeps written maintenance logs (show an example subset of a log file), and how there is there is a weather station and power meter with measurements logged to a data historian, etc. Then describe the tasks (real-world business needs/questions) that a performance engineer (the user going through the tutorials) will be trying to complete. In the process, reference all of the example data files that will be used in the tutorials.

Timeseries IV simulator class wrapper

[JOSS review] Add Statement of Need

For the JOSS review, it is stated that the package documentation AND software paper should have a Statement of Need. Currently, I do not see a statement of need in the software documentation (readthedocs); I suggest that this is added, e.g., on the main landing page.

[JOSS review] Avoid separate doc requirements.txt

I'm confused as to why there is a separate requirements.txt file for the documentation.

Is there a reason that these dependencies are not listed here:

pvOps/setup.py

Lines 54 to 56 in fe6e657

	DOCS_REQUIRE = [
	'sphinx == 2.2.0'
	]

I see no reason why they should not be included in the DOCS_REQUIRE, which lets you install documentation dependencies the standard way:

pip install pvops[docs]

Also, it's HIGHLY recommended to pin the versions of the documentation dependencies as this often ends up causing trouble down the road!

This is part of a JOSS review: openjournals/joss-reviews#5755

[JOSS review] Make .__version__ available

It's quasi-standard for Python packages to be able to return the package version using .__version__

This doesn't seem possible using the pvops package at the moment:

pvops.__version__

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pvops' has no attribute '__version__'

Having users easily retrieve the package version is important in science, thus I think this is somewhat important to implement.

This issue is part of a JOSS review: openjournals/joss-reviews#5755

Deprecated function pandas.DataFrame.append

Hi,
when running the example code in tutorial_textmodule.ipynb I get an error in cell [6]
The creation of the df_remapped_assets runs code in pvops/text/utils.py where the function DataFrame.append is used. However, this seems to be deprecated: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html?highlight=append

They advise to use pandas.concat instead.

To replicate, use pandas > 2.0

Remap attributes function must retain the unaltered attributes

Currently, if you run a df[ATTRIBUTE] through the remap_attributes function, it will only return the attributes which are being renamed. It should retain the unaltered attributes as well.

[JOSS Review] Link to example code

In your documentation you write the following:

An example implementation of all capabilities can be found in text_class_example.py for specifics, and tutorial_textmodule.ipynb for basics.

As those two tutorial files take a central role in your documentation: Could you add links to both text_class_example.py and tutorial_textmodule.ipynb? By doing so the reader of your documentation does not need to search through your github repo.

[Joss Review] Extend caption of Figure 1

Please extend the caption of figure 1.

This comment is part of a JOSS Review: openjournals/joss-reviews#5755

Timeseries filtering for common data integrity issues

Find flatlining, not-reporting, and outside nominal values, and filter.

timeseries modeling functions with regression

[JOSS review] Change CITATION file extension from .cif to .cff

The citation file has an incorrect extension (.cif).

The correct extension should be .cff. See this reference for further information.

Also, the package version specified inside the citation file is incorrect.

[JOSS review] Consider changing copyright/author

I noticed that the copyright/author on the readthedocs website is different from the main author of the JOSS paper. Perhaps it should be considered changing this to something more generic, e.g., "pvOps Developers" or to Sandia who is stated in the copyright file?

Note, I have no preference for what is provided, I merely wanted to bring attention to this to ensure it is still correct.

This is part of a JOSS review: openjournals/joss-reviews#5755

pvOps/docs/conf.py

Lines 21 to 22 in fe6e657

	copyright = "2021, Thushara Gunda"
	author = "Thushara Gunda"

Consider using bipartite layout for text.visualize.visualize_attribute_connectivity

Hi All,

Have you considered using a bipartite layout for the attribute connectivity plots produced by this function? The graph that this function creates is bipartite by definition, so it could help make the resulting visual more digestible.

All the best,
Ben

[JOSS review] Remove external licenses in NOTICE folder

I think it's quite unusual to include a folder with the licenses of dependencies.

One of the main reasons I'm against this is that it is a certainty that these licenses will be outdated. For example, the included pvlib license is no longer correct. pvlib does not do this, and given its affiliation with Sandia I do not think this is necessary here.

Perhaps @cwhanse can comment on this from a Sandia point of view?

[JOSS review] Address comment stating "This needs to be reworked"

Are you planning on addressing this comment before publishing the JOSS paper?

pvOps/pvops/iv/physics_utils.py

Lines 8 to 15 in fe6e657

	def calculate_IVparams(v, c):
	"""Calculate parameters of IV curve.
	This needs to be reworked: extrapolate parameters from linear region instead of
	hardcoded regions.
	Parameters
	----------

Add irradiance_anomalies and inverter clipping filters

[JOSS Review] Error in `tutorial_textmodule.ipynb` - supervised classification

When running the following cell I get an error.

Cell:

# Setting few cross validation splits because of few example data
results, best_model = e.classify_supervised(n_cv_splits=2, embedding='tfidf')
print('best_model', best_model)
results

Error:

UnboundLocalError: local variable 'results_df' referenced before assignment
ValueError: Invalid parameter 'normalize' for estimator RidgeClassifier(alpha=0.0). Valid parameters are: ['alpha', 'class_weight', 'copy_X', 'fit_intercept', 'max_iter', 'positive', 'random_state', 'solver', 'tol'].

Since there is some ML happening in the background it might also be related to my PC? Can you reproduce this error?

Add whatsnew in docs

update remappings_asset csv

remove duplicate entry (substation--> substation) and 2 incorrect mappings (energy storage/batter --> energy meter and switchgear --> transformer)

[JOSS review assistance] tutorial_timeseries

Feedback on https://github.com/sandialabs/pvOps/blob/master/examples/tutorial_timeseries.ipynb:

• A description of the example data is needed (each of the 3 files). I think it's ok to describe irradiance, ambient temp, and wind speed as "covariates", but a PV performance engineer might have to look up the definition of covariates (I just did...). In this case, I would also refer to those values as something like "met (meteorological) measurements" or "weather data" to help users put things in context.
• Just a suggestion, but the power vs covariates charts might be more interesting if you added a 3rd dimension with color. For example, changing the color of the markers in irrad_poa_wm2 chart based on ambient temperature, or in the other two based on irradiance.
• Repeating the visualization of all three power vs covariated after normalizing feels unecessary. Maybe consider dropping the charts before normalizing?
• For final results: The straight line over nighttime that is not at zero makes the charts hard to interpret. I'm not sure what the cleanest solution is...
  
• I suggest adding scatter plots of measured vs each model
• The "Mean squared error" and "Coefficient of determination" statistics are not immediately obvious to me. For example, it is very hard to know if 1037605.81 W is a relatively large or small error value. Including something like normalized mean bias error and normalized RMSE, with a short description of what they mean, might be helpful.
• I suggest adding a PVWatts-style model and/or PVUSA/ASTM regression model to the model performance comparison. And/or cite sources on how these models are used/validated elsewhere. I don't have a good sense for how good these models are and whether they are appropriate to use.

[JOSS review] Rename of IV package

I do not think the "IV package" is actually a package, but rather, I believe it is a subpackage.

pvOps/docs/pages/moduleguides/iv.rst

Line 1 in fe6e657

IV package

[JOSS review] Organize examples as tiles

The visualization of the examples is currently not very pleasing: https://pvops.readthedocs.io/en/latest/pages/examples.html

Instead, I suggest having a tile layout similar to what is done in pvlib python (see screenshot below).

This may require switching to the pydata sphinx theme (which I think is recommendable nonetheless)

Add IVSimulator to codebase

Fix documentation for text2time

Most of it has to do with new naming for some functions

[JOSS review] Test deprecation warnings

When running the tests, I get 13 warnings. Some of these are fine (e.g., divide by zero).

However, there are some deprecation warnings that should be fixed to avoid issues down the road.

• FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '2013-10-20 00:00:00' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
  om_df.loc[i, meta_cod] = cod_dates[i]
• FutureWarning: Series.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  prod_df.loc[:, prod_ener].fillna(method="ffill", inplace=True)
• MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap(obj) instead.
  cmap = plt.cm.get_cmap(cmap_name, len(asset_set))
• DeprecationWarning: module 'sre_constants' is deprecated
  import sre_constants

Feel free to tick off the above boxes as these are fixed.

There's also a conversion warning, but perhaps this is ok?

RuntimeWarning: some failed to converge after 100 iterations warnings.warn(msg, RuntimeWarning)

This issue is part of a JOSS review: openjournals/joss-reviews#5755

Unsupervised text document filtering algorithm to remove abnormal documents

Using embedded documents, conduct outlier detection through a isolated random forest (or other)

[JOSS review] What are the docker files used for?

I noticed the docker folder with two files - however I didn't see any mention of what that is being used for?

Add more effective baseline energy models (alternatives to IEC)

Generate asset classification model which will be available for users

Create efficient alternative to IV ParamExtractor

Use ODE solver like https://github.com/rtqichen/torchdiffeq

[JOSS Review] Add package overview to documentation page

It would be great if you give an overview of your package in your documentation page. Especially the content of Figure 1 from the paper could be reused for this purpose. Especially the two columns Type of data and Highlights of functions could be very useful on the documentation page.

Consider renaming functions in packages to avoid duplicate module names (i.e. text.preprocess and text2time.preprocess)

[JOSS review] Update abbreviations

It seems most of the terms on the abbreviations page are not defined.

Also, many of the items seem to not be abbreviations but rather symbols. Perhaps the title of this page should be changed to Terminology?

[JOSS review] Fix documentation build errors

When building the documentation, I get a few warnings. It would be nice to fix these issues.

Add example Jupyter Notebooks in notebooks with nbsphinx

https://nbsphinx.readthedocs.io/en/0.8.3/

[JOSS review] Add dates to What's New

I highly suggest adding dates in parentheses next to the release version in the What's New files.

This makes it much easier to get a sense of the development process and informs which What's New entries are important to read:

Here's an example of how pvlib does it: