solid-energy-systems / newarenda Goto Github PK

Python module and command line tool for reading and converting Neware nda and ndax battery cycling data files.

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

batteries

newarenda's Introduction

NewareNDA

Python module and command line tool for reading and converting Neware nda and ndax battery cycling files. Auxiliary temperature fields are currently supported in both formats.

Installation

To install the latest version from the PyPi package repository:

pip install --upgrade NewareNDA

To install the development branch directly from Github:

pip install git+https://github.com/Solid-Energy-Systems/NewareNDA.git@development

To install from source, clone this repository and run:

cd NewareNDA
pip install .

Usage

import NewareNDA
df = NewareNDA.read('filename.nda')

Logging

Additional test information including active mass, remarks, and BTS versions is returned via logging. The following command will print this logging information to the terminal:

import logging
logging.basicConfig()

Command-line interface:

usage: NewareNDA-cli [-h]
                     [-f {csv,excel,feather,hdf,json,parquet,pickle,stata}]
                     [-s] [-v]
                     [-l {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
                     [-c {chg,dchg,auto}]
                     in_file out_file

Script for converting Neware NDA files to other file formats. The default
output format is csv. Other formats may require installing additional
packages.

positional arguments:
  in_file               input file
  out_file              output file

options:
  -h, --help            show this help message and exit
  -f {csv,excel,feather,hdf,json,parquet,pickle,stata}, --format {csv,excel,feather,hdf,json,parquet,pickle,stata}
  -s, --software_cycle_number
                        Generate the cycle number field to match old versions
                        of BTSDA.
  -v, --version         show version
  -l {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}, --log_level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
                        Set the logging level for NewareNDA
  -c {chg,dchg,auto}, --cycle_mode {chg,dchg,auto}
                        Selects how the cycle is incremented.

Troubleshooting

If you encounter a key error, it is often the case that your file has a hardware setting that we have not seen before. Usually it is a quick fix that requires comparing output from BTSDA with values extracted by NewareNDA. Please start a new Github Issue and we will help debug.

newarenda's People

Contributors

Stargazers

Watchers

Forkers

vividh-garg balaranjan zellfei grimler91 adamopps jianglst ml-evs changbatterygroup

newarenda's Issues

KeyError: 'SvrVer'

When using NewareNDA version 2023.11.1 to read an NDAx file the code throws a KeyError: 'SvrVer'. It appears the NDAx file doesn't have this attribute or the name of the attribute may have changed in this newer NDAx file. The BTSServer is 8.0.0.323 (2023.05.31) and the Client version is 8.0.0.516(2023.05.31). Anyway we can look into the binary to see if this attribute has been changed to something else or removed? I tried this using the feature/ndax8 branch and the same issue arose.

Read a value that does not appear in the original NDAX file

Hello,

When I use this useful tool to read the NDAX file, I find one and only one row value is not aligned with the original NDAX file.

As is shown in this picture underlined by green line, the Time should be 0.0 instead of 12.0, Step should be 36 instead of 35. You can also refer to the original file screenshot.

Besides, when I read the NDAX file I got the warning " UserWarning: IMPORTANT: This ndax has missing data. The output from NewareNDA contains interpolated data! warnings.warn("IMPORTANT: This ndax has missing data. The output from " " The version of the BTS version is 8.0.0.526

Best regards,
Aaron

Bug fix: new unknown key

New unknown key in 2022.07.26 data:

01 Test data\Experiment2-10\S220502B046\S220502B046_NW_SES PROPRIETARY & CONFIDENTIAL.ndaTraceback (most recent call last):
df = NewareNDA.read(nda)
File "NewareNDA.py", line 56, in read
output.append(_bytes_to_dict(bytes))
File "NewareNDA.py", line 147, in _bytes_to_dict
multiplier = multiplier_dict[Range]
KeyError: 60000

NewareNDA Extraction Issue: Only One Row Extracted with Incorrect Values

Hello, I'm having an issue with the execution of NewareNDA. It only extracts a single line, and its values are incorrect. I'm currently working on a .nda file version 9 (130).

New log_level keyword not printing info

It's set to 'INFO' by default, but I'm not seeing the information printed to the terminal.

Missing columns

Hi I have the following issue:
when I load a ndax file in a dataframe, it doesn't show all the columns of the original ndax file.
for example I'm missing the specific discharge capacity.

Add a main function

Add a main function so NewareNDA can be used as a file conversion script. Easily exportable file formats include:
CSV, Excel, Feather, HDF, JSON, Parquet, Pickle, and STATA

Feature request: partial data read

Please consider this feature only if you could implement it with 'fseek' or similar without reading the whole file.

Add a startIndex to position the file pointer to where a record index is and read the rest of the file. Return a dataframe with indexes starting from the startIndex (instead resetting it to 0.)

For example,
index,column1,column2,column3
1, x, y, z
2, x, y, z
3, x, y, z
4, x, y, z

if the read method is called with startIndex=3, the return should be
index,column1,column2,column3
3, x, y, z
4, x, y, z

Convert 'Status' to a categorical dtype

https://pandas.pydata.org/docs/user_guide/categorical.html#categorical

Query Regarding Data Extraction from NDAX Files and Specific Capacity Information

May I kindly inquire whether it's feasible to extract information regarding the cycling layers from the ndax file? Additionally, I'm curious if the extracted data pertains to overall capacity rather than specific capacity. Would it be possible to obtain data in terms of specific capacity? thanks !

Temperature does not always match BTSDA output

The issue is that the aux_df join should occur on the Neware "Index" key rather than dataframe index. See for example S220502B012.

Read temperature field

Need an example of a file with temperature data

Regression tests for software_cycle_number and cycle_mode keywords

Using the existing test files, check the output for different keyword choices for consistency.

Reading aux data with NewareNDA as well

This issue hopes to shed some light on the format of auxiliary data in the ndax files, and to track progress in making NewareNDA read the data.

In the equipment I am working with is possible to have extra/auxiliary voltage and temperature measurements. In BTS settings it is then possible to select whether this data should be exported, and whether it should be merged with the main data or put in a separate excel sheet (when exporting to excel):

In the ndax archive the data then contains one data_Aux.ndc file, and one file per aux channel named something like YYYYMMDD_hhmmss_###_#_1.ndc, where the last digit is increased for each file.
Probably data_Aux.ndc contains info about how many aux channel files there are, and/or about their format, and the other ndc files contain the actual data.

Has anyone had any success in understanding the format of these aux ndc files?

Is it possible to export "cycle" data only

When transfer ndax data into excel, there will be several sheets, like "record", "cycle" and so on. So, is it possible to extract "cycle" data, in which only contain data of capacity, energy, start time, end time and any other data ,just like in excel from ndax file by this script?

In search of .NDA's hidden mass

I have delved into the depths of binary data extraction and have successfully found Active Mass, Remarks, and some other interesting data points for .nda files generated by Neware 4000 and 9000 instruments. I do not have access to .ndax files, so I cannot provide any information about those unfortunately.

NEWARE 4000 (and perhaps other machines)

The entered "Active Material" value exists at byte 0x00000098 (decimal: 152).
It appears that the data is stored in LE as a 4-byte uint (BTS Software does not let you enter a negative number.)
The value decoded is actually the ActiveMaterial*1000 (in other words, to extract active material, divide this decoded value by 1000)
When specifying Active Material in BTS Software, the default is mg, but there are other options for units, including some volumetric types. I do not know how changing units affects this value.
The "Remarks" section exists at byte 0x0000090D (dec: 2317)
It is simply an ASCII/UTF-8 encoded string.
I suspect that it is 100 Bytes long, but this is untested.

NEWARE 9000

The same metadata type information is stored in the .nda files, but as a footer to the data, unlike the 4000 case.
I have identified the footer as beginning as a byte string of the following:
06 00 f0 1d 81 00 03 00 61 90 71 90 02 7f ff 00
This is used to signify the beginning to the header where our relevant metadata is stored.
I have seen this exact string in a few of my datafiles, but it is very possible that at least part of this string is not static, I just have not seen that case yet.
The first 107 bytes after the signifier are not decoded yet (and I have no plans to do so at the moment.)
The next 128 bytes are ASCII text for the "Creator" or "Operator"
The next 128 bytes are ASCII text for the "P/N"
The next 128 bytes are ASCII text for the "Remark Information" or "Comments"
The next 8 bytes are LE encoded double float for the "Active Material"
There is more in the footer following this, but it does not seem important to me at the moment. There is another big chunk of ASCII, so feel free to investigate at your interest.
- Table of 9k Footer Storage Structure:
  
  Bytes after signifier |dtype | data
  1-107 |? | Unknown
  108-235|ASCII | Creator (Operator)
  236-363|ASCII | P/n
  364-491|ASCII | Remarks (Comments)
  492-499|double| Active Material

I leave implementation details to the more experienced among us. I can certainly take a stab at adding some extraction code, as I plan on doing so for my personal work in the meantime. However, I don't know whether any of these extractions should be added to the main "read()" functions, or if they should be their own methods.

I can imagine extracting the mass and adding dedicated "Specific Capacity" Columns to the main read_nda() function. It is also possible to have some separate function along the lines of a "read_metadata()" function that returns a simple dictionary with Remarks, Active Material, P/N, Run Date, etc. as we find them. Either implementation would be valuable in my opinion.

There are a few other data structures that I have found in the Neware 4000 file format. In short, they aren't useful, so I wont provide too much detail. One instance is a data structure that extracts with the same format as the main data points, but it has a different identifier ('\xAA' instead of '\x55'). These data blocks appear to be the last data point of each step. Within these data blocks, there are 8 bytes of unidentified data that appear to be values for Differential Capacity.

Automatic cycle_mode detection causing errors for NDA files with only rest states

KeyError due to missing element in multiplier_dict

Hello,
I am developing a script to read neware nda files into my database.
I have encountered the following error due a missing key/value in the multiplier dict.

Stacktrace:

2023-12-06T15:27:08.923267 [INFO] NewareNDA.read_nda: NDA version: 29
2023-12-06T15:27:08.923267 [INFO] NewareNDA.read_nda: Server: BTSServer(R3)-7.6.0.301 (2018.07.30)
2023-12-06T15:27:08.923267 [INFO] NewareNDA.read_nda: Client: BTS Client 8.0.0.474(2020.12.22)
Traceback (most recent call last):
  File "C:\Users\User\projects\project\src\prototype\main.py", line 51, in <module>
    main()
  File "C:\Users\User\projects\project\src\prototype\main.py", line 43, in main
    df = NewareNDA.read(temp_file)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\projects\project\venv\Lib\site-packages\NewareNDA\NewareNDA.py", line 30, in read
    return read_nda(file, software_cycle_number)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\projects\project\venv\Lib\site-packages\NewareNDA\NewareNDA.py", line 94, in read_nda
    output.append(_bytes_to_list(bytes))
                  ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\projects\project\venv\Lib\site-packages\NewareNDA\NewareNDA.py", line 152, in _bytes_to_list
    multiplier = multiplier_dict[Range]
                 ~~~~~~~~~~~~~~~^^^^^^^
KeyError: 1

Possible Solutions:

Add element to for key 1 with value 1e-4
This appears to be the correct multiplier for my example based off my own testing.
Unfortunately, I am not able to share the data files as it is confidential.
Set multiplier via keyword argument
df = NewareNDA.read('my_file.nda', multiplier={1:1e-4})

Please advise on next steps to move this forward.

Regards,
Steven Gallagher

_interpolate_data() function produces strange results and misleading warning

When using the read_ndax() function without interpolating (I commented out the line which calls that function), the output is like so:

But the output when using the interpolate_data() function is like so:

There are several strange things happening when interpolating:

The first few Charge_ and Discharge_Capacity values are changed to NaN values.
The Cumulative Charge Energy and Charge Capacity are summed (but not quite exactly...)
The timestamp is now non-monotonic between indices 2 -> 4 (pandas index, not Neware index).
The increase in time between (pandas) index 1 and 2 (0.6s) does not match the increase in timestamp (5s).
The Charge Capacity is positive in a CC_DChg step (pandas index 3), and then decreases within the same step to zero (pandas index 4).

Furthermore, I am a little confused about when the warning for interpolating data appears. The first clause in the _interpolate_data() function is:

nan_mask = df["Time"].notnull()

And if this is True for any row, then the warning is raised. But surely the interpolation warning (and the interpolation itself) should only happen if this is False (i.e. when none of the rows of Time are null). Then further on in the function, lots of replacing of values happens for rows where Time and TimeStamp is not null, but why are these ones being replaced? Surely these notnull() calls should be isnull()?

I have maybe misunderstood, so if this is the case, could you explain why these weird things are happening?

Thanks,
Ian

Cell with invalid records

230094-4-10-359.nda (Record 59890)
230094-4-13-359.nda (Record 60376)

Ability to read ndax format

Accessing data_log.ndc and data_es.ndc

Hi! This is not so much an "issue", but a request for help.

I am giving Neware the benefit of the doubt and really hoping that they are not interpolating data. I wonder if the missing data is contained in the data_log.ndc and data_es.ndc files? These are currently not converted from binary in this package.

I tried to convert these by adapting the functions that you have written for the other files, but I am not sure what the numbers you have used in the conversions should be e.g. Record length, header, and the numbers used here and here.

How did you find these numbers when you were creating the existing functions?

Have you tried to convert these other ndc files in the later versions of the software that have the missing data issue talked about here?

Incorrect cycle numbers

Cycles do not always match Neware. I started a new cycle at each charge step, but Neware seems to require a discharge step to have occurred since the last cycle.

Example file: B411001020_0201.nda

Minor Logging Functionality Rework - Align implementation to best practices

Intention:

Log to a specific logger labeled 'newarenda'. Common practice is for modules to log to a logger with the same name as its package. Currently, these scripts log to 'root' which inserts itself into the default logger for any application and is not recommended.
Include some warnings in the log where applicable.

Are there any other modifications to be made? Things to consider?
EDITS + ADDITIONS:

arg to set the desired log level in read() and CLI script

BTS85 charge/discharge capacity values are unrealistic

Hi there!

Could you assist me in overcoming this issue? When I run the NewareNDA on the latest files generated from BTS85, the charge/discharge capacity values are unrealistic. However, when we open these files in BTSDA 8.0.0.526, the values appear normal (please refer to the attached figure). For your convenience, I have attached an example file to this message. I am eager to resolve this issue.

BTS85-36-6-5-110-20240424.zip

Missing multiplier: add option to allow no-rescaling

Hi there, thanks for this useful package! I've been trying to integrate Neware support into navani (a wrapper for several packages that allows us to normalize battery data across all brands in our lab) and thus datalab (our group's data management platform).

Some of the test data I've been provided does not work with the current version of this package, as we are missing the instrument Range=5. I would really like to be able to use this package in a mode that is robust to missing multipliers, e.g., give a gigantic warning but do not error if the instrument scaling has not been determined, as in our case, the user can supply the desired scaling manually to be stored alongside the data file. Either this could be enabled by an option, or be the default.

I've coded this up in my fork which I can continue to use for now, but thought I should open an issue in the meantime to accompany an upcoming PR. Is this functionality you would like to add?

Cells that fail to load

R176002001C5
JA0101C3C301
JALFPLiCyc02

Cycle number calculation vs decoded from data

#3 Related to this closed cycle no. issue.
Why do you calculate cycle no. when you can just decode it from the data itself as Neware stores the cycle no. natively (as seen in the xls/xlsx files). There are many recipes/tests which involve multiple instances of charge and discharge in the same cycle. So your calculation comes out wrong in those tests.

I am currently using the code below to get the cycle number. Essentially, the correct cycle number here will be 'Cycle +1'.

[Index, Cycle] = struct.unpack('<IH', byte_stream[2:8])

EOFError, no valid records and empty 'Step' Series

I am trying to use the CLI script to export .nda files from a Neware BTS9000 to excel and have run into a few errors. On first attempt, I was getting the EOF-Error, stating that there are no valid records. Upon binary file inspection, the file did not have the specified identifier contained in it.

Based on my best guess, I set the header length to be a static 1024 bytes and bypassed the header loop. Doing this raised an IndexError in _count_changes, thus the data series (df['Step']) passed was empty. It looks like my .nda file does not save data in the same structure than this script expects. The .nda file isn't empty, as I can load it back into the BTS client (version 9.0.x.x) on a different computer and the data is present.

Any ideas of what might be going on? Is there some software setting in BTS/BTS9 that I should be using? I can send an example .nda file or two if you are willing to look into it.

Thanks!

Is cycle number stored for each record or must it be calculated?

FutureWarnings for Pandas 3.0

Cells that fail to load

Fails at the date field:
B475006001
B475006003
S 220502B010_-25degC_SES PROPRIETARY & CONFIDENTIAL

Old nda's cause NewareNDA to hang

Here's an nda (version 8) that fails to load:
https://github.com/FTHuld/neware_reader/blob/master/tests/testdata/old_nda_file.nda

Missing test data and CI

Related to #47 and an upcoming PR, there are limited contribution instructions, and it is currently impossible to run the integration tests as the test data is not provided in this repo.

Also, there is no continuous integration testing of new PRs (via e.g., GitHub actions) which can go a long way to prevent contributions and also make expectations clear.

I'm happy to help set this up (it is 5 mins work once you've done it before) if you like, but probably having the test data available is a minimum requirement!

Cells failing to load

Precision_DS_2_co

Automated regression testing

Here is a running list of NDA's that are difficult to parse:

: Edge case - no records
56_8_8_133_20220108163707.nda: Edge case - one usable record
: First Index is not 1
SNR015008002.nda: dates are stored as Unix timestamps
B478001013.nda: Final cycles contain consecutive CC_DChg steps
B448L1-07_DC.nda: First cycle contains only discharge

PerformanceWarning

NewareNDAx.py:131: PerformanceWarning: Adding/subtracting object-dtype array to TimedeltaArray not vectorized.

state_dict[Status] key error

Hello, I am having the following issue for .ndax files, receiving the error KeyError: 0
Really appreciate any help on this if possible

Traceback (most recent call last):

File ~/anaconda3/envs/NewareDataAnalysis/lib/python3.10/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
exec(code, globals, locals)

File ~/Downloads/untitled3.py:10
df1 = NewareNDA.read('B12023-SAM13.ndax')

File ~/anaconda3/envs/NewareDataAnalysis/lib/python3.10/site-packages/NewareNDA/NewareNDA.py:30 in read
return read_ndax(file)

File ~/anaconda3/envs/NewareDataAnalysis/lib/python3.10/site-packages/NewareNDA/NewareNDAx.py:28 in read_ndax
data_df = read_ndc(data_file)

File ~/anaconda3/envs/NewareDataAnalysis/lib/python3.10/site-packages/NewareNDA/NewareNDAx.py:54 in read_ndc
output.append(_bytes_to_list_ndc(bytes))

File ~/anaconda3/envs/NewareDataAnalysis/lib/python3.10/site-packages/NewareNDA/NewareNDAx.py:92 in _bytes_to_list_ndc
state_dict[Status],

KeyError: 0

Can this tool support BTS8.0 server in future？

it seems that neware update there data protocol in BTS8.0

NewareNDA-cli.py export problem with non-default index

See for example:
H009M7PS5A091
H009M7PS5A110
H009M7PS5A074

Issue with interpolating data from ndax file

Hi, having an issue with interpolating missing data from ndax files. Have not modified any of the original code. Just installed the package and am trying to run the example usage (import NewareNDA --> df = NewareNDA.read('testndax.ndax'). Consistently seeing the following issue:

/usr/local/lib/python3.10/dist-packages/pandas/core/frame.py in interpolate(self, method, axis, limit, inplace, limit_direction, limit_area, downcast, **kwargs)
11853 **kwargs,
11854 ) -> DataFrame | None:

11855 return super().interpolate(
11856 method,
11857 axis,

TypeError: super(type, obj): obj must be an instance or subtype of type

triggered by this line in the _data_interpolation(df) function

/usr/local/lib/python3.10/dist-packages/NewareNDA/NewareNDAx.py in _data_interpolation(df)
103
104 # Group by step and run 'inside' interpolation on Time
--> 105 df['Time'] = df.groupby('Step')['Time'].transform(
106 lambda x: pd.DataFrame.interpolate(x, limit_area='inside'))
107

any help is appreciated. thanks!

Some timestamps are converted to local time

NewareNDA sometimes calls datetime.fromtimestamp() which is time-zone-aware, and so some dates are converted to the user's time zone. Since no time zone information is stored in nda/ndax, these calls should probably be replaced with datetime.utcfromtimestamp() which is naive.

Some aux data missing from (some) files

Hi,

I have noticed that NewareNDA sometimes fails to extract all aux data, and instead just returns NaN after some point. Non-aux values work fine. Here is an example plot with capacity and aux temperature plotted with data from either an exported xlsx file, or from NewareNDA:

As can be seen the aux temperature is missing in NewareNDA data after roughly cycle index 250.
(I will send the data to Dan in private)

Multiplier Key Error?

Hi, having an issue loading cell data for .nda files, receiving the error KeyError: -1.
Just using the df1 = NewareNDA.read(path) command.
Any help appreciated, thanks.
Full error report below.

`---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[49], line 1
----> 1 df1 = NewareNDA.read(path)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\NewareNDA\NewareNDA.py:28, in read(file, software_cycle_number)
26 _, ext = os.path.splitext(file)
27 if ext == '.nda':
---> 28 return read_nda(file, software_cycle_number)
29 elif ext == '.ndax':
30 return read_ndax(file)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\NewareNDA\NewareNDA.py:88, in read_nda(file, software_cycle_number)
83 if len(bytes) == record_len:
84
85 # Check for a data record
86 if (bytes[0:2] == b'\x55\x00'
87 and bytes[82:87] == b'\x00\x00\x00\x00'):
---> 88 output.append(_bytes_to_list(bytes))
90 # Check for an auxiliary record
91 elif (bytes[0:1] == b'\x65'
92 and bytes[82:87] == b'\x00\x00\x00\x00'):

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\NewareNDA\NewareNDA.py:146, in _bytes_to_list(bytes)
143 if Index == 0 or Status == 0:
144 return []
--> 146 multiplier = multiplier_dict[Range]
148 # Create a dictionary for the record
149 list = [
150 Index,
151 Cycle + 1,
(...)
161 datetime(Y, M, D, h, m, s)
162 ]

KeyError: -1`

Step number incorrect

Looking at changes in 'Jump' does not work for this file:
SNR015007001.nda

Aux data is returned as float64 but could be float32 for performance

The other fields have dtypes defined in dtype_dict.