echemdata / galvani Goto Github PK
View Code? Open in Web Editor NEWRead proprietary file formats from electrochemical test stations
License: GNU General Public License v3.0
Read proprietary file formats from electrochemical test stations
License: GNU General Public License v3.0
Remove python 2 compatibility code
https://travis-ci.org/github/echemdata/galvani/builds/704756307/config
Build config validation
root: deprecated key sudo (The keysudo
has no effect anymore.)
root: missing dist, using the default xenial
root: missing os, using the default linux
Use the logging
module to write some debug output on what is happening during file loading.
This can also be used for warnings etc.
The BioLogic parser code exits from a generator by raising StopIteration
. This is no longer supported in Python 3.7 and leads to a RuntimeError
.
https://docs.python.org/3/whatsnew/3.7.html#changes-in-python-behavior
Currently the Bio-Logic .mpr parser exits with an error if it encounters an unknown column ID. This made sense at the time of writing because I had no way of knowing the data type of the new column and thus how many bytes it would take up in the file.
Looking at the file history, it seems that all new columns added in the past three years have been '<f4' (32-bit float). https://github.com/echemdata/galvani/blame/master/galvani/BioLogic.py
This means that it is probably safe to assume that any new unknown columns will also be 32-bit floats. This will make it possible to load the file despite the presence of unknown columns.
I started looking at the installed EClab.exe
(version 11.36) and found some interesting data. Apparently, all column names are stored in a sorted array at 0x143E56D and continuing:
# Maps from colID to a tuple defining a numpy dtype
# names can be read from installed EClab.exe (version 11.36 from 0x143E56D)
VMPdata_colID_dtype_map = {
#TODO: ('ox/red', TODO),
#TODO: ('error', TODO),
4: ('time/s', '<f8'),
5: ('control/V/mA', '<f4'),
6: ('Ewe/V', '<f4'),
7: ('dQ/mA.h', '<f8'),
8: ('I/mA', '<f4'), # 8 is either I or <I> ?? --> it is I/mA
9: ('Ece/V', '<f4'),
# 10: ('Aux/V', TODO),
11: ('I/mA', '<f8'),
# 12: ('log(|<I>/A|)', TODO),
13: ('(Q-Qo)/mA.h', '<f8'),
# 14: does not exist?
# 15: does not exist?
16: ('Analog IN 1/V', '<f4'),
# 17: ('Analog IN 2/V', '<f4'),
# 18: ('Analog IN 3/V', '<f4'),
19: ('control/V', '<f4'),
20: ('control/mA', '<f4'),
# 21: ('control changes', TODO),
# 22: ('log(|I/A|)', TODO),
23: ('dQ/mA.h', '<f8'), # Same as 7?
24: ('cycle number', '<f8'),
# 25: ('DQ/mA.h', TODO),
# ...
}
There are some bytes of variable length between the column names and it looks like they might contain type information. Anyways, I will work on this with low priority but I will share my findings as soon as I have more.
To allow collaborative development work, see discussion on #26
Create an organization: https://github.com/organizations/new
Transfer the galvani repo there: https://help.github.com/en/articles/transferring-a-repository
Add @bcolsen as a collaborator
When running the tests:
DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
I got tired of EC-lab's plotting functionality, so I've been working on my own program for live data plotting using EC-lab's .mpr files.
However, the plot sometimes stops updating and the error below is printed.
ValueError: Found b'\x82\xe0zOVP', expecting start of new VMP MODULE
I have no problems otherwise plotting the data, and I can restart the live plotting program and it'll work for a while until it encounters this error again.
Do you have any idea what causes this or how I could fix it? I tried looking around in the Biologic.py file, but don't really understand in which instance the error is triggered.
It appears from initial inspection that .mpr
files use local time for timestamps. Further investigation is needed.
Trying to add 'Acir/Dcir Control', but I failed, example mpr and mpt can be found at the attached links.
https://drive.google.com/file/d/1DZ7tvy2EOntFyqlfnam-CL6AhmaJGAl1/view?usp=sharing
https://drive.google.com/file/d/1BIHCRX_ja-4tqGGLpkHK-I0ESMnbOsOQ/view?usp=sharing
Pandas is significantly easier to use. On the other hand, an end user can easily convert the numpy array to pandas after receiving it as a result.
Hi, sorry to bother you again, but I think I may have found another unimplemented feature which is now causing an error.
When I try to load in the attached file, I get "NotImplementedError: column type 168 not implemented".
I am having this same problem with all the data I have recorded today. I didn't update EC-lab or change the parameters of the technique so I find it really odd that the data from today cannot be loaded when all the older data files can.
Is there any possibility to extract the test-set up information like the ones contained in the .mps from the .mpr?
Do you know if they are hidden in the file or if they can not be retrieved from the .mpr?
I was trying to read one .mpr file and it was continuously generating the following errors:
1)'NotImplementedError: Column ID 174 after column control/mA is unknown'
BioLogic.py script modified and it is now reading the .mpr file correctly:
Here below the change:
(line 186) 174: ('Ewe/V', '<f4'), added.
At the moment I am reading properly also all the other files.
Please double-check if this change can be pushed
Test_error.zip
?
Because I also have to work with Bio-Logic instruments I tried to port your source (the mpr part) to IgorPro (because I use Igor for my work on the data).
There I noticed some differences in the size of the colIDs:
4 and 6 can be both f4 (in CA scans) and f8 (in CV scans),
11 and 24 were f4 (not f8),
131 was u1 (not u2), and
435 is f4 (new?) - gives the same data as 434.
Only with these settings I got reasonable data out of the file.
I obtain different results when exporting data from EClab into text format and when using .mpr file and galvani (before loading the data in a dataframe).
Using text file, I obtain the following columns:
mode, ox/red, error, control changes, Ns changes, counter inc., Ns, time/s, dq/mA.h, (Q-Qo)/mA.h, control/V/mA, Ecell/V, I Range, Energy charge/W.h, Energy discharge/W.h, Capacitance charge/µF, Capacitance discharge/µF, I/mA, x, Q discharge/mA.h, Q charge/mA.h, Capacity/mA.h, Efficiency/%, control/V, control/mA, cycle number, P/W.
Whereas with galvani from.mpr I only obtain:
'flags', 'Ns', 'time/s', 'dQ/mA.h', '(Q-Qo)/mA.h', 'control/V/mA', Ewe/V', 'I Range', 'Q charge/discharge/mA.h', 'half cycle', 'Energy charge/W.h', 'Energy discharge/W.h', 'Capacitance charge/µF', 'Capacitance discharge/µF'
I'm unfamiliar with binary files and I found this package excellent for my purposes. The .mpr file I have is not corrupted and send no error. The missing columns are not present in the VMPdata_dtype_from_colIDs function (at least the Intensity I/mA column which I was particularly interested in,as well as P/W), so the import error must be previously in the code, such as in read_VMP_modules. My skills did not allow me to go much further than that...
Hope you can fix this!
Hi, I tried using Galvani for the first time and when trying to load the mpr file from a BioLogic potentiostat, it raises the following exception:
ValueError: time data '11.12.20' does not match format '%m/%d/%y'
ValueError: time data '11.12.20' does not match format '%m-%d-%y'
It seems like my date format uses dots instead of "/" or "-". Don't know if it has to do with my EC-Labs settings or maybe that the date format of the OS is set to using "." (which is the standard for German date format).
I changed lines 398ff. in "BioLogic.py" to
try:
tm = time.strptime(settings_mod['date'].decode('ascii'), '%m/%d/%y')
except ValueError:
try:
tm = time.strptime(settings_mod['date'].decode('ascii'), '%m-%d-%y')
except ValueError:
tm = time.strptime(settings_mod['date'].decode('ascii'), '%m.%d.%y')
and the same for the similar lines 418ff. which solved the problem. Maybe you want to integrate the additional date format in Galvani, @chatcannon.
BTW: Thanks a lot for this really nice python package! :)
Hi!
I'm doing my Masters and will probably continue with my PhD working with EC-Lab quite a bit and I've just about had it with the .mpr file format which requires me to export the data manually every single time.
I stumbled upon this seemingly amazing module and I've been trying to import data using the BioLogic.MPRfile function, but I can't seem to get it to work as I get "ValueError: Unrecognised version for data module: 3" when I run
BioLogic.MPRfile("some_file.mpr")
I have some experience with python but mainly as a tool for data analysis, so my troubles may be a result of my very limited Python knowledge.
I was hoping that you might have an idea as to how I could get this module to work so that I may import .mpr files directly into Python.
(I wasn't really sure how to reach out for help here so I opened an issue as I could find no other way of doing so. I'm sorry if this was an inappropriate way of going about it)
The "if name in (long list) elif name in (other long list)" block is unwieldy. It should be refactored to use sets or mappings.
Here is my code:
`from galvani import res2sqlite as r2s
Input_file = 'C012.res'
Output_file = 'new'
r2s.convert_arbin_to_sqlite(Input_file, Output_file)`
Here is the output:
Reading Version_Table...
Error while importing Version_Table
Traceback (most recent call last):
File "/pithy/code/arbin_import_test.py", line 19, in
r2s.convert_arbin_to_sqlite(Input_file, Output_file)
File "/usr/local/lib/python2.7/dist-packages/galvani/res2sqlite.py", line 417, in convert_arbin_to_sqlite
mdb_get_data(s3db, input_file, table)
File "/usr/local/lib/python2.7/dist-packages/galvani/res2sqlite.py", line 397, in mdb_get_data
mdb_get_data_text(s3db, filename, table)
File "/usr/local/lib/python2.7/dist-packages/galvani/res2sqlite.py", line 373, in mdb_get_data_text
mdb_sql.terminate()
UnboundLocalError: local variable 'mdb_sql' referenced before assignment
Can you help me with this error?
Thank you
I ran into a similar bug as #27, while doing Biologic.MPRfile('path/to/mpr.mpr')
ValueError: Invalid magic for .mpr file: b'\r\n<Configurati'
from
File "/Users/suweiguo/anaconda3/envs/eclab2/lib/python3.6/site-packages/galvani-0.1.0-py3.6.egg/galvani/BioLogic.py", line 314, in init
which is probably due to the very old Biologic VMP
I'm trying to open .mpr files using the galvani library.
When opening files of GCPL technique of BioLogic, everything is fine.
However, when opening files after the GCPL2 technique, I got error:
NotImplementedError: Column ID 27 after column control/mA is unknown
Looks like this file has some extra, unexpected column.
Is there a way to make this column recognizable and be able to read the file with Galvani?
Including the "build:passing" button
When I run pytest
from the project root directory, 18 test fail, all with the same error message:
____________________________________________________________ test_MPR6_matches_MPT6 _____________________________________________________________
testdata_dir = '/home/user/galvani/tests/testdata'
def test_MPR6_matches_MPT6(testdata_dir):
> mpr = MPRfile(os.path.join(testdata_dir, 'bio_logic6.mpr'))
tests/test_BioLogic.py:213:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <galvani.BioLogic.MPRfile object at 0x75c16c88b430>, file_or_path = '/home/user/galvani/tests/testdata/bio_logic6.mpr'
def __init__(self, file_or_path):
self.loop_index = None
if isinstance(file_or_path, str):
mpr_file = open(file_or_path, 'rb')
else:
mpr_file = file_or_path
magic = mpr_file.read(len(MPR_MAGIC))
if magic != MPR_MAGIC:
> raise ValueError('Invalid magic for .mpr file: %s' % magic)
E ValueError: Invalid magic for .mpr file: b'version https://git-lfs.github.com/spec/v1\noid sha25'
galvani/BioLogic.py:382: ValueError
I was trying to read one .mpr file and it was continuously generating the following errors:
1)'NotImplementedError: Column ID 69 after column (Q-Qo)/mA.h is unknown'
BioLogic.py script modified and it is now reading the .mpr file correctly:
Here below the change:
(line 163) 69: ('R/Ohm', '<f4'), added.
2)'NotImplementedError: Column ID 26 after column Energy/W.h is unknown'
BioLogic.py script modified and it is now reading the .mpr file correctly:
Here below the change:
(line 155) 26: ('Rapp/Ohm', '<f4'), added.
[see attached file
At the moment I am reading properly also all the other files.
Please double-check if this change can be pushed?
Hi,
Thanks for your code! I would like to use the first part to convert the MPT file into an array.
However I get the following error when calling MPTfile , and I really don't understand what to change :( Would you tell me what is going on?
error line : str3 = lambda b: str(b, encoding='ascii')
error message : UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 165: ordinal not in range(128)
Thank you.
Marion
Hello,
I'm getting a ValueError from the date pulled by BioLogic.py. The weird thing is it worked on files generated yesterday but not today, and we haven't changed anything. I believe the data file was generated by version 11.02 of EC-labs. The full error is below. Thanks!
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.7/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/FREDsense/automated_analysis/WebFrontEnd.py", line 84, in ec_assay
output = runAnalysis(file_filenames, timestamp)
File "/home/FREDsense/automated_analysis/scripts/Peak_Analysis.py", line 92, in runAnalysis
mpr = MPRfile(filename)
File "/home/FREDsense/.local/lib/python3.7/site-packages/galvani/BioLogic.py", line 353, in init
tm = time.strptime(str3(settings_mod['date']), '%m/%d/%y')
File "/usr/lib/python3.7/_strptime.py", line 571, in _strptime_time
tt = _strptime(data_string, format)[0]
File "/usr/lib/python3.7/_strptime.py", line 359, in _strptime
(data_string, format))
ValueError: time data '03-14-19' does not match format '%m/%d/%y'
I'm trying to run the convert_arbin_to_sqlite
method on one of my .res files and also specify the output file to be in a .s3db format. However, the code always fails on that line and provides me with this error:
RuntimeError: Could not locate the 'mdb-export' executable. Check that mdbtools is properly installed.
I then pip installed mdbtools and ran a pip freeze to see that I have version 0.3.14 installed. I also tried including an import mdbtools
statement at the top of my file to see if that fixed the problem, but then I get this error:
ModuleNotFoundError: No module named 'MDButils'
I'm not sure why this is happening, so I'd appreciate any assistance or guidance from any of you. Thank you so much.
Since galvani
is primarily designed to solve an "upstream" issue in electrochemical data analysis, I think galvani
is especially powerful when combined with other electrochemical data analysis packages. Unfortunately, the GPL-3.0 copyleft license is restrictive, which limits its utility for these contexts. See this issue as an example.
Would the original authors consider releasing this code with a more permissive license such as MIT? There is some discussion of the legal issues involved on SO, see here as an example. Thanks!
This project uses setup.py
which still gets legacy support. According to the documentation, modern Python projects should use pyproject.toml
instead.
Checking tools to consider:
flake8 plugins:
e.g. using a C extension
Add machine-readable copyright and licensing metadata.
I've been looking at the past couple of issues regarding unparsable columns due to not enough colID
s present in the dictionaries in the BioLogic.py source code. I followed the procedures in several of these issues and was able to retrieve some more column headers by adding new colIDs myself. When looking at my .mpt file, I notice that there are around 8-9 additional columns present in my file which aren't present in the dictionaries. I was successfully able to retrieve two columns from those because I kept getting a NotImplementedError
along with a colID
that I had to enter in the dictionary. However, after I got these two columns, I stopped receiving NotImplementedError
s for some reason, even though from the corresponding .mpt file I know that there are more columns I need to parse which aren't found in the BioLogic.py dictionaries. When I run mpr_object.dtype.names
to print out all the columns which got parsed, I get the following tuple:
('flags', 'Ns', 'I Range', 'time/s', 'control/V/mA', 'Ewe/V', 'I/mA', 'dQ/mA.h', '(Q-Qo)/mA.h', 'Energy/W.h', 'Q charge/discharge/mA.h', 'half cycle', 'Temperature/°C', 'Energy charge/W.h', 'Energy discharge/W.h', 'Capacitance charge/µF', 'Capacitance discharge/µF')
However, when I check through my .mpt file, there are more columns past the last one which are left to be parsed but aren't being parsed for some reason. I also am not receiving any additional NotImplementedError
s for me to be able to reverse engineer the remaining columns. One thing I did notice, however, is that in the .mpt file, the next column which isn't being parsed and is located immediately after Capacitance discharge/µF
called x
. It seems like this is the only column name in the file which is just a char and not a string, so could there be some fundamental comparison issue in the source code that could possibly explain why all the columns past x
aren't being examined/parsed by the BioLogic.py code?
Please let me know if this is an issue anyone has received before and if there are possible suggestions to fixing it. Also let me know if you want me to send an example file/output if you're still not sure what my issue is. Thank you!
Currently the MPR parsing code raises Exceptions in any case where it gets data it does not understand. In many cases the user might prefer parsing to continue with possibly incorrect metadata rather than failing altogether.
Currently the tox tests run flake8
and reuse lint
multiple times - once for every python version tested. Instead, the linting tools should only run once, in a separate tox environment.
I am trying to parse a .res file for a project with the convert_arbin_to_sqlite()
function in res2sqlite.py; however, I keep receiving an sqlite3.OperationalError
.
This is the code snippet I am trying to run:
res.convert_arbin_to_sqlite(infilepath, outfilepath) # outfilepath is a .s3db file
connection = sqlite3.connect(outfilepath)
cursor = connection.execute('SELECT * FROM Global_Table')
data = cursor.fetchall()
However, after trying to run this, I keep receiving the following error message:
sqlite3.OperationalError: table Global_Table has no column named Log_Can_BMS_Data_Flag
I checked the res2sqlite.py source code, specifically the part where the Global_Table
is created (starting from Line 37), and I notice that there are several columns, the first one being Log_Can_BMS_Data_Flag
, which seem to be read in by my .res file but aren't present in the res2sqlite.py source code during the Global_Table
creation process.
Has anyone received this kind of error before, and if so, can I please receive some assistance on it? Thanks!
In the Biologic.MPRfile class this error seems to arise on certain files:
ValueError: Invalid magic for .mpr file: b'<?xml version="1.0" encoding="UTF-8"?>\r\n<Configurati'
This is generated from BT-Lab version 1.65.
The column with id 7
should be named dq/mA.h
, not dQ/mA.h
. The manual says (excerpt):
Variables Description | Variable name/unit |
---|---|
Common Variables | |
dq : charge increment between two recorded values | dq/mA.h |
Additional variables | |
dQ: charge on a potential step | dQ/mA.h |
I am asking my boss if I can add an example file to this issue.
Once #38 is merged then we should update the pip package so that the bug fixes are available to users.
I have a files from a my new biologic that has duplicate columns. The columns have the same ID and data. As a result the file errors saying the dtype doesn't match the length of data.
This happens because the ordered dict only registers one of the columns.
Since neither a dictionary nor a numpy recarray can have the same keys this raises a problem.
I have a PR to fix this.
I'm new to python, how do I run the code? Where to put the .mpt-File (from a BioLogic device) to translate? Thanks in advance!
Currently the test data files are stored on figshare.com and retrieved with wget in get_testdata.sh
Now that github supports storing data files with git-lfs, this could be an attractive alternative.
Hi! I'm trying to open .mpr files using the following code
from galvani import BioLogic as BL
import pandas as pd
mpr = BL.MPRfile(path)
df = pd.DataFrame(mpr.data)
df.head()
however, I always get the error
NotImplementedError: column type 471 not implemented
Am I doing something wrong?
When converting an mpr file to an mpt file, there is a lot of header information, e.g.:
EC-Lab ASCII FILE
Nb header lines : 191
Galvano Electrochemical Impedance Spectroscopy
Run on channel : 1 (SN 36210)
User :
CE vs. WE compliance from -10 V to 10 V
Electrode connection : standard
Ewe ctrl range : min = -10.00 V, max = 10.00 V
Safety Limits :
Ewe min = -0.50 V
Ewe max = 2.80 V
for t > 10 ms
Do not start on E overload
Acquisition started on : 08/11/2021 16:03:59.530
Loaded Setting File : C:\BT-Lab\Daten\PT\Gedruckte Ti PTLs\Ui-Kennlinie.mps
Saved on :
File : 210811_N115_T80_p1_Probe11_02_GEIS_C01.mpr
Directory : C:\EC-Lab\PT\gedruckte PTLs\
Host : 130.75.205.160
Device : SP-150 (SN 1235)
Address : USB
EC-Lab for windows v11.36 (software)
Internet server v11.36 (firmware)
Command interpretor v11.36 (firmware)
Electrode material :
Initial state :
Electrolyte :
Comments :
Current amplifier : SP-150B-20A
Electrode surface area : 0.001 cm�
Characteristic mass : 0.001 g
Equivalent Weight : 0.000 g/eq.
Density : 0.000 g/cm3
Cycle Definition : Charge/Discharge alternance
It would be really useful if this project could read these information from mpr files. Specifically, I really need the experiment method (Galvano Electrochemical Impedance Spectroscopy
in the above example`).
After solving the problems with encoding and missing column headers I end up with a single array for the EIS measurements.
galvani test.txt
Is it possible to extract the data for single measurements?
In the header meta data is present for every measurement.
Hi,
get a TypeError when using galvani with numpy v1.12.0.
Traceback (most recent call last): File "test.py", line 15, in mpt1, comments = MPRfile(os.path.join(testdata_dir, 'test.mpr')) File "/usr/local/lib/python3.5/dist-packages/galvani-0.0.1a1-py3.5.egg/galvani/BioLogic.py", line 298, in __init__ TypeError: only integer scalar arrays can be converted to a scalar index
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.