physiopy / phys2bids

Python3 library to format physiological files in BIDS. At the moment, it supports Acqknowledge and Labchart. BrainHack participants, check the issues with the BrainHack labels!

physiopy

Metapackage for physiopy repository

phys2bids's Issues

Internal I/O object/schema

So far, the code was set to adapt to whatever input was provided and process it to an output on a case to case base.
This is not a good approach, not only due to the flexibility we need to offer, but also from a coding perspective.

It's much better to create an object ("Output object"), that contains not only the data but a schema that can be used to:
a. output information about the file that can be used by the user to properly set the input.
b. get the name of the file and the info for the json file.

In this way, the main code would operate on this object, while another family of scripts (interfaces?) would work on populating the object schema based on the type of input and the information embedded in it.

You can think about the new object as composed of a numpy matrix (data), some lists (values that are defined channel by channel) and some variables that contain general information.

This said, what should be the adopted schema (i.e., what information do we need to carry around)?
We know that BIDS has certain requirements, so those have to be in the schema.
However, we can expand the schema thinking about what we'd like to have there for general purposes.

Write docstrings

Add numpy docstrings to functions

Improve heuristics rules

Detailed Description

At the moment, the only thing used during heuristics to get the new filename is the name of the input of phys2bids. It's a very simple, basic way to BIDSify files, but heuristic statements could use much more information than just the name (or instead of the name)! For instance, the number of channels, their content, their lenght, ...

Possible Implementation

• Have a look at heudiconv and its heuristics
• Tinker with the heuristics to improve them!

phys2bids and/or physiopy logos

Maybe this is not the most important point ever, but having a nice figure that represents the work we're doing is incredibly nice.
Luckily, we have a fellow in Donostia that likes doing this sort of things - and does them well!
He sort of volunteered to give us a logo for pys2bids and/or physiopy.
@danalclop wanna take over this issue?

Change floats print format

Currently, any print of a float in a file has a format with many decimals.
It would be better to have a formatted print with max 4 decimals.

This doesn't affect the data, only the information in the json and in the log.

Add multifrequency support to labchart txt files

Once #54 is finish we have to add support for files with different frequency sampling in the recordings, we have to infer the frequency from the channels since labchart doesn't specify in the header

Error output for files with no header

When testing the code, I inputted a file with no header information by accident. The error message I got was not very clear though, see the UserTesting doc for info.

Running heuristic on txt file outputs folders but not files

File extension is .txt
File exists
File Example.txt contains:

Interval=	0.001 s

ChannelTitle=	Trigger	CO2	O2	Pulse

UnitName=	*	mmHg	mmHg	*

Preparing BIDS output using heur_ex.py

Improving trigger pictures.

So far, the trigger picture created around this line have a simple title, and show the expected start and expected end of the sequence.
It would be nice to:

• add the name of the corresponding tsv output file in the title
• add a third subplot in the second row, that shows the last trigger
• rescale differently the Y axis in the first row, maybe the higher between twice the max value in the trigger array and three times the threshold?

Create testing heuristics

We need heuristics that could be used for testing on the data.

Create unit tests for phys2bids.py

@vinferrer , @eurunuela , you were interested in this one?

default channel trigger doesn't work for txt files

Hi, doing some testing , i have realized that the default of -chtrig is not vaild for labchart txt files since the trigger channel happens to be index 1 no index 0, should we change default or only correct it when txt files are used?

Is this the behaviour we want? is_valid function returns input variable

As I'm working on the unit tests, I've just seen that the is_valid function returns the input variable. Doesn't it make more sense for it to be a void function? If the variable is not valid, it could raise an error, but do nothing otherwise.

https://github.com/smoia/phys2bids/blob/f2d7d52e4e9d9d963f98aeb8809c81660e2f1794/phys2bids/physio_obj.py#L37

Tests failing due to errors while executing flake8

I know that @vinferrer is looking at that,

The issue is that all the tests we have in travis fail due to a couple of errors in linter.
The first couple of tests (the linters themselves) are obvious to solve.
@rmarkello any idea on why the other tests fail with errors related to flake8?

Add txt support

Similar to #41 , but at the very least we should be able to process a txt file with a header coming from labchart.
On OSF there's this one.
Bonus: be able to run the txt file in this repo (folder tests/data), that comes from acqknowledge.

This requires setting up a new script in interfaces, similar to the existing acq.py.

Add all-contributors to this repo

Add support for txt files

With the new code, txt is not supported at the moment.

EDIT: txt is partially supported.
The possible txt formats are (tick what's covered):

• LabChart, with header
• AcqKnowledge, with header
• LabChart, without header
• AcqKnowledge, without header
• LabChart, with header and comment
• ...

[BUG] Cannot delete array elements in Blueprint_Output.delete_at_index()

Line raising the error:

phys2bids/phys2bids/physio_obj.py

Line 397 in 46f5239

del(self.timeseries[idx])

Error:

blueprint_out.delete_at_index(1)...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/phys2bids/phys2bids/tests/test_physio_obj.py in 
----> 1 blueprint_out.delete_at_index(1)
      2 assert blueprint_out.ch_amount == num_channnels - 1

~/phys2bids/phys2bids/physio_obj.py in delete_at_index(self, idx)
    395             `idx` gets deleted
    396         """
--> 397         del(self.timeseries[idx])
    398         del(self.ch_name[idx])
    399         del(self.units[idx])

ValueError: cannot delete array elements

Split input in different outputs based on the sample frequency of the channels

BIDS wants each *_physio.tsv.gz file to have the same frequency internally.
When a input with different frequencies is given, the program should be able to export it in different outputs.

Add testing files

In order to be able to test the code (and have people test the code), we would need samples of data that work with the code. I have acquired a two minutes sample, with physiological recordings and the MRI trigger, and saved it in many different formats to be able to integrate them.

Could you guys in Chicago create, acquire or share a mock file with labchart and one with the respiract, that can be released publicly, and save it in different ways (as txt, matlab matrix, with/without header, ...)?
It doesn't have to be long (actually, short is better).

Object is not subscriptable in BlueprintInput.rename_channels

https://github.com/smoia/phys2bids/blob/f2d7d52e4e9d9d963f98aeb8809c81660e2f1794/phys2bids/physio_obj.py#L167

See the error:

~/phys2bids/phys2bids/physio_obj.py in rename_channels(self, new_names, ch_trigger)
    165 
    166         if 'trigger' in new_names:
--> 167             del(new_names[new_names.index['trigger']])
    168         elif ch_trigger:
    169             del(new_names[ch_trigger])

TypeError: 'builtin_function_or_method' object is not subscriptable

Adding documentation on how to use phys2bids

Similar to #12 , but we need to explain at least how to:

• Obtain the info of the file in order to set up a real "run" (-info)
• Set up an heuristic file (heur.py)
• Run the program itself

Before OHBM.

0- vs 1-indexing the `run` keyword in `use_heuristic()`

In #76 (comment) a brief discussion was started about whether the run keyword should be 0- or 1-indexed when users provide a heuristic to convert their data to BIDS format. We are currently using a 0-indexing approach; I personally think a 1-indexing approach makes more sense here, but am open to discussion on this point!

@smoia found a relevant portion of the BIDS spec that touches on this, but the spec is not prescriptive on this point. I'm happy to hear others thoughts and go about issuing a change if it's decided that's warranted!

Deal with "non-required" packages for interfaces.

One of the issues of having to deal with multiple type of files in input is that some of them (acqknowledge, matlab) requires libraries to load the content of the files.

However, such libraries are necessary only reading those files, and it's possible that a succesful run of the program doesn't require them.

It's easy to import them only if needed, but the question is:
should they be required (hence automatically installed) during the setup of phys2bids, or should we throw an error if they're missing and ask to install them later?

phys2bids No module named 'phys2bids.physiobids'

Hi guys,

So I basically installed all the modules required for phys2bids in python3 and executed with python3 the setup.py file and when I tried the command from the console this funny error came out:
phys2bids
Traceback (most recent call last):
File "/usr/local/bin/phys2bids", line 11, in
load_entry_point('phys2bids==0.4.0', 'console_scripts', 'phys2bids')()
File "/home/vicente/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/vicente/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 2852, in load_entry_point
return ep.load()
File "/home/vicente/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 2443, in load
return self.resolve()
File "/home/vicente/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 2449, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
ModuleNotFoundError: No module named 'phys2bids.physiobids'
I have no absolute idea why is this happenning.

First problem

Making heuristic functions a simple set of if elif else statements

Detailed Description

At the moment, heuristic files contain a bunch of code that shouldn't be changed by the user (e.g. from here).
It would be better to move that code outside of the heur file, so that the only content of those scripts is a simple definition that reads an input, goes through a bunch of if statements to set up the labels necessary for the name of the file, and returns those labels.

Context / Motivation

This would simplify the heuristic files on one hand, and help the integration of phys2bids in other BIDSificators programs such as BIDScoin (as in #199).

Possible Implementation

• Move all the non-if..elif..else code into the function use_heuristics.

Add license info in files

Add info about the license in all the main files, as per the end of the GNUv3 license

New all-contributors issue

Summary

Just a matter of using the all-contributor bot.

## Additional detail

## Next Steps

* 
* 
* 

[BUG] check_input_ext doesn't work with double extensions

As the title says, the method does not work with double extensions when the extension given is not .gz. Have a look at the example below:

check_input_ext('help.txt.gz', 'py') != 'help.py'

setup.cfg should be updated to follow the name of the organization

There are a couple of lines that reference should reference the physiopy organization instead of a personal GitHub project. See the lines here.

Context / Motivation

One would expect to find physiopy references everywhere in the project, which makes it somehow incoherent to find a reference to a personal GitHub project. This change would make the package more coherent with what physiopy is as a community.

Possible Implementation

It should just be a matter of changing the urls (please correct me if I'm wrong).

Correcting linter errors

We have a bunch of errors and warnings in the linter test that makes it fail.
While I don't see any problem for Warnings (but maybe I'm wrong), we can't have Errors.
looking at Travis CI details, I see most of them are in interfaces/txt.py. @vinferrer , can you check that your scripts pass lintering for Travis CI?

Check the minimal version of Numpy and Pandas required by the code

At the moment, the earliest version installed in my systems is specified.
Maybe earlier versions would work as well.

Adding more all-contributors to phys2bids

After #46 bot malfunciton, let's try again.

Phys Data Best Practices Documentation

Phys data best practices documentation outline for OHBM abstract here.

Feel free to edit as you see fit.

Standard Acquisition(?)

Goes together with #23 .

One thing that we can do to facilitate the process for the user is to think about "standard cases" that could act as a blueprint for the acquisition of physiological data.

E.g.:

• Save the trigger in channel 1
• Use x sampling frequency
• Use y unit of measure
• ...

Keeping in mind that we want to be as flexible as possible, we can offer heuristics for particular cases that would work almost automatically.
If we want to offer them, what would be good examples of "standard cases"?

Add channel plots

It's easy to know which channel is which if its timeseries is plotted.
There's already a function to do so in viz.py (plot_channel).

We need to have the function going automatically on all the channels, if the user requires info about the file.

Contribution guidelines

I started re-drafting a contribution guidelines.
Please have a look here:
https://docs.google.com/document/d/1FCk3BPp6UZePSf44NDL_4CQeMV4gvQUfr1wbLU9mlfw/edit?usp=sharing

You might need to ask permission to comment it.
Deadline for this is 14/12.

Add user documentation

The library needs a readthedocs with explanations on usage, bids format and physio files.

Refactor phys2bids.use_heuristic

In phys2bids.py, use_heuristic needs a bunch of f-string rewritings, and the initialisation of fldr should use os instead.

[DOC] Adding Code of Conduct to project

In order to follow the recommended community guidelines (see https://github.com/physiopy/phys2bids/community), we should add a Code of conduct to our project.

How about we draft it in the following Google Docs?

https://docs.google.com/document/d/1qDdDnuADu87zjxsDEjJpFzGmjETWsK15XQcUlieKCTg/edit?usp=sharing

OHBM 2020 abstract

As discussed, we should produce an abstract for a software demonstration in OHBM.

Currently, Molly and I are taking the lead on this, but any input is well accepted!

Reorganise the script folder & files

The actual code is too long, and could be split in different files and better organised in folders.
One nice way to split the file could be:

• main
• utilities
• visualisations
• bids

Or also split processing of .txt files and .acq files

Chopping files semi-automatically

A lot of researchers will have one physiological file for the whole scan session, which will need to be chopped into separate files corresponding to each scan to be analyzed. Offering a way of semi-automatically chopping these files, and allowing phys2bids to output multiple physiological files from the same input file, would be useful.

README - instructions for package install

It may be obvious to regular python users, but it may help to instruct the user to clone the repo into the location where their python packages are located. For me, I cloned it elsewhere first, and then when I later renamed a directory the path to the package was wrong. So just some extra text in the REAME suggesting the location of where this should go (i.e. for me on a macOS it was /usr/local/lib/python3.7/site-packages ). If there is a reason not to put it in site-packages, let me know.

BlueprintOutput.delete_at_index() does not update the number of channels in object

Possible solution:

def delete_at_index(self, idx):
    self.timeseries = np.delete(self.timeseries, idx, axis=0)
    self.ch_amout -= 1
    del(self.ch_name[idx])
    del(self.units[idx])

Add logger

Add license information in all the scripts and in the data folder.

We should add the license information in all the scripts and in the readme(s).
I think that the best solution would be to:

• add a string in the initial docstring of each script, that contains:
  Copyright 2019, The Phys2BIDS community. Please scroll to bottom to read full license.
• add the full boilerplate (line 189 in LICENSE ) to the end of each script, and to the end of the README
• add a README file in tests/data that contains: This files are under the license of the phys2bids package. and then the boilerplate.

If you have better ideas/think it's not necessary say it.
Otherwise, it's just about adding multi-line comments in each file, hence a very good first issue.

Add support for matlab files

Detailed Description

We have some labchart-exported matlab files in our OSF. We could implement an interface function that reads matlab files, in order to support yet another type of data!

Possible Implementation

• Look at the functions in interfaces
• Using scipy or h5py (or both), write an itnerface function to read Labchart data in matlab format.
• Write a test for such interface function!

Do User Testing on refactored code

While #42 deals with unit testing and negative testing, this calls for user testing.

As soon as #38 is merged, we can start trying it out on the acqknowledge testing files that we have at disposal, to see how to improve the code!

The new code works only on acq

I guess team Chicago was interested in this?
@RayStick @AyyagariA @kristinazvolanek

Add documentation on collaboration