The ipa-project-template from fmi-faim

Keep an up-to-date .pre-commit-config.yaml in template subdir

Our current pre-commit.ci workflow updates the .pre-commit-config.yaml in the repository root, but doesn't check the file inside the template subfolder.

Also, we should consider using ruff (both check and format), remove redundant others (e.g. black), and maybe add a ruff.toml to the project template (if we want to deviate from defaults).

Investigate if ruff covers black.

Use pixi in template

We should replace the miniforge/conda environments in the template with pixi.

Add documentation about git versioning

It would be nice to have a guide on how to put a new IPA project under version control with git and pushing it to GitHub.

Move documentation to MkDocs

I just tried out MkDocs and it is much easier to use! We should move this and the template default documentation to MkDocs and potentially follow https://diataxis.fr/ for the content structure.

Minimize logging boilerplate

The current code to create a logger is redundant and takes quite some space:

ipa-project-template/template/ipa/s01_preprocessing/run_preprocessing.py

Lines 14 to 33 in 95015a1

    
           def _create_logger(name: str) -> logging.Logger: 
        
               """ 
        
               Create logger which logs to <timestamp>-<name>.log inside the current 
        
               working directory. 
        
               Parameters 
        
               ---------- 
        
               name 
        
                   Name of the logger instance. 
        
               """ 
        
               logger = logging.Logger(name.capitalize()) 
        
               now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") 
        
               handler = logging.FileHandler(f"{now}-{name}.log") 
        
               handler.setLevel(logging.INFO) 
        
               formatter = logging.Formatter( 
        
                   "%(asctime)s - %(name)s - %(levelname)s - %(message)s" 
        
               ) 
        
               handler.setFormatter(formatter) 
        
               logger.addHandler(handler) 
        
               return logger

Identical duplicates

ipa-project-template/template/ipa/s02_segmentation/run_segmentation.py

Lines 14 to 33 in 95015a1

    
           def _create_logger(name: str) -> logging.Logger: 
        
               """ 
        
               Create logger which logs to <timestamp>-<name>.log inside the current 
        
               working directory. 
        
               Parameters 
        
               ---------- 
        
               name 
        
                   Name of the logger instance. 
        
               """ 
        
               logger = logging.Logger(name.capitalize()) 
        
               now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") 
        
               handler = logging.FileHandler(f"{now}-{name}.log") 
        
               handler.setLevel(logging.INFO) 
        
               formatter = logging.Formatter( 
        
                   "%(asctime)s - %(name)s - %(levelname)s - %(message)s" 
        
               ) 
        
               handler.setFormatter(formatter) 
        
               logger.addHandler(handler) 
        
               return logger

ipa-project-template/template/ipa/s03_feature_extraction/run_feature_extraction.py

Lines 14 to 33 in 95015a1

    
           def _create_logger(name: str) -> logging.Logger: 
        
               """ 
        
               Create logger which logs to <timestamp>-<name>.log inside the current 
        
               working directory. 
        
               Parameters 
        
               ---------- 
        
               name 
        
                   Name of the logger instance. 
        
               """ 
        
               logger = logging.Logger(name.capitalize()) 
        
               now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") 
        
               handler = logging.FileHandler(f"{now}-{name}.log") 
        
               handler.setLevel(logging.INFO) 
        
               formatter = logging.Formatter( 
        
                   "%(asctime)s - %(name)s - %(levelname)s - %(message)s" 
        
               ) 
        
               handler.setFormatter(formatter) 
        
               logger.addHandler(handler) 
        
               return logger

I think we could improve this by migrating it into some utility function, so we can import create_logger and call create_logger(name) from any ipa step.
In addition, I'd like to use a basic decorator to have the function arguments logged by default (along the lines of https://ankitbko.github.io/blog/2021/04/logging-in-python/):

my_logger = create_logger(name="processing")

@log(logger=my_logger)
def run_processing(input_dir: str, output_dir: str):
	# do some processing
	my_logger.info("Done!")

... which would log the values of input_dir and output_dir, without having to put extra code into the run_processing function.

@tibuch what do you think?

Link back to GitHub repository from documentation

If you browse to https://fmi-faim.github.io/ipa-project-template/, there's no obvious way to get to this GitHub repository if you're interested in looking at the source.

Check available space in pixi cache_dir before proceeding

To prevent usage of the project code without initializing (i.e. running init.sh at the beginning of your session), we could introduce a task that checks for a minimum available space in the cache_dir folder of the pixi configuration, and warns the user if they're running with a "wrong" cache directory.

For example, this bash snippet gives the current free space on the file system where cache_dir is located:

pixi info --json | grep -o '"cache_dir": "[^"]*' | grep -o '[^"]*$' | tr -d '\n' | xargs -0 -I {} df -P {} | tail -1 | awk '{print $4}'

We could use this - or a (platform-independent) Python equivalent from a minimal environment in pixi - to fail early if there's not enough space available. What do you think, @tibuch?

Consider unifying config.yml across steps

Having a single config.yml for all steps of a workflow would have several benefits:

Avoid redundant definition of common arguments (e.g., zarr plate directory, project folder, etc.).
Reuse the config file for possible (prefect, nextflow, ...) workflows/pipelines built across all steps.

Should source_status task fail when there are untracked changes?

Currently, I get a warning when there are untracked changes, thanks to the source_status task, but the dependent tasks continue.

@tibuch did you intend that source_status blocks on untracked changes, or was it meant merely as a warning?

fmi-faim / ipa-project-template Goto Github PK

ipa-project-template's People

Contributors

Stargazers

Watchers

Forkers

ipa-project-template's Issues

Keep an up-to-date .pre-commit-config.yaml in template subdir

Investigate if ruff covers black.

Use pixi in template

Add documentation about git versioning

Move documentation to MkDocs

Minimize logging boilerplate

Link back to GitHub repository from documentation

Check available space in pixi cache_dir before proceeding

Consider unifying config.yml across steps

Should source_status task fail when there are untracked changes?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def _create_logger(name: str) -> logging.Logger:
	"""
	Create logger which logs to <timestamp>-<name>.log inside the current
	working directory.

	Parameters
	----------
	name
	Name of the logger instance.
	"""
	logger = logging.Logger(name.capitalize())
	now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
	handler = logging.FileHandler(f"{now}-{name}.log")
	handler.setLevel(logging.INFO)
	formatter = logging.Formatter(
	"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
	)
	handler.setFormatter(formatter)
	logger.addHandler(handler)
	return logger