allenneuraldynamics / aind-data-transfer-models

Repository for data models used in aind-data-transfer-service

aind-data-transfer-models

Usage

Installation

• From pypi, run

pip install aind-data-transfer-models

• From git, in the root directory, run

pip install -e .

To develop the code, run

pip install -e .[dev]

Contributing

Linters and testing

There are several libraries used to run linters, check documentation, and run tests.

• Please test your changes using the coverage library, which will run the tests and log a coverage report:

coverage run -m unittest discover && coverage report

• Use interrogate to check that modules, methods, etc. have been documented thoroughly:

interrogate .

• Use flake8 to check that code is up to standards (no unused imports, etc.):

flake8 .

• Use black to automatically format the code into PEP standards:

black .

• Use isort to automatically sort import statements:

isort .

Pull requests

For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:

<type>(<scope>): <short summary>

where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:

• build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
• ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
• docs: Documentation only changes
• feat: A new feature
• fix: A bugfix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• test: Adding missing tests or correcting existing tests

Semantic Release

The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):

Commit message	Release type
fix(pencil): stop graphite breaking when too much pressure applied	Patch Fix Release, Default release
feat(pencil): add 'graphiteWidth' option	Minor Feature Release
perf(pencil): remove graphiteWidth option BREAKING CHANGE: The graphiteWidth option has been removed. The default graphite width of 10mm is always used for performance reasons.	Major Breaking Release (Note that the BREAKING CHANGE: token must be in the footer of the commit)

aind-data-transfer-models's Issues

Clarify `aind-metadata-mapper` dependency

I am very much puzzled by this dependency:

aind-data-transfer-models/pyproject.toml

Line 25 in 40dd640

'aind-metadata-mapper==0.15.0'

I cant seem to find any references to it in the code, and it is rather annoying as it carries several strict pinned dependencies that make interoperability between aind tools very difficult to maintain.

  "aind-data-schema==0.36.0",
  "aind-data-schema-models==0.1.7",
  "pydantic-settings>=2.0",

Is this dependency really necessary?

upgrade to use aind-metadata-mapper0.16.0

Conflicting dependencies between `aind-data-schema` and `aind-data-transfer-models`

When trying to install the watchdog service with the latest version of aind-data-schema I got the following error when attempting to bootstrap the environment:

The conflict is caused by:
    aind-data-schema 0.38.5 depends on aind-data-schema-models>=0.1.7
    aind-data-transfer-models 0.5.1 depends on aind-data-schema-models==0.1.1

For reference, these are the versions I am specifying:

    'aind-watchdog-service@git+https://github.com/AllenNeuralDynamics/aind-watchdog-service@development',
    'aind-data-schema==0.38.5'

As of the date of this issue, the development branch targets 49092f2.

Set user email and notification types on BasicUploadConfigs class

Is your feature request related to a problem? Please describe.
Airflow parses the BasicUploadConfigs for each pipeline run.

Describe the solution you'd like
It will be nice if the user email is set in the BasicUploadConfigs class in addition to the SubmitJobRequest. The email in the SubmitJobRequest can be used to set all the other ones.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add Optional field to allow a user to define GatherMetadataJob config

User story

As a user, I want to define session settings in the data-transfer-service.

Acceptance criteria

• Import aind-metadata-mapper
• Add an optional field for metadata configs
• Given a user creates a BasicUploadConfigs model, when the model is being validated, then any missing fields in the metadataconfigs class will be filled in using information from the required BasicUploadConfigs fields.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add input data mount

Is your feature request related to a problem? Please describe.
When a user needs to run a custom pipeline, they need to define an input data mount in addition to the process_capsule_id field

Describe the solution you'd like
An input_data_mount field

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add code ocean trigger configs into BasicUploadJobConfigs

Is your feature request related to a problem? Please describe.
Right now, a separate process parses settings from BasicUploadJobConfigs into TriggerConfigModel

Describe the solution you'd like
Add optional codeocean_settings field to BasicUploadJobConfigs that is backwards compatible (i.e., it will fill in unset fields in TriggerConfigModels using existing fields in BasicUploadJobConfigs.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

ModalityConfigs should use V0036JobProperties instead of V0036JobSubmission

Describe the bug
A user should be able to add custom slurm configs when running modality compression scripts. However, the slurm settings is using V0036JobSubmission, which is different.

Add cap to the number of jobs a user can submit at once

Is your feature request related to a problem? Please describe.
We should add a cap to the number of jobs a user can submit in a single request to avoid blocking the queue for other users without confirmation

Describe the solution you'd like
Add a max_items of 1000 to the list in the submit_job_request

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add metadata_dir_force to fill_in_metadata_configs

Is your feature request related to a problem? Please describe.
metadata_dir_force doesn't proliferate to metadata_configs

Describe the solution you'd like
It can be added in the fill_in_metadata_configs validator

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add SubmitJobRequest class

Is your feature request related to a problem? Please describe.
Currently, a lot of code is being done to translate the configs set in the ui into the request being sent to the backend. We should explicitly manage the configs in a class.

Describe the solution you'd like
Add a SubmitJobRequest class with the following fields:

• user_email
• notificiation_type
• List[BasicUploadJobConfigs]

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Use str instead of PurePosixPath for GatherMetadataJob Settings class

Describe the bug
Errors arise when using PurePosixPaths in the GatherMetadataJob settings class

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Allow user to define a config file without validation

Is your feature request related to a problem? Please describe.
If a user defines a config file for the session settings, the service will try to parse the config file. Since the service doesn't have direct access to VAST, we should allow a user to point to a session config file without validation.

Describe the solution you'd like
Add a wrap validator to allow the session settings to allow for just a single user_settings_config_file field.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Upgrade to use aind-data-schema v1.0

User story

As a user, I want to use the latest aind-data-schema version, so I can attach the latest metadata to data assets on upload.

Acceptance criteria

• Update versions of aind-data-schema
• Update unit tests appropriately

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

May depend on AllenNeuralDynamics/aind-metadata-mapper#121 being completed first