User story

As an engineer, I want to publish a docker image so that I can run the code in the cloud.

Acceptance criteria

• Given a branch is merged into main, then github action will build and publish a docker image.
• Figure out where to publish docker image.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Describe the bug
Missing output file(s) capsule/results/* expected by process `capsule_aind_data_asset_indexer_docdb_1 (capsule-3506143)

To Reproduce
Steps to reproduce the behavior:

1. Run Mat Views pipeline
2. Open output file
3. See Error

User story

As a user I want to ensure that aind-data-access-api is fully added on the next upgrade so that I can access the latest changes

Acceptance criteria

• Change pyproject.toml to use the full aind-data-access-api

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Describe the bug
In the past, "tags" used to always be in the response returned from Code Ocean.

To Reproduce
Steps to reproduce the behavior:

1. Run response = co_client.search_all_data_assets() and then results = response.json()["results"]
2. Notice that some of the latest results don't have a "tags" field.

Expected behavior
In the past, the tags was always present. We'll have to update the code to handle this new behavior.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

User story

As a user, I want to see processed data in the docdb. The tags are being changed from "processed" to "derived" so we need to update that here.

Acceptance criteria

• We should keep "processed" for legacy purposes.
• We need to add aind_data_schema.data_description.DataLevel.derived also

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Describe the bug
The SmartSPIM report shows the following records:

These were for runs that have since been deleted from S3, and as far as I can tell they also do not exist in Code Ocean.

To Reproduce
Steps to reproduce the behavior:

1. Go to the SmartSPIM dashboard
2. Filter to subject_id 695464.
3. See all the zombie assets.

Expected behavior
Only real data should appear in docdb :).

User story

As a software engineer, I want to add a check in S3 crawler script, so I can catch empty dataframes before merging them.

Acceptance criteria

• Check for empty dataframes before merging

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user I want to use the latest versions of the dependencies so that I can access the latest changes

Acceptance criteria

• Change the docker file
• Change pyproject.toml to use the latest dependencies

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

aind-data-access-api needs full option

Describe the bug
Missing output file(s) capsule/results/* expected by process `capsule_aind_data_asset_indexer_docdb_1 (capsule-1936612)

To Reproduce
Steps to reproduce the behavior:

1. Run Mat Views pipeline
2. Open output file
3. See Error

User story

As an engineer, I want a script to do sanity checks on DocDB.

Acceptance criteria

• Given script is run, then all data asset records in S3 are downloaded
• Given downloaded data asset records, check whether metadata file exists and create a table
• Given table, save table in redshift

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Figure out how we want to log discrepancies.

User story

As an engineer, I want DocDB to be updated to match S3.

Acceptance criteria

• Given table in redshift with records info, compare with records in DocDB.
• Given discrepancies, update DocDB and log.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to index processed data assets stored in aws buckets other than the Code Ocean results bucket, so I can analyze data from there.

Acceptance criteria

• When a run is executed, data tagged with processed and are in s3 buckets will be added to the document store.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to update the document store with data asset metadata, so I can search them more easily.

Acceptance criteria

• When the capsule is triggered, the document store will be populated.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Maybe this will be moot in with future releases of Code Ocean?

User story

As a user, I want to optionally pass an ec2 instance's assumed role's aws credentials, so I can run the capsule via pipelines.

Acceptance criteria

• If the default aws credentials attached to a capsule are not found, then it will attempt to use the ec2 instance's credentials.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

ValueError: You are trying to merge on object and float64 columns for key 's3_prefix'. If you wish to proceed you should use pd.concat

More can be found on cloudwatch logs in prod:

https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fecs$252FAindDataAssetIndexerTD/log-events/ecs$252FDataAssetIndexer$252Fd67c97b6dec94bc09d13546595570eb3$3Fstart$3D1705377646996

User story

As a developer, I want to have checks on tests and code formats, so I can maintain the code base easier.

Acceptance criteria

• When a Pull Request is opened, tests and linters will run automatically.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.