molgenis / molgenis

MOLGENIS - for scientific data: management, exploration, integration and analysis.

THIS MOLGENIS VERSION IS IN ARCHIVE MODE. PLEASE USE NEXT GENERATION AT MOLGENIS-EMX2

Welcome to MOLGENIS

MOLGENIS is a collaborative open source project on a mission to generate great software infrastructure for life science research.

Develop

MOLGENIS has a frontend and a backend. You can develop on them separately. When you want to develop an API and and App simultaneously you need to checkout both.

• Develop backend
• Develop frontend

Useful links

• Documentation
• Organisation

Deploy MOLGENIS

• Deploy MOLGENIS

molgenis's Issues

After importing for a while the session is invalidated

After the importing is finished this error occurs. This is with big datasets, which take some time before the importing is done.
A white screen is shown in the application.

This is the error message in the catalina.out
org.molgenis.framework.db.DatabaseException: setAttribute: Session [47D91C1DC369AF8117F2A9AB36EB4C5C] has already been invalidated

URL not found always returns 200OK

When a URL can't be mapped to a controller it is send to the molgenis FrontController. When the molgenis FrontController can't map the URL to a controller an empty page with 200OK is returned instead of a 404

Molgenis generator ignores model field attribute 'length'

The model attribute 'length' of tag 'field' does not result in generated @Length annotations for the entitities (JPA backend)

MolgenisDatabasePopulatorGen in molgenis-core depends on omx-core and omx-auth modules

The MolgenisDatabasePopulatorGen in molgenis-core depends on the omx-core and omx-auth modules.
Can we move the generator to the omx-auth module?

Validate DataSet matrix during import

Check values (ObservableValue) of DataSets if they are in the correct format according to the datatype specified in the ObservableFeature and show error if incorrect. Now all values are saved as string and are not validated. This can lead to problems later on for example during indexing for elastic search and viewing in the DataSetViewer

Importer: validator only accepts Excel

UploadWizardPage in molgenis-omx-importer uses ImportWizardExcelPrognosis for input instead of a generic validator based on TableReaderFactory. As a result import of other tabular formats fails.

Missing images for save and cancel buttons

The save and cancel buttons of the add record popup are missing

Feature filter dialog in data explorer is displayed multiple times

If there is already a filter selected, the 2nd filter pop up dialog is displayed several times, creating multiple layers (that is why the background is really gray instead of blurry).

Mapper createList returns empty list

Generated mappers classes (JDBC, JPA and others) generate createList(int nr) method that returns empty list of entities instead of a list with "nr" entities. Behavior is not documented, but most likely a bug.

Exporting data out dataviewer very slow

500 rows takes 7:30 minutes.

Harmonization of datatypes in molgenis

We have FieldType (and MolgenisFieldTypes.FieldTypeEnum) for entities.

For ObservableFeature we have [xref,string,categorical,nominal,ordinal,date,datetime,int,code,image,decimal,bool,file,log,data,exe]. But this is not enforced, DataType is just a string field.

We must have one set of types wich we use everywere and it must be static typed and validated

Protocol viewer: Excel download does not work in IE8

Core issue (GUI): When adding entity_A to entity_B, entity_B will NOT be selected automatically.

FOr example: In the browser I have a project selected, I want to add a sample. THis results in a new form used to enter values of that sample. I expect the project_value to be selected automatically. But it doesn't .....

EntityManagerFactory instances not closed

EntityManagerFactory instances in org.molgenis.framework.db.jpa.JpaDatabase are never closed resulting in memory leaks (see #136)

Data explorer: always show the target identifier

If we select a column in the database we don't know which target it is, so we should have a system that automatically selects the target identifier when 1 or more features are selected in the tree.

Check if decorator exists when validating model

Model input="file" not working with JPA

Model input="file" is not working with JPA because of the problem described in:
http://stackoverflow.com/questions/14771399/hibernate-4-calls-setbinarystream-that-is-not-supported-in-c3p0

The most recent hibernate-c3p0 is still based on 0.9.1 so upgrading to 0.9.2 as suggested in article above is not possible.

Dropdown fields in entity panels not working

Setting permissions as an admin is not possible anymore, these are now system entities.

DataSetTable performance for large data sets

DataSetTable.getRows() stores the whole table in memory, which causes performance issues and out of memory errors.

Delete a record from standard molgenis forms fails

Sometimes when you delete a record with the standard molgenis forms you get a success message but the record is not deleted from the database

Hard enters in a cell messes up the import

The import can't handle the hard enters in a cell

Styling of the dataexplorer at lower resolutions

At lower screen resolutions the dataexplorer does not look correctly and is not easy to use.

Hudson build dependency order not ok

Hudson build dependency order not ok.

Web app redeploy memory leaks

Redeploying a web app results in memory leaks:

0 WARN [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] HHH000208: org.hibernate.connection.C3P0ConnectionProvider has been deprecated in favor of org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider; that provider will be used instead.
110 WARN [com.mchange.v2.c3p0.management.ActiveManagementCoordinator] A C3P0Registry mbean is already registered. This probably means that an application using c3p0 was undeployed, but not all PooledDataSources were closed prior to undeployment. This may lead to resource leaks over time. Please take care to close all PooledDataSources.

Due to these memory leaks multiple redeploys will result in out of memory errors, especially inconvenient for nightly deployed applications.

Entity __Type field should be a system field

Copied from molgenis-legacy repository:

The Entity __Type field should be a system field.
At the time of writing applications such as XQTL depend on availability of __Type in the UI. These dependencies need to be resolved before we can make __Type a system field.

Move mock object to test jars

Create a test jars to host mock objects and other common test code

Harmonize different methods to store files

Currently you can use the input="file" type to write files as blobs in the database or you can use a MolgenisFile entity which in combination with "a plugin" stores the file meta-data in the database and stores the file outside of the database.

Protocol viewer: view action fails for large selections

Same problem as #59 but requires a different solution

Protocol viewer: clear search twice without selection incorrect

Reproduce:
perform a search
press clear --> tree collapses
press clear --> tree expands = incorrect behavior

Protocol viewer: Excel download fails in case of large feature selection

The Excel download in the protocol viewer fails when the list of selected feature is large due to the fact that the selection is included in the request URL instead of the request body. Solution: do a POST instead of GET, put the selected features in the request body

DotDocMinimalGen errors when "dot" not available

Get rid of the following errors during generation:

0 ERROR [org.molgenis.generators.doc.DotDocMinimalGen] Generation of graphical documentation failed: return code Cannot run program "dot": CreateProcess error=2, The system cannot find the file specified. Install GraphViz and put dot.exe on your path.
0 [Thread-4] ERROR org.molgenis.generators.doc.DotDocMinimalGen - Generation of graphical documentation failed: return code Cannot run program "dot": CreateProcess error=2, The system cannot find the file specified. Install GraphViz and put dot.exe on your path.

Generate WebAppInitializer based on molgenis.properties

Retrieve molgenis frontcontroller mappings from molgenis.properties and add to WebAppInitializer

Importer: updating DataSet does not work

Importing data using the "Add, update existing" option fails when updating a dataset entity:
java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]

Reproduce using the following omx data:
identifier name protocolused_identifier dataset_test dataset_test protocol_test

Reintroduce recaptchas in molgenis-omx-auth

During migration of the molgenis authentication modules from the molgenis_apps-legacy to the molgenis repository the recaptcha functionality was removed because the library was not available in the maven repository.

Define cache response headers

Define cache response headers for all resources. Currently the headers are defined in such a way that an explicit browser refresh (for example javascript) might be necesarry).

Data explorer: do not perform search if no data items selected

Do not perform a search in the data explorer if no data items are selected (resulting in the pager and number of search results not being displayed). Instead display a message that tells the user to first select one or more data items.

Omx app: test dataset contains unknown field ontology_ontologyterm

The example dataset example_omx2.2.xls reports an unknown field ontology_ontologyterm. Fix the field or remove the field from the example dataset.

Remove feature from protocol

When a feature is removed from a protocol, the observedvalues are not removed from the DataSet. They are stil visible in the DataSetViewer.

See: removed 'organism', still visible in the viewer:

MolgenisModel dependency resolving incorrect

Entity dependency resolving in MolgenisModel fails in the following case:
Entity A extends B and entity A has field B

Example:

*** Remove field B from entity A
// entities added in import order ENTITIES_IMPORTABLE = new LinkedHashMap<String, EntityImporter>(); ENTITIES_IMPORTABLE.put("institute", new InstituteEntityImporter()); ENTITIES_IMPORTABLE.put("investigation", new InvestigationEntityImporter()); ENTITIES_IMPORTABLE.put("molgenisfile", new MolgenisFileEntityImporter()); ENTITIES_IMPORTABLE.put("runtimeproperty", new RuntimePropertyEntityImporter()); ENTITIES_IMPORTABLE.put("characteristic", new CharacteristicEntityImporter()); ENTITIES_IMPORTABLE.put("observationtarget", new ObservationTargetEntityImporter()); ENTITIES_IMPORTABLE.put("individual", new IndividualEntityImporter()); ENTITIES_IMPORTABLE.put("ontology", new OntologyEntityImporter()); ENTITIES_IMPORTABLE.put("species", new SpeciesEntityImporter()); ENTITIES_IMPORTABLE.put("genome", new GenomeEntityImporter()); ENTITIES_IMPORTABLE.put("chromosome", new ChromosomeEntityImporter()); ENTITIES_IMPORTABLE.put("gene", new GeneEntityImporter()); ENTITIES_IMPORTABLE.put("protein", new ProteinEntityImporter()); ENTITIES_IMPORTABLE.put("proteindomain", new ProteinDomainEntityImporter()); ENTITIES_IMPORTABLE.put("exon", new ExonEntityImporter()); ENTITIES_IMPORTABLE.put("ontologyterm", new OntologyTermEntityImporter()); ENTITIES_IMPORTABLE.put("accession", new AccessionEntityImporter()); ENTITIES_IMPORTABLE.put("person", new PersonEntityImporter()); ENTITIES_IMPORTABLE.put("study", new StudyEntityImporter()); ENTITIES_IMPORTABLE.put("citation", new CitationEntityImporter()); ENTITIES_IMPORTABLE.put("submission", new SubmissionEntityImporter()); ENTITIES_IMPORTABLE.put("contribution", new ContributionEntityImporter()); ENTITIES_IMPORTABLE.put("studydetails", new StudyDetailsEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypeproperty", new PhenotypePropertyEntityImporter()); ENTITIES_IMPORTABLE.put("samplepanel", new SamplePanelEntityImporter()); ENTITIES_IMPORTABLE.put("assayedpanel", new AssayedPanelEntityImporter()); ENTITIES_IMPORTABLE.put("variant", new VariantEntityImporter()); ENTITIES_IMPORTABLE.put("observablefeature", new ObservableFeatureEntityImporter()); ENTITIES_IMPORTABLE.put("protocol", new ProtocolEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypemethod", new PhenotypeMethodEntityImporter()); ENTITIES_IMPORTABLE.put("dataset", new DataSetEntityImporter()); ENTITIES_IMPORTABLE.put("panel", new PanelEntityImporter()); ENTITIES_IMPORTABLE.put("experiment", new ExperimentEntityImporter()); ENTITIES_IMPORTABLE.put("gwasexperiment", new GWASExperimentEntityImporter()); ENTITIES_IMPORTABLE.put("usedmarkerset", new UsedMarkerSetEntityImporter()); ENTITIES_IMPORTABLE.put("frequencycluster", new FrequencyClusterEntityImporter()); <-- field B ENTITIES_IMPORTABLE.put("genotypefrequency", new GenotypeFrequencyEntityImporter()); ENTITIES_IMPORTABLE.put("allelefrequency", new AlleleFrequencyEntityImporter()); ENTITIES_IMPORTABLE.put("significance", new SignificanceEntityImporter()); ENTITIES_IMPORTABLE.put("effectsize", new EffectSizeEntityImporter()); ENTITIES_IMPORTABLE.put("selectioncriteria", new SelectionCriteriaEntityImporter()); ENTITIES_IMPORTABLE.put("category", new CategoryEntityImporter()); ENTITIES_IMPORTABLE.put("observationset", new ObservationSetEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypevalue", new PhenotypeValueEntityImporter()); ENTITIES_IMPORTABLE.put("observedvalue", new ObservedValueEntityImporter()); ENTITIES_IMPORTABLE.put("panelsource", new PanelSourceEntityImporter());

*** Keep field B in entity A
// entities added in import order ENTITIES_IMPORTABLE = new LinkedHashMap<String, EntityImporter>(); ENTITIES_IMPORTABLE.put("institute", new InstituteEntityImporter()); ENTITIES_IMPORTABLE.put("investigation", new InvestigationEntityImporter()); ENTITIES_IMPORTABLE.put("molgenisfile", new MolgenisFileEntityImporter()); ENTITIES_IMPORTABLE.put("runtimeproperty", new RuntimePropertyEntityImporter()); ENTITIES_IMPORTABLE.put("characteristic", new CharacteristicEntityImporter()); ENTITIES_IMPORTABLE.put("observationtarget", new ObservationTargetEntityImporter()); ENTITIES_IMPORTABLE.put("individual", new IndividualEntityImporter()); ENTITIES_IMPORTABLE.put("ontology", new OntologyEntityImporter()); ENTITIES_IMPORTABLE.put("species", new SpeciesEntityImporter()); ENTITIES_IMPORTABLE.put("genome", new GenomeEntityImporter()); ENTITIES_IMPORTABLE.put("chromosome", new ChromosomeEntityImporter()); ENTITIES_IMPORTABLE.put("gene", new GeneEntityImporter()); ENTITIES_IMPORTABLE.put("protein", new ProteinEntityImporter()); ENTITIES_IMPORTABLE.put("proteindomain", new ProteinDomainEntityImporter()); ENTITIES_IMPORTABLE.put("exon", new ExonEntityImporter()); ENTITIES_IMPORTABLE.put("ontologyterm", new OntologyTermEntityImporter()); ENTITIES_IMPORTABLE.put("accession", new AccessionEntityImporter()); ENTITIES_IMPORTABLE.put("person", new PersonEntityImporter()); ENTITIES_IMPORTABLE.put("study", new StudyEntityImporter()); ENTITIES_IMPORTABLE.put("citation", new CitationEntityImporter()); ENTITIES_IMPORTABLE.put("submission", new SubmissionEntityImporter()); ENTITIES_IMPORTABLE.put("contribution", new ContributionEntityImporter()); ENTITIES_IMPORTABLE.put("studydetails", new StudyDetailsEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypeproperty", new PhenotypePropertyEntityImporter()); ENTITIES_IMPORTABLE.put("samplepanel", new SamplePanelEntityImporter()); ENTITIES_IMPORTABLE.put("assayedpanel", new AssayedPanelEntityImporter()); ENTITIES_IMPORTABLE.put("variant", new VariantEntityImporter()); ENTITIES_IMPORTABLE.put("observablefeature", new ObservableFeatureEntityImporter()); ENTITIES_IMPORTABLE.put("protocol", new ProtocolEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypemethod", new PhenotypeMethodEntityImporter()); ENTITIES_IMPORTABLE.put("dataset", new DataSetEntityImporter()); ENTITIES_IMPORTABLE.put("panel", new PanelEntityImporter()); ENTITIES_IMPORTABLE.put("experiment", new ExperimentEntityImporter()); ENTITIES_IMPORTABLE.put("gwasexperiment", new GWASExperimentEntityImporter()); ENTITIES_IMPORTABLE.put("usedmarkerset", new UsedMarkerSetEntityImporter()); ENTITIES_IMPORTABLE.put("significance", new SignificanceEntityImporter()); ENTITIES_IMPORTABLE.put("effectsize", new EffectSizeEntityImporter()); ENTITIES_IMPORTABLE.put("selectioncriteria", new SelectionCriteriaEntityImporter()); ENTITIES_IMPORTABLE.put("category", new CategoryEntityImporter()); ENTITIES_IMPORTABLE.put("panelsource", new PanelSourceEntityImporter()); ENTITIES_IMPORTABLE.put("frequencycluster", new FrequencyClusterEntityImporter()); <-- field B at different position ENTITIES_IMPORTABLE.put("genotypefrequency", new GenotypeFrequencyEntityImporter()); ENTITIES_IMPORTABLE.put("allelefrequency", new AlleleFrequencyEntityImporter()); ENTITIES_IMPORTABLE.put("phenotypevalue", new PhenotypeValueEntityImporter()); ENTITIES_IMPORTABLE.put("observationset", new ObservationSetEntityImporter()); ENTITIES_IMPORTABLE.put("observedvalue", new ObservedValueEntityImporter());

Implement date filter in dataexplorer

Dates are now treated as string, so you can not sort.
Implement date filter with date picker.

In molgenis multiple dat formats are used.
We should have one used everywhere in molgenis

Data explorer: use multiple pages of Entity REST API results

The data explorer currently only uses one page of Entity REST API results to:

• fill dataset select
• fill features of a protocol in the tree

Use all pages of the Entity REST API.
How to deal with large numbers of features per protocol?

Fields should be system fields if entity is a system entity

Unique not honored

When i add a unique tag to my entity i can stil add entites with the same value for that field.
No unique key is generated.

Using unique entity field combinations as input for a cmbBox results in a broken filter.

For example: When using a unique field which is a combination of two entitiy fields. The combined value will be displayed in a comboBox in the GUI (which is nice). Next when you try to filter (by typing a search value) this will result in an empty resultlist.

Database connection time out exceptions

Database connection errors occur overnight resulting in the web application not working:

595511526 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] SQL Error: 0, SQLState: 08003
595511526 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] No operations allowed after connection closed.
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: No operations allowed after connection closed.

Capitalization of model field name with underscore

results in Entity.setGdna_Start instead of Entity.setGdna_start (mind the capitalization here). This causes problems when writing generators.

Note: After fixing this bug set generate_rest to true in the omicsconnect app module

Development of (simple) user specific GUI functionality is very complex.

It's very hard to implement user controls and add specific events to the controls. ALso the only way to implment constraints now is to use decorators which in turn use DatabaseExceptions while I only want these constraints to apply on the clientside, not on the serverside.

App omicsconnect: importing dataset matrix fails

Importing a dataset matrix using the importer using the omicsconnect app results in the following exception while adding a ObservedValue:
"Field 'Identifier' doesn't have a default value"

Importing the same dataset using another application does not cause an exception.

Possible cause is related to the model extensions used in the omicsconnect app and/or the fact that the omicsconnect model results in warnings/errors during generation (cyclic dependencies).

Styling of the loginscreen

The login screen isn't style correctly.
See screenshot

Security decorators do not secure complete decorator API

The security decorators do not override all methods in decorators resulting in a security flaw