gluufederation / oxorm Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 1.0 6.58 MB

License: MIT License

Java 100.00%

oxorm's People

Contributors

Stargazers

Watchers

Forkers

uprightech

oxorm's Issues

chore: avoid eliding milliseconds (if zero) when retrieving a Date as a String

If an attribute is defined as Date in schema and retrieved as a String in the ORM bean, the milliseconds are elided if they are zero. See ticket 11018

Example:
2001-01-02T12:13:14.000 (Date) -> "2001-01-02T12:13:14" (String)

Specify Different Write / Read LDAP servers in ldap.properties

We can offload some of the read traffic to read-only replicas. This doesn't make sense for writes, because all writes get replicated anyway.

Broader support for lowercase filter in couchbase

Currently we support LOWER in equality, in SCIM there is need for other operators.

Can we add so the following N1QL can be produced? (value is a string):

WHERE (LOWER(attribute) <= value)
WHERE (LOWER(attribute) >= value)
WHERE (LOWER(attribute) LIKE "initial%value%final")

feat: add option to skip`lower` SQL statements

RDBS aurora/mysql (mysql 5.7) not supports lower statement in filters. We need to add new option option.statement.use_filter=true/false to allow skip lower statements in search requests.

feat: update to conform Couchbase SDK 3.x

Map LDAP credentials to backend

With the upgrade to OpenLDAP, it's possible we will have to use different credentials to access different backends. For example, instead of using cn=directory manager for everything, we may need to use cn=manager,o=gluu and cn=manager,o=site respectively for these different backends.

Use dedicated analytics server to do SCIM search

CE 4.0 uses adaptive index which allows to do any SCIM search. But there is one little price of using these indexes:

A generic adaptive index (on SELF) will be qualified for all queries on the keyspace. So, when using with other GSI indexes, this will result in more IntersectScan operations for queries that qualify other non-adaptive indexes. This may impact query performance and overall load on query and indexer nodes. To alleviate the negative effects, you may want to specify the USE INDEX clause in SELECT queries whenever possible.

These indexes led to 7% performance decrease and 5-10% CPU extra load.

We should try to use analytics service instead of adaptive indexes to check if this solution is quicker

Support MySQL with MariaDB engine

MariaDB engine supports JSON fields and operations with JSON values. But instead of JSON, it uses the LONGTEXT alias. As result, the ORM can't determine from the DB metadata for which fields we use JSON. We need to add a metadata table to provide additional information about JSON fields. The table should have the name: INFORMATION_SCHEMA_COLUMNS. Required columns are: TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE. Information about column types we can take from the execution result SELECT * FROM information_schema.columns

feat: Add PostgreSQL support

Add property to allow the Couchbase SDK traffic compression option

This will reduce network traffic

Support child tables in MySQL ORM

For most case JSON type is enough to store multi valued types. But if we needed to index them properly without filter whole table on request we need to allow to store multi-valued attributes in separate child tables. We support similar approach with Spanner DB. We can reuse idea which we used to write this part for MySQL ORM.

ORM : add support for conditional delete

We should support conditional remove.

E.g. DELETE FROM bucket WHERE ( oxDeletable = TRUE AND oxAuthExpiration <= "2019-08-22T16:59:50.553+0000" )

Idea:
public <T> boolean deleteEntries(String baseDN, Class<T> entryClass, Filter filter)

Custom LDAP filter does not handle NOT and Extensible search

When I want to add a custom LDAP filter to Cache Refresh on oxTrust, it results a ArrayOutOfBounds Exception.
I wanted to filter out disabled accounts with the following additional filter:
(!(useraccountcontrol:1.2.840.113556.1.4.803:=2))

After looking into the code of org.gluu.search.filter.Filter class, it does not handle Extensible match, unlike the com.unboundid.ldap.sdk.Filter. And the org.gluu.persist.ldap.impl.LdapFilterConverter class is not handling the NOT type correct, as it retrieves the Filter components from a field that is setting it to an empty array, than it get's the genericFilters[0], which is NULL, as the array length is 0.

Please check and fix. I've fixed as a pilot locally like this, by separating the NOT type from the AND and OR and converted the Extensive type to RAW.

`public com.unboundid.ldap.sdk.Filter convertToLdapFilter(Filter genericFilter) throws SearchException {
FilterType type = genericFilter.getType();

    if ((FilterType.AND == type) || (FilterType.OR == type)) {
        Filter[] genericFilters = genericFilter.getFilters();
        com.unboundid.ldap.sdk.Filter[] ldapFilters = new com.unboundid.ldap.sdk.Filter[genericFilters.length];

        if (genericFilters != null) {
            for (int i = 0; i < genericFilters.length; i++) {
                ldapFilters[i] = convertToLdapFilter(genericFilters[i]);
            }

            if (FilterType.AND == type) {
                return com.unboundid.ldap.sdk.Filter.createANDFilter(ldapFilters);
            } else if (FilterType.OR == type) {
                return com.unboundid.ldap.sdk.Filter.createORFilter(ldapFilters);
            }
        }
    }

    if (FilterType.NOT == type) {
        return com.unboundid.ldap.sdk.Filter.createNOTFilter(convertToLdapFilter(genericFilter));
    }`

`protected Filter convertRawLdapFilterToFilterImpl(com.unboundid.ldap.sdk.Filter ldapFilter) throws SearchException {
byte type = ldapFilter.getFilterType();

    if ((com.unboundid.ldap.sdk.Filter.FILTER_TYPE_AND == type) || (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_OR == type)) {
    	com.unboundid.ldap.sdk.Filter[] ldapFilters = ldapFilter.getComponents();
        Filter[] genericFilters = new Filter[ldapFilters.length];

        if (ldapFilters != null && ldapFilters.length > 0) {
            for (int i = 0; i < ldapFilters.length; i++) {
            	genericFilters[i] = convertRawLdapFilterToFilterImpl(ldapFilters[i]);
            }

            if (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_AND == type) {
                return Filter.createANDFilter(genericFilters);
            } else if (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_OR == type) {
                return Filter.createORFilter(genericFilters);
            }
        }
    }
    
    if (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_NOT == type) {
    	return Filter.createNOTFilter(convertRawLdapFilterToFilterImpl(ldapFilter.getNOTComponent()));
    }

.....
.....
.....

    if (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_SUBSTRING == type) {
        return Filter.createSubstringFilter(ldapFilter.getAttributeName(), ldapFilter.getSubInitialString(),
                ldapFilter.getSubAnyStrings(), ldapFilter.getSubFinalString());
    }
    
    if (com.unboundid.ldap.sdk.Filter.FILTER_TYPE_EXTENSIBLE_MATCH == type) {
    	return new Filter(FilterType.RAW, ldapFilter.toNormalizedString());
    }

    throw new SearchException(String.format("Unknown filter type '%s'", type), com.unboundid.ldap.sdk.ResultCode.PROTOCOL_ERROR_INT_VALUE);`

Specify "failover" | "round robin" connection pool strategy in ldap.properties

If we are using mirrored masters, it's important that we send writes to just one server.

Review Couchbase indexes to check where we can drop objectClass = "XYZ" from them

WHERE objectClass = "XYZ" is not needed if we store in bucket document of one type only

Find and count throw exception if base entry does not exist

Trying to compute the number of fido entries under a user branch, the following is raised:

org.gluu.persist.exception.EntryPersistenceException: Failed to calculate the number of entries with baseDN: ou=fido,inum=A8F2-DE1E.D7FB,ou=people,o=gluu, filter: (&(&(objectClass=organizationalUnit))(&(objectClass=top)(objectClass=organizationalUnit)))
	at org.gluu.persist.ldap.impl.LdapEntryManager.countEntries(LdapEntryManager.java:780) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.impl.LdapEntryManager.countEntries(LdapEntryManager.java:734) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.impl.BaseEntryManager.countEntries(BaseEntryManager.java:191) ~[oxcore-persistence-core-4.0.rc3.jar:?]
      ....
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:403) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.search(LdapOperationsServiceImpl.java:305) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.impl.LdapEntryManager.countEntries(LdapEntryManager.java:776) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	... 95 more
Caused by: com.unboundid.ldap.sdk.LDAPSearchException: The search base entry 'ou=fido,inum=A8F2-DE1E.D7FB,ou=people,o=gluu' does not exist
	at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3772) ~[unboundid-ldapsdk-4.0.7.jar:4.0.7]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:373) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.search(LdapOperationsServiceImpl.java:305) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.impl.LdapEntryManager.countEntries(LdapEntryManager.java:776) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	... 95 more

IMO that should not be an error: when the base of search does not exist, the call should return 0.

About the same applies for find:

org.gluu.persist.exception.EntryPersistenceException: Failed to find entries with baseDN: ou=fido2_register,inum=A8F2-DE1E.D7FB,ou=people,o=gluu, filter: (&(&(objectClass=oxFido2RegistrationEntry))(&(oxStatus=registered)(personInum=A8F2-DE1E.D7FB)))
	at org.gluu.persist.ldap.impl.LdapEntryManager.findEntries(LdapEntryManager.java:441) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.impl.BaseEntryManager.findEntries(BaseEntryManager.java:150) ~[oxcore-persistence-core-4.0.rc3.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: org.gluu.persist.exception.operation.SearchException: The search base entry 'ou=fido2_register,inum=A8F2-DE1E.D7FB,ou=people,o=gluu' does not exist
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:427) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.search(LdapOperationsServiceImpl.java:305) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.impl.LdapEntryManager.findEntries(LdapEntryManager.java:438) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	... 73 more
Caused by: com.unboundid.ldap.sdk.LDAPSearchException: The search base entry 'ou=fido2_register,inum=A8F2-DE1E.D7FB,ou=people,o=gluu' does not exist
	at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3772) ~[unboundid-ldapsdk-4.0.7.jar:4.0.7]
	at com.unboundid.ldap.sdk.AbstractConnectionPool.search(AbstractConnectionPool.java:2056) ~[unboundid-ldapsdk-4.0.7.jar:4.0.7]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:425) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.search(LdapOperationsServiceImpl.java:305) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	at org.gluu.persist.ldap.impl.LdapEntryManager.findEntries(LdapEntryManager.java:438) ~[oxcore-persistence-ldap-4.0.rc3.jar:?]
	... 73 more

Paged search with AWS MySQL fails

Add option to exclude objectClass filter from queries for specified Couchbase buckets

We need to add option:

buckets.option.disable-entry-type-filter: jans_user, jans_token, jans_session

This should enable automatic objectClass = "<name>" expression removal when bucket in list above.

This needed for cases when bucket contains entries of one type only.

Dates stored as local when server is in a time zone other than UTC when in couchbase

It is necessary to store dates like oxCreationTimestamp and updatedAt in ISO format with time zone UTC

Class org.gluu.persist.couchbase.impl.CouchbaseEntryManager uses Java SimpleDateFormat with format yyyy-MM-dd'T'HH:mm:ss.SSS (see encodeTime method) and this is correct as long as server timezone is UTC. When this is not the case, the date will be stored as a local date which is shifted.

In SCIM we use oxCreationTimestamp and updatedAt to build query filters and it is assumed that datetimes have no offset, ie. expressed in UTC

feat: add json_valid valid check to MariaDB JSON column to determione

Reference and description: JanssenProject/jans#4990

Support a way to accurately express less than and greater than filters

I found an issue when handling multivalued attributes in CB. It seems that strict inequality (<, >) filters cannot be accurately defined by reusing <=, >=, =, !

Example:

If chimp is a custom multivalued attribute and I want to query entries where chimp > 0 one thinks of using:

Filter.createANDFilter(
	Filter.createNOTFilter(Filter.createEqualityFilter(chimp, 0).multiValued(true)),
        Filter.createGreaterOrEqualFilter(chimp, 0).multiValued(true)
)

which produces

( objectClass = "gluuPerson" ) AND ( ( NOT 0 IN chimpn ) AND ANY chimpn_ IN chimpn SATISFIES chimpn_ >= 0 END )

If my data looks like

user1 = {
   "chimp": [0,1],
   ...
}

user2 = {
   "chimp": [1,2],
   ...
}

execution of the query only outputs user2 and rejects user1 because it contains 0.

I propose to add createGreaterFilter and createLessFilter methods to Filter interface and delegate converters the task of building the appropriate expressions.

Extends Couchbase ORM to use Analytics services

Couchbase Analytics service allows to to search without creating indexes. It's dedicates server which do search without data update. This will help us with SCIM which uses custom attributes. And as result requires more and more indexes for proper work.