codice / ddf

DDF Distributed Data Framework - an open source, modular integration framework.

Distributed Data Framework (DDF)

Distributed Data Framework (DDF) is an open source, modular integration framework.

Features

• Standardization
  • Building on established Free and Open Source Software (FOSS) and open standards avoids vendor lock-in
• Extensibility
  • Capabilities can be extended by developing and sharing new features
  • Built on top of Apache Karaf for OSGi support
  • Apache Camel and Apache CXF integration
• Flexibility
  • Only features required can be deployed
• Federated Open Geospatial Consortium (OGC) filter powered metadata catalog
  • Enterprise Service Bus (ESB) performs transformation between query standards
  • Apache Solr integration
    • Well Known Text (WKT) indexing and search for spatial awareness
    • Full text search
    • XML indexing
  • Open Geospatial Consortium (OGC) KML, CSW, and WFS federated services
  • OpenSearch federated services
  • REST API for catalog operations
  • Integrated content framework to store actual products associated with the indexed metadata
  • Tika parser for extracting metadata from common file formats (Office, PDF, etc)
  • Plugin support for pre and post processing on all operations
  • Eventing for notifications
  • Metrics
• Security
  • Web Service Security (WSS) functionality that comes with DDF is integrated throughout the system
    • SAML 2.0 Web Browser SSO Profile with included IdP server and client
    • SAML ECP Profile
    • SAML Security Token Service (STS) based on WS-Trust
    • Automatic protection and Single Sign On (SSO) for web applications without modifying the application itself
    • Extensible PDP with XACML 3.0 support for authorization decisions
    • LDAP integration
      • Included OSGi enabled OpenDJ LDAP server
    • X.509 authentication
    • Basic authentication
    • SAML authentication
    • Guest login support
    • WS-Security, WS-SecurityPolicy, WS-Policy, WS-Trust, WS-SecureConversation, WS-Addressing
  • Provides a pluggable and extensible Security Framework (a set of APIs that define the integration with the DDF framework)
  • Provides Security Service reference implementations for a realistic end-to-end use case.
  • Role and Attribute based access control
  • Attribute based filtering for searches performed throughout the system
  • Federated identity through metadata catalog
• Search user interface [DDF UI (https://github.com/codice/ddf-ui)
  • 3D WebGL map based on Cesium
  • 2D map based on OpenLayers 3
  • USNG/MGRS grid support
  • GeoNames geocoder integrated into both maps
  • Upload and edit capability
  • Saved workspaces (searches and metadata artifacts)
• Admin Web user interface
  • Web based install wizard
  • Application grid to organize configurations
  • Pluggable configuration pages for applications to simplify configurations for complex scenarios
  • Metrics web application to view up to date system metrics
• Simplicity of installation and operation
  • Unzip and run
  • Configuration and Installation via a modern Admin Web console
• Simplicity of Development
  • Build simple Java Objects and wire them in via a choice of dependency injection frameworks
  • Make use of widely available documentation and components for DDF's underlying technologies
  • Modular development supports multi-organizational and multi-regional teams

Building

What you need

• Install OpenJDK 11.
• Make sure that your JAVA_HOME environment variable is set to the newly installed JDK location, and that your PATH includes %JAVA_HOME%\bin (Windows) or $JAVA_HOME$/bin (*NIX).
• Install Maven 3.6.3 (or later). Make sure that your PATH includes the MVN_HOME/bin directory.
• Set the MAVEN_OPTS variable with the appropriate memory settings

Optional

• If you do not wish to run formatting from the commandline (see below) you may use an IDE to format the code for you with the google-java-format plugins.
  • https://github.com/google/google-java-format
    • IntelliJ: https://plugins.jetbrains.com/plugin/8527
    • Eclipse: https://github.com/google/google-java-format/releases/download/google-java-format-1.3/google-java-format-eclipse-plugin-1.3.0.jar

How to build

In order to run through a full build, be sure to have a clone for the ddf repository and optionally the ddf-support repository (NOTE: daily snapshots are deployed so downloading and building each repo may not be necessary since those artifacts will be retrieved.):

git clone git://github.com/codice/ddf.git
git clone git://github.com/codice/ddf-support.git (Optional)

Change to the root directory of the cloned ddf repository. Run the following command:

mvn install

This will compile DDF and run all of the tests in the DDF source distribution. It usually takes some time for maven to download required dependencies in the first build. The distribution will be available under "distribution/ddf/target" directory.

How to release

See https://github.com/codice/ddf/wiki/Release-Process for documentation on how to perform a release.

Formatting

If during development the build fails for formatting, eg:

Failed to execute goal com.coveo:fmt-maven-plugin:1.8.0:check (default-cli) on project <module name>: Found <number> non-complying files, failing build

You can run the formatter with the command:

mvn fmt:format

For additional information about building DDF, see Building DDF.

How to Run

• Start SolrCloud based on the README in <ddf_repo_src>/distribution/docker/solrcloud
• Unzip the distribution.
• Run the executable at <distribution_home>/bin/ddf.bat or <distribution_home>/bin/ddf

Additional information

Find documentation about DDF at codice.org.

Discussions can be found on the Announcements forum, Users forum, and Developers forum.

For changes between versions and known issues, please read the release notes.

If you find any issues with DDF, please submit an issue.

For information on contributing to DDF see Contributing to Codice.

The DDF Website also contains additional information at http://ddf.codice.org.

Many thanks for using DDF.

-- The Codice DDF Development Team

Copyright / License

Copyright (c) Codice Foundation

This is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. A copy of the GNU Lesser General Public License is distributed along with this program and can be found at http://www.gnu.org/licenses/lgpl.html.

ddf's Issues

Experiment with PR templates

Description

We are currently working on a multi-issue experiment #4289 to improve workflow and transparency. It's reasonable to assume we could get similar process improvements by incorporating multiple PR templates as well.

I'm not quite sure what/how many is appropriate for PRs and would like to see some conversation on the topic. Off the top of my head, I think templates for

• Bug fix (dependency upgrades would fit in here as well, I think?)
• Documentation
• Improvement (and/or new feature)
• Repository meta

would work well. These are granular enough that we could include different body elements, tags, and even assignees in the case of Docs and Repo meta where there are committers who actively work those very specific areas long term. They are not so granular, I think, that we're making noise for no value.

Automatic logout fails to create logout request

Description

Severity = High

Automatic logout fails to create logout request.
After the timer shows up that the user will be logged out within x seconds and gets to 0 and redirected, the following message is shown:

You are now signed out.
You can return to the landing page.
Failed to create logout request

Steps to Reproduce

1. Install and login through the UI
2. Wait and be idle on the UI until session expiring countdown executes
3. Have chance of being redirected to a successful logout, or receive above message

Expected behavior :

Redirected to a successful logout and receive user logged out successfully

Actual behavior:

Receive mentioned message and the user is not actually logged out

Reproduces how often:

about 5-10%

Affects versions

2.13.6
2.14.0

Additional Information

Stack trace of null pointer exception:

java.lang.NullPointerException: null
        at ddf.security.assertion.impl.SecurityAssertionImpl.parseToken(SecurityAssertionImpl.java:168) ~[242:security-idp-client:2.13.6.SNAPSHOT]
        at ddf.security.assertion.impl.SecurityAssertionImpl.<init>(SecurityAssertionImpl.java:145) ~[242:security-idp-client:2.13.6.SNAPSHOT]
        at ddf.security.assertion.impl.SecurityAssertionImpl.<init>(SecurityAssertionImpl.java:128) ~[242:security-idp-client:2.13.6.SNAPSHOT]
        at org.codice.ddf.security.idp.client.LogoutRequestService.getIdpSecurityAssertion(LogoutRequestService.java:517) ~[242:security-idp-client:2.13.6.SNAPSHOT]
        at org.codice.ddf.security.idp.client.LogoutRequestService.sendLogoutRequest(LogoutRequestService.java:186) [242:security-idp-client:2.13.6.SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193) [125:org.apache.cxf.cxf-rt-frontend-jaxrs:3.2.6]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103) [125:org.apache.cxf.cxf-rt-frontend-jaxrs:3.2.6]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [121:org.apache.cxf.cxf-core:3.2.6]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:225) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [19:javax.servlet-api:3.1.0]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276) [134:org.apache.cxf.cxf-rt-transports-http:3.2.6]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.codice.ddf.pax.web.jetty.SecurityJavaSubjectFilter.doFilter(SecurityJavaSubjectFilter.java:60) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.codice.ddf.pax.web.jetty.ResponseFilter.doFilter(ResponseFilter.java:96) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.codice.ddf.pax.web.jetty.CsrfFilter.doFilter(CsrfFilter.java:181) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.codice.ddf.pax.web.jetty.ClientInfoFilter.doFilter(ClientInfoFilter.java:72) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215) [177:org.eclipse.jetty.websocket.server:9.4.11.v20180605]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:513) [78:org.eclipse.jetty.security:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [81:org.eclipse.jetty.servlet:9.4.11.v20180605]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [185:org.ops4j.pax.web.pax-web-jetty:7.2.4.CODICE]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.Server.handle(Server.java:531) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [80:org.eclipse.jetty.server:9.4.11.v20180605]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [72:org.eclipse.jetty.io:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [83:org.eclipse.jetty.util:9.4.11.v20180605]
        at java.lang.Thread.run(Thread.java:748) [?:?]```

Migrate the result-sort component to React

Description

This change will migrate the result-sort component to React

Expected behavior:

• the expected behavior of result sort should remain the same

Version

2.14.0

Additional Information

Add Artifact Size Enforcer Plugin to DDF builds

This PR - codice/ddf-support#54 - in the DDF-Support repository adds a build enforcer plugin to check/validate that that size of generated build artifacts does not exceed a threshold. Using it in DDF will allow us to monitor and maintain the size of each artifact and help us trim down the resulting distribution size.

Remove feature installation profiles from the installer

Description

The different profiles in the installer are ambiguous and not heavily used. Instead of making these profiles an option in the installer, we can install a default profile on boot. This would provide the following benefits:

• The system would take a bit longer to boot but the overall installation time should improve greatly since karaf will optimize for all the bundles being started at once. Some manually testing I’ve done is showing about a 40 sec speedup.

• Itests currently install individual apps instead of one of these profiles. This change would make our itests more consistent with a production environment.

• There are several places where we use the word “profile” in ddf, each meaning something different. Removing one of these definitions would add clarity.

• This will allow us to remove the profile & app features, making our features much more flat and well defined. It would also allows us to uninstall features again.

• Once the system has booted, you can reach other areas of the system without going through the installer, which will improve development time.

Expected behavior:

The "profile selection" stage of the installer is removed and the features composing the profile feature are added to the bootFeature section.

Version

Latest (2.14.0-SNAPSHOT)

Additional Information

Upgrade Geotools to latest version

Description

Geotools needs to be upgraded to the latest version, 20.1.

Upgrades will also be made for the dependencies that interact with geotools.

Version

DDF-2.14

OpenSearchSource doesn't have a configurable poll interval

Description

Improvement: OpenSearchSource doesn't have a configurable poll interval

We have a server admin that would like us to stop polling him every minute. Other sources allow you to change the poll interval, but not this one.

Issue is not a bug.

Steps to Reproduce (if issue is bug)

1. [First Step]
2. [Second Step]
3. [and so on...]

Expected behavior:

[What you expect to happen]

Actual behavior:

[What actually happens]

Reproduces how often:

[What percentage of the time does it reproduce?]

Version

[Application version]

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Query Representation Not Preserved for Bounding Box & Keyword

Description

One common complaint we have received is the lack of preservation pertinent to queries being saved from Intrigue.

Two Main Issues:

• Queries saved from a keyword re-render as a polygon or multipolygon (keyword metadata is lost and not preserved)
• Queries saved from bounding box re-render as polygon (bbox corner points are not preserved and fail to reconstruct the correct bbox mode)

This user experience is strange and causes a lot of confusion due to lack of preserved information related to the original query.

This ticket proposes the UI to re-interpret the construction of the original query utilizing the filter-tree JSON approach instead of solely reconstructing off the persisted WKT from the preserved CQL. There are many benefits to this approach:

1. We get a richer format that can utilize the geojson spec, but also enhance the metadata we store about a query

2. We get direct 1:1 interpretation from original query structure to and from intrigue. Currently, intrigue interprets from CQL, therefore, it must use a tree-walk interpreter to re-render the correct data before actually rendering onto Intrigue. With the filter tree JSON approach, we can stop doing any CQL conversion before rendering onto Intrigue. This will allow us to start moving in the direction of dropping support for the CQL parser.

As a DDF developer, I would like to be able to assign myself to an issue when I begin work on it

Description

Currently, there is no way for a contributor to indicate they are working on an issue short of leaving a comment on the issue and hoping a committer notices it and adds them as an Assignee. Ideally, a contributor would be able to self-assign when they decide to pick up an issue.

Expected behavior:

When a contributor (who is not a committer) wants to start work on an issue, they have some mechanism by which they can self-assign to the issue so that they do something and they end up assigned to the issue.

Version

N/A

Additional Information

Remove keyczar and replace with custom encryption/decryption logic

Description

Keyczar is currently insecure: https://github.com/google/keyczar#use-of-sha-1-and-1024-bit-dsa. As such, it needs to be removed and replaced with custom logic (after further research, we have decided to try out Tink) that will do the encryption/decryption instead.

Version

2.13.6

The Query Settings extension should empty the extensions region when no extension is returned

Description

As a developer, I want the query settings extension to empty the extension region when the getExtensions call returns nothing. At the moment, it will leave behind the previous extensions, which puts pressure on downstream to do things like return empty views or try to manipulate the region directly.

Expected behavior:

If getExtensions returns nothing, the existing extension region should be emptied.

Version

2.14.0

Map Location Search Dropdown has overlapped results

Description

When trying to search for a location in Intrigue, the dropdown results seem to render on top of each other which makes the dropdown hard to use. It's possible to click a dropdown item without being able to read what it is.

Steps to Reproduce

1. Open up Intrigue and make sure you have the map visualization
2. Type in the "Enter a region, country, or city" dropdown and notice that the results show up like this:
   
   

Expected behavior:

Dropdown items render one after another and are readable.

Actual behavior:

Dropdown items render on top of each other.

Reproduces how often:

Seems to happen 100% of the time for me

Version

Master branch/untested with backport branch

Notes

I'm not sure what the root cause is here. Can anyone else try to reproduce it?
(Duplicate of https://codice.atlassian.net/browse/DDF-4482)

Add Code of Conduct to DDF

Add a standard CoC to DDF for contributors.

On the release branch, suppress Karaf's XXE vulnerability and update the release notes

Description

A CVE just popped up that is blocking 2.13.x builds (doesn't affect DDF master because it's been upgraded to Karaf 4.2.2). Suppressing this CVE since it's mitigated by file system hardening. Will also update the release notes to be explicit about the vulnerability and point to the hardening guide.

[ERROR] One or more dependencies were identified with vulnerabilities that have a CVSS score greater than or equal to '7.0':
00:26:45 [ERROR] 
00:26:45 [ERROR] org.apache.karaf.jaas.boot-4.2.1.jar: CVE-2018-11788
00:26:45 [ERROR] 
00:26:45 [ERROR] See the dependency-check report for more details.

Build reference: https://jenkins.codice.org/job/DDF-Jobs/job/2.13.x/job/pr/job/Linux/597/console

Expected behavior (if applicable):

N/A

Version

2.13.6-SNAPSHOT

Additional Information

N/A

Error pops up when you share a search form

Description

When you go to share a search form, and click apply an error message will pop up saying the save has failed. However, the sharing was actually successful. For example, if user A shares a form with user B, user A will see an error message but user B will have access to the form.

Steps to Reproduce (if issue is bug)

1. Create a new search form
2. Attempt to share, error message will appear even when sharing is successful

Expected behavior:

The search form should be saved, along with a "Save Successful" message

Actual behavior:

Search form not saved and an error message appears

Reproduces how often:

Behavior can be reproduced every time

Version

[Application version]
2.14

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Update CDM (in place) webdav implementation so it gracefully handles network disconnects

Description

Upon a network disconnect the current webdav implementation will stop all processing when an error occurs and throws an error to the console. The problem occurs that the state, at least when the metacards are being created, only updates once the entire operation finishes. This can lead to inconsistent states where multiple metacards exist per webdav entry. This is also a problem with delete and potentially a problem for modifications.

Steps to Reproduce (if issue is bug)

1. Start up a webdav server, (follow the directions here to set it up over Tomcat)
2. set a breakpoint in the code @ DavAlterationObserver#177
3. Set up CDM to point to the webdav server
4. Ingest one or two metacards and then shutdown the tomcat server. (Press play one or two times)
5. wait until the error is thrown
6. Start the tomcat server again
7. Query for the metacard, and note there are two of them.

Expected behavior:

The state gets updated after a process successfully completes.
There is only one metacard after a disconnect and reconnect.

Actual behavior:

The state gets updated at the end of the batch.
There are multiple metacards after a disconnect and reconnect

Reproduces how often:

100%

Version

This is a problem on both the [2.13.x], and [master] branches, along with any branch cut from those.

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Reorder documentation for URL Resource Reader to match CDM's layout

Description

Currently, the contents of CDM 7.6.5 and the URL Resource Reader 22.3.1 are very similar, but the organization of URLRR's documentation could be considered confusing.

URLRR's contents should be ordered to match CDM's contents. The permissions section should be moved to appear before the app configuration section. Additionally, the permissions section should not have its own major heading (22.4.). It should fall under the 22.3.1 tree.

This may make configuring a CDM to monitor in place easier to follow.

Version

2.13.6
2.14.0

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Migrate metacard-interactions to react

Description

Migrate metacard-interactions to react to clean up the code and make it more performant.

Expected behavior:

Functionality remains the same

Version

2.14.0-SNAPSHOT

Rename "Catalog UI Search" to "Catalog Search UI" in configurations

Description

I propose rewording "Catalog UI Search" to "Catalog Search UI" the Search UI configurations:

There are currently 2 different Catalog search UIs included in DDF: Intrigue and Simple Search UI. There isn't only one "Search UI", and I don't think there will be other types of UIs for Catalog than Search UIs. I think this rename will make this fact clearer and be obvious that the configurations affect Intrigue, not "the only search UI".

Expected behavior:

• The Admin Console and the Documentation are updated to make clear that there are 2 search UIs: Simple Search UI and Catalog Search UI (a.k.a Intrigue).
• Intrigue configurations should be designated to indicate that they affect Intrigue, not all Search UIs/the Search UI.

Version

2.14.0

Additional Information

This ticket was spawned from #4291.

Thumbnail Images are loaded even when they are not rendered on the page

Description

Thumbnail Images are loaded even when they are not rendered on the page. This can be verified by open the browser debugger and monitoring the network traffic. With no table view loaded, and the left column of results set to list view (as opposed to gallery view), there is no place that the thumbnails are rendered to the user upon receiving a query response. However, the UI attempts to download the first 25 thumbnails anyway.

Ideally, these calls would not happen until the thumbnails are to actually be rendered on the page.

Steps to Reproduce (if issue is bug)

1. [First Step]
2. [Second Step]
3. [and so on...]

Expected behavior:

Thumbnails are loaded on demand

Actual behavior:

Thumbnails are loaded like they're going out of style

Reproduces how often:

100% of the time

Version

2.14.0

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Allow user the ability to set a default list

Description

A user can create a number of lists in the List tab in intrigue. Allow the user to set one of the lists as a default list so that it will appear at the top of the list.

Expected behavior:

The user will be able to set a list as default and that list will move to the top of the list.

In the image above, ccccc was selected as the default.

Actual behavior:

Currently there isn't a way to set a default list.

Reproduces how often:

[What percentage of the time does it reproduce?]

Version

[Application version]

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Negative decimal coordinates are off by one in the tooltip

Description

See screenshot. Stored latitudes and longitudes differ from what is displayed in the tooltip by one degree. Only affects negative values.

Steps to Reproduce

1. Set coordinate format to decimal
2. Place a breakpoint on line 114 of map-info.view.js
3. Move your mouse onto the map, somewhere south of the equator in the western hemisphere
4. Check the stored values of latitude and longitude in the console
5. Resume script execution
6. Check the displayed values of latitude and longitude in the tooltip

Expected behavior:

Negative decimal coordinates displayed in the tooltip should match the stored coordinates

Actual behavior:

Negative decimal coordinates displayed in the tooltip are off from the stored coordinates by one degree

Reproduces how often:

100%

Additional Information

Guest ingests should not include IP Address as part of Guest Principal name

This is a duplicate of this Jira issue for testing purposes

The Guest Principal is created with a name of the form Guest@[IP_ADDR] which leaks information about Guest access when files are ingested and the PoC field is filled in. We should obfuscate the IP Addresses of Guest Principals.

• Update the STS Guest Validator to set the Principal name to Guest@[RANDOM]
• Write a security audit record of the form Mapping IP Addr [XXXX] to Internal ID [200f7e55Xda24X4ec3X8e9fX2bad7393fd53]
• Prevent a small secondary issue where Guest@ is sometimes prepended twice to the id on the principal's name
• Cache the generated token for a reasonable period so it can be reused instead of a new random id being created and logged on each Guest access

Implement a DoSFilter in DDF

Description

DDF needs a filter to prevent DoS attacks. Configure the filter to allow a reasonable number of requests and to not interfere with the rapid requests made when testing.

Version

Apply fix to 2.13.6 and 2.14.0

Migrate Map Components to React - Home and Set Home Buttons

Description

This change will migrate the home and set home buttons to React and implement them using a split button drop down.

Expected behavior:

• the Home and Set Home buttons will be displayed as a split button drop down.
• clicking "Home" returns the user to the pan position and zoom level previously saved.
• clicking "Set Home" sets the current pan position and zoom level as the home position.

Version

2.14.0

Additional Information

None.

Round location data from ingested files to 6 digits

Description

Round ingested location data to 6 degrees of decimal precision.

A PR was recently merged to limit UI of location data to 6 degrees of decimal precision. Previously up to 14 digits were allowed, but that level of precision was deemed unnecessary and it caused problems throughout the system, including readability issues. The previously mentioned PR only restricted user input via the UI; location data ingested via files is still allowed to have more than 6 digits of decimal precision.

The scope of this issue would be to round location data ingested via files such that the location data on any given metacard will only have 6 digits of precision.

NOTE: This issue will most likely impact downstream projects.

[Description of the issue]

Steps to Reproduce (if issue is bug)

1. [First Step]
2. [Second Step]
3. [and so on...]

Expected behavior:

[What you expect to happen]

Actual behavior:

[What actually happens]

Reproduces how often:

[What percentage of the time does it reproduce?]

Version

[Application version]

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

As a user, I would like to enter Degrees, Minutes, Seconds coordinates into the gazetteer search box so I can pan to a certain location quickly

Description

Copying a DDMMSS coordinate (06°09′13″S 136°31′35″E) from the map and pasting it in the pan/zoom does not work. It appears we do not currently support this format in the pan & zoom capability. The result instead if it being treated as a series of 6 numbers, and being recognized as 3 lat/lon decimal pairs that can be used to define a bounding box, which may be nothing like the originally defined point.

Expected behavior:

Ideally the pan and zoom feature would recognize the DDMMSS format and provide that as the top option.

Version

Additional Information

Update lodash and plotly to newer versions

DDF is using out of date versions of these dependencies and needs to be updated.

Make WMTS related errors be named consistently and correctly within open layers

Description

We have been getting complaints about inconsistencies within our error messages, specifically referring to WMTS as WMT. This has caused confusion in our downstream projects as we refer to the same types of layers as WMTS elsewhere and WMT here.

This is a very small improvement to update these messages by making them read as "WMTS" instead of "WMT".

Example:

ddf/ui/packages/catalog-ui-search/src/main/webapp/js/controllers/ol.layerCollection.controller.js

Line 68 in 15f2049

throw new Error('WMT map layer source has no layers.')

https://en.wikipedia.org/wiki/Web_Map_Tile_Service

Expected behavior:

Have error messages read as WMTS in place of WMT.

Querying by attribute before schema fields are cached throws an exception

Description

Querying by attribute before schema fields are cached throws an exception

Steps to Reproduce

1. Start DDF.
2. Query by attribute with float type.

Expected behavior :

Query by attribute and get 0 results.

Actual behavior:

Query by attribute and an exception is thrown.

Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://localhost:8994/solr/catalog: Invalid Number: 1.1 for field ext.float_attribute_int

Reproduces how often:

100% before ingesting data.

Affects versions

2.14.0-SNAPSHOT

Transformer blacklist for metacard and query response transformer export actions

Description

• Create an admin configurable blacklist for all metacard and query response transformers.
• Exporting individual query results, result sets, and actions will reflect this change.

Purpose:

• Downstream projects will be able to control which transformers appear as export options. This level configuration can be used to enhance user experience.
• The query response transformers have this functionality via a service property. Configuration as opposed to customization is preferred for this type of operation.
• If this issue is not resolved then broken or otherwise unwanted transformers will continue to show up.

Expected behavior:

• Blacklisted transformers will not show up as export options.

Actual behavior:

• All transformers transformers show up as export options.

Version

2.14.0

Additional Information

None

As an administrator, I'd like finer control on who can see warnings and errors

Description

After a merge, merge lost the ability for admins to say "I want no users to see my machine's errors and/or warnings, but I want my users to see errors/warnings from other machines if the other machine wants to show me errors/warnings".

Expected behavior (if applicable):

Version

Master/2.13.x

Additional Information

Optimize MetacardTypeImpl getAttributeDescriptor

Description

The getAttributeDescriptor method of MetacardTypeImpl is fairly inefficient as it loops through all descriptors on a metacard until it finds the one that matches the name.

Steps to Reproduce (if issue is bug)

N/A

Expected behavior:

N/A

Actual behavior:

N/A

Reproduces how often:

When ingesting large amounts of data, this method shows up as one of the hot spots in the code.

Version

2.13

Additional Information

N/A

Refactor Location Radio Buttons to Dropdown

Description

Currently, as we extend the location component to support more items, BBOX, polygon etc. As the number of items in this section grows, the UI starts to look strange.

Given that this plugin is extensible and used downstream, the numbers of items has already started to grow and makes for a poor user experience.

This UI enhancement seeks to remediate this problem by placing everything in a scrollable dropdown.

Current

Inevitable Issues (Mockup)

Proposed Change

Steps to Reproduce (if issue is bug)

1. Search
2. Go to location and select Somewhere Specific
3. See Radio buttons to be replaced with above image

Treat Basic / Text Search Forms as First Class Components

Description

Treat Basic / Text Search Forms as First Class Citizens. With the introduction of custom search forms, these were moved under the search form management view, which created a poor user experience since the user would have to click more to conduct a search.
[Description of the issue]

Expected behavior:

[What you expect to happen]
The Basic / Text Search forms should be available from the initial dropdown.

Actual behavior:

[What actually happens]
The Basic / Text Search forms are buried in an additional dropdown.

Version

[Application version]
2.14

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.
Original PR that introduced the poor UX changes : #3004

Update the documentation for ClaimsHandlerManager to reflect correct type for url property

Description

The url property on ClaimsHandlerManager was recently changed from String to Array[String] but the documentation was not updated to match.

Expected behavior (if applicable):

The documentation should match the configuration.

Version

Master & 2.13.5

Additional Information

Relates to work performed for https://codice.atlassian.net/browse/DDF-4090

Move misplaced content from Integrator Documentation

Description

Some of the pre-existing content in the Integrating section of the documentation would be more useful for other audiences. This ticket is to relocate that content appropriately

Version

2.14.0

Additional Information

Any additional information, configuration or data that might be necessary
to reproduce the issue.

Add support for preQueryPlugin to be async

Description

Add support for the frontend preQueryPlugin in the Query model to be an async function.
This will give more flexibility to what the plugin can do. For example, this will allow the plugin to fetch data before adding the data/payload to the search request.

This will not affect the PreQueryPlugin in the catalog framework

Special care should be taken to ensure that functions that use the return value of startSearch in the Query Model handle the returned promise gracefully

Expected behavior:

Searches to still operate the same from the user perspective

Version

2.14.0

Additional Information

Any additional information, or data that might be useful to know about

Windows builds produce a distribution with unusable shell scripts

Description

Building a distribution on a windows machine results in an assembly containing shell scripts with CRLF. The distribution zip should instead be consistent regardless of what OS it was built on.

Steps to Reproduce

1. Build distribution on windows
2. Run the distribution on linux (or use the docker image produced by the build)
3. Observe that the bin scripts do not work due to CRLF

Expected behavior :

Bin scripts should work correctly when produced from a windows machine

Actual behavior:

Bin scripts do not work unless dos2Unix is run on them

Reproduces how often:

Any time a windows built zip is used on linux

Upgrade Karaf to the latest version [4.2.6]

Description

Upgrade Karaf to version 4.2.3

Version

Master

Additional Information

Current workarounds are commented below that will be fixed upon upgrading to Karaf 4.2.3.
When upgrading these workarounds should be removed.

Fix CSW ingest and query performance hot spots

Description

Fix profiled hotspots during CSW ingest and query operations.

Expected behavior:

Increased efficiency for ingest and query operations via CSW endpoint.

Version

2.14.0-SNAPSHOT

Additional Information

Add space around the actions / buttons belonging to the map

As a user, space around the buttons / actions will help me to associate them with the map and is more aesthetically pleasing.

Description

Why is this change being proposed?
It helps users understand where the actions belong and provides some breathing room.

Who would want this change implemented?
Users

Current behavior:

Expected behavior:

Version

2.14.0

Issue Template improvements

Description

Improving the quality of the issue templates by making them easier to use, and more informative for both developers and committers of this repository.

Additional Information

While we're still in the pilot program using git for our ticket tracking, this ticket will remain open.

If you're putting up a PR for this issue remove fixes before this ticket number.

reorganize endpoint documentation by task

Description

Reorder the documentation in Integrating > Endpoints to group the endpoints according to their uses:

• Ingest endpoints
• CRUD endpoints
• Query endpoints
• Content Retrieval endpoints
• Publication/Subscription endpoints

Version

2.14.0

Web apps embedded in the source action iframe do not work as expected

assign to @rymach

Description

See steps

Steps to Reproduce

1. add a source action pointing to some webapp url
2. click the source action on the sources page
3. attempt to interact w/ the webapp in the iframe
4. errors appear in console, webapp doesn't work

Expected behavior :

webapp works

Actual behavior:

webapp doesn't work,
errors in console

Reproduces how often:

100%

Platform and environment:

N/A

Affects versions

all

Additional Information

Migrate query result export actions from marionette to react

Description

• Currently the export actions for a query result are written with Marionette and to help in the continued effort towards a React migration this component should be migrated as well.
• This change is being proposed because some other work is being done in this export area so it makes sense to migrate this component now.

Expected behavior:

• The expected behavior for this improvement is to behave exactly as it did before this migration.

Version

2.14.0

Additional Information

• None

Update Pull request template to link to and support GH Issues pilot

Description

Update the PR Template so that it includes references to the documentation in the GH Wiki regarding the pilot of using GH Issues. Also make a small update to support linking back to GH Issue and future workflow automation with the Fixes: #___ methodology.

Additional Information

For more information about the pilot program, see the thread in the DDF Developers' Group.

Upgrade c3p0 to newer version

Description

Upgrades c3p0 to newer version.

Additional Information

There's a newer patch version with better XML processing.

Move configuration defaults from .config files to classes

Description

• Classes will be created in a valid state instead of initializing before setting required properties. These classes will also be able to be unit tested in their valid/default state.
• Features will no longer need to drop .config files, reducing the total number of .config files and simplifying feature definitions. These .config files also cause default configurations to appear as custom configurations in the Admin Console:
  
• Downstream projects can drop .config files containing only the properties that are different from the defaults defined in DDF.

Expected behavior:

• Dropping a .config file that contains not all of the properties for a config should successfully update the properties but keep the defaults for all not-included properties.
• Configurations appear to set to the defaults in the Admin Console:
  

Version

2.14.0

Additional Information

none

Intrigue's WebSockets implementation does not check if user is still logged in

Description

If a user has the Intrigue "Search" view up in the browser and logs out in a separate tab, they are still able to modify and execute queries. This is because, after the initial authentication, the WebSocket requests do not check if the user still has a token in that HttpSession.

Steps to Reproduce

1. Go to Intrigue and go into a workspace.
2. Execute a search (wildcard is fine)
3. In a separate browser tab, pull up the AdminUI and logout.
4. Go back to Intrigue and edit the open query and/or execute it.

Expected behavior :

The query cannot be executed once the user is logged out.

Actual behavior:

The query is executed and relevant results returned.

Reproduces how often:

100%

Platform and environment:

N/A

Affects versions

2.13.6 and master

Additional Information

N/A