Log4j vulnerability about snow-owl

[cmark]

Yeah, that's the 8.x build (Codecov sometimes misbehaves, could not figure out why so far). The 7.x builds are running on our internal CI servers and they have been completed 30 minutes now. Dockerhub refreshes itself quite slowly, I recommend pulling the image anyway via docker pull b2ihealthcare/snow-owl-oss:7.x and checking its image ID. It should be f0a33069ff08.

I hope this helps.

Cheers,
Mark

[cmark]

Hi @mattecasu,

Snow Owl is unaffected of this vulnerability (it uses a different logging library than log4j), but Elasticsearch as storage and search backend is somewhat vulnerable, RCE is not possible, but information leakage in certain cases yes, please refer to this site: https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/.
Whether you are running your Snow Owl instance in embedded mode or connecting it to an external Elasticsearch cluster, it is recommended to set the -Dlog4j2.formatMsgNoLookups=true command-line argument for both Snow Owl (just in case) and Elasticsearch and restart the service(s). This eliminates the possibility of information leakage and mitigates the issue while we are in the process of releasing an official patch.

All three dev streams (6.x, 7.x and 8.x) have received their respective patches already back in December before the holidays, but due to other issues (see #958) with the latest 7.16.2 Elasticsearch we couldn't release the 7.x and 8.x streams that time, only the 6.x patch. Also, it is very likely that another Elasticsearch patch (7.16.3 that contains log4j 2.17.1) will be released soon and we have decided to wait for it before we perform an actual release.

The currently planned release date for Snow Owl 7.18.1 and 8.1.0 is the 21st of January.

Cheers,
Mark

[cmark]

Hi @mattecasu,

Snow Owl 7.18.1 is out, which depends on Elasticsearch 7.16.3 and resolves any potential security vulnerability around log4j2.
Please find the docker image here.
If you have any further questions or issues feel free to contact us.

Cheers,
Mark

[mattecasu]

Dear Mark,

thanks for the patched tags - we are having an issue when using 7.18.1, I suspect it's not compatible with the latest AWS ElasticSearch clusters version (that is on 7.10). It seems like it cannot contact the cluster or find the indexes ("cluster not available" and also Couldn't check the existence of ES index 'locks-lock' ).
[Are you aware of this? Or maybe we are not considering something]

By the way: I confirmed 7.18.0 instead worked fine (which is on ES 7.10)

[cmark]

Hi @mattecasu,

Do you have the full stack trace?
It could be a compatibility issue between the Elasticsearch 7.16.3 client and the AWS Elasticsearch 7.10 service.

Cheers,
Mark

[aelred]

Hi @cmark, here's the full stacktrace we're getting:

[2022-01-31 16:28:06.733] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] INFO  elastic-snowowl - Connecting to Elasticsearch cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com', connect timeout: 2000 ms, socket timeout: 120000 ms.
[2022-01-31 16:28:07.222] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] INFO  index.locks - Preparing 'locks' indexes...
[2022-01-31 16:29:07.275] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] ERROR snowowl - Couldn't check the existence of ES index 'locks-lock'.
com.b2international.index.IndexException: Couldn't check the existence of ES index 'locks-lock'.
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:150)
	at com.b2international.index.es.admin.EsIndexAdmin.create(EsIndexAdmin.java:171)
	at com.b2international.snowowl.core.locks.DefaultOperationLockManager.<init>(DefaultOperationLockManager.java:85)
	at com.b2international.snowowl.core.internal.locks.LockPlugin.preRun(LockPlugin.java:43)
	at com.b2international.snowowl.core.setup.Plugins.preRun(Plugins.java:88)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:291)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:272)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:260)
	at com.b2international.snowowl.server.product.SnowOwlServerActivator.start(SnowOwlServerActivator.java:52)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)
	at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
	at org.eclipse.osgi.container.Module.doStart(Module.java:605)
	at org.eclipse.osgi.container.Module.start(Module.java:468)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1845)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1838)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1781)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1743)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1665)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: com.b2international.commons.exceptions.BadRequestException: Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available.
	at com.b2international.index.es.client.EsClientBase.checkAvailable(EsClientBase.java:101)
	at com.b2international.index.es.client.http.IndicesHttpClient.exists(IndicesHttpClient.java:60)
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:148)
	... 26 common frames omitted

(I slightly redacted our Elasticsearch URL)

[cmark]

Hi @aelred,

Could you please try the latest 7.x image from Dockerhub (it should be available at any minute now)? That has a change that will print out the cluster diagnostic message next to the Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available. error, so we can see what is going on under the hood and hopefully get a fix for it quickly.

Thank you,
Mark

[aelred]

Thanks Mark, we'll give it a try when it's pushed up!

It looks like the build failed though:

->  Pinging Codecov
https://codecov.io/upload/v4?package=github-action-1.0.6&token=<hidden>&package=github-action-1.0.6&token=&branch=8.x&commit=f1bc20cf61320f7162c7bb20853e5915e33fc699&build=1773785924&build_url=https%3A%2F%2Fgithub.com%2Fb2ihealthcare%2Fsnow-owl%2Factions%2Fruns%2F1773785924&name=&tag=&slug=b2ihealthcare%2Fsnow-owl&service=github-actions&flags=&pr=&job=Java%20CI&cmd_args=n,F,Q,Z
{'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
404
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  575k  100   171  100  575k   1195  4025k --:--:-- --:--:-- --:--:-- 4027k
    {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
Error: Codecov failed with the following error: The process '/usr/bin/bash' failed with exit code 1

https://github.com/b2ihealthcare/snow-owl/runs/5009331342?check_suite_focus=true

[aelred]

Oh you're right! Sorry, I was looking at the wrong thing.

With the new 7.x we get: Invalid or missing build flavor [oss].

Here's the full stack trace:

[2022-01-31 18:18:52.724] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] INFO  elastic-snowowl - Connecting to Elasticsearch cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com', connect timeout: 2000 ms, socket timeout: 120000 ms.
[2022-01-31 18:18:53.030] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] INFO  index.locks - Preparing 'locks' indexes...
[2022-01-31 18:19:53.041] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] ERROR snowowl - Couldn't check the existence of ES index 'locks-lock'.
com.b2international.index.IndexException: Couldn't check the existence of ES index 'locks-lock'.
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:150)
	at com.b2international.index.es.admin.EsIndexAdmin.create(EsIndexAdmin.java:171)
	at com.b2international.snowowl.core.locks.DefaultOperationLockManager.<init>(DefaultOperationLockManager.java:85)
	at com.b2international.snowowl.core.internal.locks.LockPlugin.preRun(LockPlugin.java:43)
	at com.b2international.snowowl.core.setup.Plugins.preRun(Plugins.java:88)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:291)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:272)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:260)
	at com.b2international.snowowl.server.product.SnowOwlServerActivator.start(SnowOwlServerActivator.java:52)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)
	at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
	at org.eclipse.osgi.container.Module.doStart(Module.java:605)
	at org.eclipse.osgi.container.Module.start(Module.java:468)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1845)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1838)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1781)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1743)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1665)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: com.b2international.commons.exceptions.BadRequestException: Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available. Diagnosis: 'The cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' reported an error: 'Invalid or missing build flavor [oss]''
	at com.b2international.index.es.client.EsClientBase.checkAvailable(EsClientBase.java:102)
	at com.b2international.index.es.client.http.IndicesHttpClient.exists(IndicesHttpClient.java:60)
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:148)
	... 26 common frames omitted

[mattecasu]

Hello Mark,
we think it's probably this? elastic/elasticsearch#76091
The thing is that the AWS-provided ES is probably used by many (including us 😅), so it would be best for Snowowl to make sure the ES client adopted can query it (hopefully AWS will soon bump their version..).

As for log4j, one way is to:

• exclude any old log4j group
• include log4j-1.2-api (translates versione 1 to version 2)
• optionally include a bridge to something else, like log4j-to-slf4j

[cmark]

Hi @mattecasu,

Interesting, according to the official Elasticsearch docs, it should be backward compatible with the older 7.x versions, but apparently that's not true.
Elastic has changed Elasticsearch's licensing from Apache license starting from 7.11.0, none of the other Elasticsarch hosting providers will ever going to upgrade their version above 7.10.2 due to that, because the new license forbids hosting like that. AWS also forked the project and created OpenSearch as an alternative, which is interesting and sad at the same time.

Anyway, we are going to debug this and hopefully get a fix for it sometime soon, in the meantime I recommend a rollback to 7.18.0 version which is able to communicate with your ES cluster without an issue and configure the mentioned -Dlog4j2.formatMsgNoLookups=true to mitigate the security vulnerability around log4j.

I hope this helps, if you have any questions please let us know.

Cheers,
Mark

[aelred]

Thanks Mark, we'll apply the configuration for now!

[mattecasu]

Hello again Mark,

did you find a way to make the patched versions compatible with AWS's ElasticSearch? Just asking to understand if you have it on the roadmap.

[cmark]

Hi @mattecasu,

Not yet, we could fix it by altering the Elasticsearch source somehow but that could easily turn into a licensing issue which we would like to avoid. Feel free to open another ticket that tracks the compatibility of Snow Owl with the AWS Elasticsearch service, which only fixes the client-side part of the Log4j2 vulnerability, the AWS Elasticsearch service will still offer Elasticsearch 7.10.2, which is somewhat vulnerable to information leakage attacks (but not to RCE attacks).

I still recommend either switching to Elastic Cloud as Elasticsearch hosting solution or hosting the Elasticsearch service on your own. Please refer to the AWS documentation to see the supported Elasticsearch versions (up to Elasticsearch OSS 7.10).

I hope this helps. Feel free to reach out to us if you have any further questions.

Cheers,
Mark