Coder Social home page Coder Social logo

presto-hadoop-apache2's People

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

mtdata ronald-nyami lipengyu mombergm tomz mandusm billonahill losipiuk teradata arhimondr z-york dharmeshkakadia wubiaoi bluestalker pi9nc mlibing eventbrite 92chabuduo 8899man arovit yingsu00 miguelweezardo ankitdixit praveengarg07 luohao jesboat aweisberg highker jainxrohit caithagoras slijian beinan amitsadaphule isabella232 leeebai ahana-inc imjalpreet lliiun-z rajatrj20 carrey-feng v-jizhang agrawalreetika ayan07 junluo 7c00 chunxutang feloxx plotor bluerocly roneymathew

presto-hadoop-apache2's Issues

Adding support for ppc64le

ENV:
arch=ppc64le
os=RHEL7.6

As mentioned in the issue prestodb/presto#14549, in order to get the tests for presto-hive, presto-geospatial, presto-spark-base, etc. to PASS and to get the presto-server execute successfully with hive catalog, I needed to replace the libhadoop.so, libsnappy.so in hadoop-apache2-2.7.4-7.jar.

So, in order to get that fixed at the source, I need to get the libraries which are compatible with RHEL/CentOS 7.6 added to this repo. I have a query regarding that:

@electrum as a part of this commit, the two libraries were added to this repo (maybe compatible with ubuntuxx.xx, but the don't work for RHEL 7.6). I'd like to request some guidance from you as to what would be the best way to get the libraries compatible with RHEL/CentOS 7.6 added to this repo?

Windows Support: Hadoop fails to load Loaded Libraries

Windows tries to load the native hadoop binaries which do not exist. It tries to load them from a wide range of folder because Windows returns Windows-<Version> from os.name. After changing this to use 'Windows' as the os.name for all Windows versions and the addition of the native libraries to /nativelib/Windows_amd64, HadoopNative correctly loads the libraries from the temp files that are created. Unfortunately, when loadAllCodecs runs it subsequently fails with the following error:

java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:60)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:51)
Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68)
        at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
        at com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:73)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:55)

Changing HadoopNative.requireHadoopNative to create a temporary directory and making loadLibrary copy the embedded files with the original names into this directory allows the libraries to correctly loaded.

mac m1 library not found

I used macbook air(m1) and got an error: library not found: /nativelib/Mac_OS_X-aarch64/libhadoop.dylib
What should i do, thank you！

FileSystem closed exception 

com.facebook.presto.spi.PrestoException: java.io.IOException: Filesystem closed
        at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:159)
        at com.facebook.presto.hive.HivePageSource.getNextPage(HivePageSource.java:204)
        at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:248)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:378)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:534)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:176)
        at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:137)
        ... 10 more
Caused by: java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:822)
        at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672)
        at java.io.FilterInputStream.close(FilterInputStream.java:181)
        at com.facebook.presto.hive.orc.HdfsOrcDataSource.close(HdfsOrcDataSource.java:42)
        at com.facebook.presto.orc.OrcRecordReader.close(OrcRecordReader.java:281)
        at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:173)
        ... 11 more

java.io.IOException: Filesystem closed 

com.facebook.presto.spi.PrestoException: Error reading from hdfs://.... at position 3
	at com.facebook.presto.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:76)
	at com.facebook.presto.orc.AbstractOrcDataSource.readFully(AbstractOrcDataSource.java:105)
	at com.facebook.presto.orc.CachingOrcDataSource.readCacheAt(CachingOrcDataSource.java:78)
	at com.facebook.presto.orc.CachingOrcDataSource.readFully(CachingOrcDataSource.java:96)
	at com.facebook.presto.orc.CachingOrcDataSource.readFully(CachingOrcDataSource.java:85)
	at com.facebook.presto.orc.StripeReader.readStripeFooter(StripeReader.java:365)
	at com.facebook.presto.orc.StripeReader.readStripe(StripeReader.java:109)
	at com.facebook.presto.orc.OrcRecordReader.advanceToNextStripe(OrcRecordReader.java:446)
	at com.facebook.presto.orc.OrcRecordReader.advanceToNextRowGroup(OrcRecordReader.java:399)
	at com.facebook.presto.orc.OrcRecordReader.nextBatch(OrcRecordReader.java:348)
	at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:135)
	at com.facebook.presto.hive.HivePageSource.getNextPage(HivePageSource.java:197)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:265)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:229)
	at com.facebook.presto.operator.Driver.processInternal(Driver.java:337)
	at com.facebook.presto.operator.Driver.lambda$processFor$6(Driver.java:241)
	at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:614)
	at com.facebook.presto.operator.Driver.processFor(Driver.java:235)
	at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
	at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at com.facebook.presto.execution.executor.LegacyPrioritizedSplitRunner.process(LegacyPrioritizedSplitRunner.java:23)
	at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:485)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.RuntimeException: java.io.IOException: Filesystem closed
		at com.google.common.base.Throwables.propagate(Throwables.java:240)
		at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:177)
		at com.facebook.presto.hive.orc.OrcPageSource.closeWithSuppression(OrcPageSource.java:200)
		at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:154)
		... 14 more
	Caused by: java.io.IOException: Filesystem closed
		at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
		at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:710)
		at java.io.FilterInputStream.close(FilterInputStream.java:181)
		at com.facebook.presto.hive.orc.HdfsOrcDataSource.close(HdfsOrcDataSource.java:55)
		at com.facebook.presto.orc.CachingOrcDataSource.close(CachingOrcDataSource.java:125)
		at com.facebook.presto.orc.OrcRecordReader.close(OrcRecordReader.java:318)
		at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:174)
		... 16 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
	at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1419)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1410)
	at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:78)
	at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:107)
	at com.facebook.presto.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:64)
	... 24 more

Hi , guys
I found with patch "File system finalizer #23" , when I start presto server first and do some query , It always happened exception like above . The reason I guess when privateCredentials not equals , the filesystem become PhantomReference and do clean , so i think we should cache filesystem .

Presto Hive plugin is missing libraries needed for Hadoop KMS

When Hadoop KMS is used, Presto Hive plugin calls org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL which fails with the current master.

Hadoop KMS calls use URIBuilder class which is missing in shaded version of hadoop libraries.

I have been able to fix this issue in https://github.com/vlkodev/presto-hadoop-apache2/commit/06b7b29060c40ec22318860696b6ab1677b5f84d

Steps I have done:

  1. Included httpclient library which contains URIBuilder class.
  2. After step 1 jackson dependencies were still missing.
  3. Included (removed exclusion) jackson-core-asl, jackson-mapper-asl

Would you consider pulling my change to include missing libraries?

URIBuilder class missing:

2016-06-20T12:13:24.190Z        ERROR   remote-task-callback-7  com.facebook.presto.execution.StageStateMachine Stage 20160620_121321_00003_nfw2i.1 failed
java.lang.NoClassDefFoundError: com/facebook/presto/hadoop/$internal/org/apache/http/client/utils/URIBuilder
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:433)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:773)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
        at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
        at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
        at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
        at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
        at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
        at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
        at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
        at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
        at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
        at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.facebook.presto.hadoop.$internal.org.apache.http.client.utils.URIBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:102)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Jackson dependencies missing:

java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:216)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:513)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:779)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
        at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
        at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
        at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
        at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
        at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
        at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
        at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
        at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
        at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
        at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.ObjectMapper
        at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:106)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 32 more

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.