prestodb / presto Goto Github PK

The official home of the Presto distributed SQL query engine for big data

License: Apache License 2.0

Java 95.75% Makefile 0.03% Python 0.07% HTML 0.06% JavaScript 1.65% ANTLR 0.05% Shell 0.10% CSS 0.05% Thrift 0.02% PLSQL 0.01% CMake 0.06% Batchfile 0.01% C++ 2.00% Mustache 0.03% NASL 0.02% Roff 0.09% Dockerfile 0.01%

big-data data hadoop hive java lakehouse presto query sql

presto's People

Contributors

Stargazers

Watchers

Forkers

dain erichwang martint hgschmie cberner nileema davidhagar-zz rishidwivedi rvs tglines serdary bigmlcom rocio keneo prasanthj instinktech josephmisiti hipjim paulsm bownux souravzzz quyf lianhuiwang lukehan imtruby joydesigner changguanghua sxhao zcfrank1st olivererwang xymbian jimolucy lgsonic nthf johnwonder hzrandd waoywssy guitarmind wujunshen ryanmmmmm tomdev2008 guoyao lrq1988 schatt jyzhao2006 baeeq dayrow90 kiora1120 missdiog xzjxylophone wanggao vshanyiao mohankreddy daftshady gbtyy linqiang alienfeel akshayms nodepycmp invinciblejha zhenghua00 linw makeclan tatuzero mvc4 xuxiandi getting xiaowenhao111 jxsdzht2010 mrlocker nksurvive lsmushroom robinup sjtufighter vdt yuhulian yubobo dongpf zwqjsj0404 myohannes junpingdu abpin microinfo shivzone frank636 bitisony kahoku martianzhang monolithic wwarren mmallad jjj09090 jeffreywugz aodota thebennos wangperry gijs xiexi galois17 lkunemail

presto's Issues

Add support for Snappy codec

The Snappy codec should work but the native library is not loaded. We need to load it in HadoopNative and make sure it is in the list of configured codecs.

Exception in console if coordinator restarts while query is planning

2013-01-14T20:51:11.677-0800    DEBUG   main    com.facebook.presto.cli.StatusPrinter   error printing status
java.lang.RuntimeException: Server refused connection: http://10.170.206.23:8080/v1/query/11
    at io.airlift.http.client.FullJsonResponseHandler.handleException(FullJsonResponseHandler.java:54) ~[presto-standalone.jar:0.1-SNAPSHOT]
    at io.airlift.http.client.FullJsonResponseHandler.handleException(FullJsonResponseHandler.java:34) ~[presto-standalone.jar:0.1-SNAPSHOT]
    at io.airlift.http.client.ApacheHttpClient.execute(ApacheHttpClient.java:171) ~[presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.server.HttpQueryClient.getQueryInfo(HttpQueryClient.java:112) ~[presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.StatusPrinter.printInitialStatusUpdates(StatusPrinter.java:72) ~[presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.Query.renderQueryOutput(Query.java:60) [presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.Query.renderOutput(Query.java:50) [presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.Console.process(Console.java:111) [presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.Console.runConsole(Console.java:91) [presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.cli.Console.run(Console.java:41) [presto-standalone.jar:0.1-SNAPSHOT]
    at com.facebook.presto.Main.main(Main.java:34) [presto-standalone.jar:0.1-SNAPSHOT]

Failed to select any array element from data of array type using JDBC

I created a table with data of array type

create table x (
reasons array<string>,
...
) row format delimited fields terminated by ',' ESCAPED BY '' COLLECTION ITEMS
TERMINATED BY '\002' stored as textfile;

Loaded data into it and use hive to query, all work fine with queries such as

select reasons[0] from x;

Using JDBC to query the array-typed data by "SELECT reasons FROM hive.default.x", it works fine:

["No_Week","Geo","Lg Mismatch"]

But it fails when using JDBC to query any array element such as "reasons[0]"

stmt.executeQuery("SELECT reasons[0] FROM hive.default.x");

Exception in thread "main" java.sql.SQLException: Query failed (#20131120_222414_00032_89y4z): line 1:39: no viable alternative at character '['
at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1568)
at com.facebook.presto.jdbc.PrestoResultSet.getColumns(PrestoResultSet.java:1519)
at com.facebook.presto.jdbc.PrestoResultSet.(PrestoResultSet.java:78)
at com.facebook.presto.jdbc.PrestoStatement.executeQuery(PrestoStatement.java:48)

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/facebook/presto/cli/Presto : Unsupported major.minor version 51.0

Hi facebook presto team:
I am using presto-0.56.Now I have configed presto server and presto discovery service ,but when I start presto server the console print error informations as below:

[hadoop@master bin]$ ./launcher run
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/facebook/presto/server/PrestoServer : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.facebook.presto.server.PrestoServer. Program will exit.
[hadoop@master bin]$

Our hadoop environment is JDK1.6_45+CDH5 and CHD5-Hive-0.11 and CHD5-HBase-0.95.2,please give me some helps ,I want to know whether this error message is caused by a problem in version.Thank you very much.

Hbase code

The documentation mentions Hbase support and I thought I saw some code in the repo at some point (but not anymore).

Could someone please point me to the Hbase code? Thank you.

Add an info-only HTTP query endpoint

Presto currently combines the ability to fetch query progress info with the fetching of the results. Presto should add another HTTP end point that allows fetching of query progress info, but without returning data and without counting as a heartbeat.

Doesn't work with HA Namenodes (CDH 4.3)

I tried to set it up today and I was greeted with this error when I tried to uses the hive table we have setup:

2013-11-06T15:32:19.381-0800    ERROR   Stage-20131106_233217_00001_rwbfh.1-378 com.facebook.presto.execution.SqlStageExecution Error while starting stage 20131
106_233217_00001_rwbfh.1
java.lang.RuntimeException: java.io.IOException: java.lang.IllegalArgumentException: java.net.UnknownHostException: sf-cluster
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-15.0.jar:na]
        at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:433) ~[na:na]
        at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:392) ~[na:na]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-15.0.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-15.0.jar:na]
        at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:463) [presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlStageExecution.access$300(SqlStageExecution.java:80) [presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlStageExecution$5.run(SqlStageExecution.java:435) [presto-main-0.52.jar:0.52]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_06]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_06]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_06]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_06]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_06]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_06]
Caused by: java.io.IOException: java.lang.IllegalArgumentException: java.net.UnknownHostException: sf-cluster
        at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:71) ~[na:na]
        at com.facebook.presto.hive.ForwardingPath.getFileSystem(ForwardingPath.java:47) ~[na:na]
        at com.facebook.presto.hive.FileSystemWrapper$1.getFileSystem(FileSystemWrapper.java:78) ~[na:na]
        at com.facebook.presto.hive.HiveSplitIterable.loadPartitionSplits(HiveSplitIterable.java:181) ~[na:na]
        at com.facebook.presto.hive.HiveSplitIterable.access$100(HiveSplitIterable.java:73) ~[na:na]
        at com.facebook.presto.hive.HiveSplitIterable$2.call(HiveSplitIterable.java:154) ~[na:na]
        at com.facebook.presto.hive.HiveSplitIterable$2.call(HiveSplitIterable.java:149) ~[na:na]
        ... 5 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: sf-cluster
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414) ~[na:na]
        at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164) ~[na:na]
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129) ~[na:na]
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:448) ~[na:na]
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:410) ~[na:na]
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:128) ~[na:na]
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308) ~[na:na]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87) ~[na:na]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342) ~[na:na]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324) ~[na:na]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351) ~[na:na]
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194) ~[na:na]
        at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:92) ~[na:na]
        at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:87) ~[na:na]
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721) ~[guava-15.0.jar:na]
        at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:68) ~[na:na]
        ... 11 common frames omitted
Caused by: java.net.UnknownHostException: sf-cluster
        ... 33 common frames omitted

I tried to add the hadoop config files to the classpath but made no difference from the look at the stack trace its trying to create a NonHAProxy to connect. This is on a CDH4.3 cluster with the Cloudera recommended way of setting up HA Namenodes.

Optimizer can remove all functions from aggregation operator

The following query causes an ArrayIndexOutOfBoundsException:

select 1 from (select 1 x) join (select 1 x, count(*)) using (x)

java.lang.ArrayIndexOutOfBoundsException: 0
    at com.facebook.presto.operator.Page.<init>(Page.java:31)
    at com.facebook.presto.operator.AggregationOperator.getOutput(AggregationOperator.java:174)
    at com.facebook.presto.operator.Driver.process(Driver.java:214)
    at com.facebook.presto.operator.Driver.processFor(Driver.java:243)
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:603)
    at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:424)
    at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:544)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Schema & Catalog name through JDBC Connection URL

JDBC Connection URL format to provide Schema & Catalog name through URL. As per my understanding currently if Catalog name & Schema name is not present in the sql 'default' schema & catalog name is used.

Exception when quitting result pager

java.lang.InterruptedException: null
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1038) ~[na:1.7.0_06]
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) ~[na:1.7.0_06]
    at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) ~[na:1.7.0_06]
    at com.facebook.presto.execution.PageBuffer.addPage(PageBuffer.java:199) ~[main.jar:0.1-SNAPSHOT]
    at com.facebook.presto.server.HttpQuery$PageResponseHandler.handle(HttpQuery.java:158) [main.jar:0.1-SNAPSHOT]
    at com.facebook.presto.server.HttpQuery$PageResponseHandler.handle(HttpQuery.java:108) [main.jar:0.1-SNAPSHOT]
    at io.airlift.http.client.ApacheHttpClient$1.handleResponse(ApacheHttpClient.java:134) [http-client-experimental-0.68-SNAPSHOT.jar:na]
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070) [httpclient-4.2.2.jar:4.2.2]
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044) [httpclient-4.2.2.jar:4.2.2]
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1035) [httpclient-4.2.2.jar:4.2.2]
    at io.airlift.http.client.ApacheHttpClient.execute(ApacheHttpClient.java:122) [http-client-experimental-0.68-SNAPSHOT.jar:na]
    at io.airlift.http.client.AsyncHttpClient$HttpExecution.call(AsyncHttpClient.java:113) [http-client-experimental-0.68-SNAPSHOT.jar:na]
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_06]
    at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_06]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_06]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_06]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_06]

Fix handling of Hive bucketed tables

The bucketing code assumes that tables declared as being bucketed actually have the correct file layout in HDFS. HiveSplitIterable currently walks the directly structure as normal when bucketing is enabled and skips any files that don't match the bucket file. It should do this instead:

Verify that the directory has exactly the right number of files
Verify that the directory has no subdirectories
If all checks pass, return splits for the bucket file
Otherwise, ignore bucketing and return splits for all files as normal

The bucket file is determined by sorting the filenames lexicographically and returning the Nth file.

when executing presto-client-cli command for query ,it show Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap space

Hi friends:
Now I can't open the page https://groups.google.com/forum/#!forum/presto-users ,so show my question here.
I have started hiveserver and started presto-server on a machine with commands below:
hive --service hiveserver -p 9083
./launcher run
When I use the presto-client-cli command ./presto --server localhost:9083 --catalog hive --schema default ,the console shows presto:default>,input the command as show tables the console prints Error running command: java.nio.channels.ClosedChannelException,
and the hiveserver console print as below:
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

my configuration file below:
node.properties
node.environment=production
node.id=cc4a1bbf-5b98-4935-9fde-2cf1c98e8774
node.data-dir=/home/hadoop/cloudera-5.0.0/presto-0.56/presto/data

config.properties
coordinator=true
datasources=jmx
http-server.http.port=8080
presto-metastore.db.type=h2
presto-metastore.db.filename=/home/hadoop/cloudera-5.0.0/presto-0.56/presto/db/MetaStore
task.max-memory=1GB
discovery-server.enabled=true
discovery.uri=http://slave4:8080

jvm.config
-server
-Xmx16G
-XX:+UseConcMarkSweepGC
-XX:+ExplicitGCInvokesConcurrent
-XX:+CMSClassUnloadingEnabled
-XX:+AggressiveOpts
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
-XX:PermSize=512M
-XX:MaxPermSize=512M
-XX:ReservedCodeCacheSize=512M
-Xbootclasspath/p:/home/hadoop/cloudera-5.0.0/presto-0.56/presto-server-0.56/lib/floatingdecimal-0.1.jar

log.properties
com.facebook.presto=DEBUG

catalog/hive.properties
connector.name=hive-cdh4
hive.metastore.uri=thrift://master:9083

HADOOP ENVIRONMENT IS CDH5+CDH5-HIVE-0.11+PRESTO-0.56

please help me ,thanks.

Stuck threads in HiveSplitIterable

When running the verifier I noticed two queries stuck in HiveSplitIterable:

at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000065a9249a8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at com.facebook.presto.hive.HiveSplitIterable.loadPartitionSplits(HiveSplitIterable.java:178)
at com.facebook.presto.hive.HiveSplitIterable.access$100(HiveSplitIterable.java:77)
at com.facebook.presto.hive.HiveSplitIterable$2.run(HiveSplitIterable.java:158)
at com.facebook.presto.hive.util.BoundedExecutor.executeOrMerge(BoundedExecutor.java:82)
at com.facebook.presto.hive.util.BoundedExecutor.access$000(BoundedExecutor.java:41)
at com.facebook.presto.hive.util.BoundedExecutor$1.run(BoundedExecutor.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

The only reason I can see for this would be markerQueue.finish() throwing, but I don't understand how this can happen.

Presto - Inclusion of varchar(64), etc. in Hive table causes error

I have an HDFS/Hive instance set up using MySQL as a metastore.

Running the following version of Hadoop: hadoop-1.2.1
Running the following version of Hive: hive-0.12.0
Running the following version of Presto Server: presto-server-0.54
Running the following version of Discovery Server: discovery-server-1.16
Using the following Presto CLI: presto-cli-0.54-executable.jar

I have a Hive Table created as follows:

[localhost:10001] hive> describe stats;
A int                     None
B smallint             None
C tinyint               None
D tinyint                None
E varchar(64)         None
F smallint                None
G int                     None
H bigint                  None
I int                     None
J bigint                  None
K int                     None
L bigint                  None
hour                    int                     None

# Partition Information
# col_name              data_type               comment

hour                    int                     None

The table appears to work well in Hive. It is correctly populated with data. However, when I attempt to query the table via Presto, I see errors:

presto:default> describe stats;

Query 20131205_215319_00040_y5j69, FAILED, 1 node
Splits: 2 total, 0 done (0.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20131205_215319_00040_y5j69 failed: MetaException(message:java.lang.IllegalArgumentException Error: type expected at the position 29 of 'int:smallint:tinyint:tinyint:varchar(64):smallint:int:bigint:int:bigint:int:bigint' but 'varchar' is found.)

presto:default> select count(1) from stats;
Query 20131205_215340_00041_y5j69 failed: MetaException(message:java.lang.IllegalArgumentException Error: type expected at the position 29 of 'int:smallint:tinyint:tinyint:varchar(64):smallint:int:bigint:int:bigint:int:bigint' but 'varchar' is found.)

There is not a lot of debug information:

2013-12-05T21:54:59.748+0000    DEBUG   query-scheduler-33      com.facebook.presto.execution.QueryStateMachine Query 20131205_215459_00042_y5j69 is PLANNING
2013-12-05T21:54:59.749+0000    DEBUG   query-scheduler-34      com.facebook.presto.execution.QueryStateMachine Query 20131205_215459_00042_y5j69 is FAILED

Presto appears to work correctly with tables that do not contain 'varchar' types:

presto:default> select count(1) from other_stats;

 _col0
-------
  2916
(1 row)

Query 20131205_224304_00043_y5j69, FINISHED, 1 node
Splits: 730 total, 730 done (100.00%)
0:05 [2.92K rows, 102KB] [563 rows/s, 19.7KB/s]

Does Presto not support 'varchar' field types?

Query fails when hive table has a boolean field

Query 0 failed:
java.lang.IllegalArgumentException: primitive not supported: BOOLEAN
  at com.facebook.presto.hive.HiveClient.getPrimitiveType(HiveClient.java:657)
    at com.facebook.presto.hive.HiveClient.getSchemaFields(HiveClient.java:636)
    at com.facebook.presto.hive.HiveClient.getTableSchema(HiveClient.java:155)
    at com.facebook.presto.metadata.ImportMetadata$1.call(ImportMetadata.java:153)
    at com.facebook.presto.metadata.ImportMetadata$1.call(ImportMetadata.java:148)
    at com.facebook.presto.util.RetryDriver.run(RetryDriver.java:83)
    at com.facebook.presto.util.RetryDriver.runUnchecked(RetryDriver.java:110)
    at com.facebook.presto.util.RetryDriver.runUnchecked(RetryDriver.java:104)
    at com.facebook.presto.metadata.ImportMetadata.getTableSchema(ImportMetadata.java:147)
    at com.facebook.presto.metadata.ImportMetadata.getTable(ImportMetadata.java:44)
    at com.facebook.presto.metadata.MetadataManager.getTable(MetadataManager.java:57)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitTable(Analyzer.java:514)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitTable(Analyzer.java:488)
    at com.facebook.presto.sql.tree.Table.accept(Table.java:23)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:9)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitAliasedRelation(Analyzer.java:541)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitAliasedRelation(Analyzer.java:488)
    at com.facebook.presto.sql.tree.AliasedRelation.accept(AliasedRelation.java:44)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:9)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitJoin(Analyzer.java:569)
    at com.facebook.presto.sql.analyzer.Analyzer$RelationAnalyzer.visitJoin(Analyzer.java:488)
    at com.facebook.presto.sql.tree.Join.accept(Join.java:53)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:9)
    at com.facebook.presto.sql.analyzer.Analyzer$StatementAnalyzer.visitQuery(Analyzer.java:134)
    at com.facebook.presto.sql.analyzer.Analyzer$StatementAnalyzer.visitQuery(Analyzer.java:100)
    at com.facebook.presto.sql.tree.Query.accept(Query.java:81)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:9)
    at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:97)
    at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:91)
    at com.facebook.presto.execution.SqlQueryExecution.analyseQuery(SqlQueryExecution.java:198)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:159)
    at com.facebook.presto.execution.SqlQueryManager$QueryStarter.run(SqlQueryManager.java:299)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Pass schema properties to input format

The full schema properties should be added to the JobConf in Hive RecordSet before calling inputFormat.getRecordReader so the input format can access all partition metadata

Client library for Ruby and Python

Background

I created Presto client library for Ruby and Python.
They're native client library that use HTTP connection to send requests to Presto.

Ruby: https://github.com/treasure-data/presto-client-ruby
Python: https://gist.github.com/frsyuki/8296022

Problem

Do you think version management (& release cycle) should be done at this repository (facebook/presto.git)?
In other words, do you think should the code in this facebook/presto.git repository?

There is an advantage to have only one repository to manage both server (Presto) and client libraries (presto-client-{ruby,python}). It becomes easy to synchronize changes to a client library when REST API is changed.

However, in my opinion, it's too much to include Ruby or Python libraries. Because client libraries often need to solve language-specific problems such as type conversion. Or creators can relatively easily add innovation in the design of the client library if the repository & ownership is separated to another community.
It's ok to include Java client library in this facebook/presto.git because it can share a lot of code with the server implementation.

other comments

I packaged ruby client library as a RubyGem.
Implementation is still beta quality. Little document. I can work on improving the code but I welcome if someone wants to improve the code.

I'm not a pythonista. I don't know how to package library for python. I don't have intention to improve the quality any more. I'd like someone to take over the project.

JSON_ARRAY_GET throws NPE for some JSON

presto:default> select JSON_ARRAY_GET('["a", null,  "c"]', 1);
Query 20131220_190434_00004_ff4ww failed: string is null
java.lang.NullPointerException: string is null
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:209)
    at io.airlift.slice.Slices.copiedBuffer(Slices.java:90)
    at io.airlift.slice.Slices.utf8Slice(Slices.java:98)
    at com.facebook.presto.operator.scalar.JsonFunctions.jsonArrayGet(JsonFunctions.java:25

presto:default> select JSON_ARRAY_GET('["a", ["b"],  "c"]', 1);
Query 20131220_190823_00010_ff4ww failed: string is null
java.lang.NullPointerException: string is null
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:209)
    at io.airlift.slice.Slices.copiedBuffer(Slices.java:90)
    at io.airlift.slice.Slices.utf8Slice(Slices.java:98)
    at com.facebook.presto.operator.scalar.JsonFunctions.jsonArrayGet(JsonFunctions.java:252)

NPE when handling CTRL-C

Exception in thread "SIGINT handler" java.lang.NullPointerException
    at com.facebook.presto.server.HttpQueryClient.cancelLeafStage(HttpQueryClient.java:184)
    at com.facebook.presto.server.HttpQueryClient.cancelLeafStage(HttpQueryClient.java:178)
    at com.facebook.presto.cli.Query.cancelLeafStage(Query.java:102)
    at com.facebook.presto.cli.Query$1.handle(Query.java:47)

Column types missing in EXPLAIN (TYPE DISTRIBUTED)

presto> explain (type distributed) select 'test' a;
                                   Query Plan                                   
--------------------------------------------------------------------------------
 - Output[a]                                                                    
         a := expr                                                              
     - Exchange[[0]] => [expr:varchar]                                          
         - Sink[6] => [expr:varchar]                                            
             - Project => [expr:varchar]                                        
                     expr := 'test'                                             
                 - TableScan[dual:di, original constraint=true] => [dummy:null] 
                         dummy := dual:dummy

Compare to EXPLAIN (TYPE LOGICAL):

presto:di> explain (type logical) select 'test' a;
                                Query Plan                                 
---------------------------------------------------------------------------
 - Output[a]                                                               
         a := expr                                                         
     - Project => [expr:varchar]                                           
             expr := 'test'                                                
         - TableScan[dual:di, original constraint=true] => [dummy:varchar] 
                 dummy := dual:dummy

Add Snappy native codec

When attempting to use Snappy compressed data, we get the following exception:

2013-11-11T08:26:33.828+0000    ERROR   SplitRunner-10-62       com.facebook.presto.execution.TaskExecutor      Error processing Split 20131111_082510_00012_b63fn.1.0 0 0.00ns
java.lang.RuntimeException: Error opening Hive split hdfs://.../000189_0 (offset=0, length=4903730) using org.apache.hadoop.hive.ql.io.RCFileInputFormat: native snappy library not available: SnappyCompressor has not been loaded.
        at com.facebook.presto.hive.HiveRecordSet.createRecordReader(HiveRecordSet.java:190) ~[na:na]
        at com.facebook.presto.hive.HiveRecordSet.cursor(HiveRecordSet.java:111) ~[na:na]
        at com.facebook.presto.spi.classloader.ClassLoaderSafeRecordSet.cursor(ClassLoaderSafeRecordSet.java:46) ~[presto-spi-0.52.jar:0.52]
        at com.facebook.presto.operator.RecordProjectOperator.<init>(RecordProjectOperator.java:45) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.split.RecordSetDataStreamProvider.createNewDataStream(RecordSetDataStreamProvider.java:46) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.split.DataStreamManager.createNewDataStream(DataStreamManager.java:61) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.operator.AbstractScanFilterAndProjectOperator.addSplit(AbstractScanFilterAndProjectOperator.java:95) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.operator.Driver.addSplit(Driver.java:166) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.operator.Driver.updateSource(Driver.java:142) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlTaskExecution.createDriver(SqlTaskExecution.java:460) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlTaskExecution.access$400(SqlTaskExecution.java:73) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlTaskExecution$2.apply(SqlTaskExecution.java:333) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlTaskExecution$2.apply(SqlTaskExecution.java:329) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.initialize(SqlTaskExecution.java:591) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.initializeIfNecessary(TaskExecutor.java:395) ~[presto-main-0.52.jar:0.52]
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:543) ~[presto-main-0.52.jar:0.52]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
        at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68) ~[na:na]
        at org.apache.hadoop.io.compress.SnappyCodec.createDecompressor(SnappyCodec.java:196) ~[na:na]
        at org.apache.hadoop.hive.ql.io.CodecPool.getDecompressor(CodecPool.java:35) ~[na:na]
        at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1314) ~[na:na]
        at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1154) ~[na:na]
        at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1133) ~[na:na]
        at org.apache.hadoop.hive.ql.io.RCFileRecordReader.<init>(RCFileRecordReader.java:102) ~[na:na]
        at org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57) ~[na:na]
        at com.facebook.presto.hive.HiveRecordSet$1.call(HiveRecordSet.java:185) ~[na:na]
        at com.facebook.presto.hive.HiveRecordSet$1.call(HiveRecordSet.java:180) ~[na:na]
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:85) ~[na:na]
        at com.facebook.presto.hive.HiveRecordSet.createRecordReader(HiveRecordSet.java:179) ~[na:na]
        ... 18 common frames omitted

Add implementation for more window function (lag() and lead())

http://prestodb.io/docs/current/functions/window.html

LAG and LEAD function are missing

Syntax should be this :
LAG (value_expression [,offset] [,default]) OVER ([partition_clause] order_by_clause)
LEAD (value_expression [,offset] [,default]) OVER ([partition_clause] order_by_clause)

More info here
http://waynepmckinney.com/blog/2013/04/29/lead-lag-window-functions/

https://issues.apache.org/jira/browse/HIVE-896

InvalidProtocolBufferException CDH5 Presto-0.57

Hi facebook team:
Now I execute one sql querying of presto from datasource hive on cdh5,but the console prints exception informations as below:
java.lang.RuntimeException: java.io.IOException: Failed on local exception: com.facebook.presto.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "slave4.sunrise.com/192.168.0.9"; destination host is: "slave4":8020;
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-15.0.jar:na]
at com.facebook.presto.hive.HiveSplitSourceProvider$HiveSplitSource.getNextBatch(HiveSplitSourceProvider.java:481) ~[na:na]
at com.facebook.presto.execution.SqlStageExecution.scheduleSourcePartitionedNodes(SqlStageExecution.java:637) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:556) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution.access$300(SqlStageExecution.java:95) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution$5.run(SqlStageExecution.java:528) [presto-main-0.57.jar:0.57]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: Failed on local exception: com.facebook.presto.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "slave4.sunrise.com/192.168.0.9"; destination host is: "slave4":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1229) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at com.sun.proxy.$Proxy157.getListing(Unknown Source) ~[na:na]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:441) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at com.sun.proxy.$Proxy158.getListing(Unknown Source) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1526) ~[na:na]
at org.apache.hadoop.hdfs.DistributedFileSystem$1.(DistributedFileSystem.java:466) ~[na:na]
at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:457) ~[na:na]
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1760) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at com.facebook.presto.hadoop.HadoopFileSystem.listLocatedStatus(HadoopFileSystem.java:30) ~[na:na]
at com.facebook.presto.hive.util.AsyncRecursiveWalker.doWalk(AsyncRecursiveWalker.java:70) ~[na:na]
at com.facebook.presto.hive.util.AsyncRecursiveWalker.access$000(AsyncRecursiveWalker.java:31) ~[na:na]
at com.facebook.presto.hive.util.AsyncRecursiveWalker$1.run(AsyncRecursiveWalker.java:58) ~[na:na]
at com.facebook.presto.hive.util.SuspendingExecutor$1.run(SuspendingExecutor.java:67) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.executeOrMerge(BoundedExecutor.java:82) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.access$000(BoundedExecutor.java:41) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor$1.run(BoundedExecutor.java:53) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.executeOrMerge(BoundedExecutor.java:82) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.access$000(BoundedExecutor.java:41) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor$1.run(BoundedExecutor.java:53) ~[na:na]
... 3 common frames omitted
com.facebook.presto.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status
at com.facebook.presto.hadoop.shaded.com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81) ~[na:na]
at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.buildParsed(RpcPayloadHeaderProtos.java:1094) ~[na:na]
at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.access$1300(RpcPayloadHeaderProtos.java:1028) ~[na:na]
at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:986) ~[na:na]
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:938) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836) ~[hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:na]
2014-01-21T10:17:06.952+0800 INFO query-scheduler-1 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20140121_021704_00000_vf3qr :: elapsed 1909.00ms :: planning 499.83ms :: scheduling 1409.00ms :: running 0.00ms :: finishing 1409.00ms :: begin 2014-01-21T10:17:04.983+08:00 :: end 2014-01-21T10:17:06.892+08:00
2014-01-21T10:17:06.994+0800 DEBUG task-notification-0 com.facebook.presto.execution.TaskStateMachine Task 20140121_021704_00000_vf3qr.0.0 is CANCELED
2014-01-21T10:32:07.093+0800 DEBUG query-management-1 com.facebook.presto.execution.SqlQueryManager Remove query 20140121_021704_00000_vf3qr
2014-01-21T10:32:07.281+0800 DEBUG query-purger-0 com.facebook.presto.server.StatementResource Removed expired query 20140121_021704_00000_vf3qr

I want to know whether it is caused by incompatible versions of protobuf.
Thanks.

Duration serialization relies on locale-dependent float formatting

Deserialization is not local-dependent and thus fails if the float contains a "," instead of a ".".

Several com.facebook.presto.operator.Test*Stats are failing due to this reason.

Null pointer exception in SqlStageExecution

2014-01-22T17:18:24.135-0800 DEBUG Stage-20140123_011823_00017_xrsjj.1-402 com.facebook.presto.execution.SqlStageExecution Error while starting stage in done query 20140123_011823_00017_xrsjj.1
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333) ~[na:1.7.0_45]
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988) ~[na:1.7.0_45]
at com.facebook.presto.execution.SqlStageExecution.scheduleSourcePartitionedNodes(SqlStageExecution.java:646) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:556) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution.access$300(SqlStageExecution.java:95) [presto-main-0.57.jar:0.57]
at com.facebook.presto.execution.SqlStageExecution$5.run(SqlStageExecution.java:528) [presto-main-0.57.jar:0.57]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Need to handle missing hdfs blocks

java.lang.RuntimeException: org.apache.hadoop.fs.BlockMissingException: Could not obtain block: blk_1667727741683600668_5887206098 file=/user/facebook/warehouse/time_spent_by_page_type/ds=2012-10-03/000000_1
    at com.facebook.presto.hive.shaded.com.google.common.base.Throwables.propagate(Throwables.java:155)
    at com.facebook.presto.hive.HiveRecordCursor.advanceNextPosition(HiveRecordCursor.java:115)
    at com.facebook.presto.ingest.ImportPartition$ImportRecordCursor.advanceNextPosition(ImportPartition.java:59)
    at com.facebook.presto.ingest.RecordProjectOperator$RecordProjectionOperator.computeNext(RecordProjectOperator.java:99)
    at com.facebook.presto.operator.AbstractPageIterator.tryToComputeNext(AbstractPageIterator.java:137)
    at com.facebook.presto.operator.AbstractPageIterator.hasNext(AbstractPageIterator.java:130)
    at com.facebook.presto.operator.AggregationOperator.iterator(AggregationOperator.java:78)
    at com.facebook.presto.operator.FilterAndProjectOperator.iterator(FilterAndProjectOperator.java:57)
    at com.facebook.presto.execution.SqlTaskExecution$SplitWorker.call(SqlTaskExecution.java:275)
    at com.facebook.presto.execution.SqlTaskExecution$SplitWorker.call(SqlTaskExecution.java:227)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at com.facebook.presto.concurrent.FairBatchExecutor.trigger(FairBatchExecutor.java:138)
    at com.facebook.presto.concurrent.FairBatchExecutor.access$100(FairBatchExecutor.java:38)
    at com.facebook.presto.concurrent.FairBatchExecutor$2.run(FairBatchExecutor.java:102)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.hadoop.fs.BlockMissingException: Could not obtain block: blk_1667727741683600668_5887206098 file=/user/facebook/warehouse/time_spent_by_page_type/ds=2012-10-03/000000_1
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:3447)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:3167)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:3391)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:3269)
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at org.apache.hadoop.hive.ql.io.RCFile$Reader.readRecordLength(RCFile.java:1389)
    at org.apache.hadoop.hive.ql.io.RCFile$Reader.nextKeyBuffer(RCFile.java:1428)
    at org.apache.hadoop.hive.ql.io.RCFile$Reader.next(RCFile.java:1592)
    at org.apache.hadoop.hive.ql.io.RCFileRecordReader.next(RCFileRecordReader.java:98)
    at org.apache.hadoop.hive.ql.io.RCFileRecordReader.next(RCFileRecordReader.java:85)
    at org.apache.hadoop.hive.ql.io.RCFileRecordReader.next(RCFileRecordReader.java:39)
    at com.facebook.presto.hive.HiveRecordCursor.advanceNextPosition(HiveRecordCursor.java:106)
    ... 16 more

Add flag to force local scheduling of hive splits

When processing hive splits, Presto will sometimes choose to schedule the processing on a node that is not collocated with the data. There should be a flag to disable this behavior.

Presto - Issue communicating with Hadoop/HIVE

Hello.

I have an HDFS/Hive instance set up using MySQL as a metastore.

Running the following version of Hadoop: hadoop-2.2.0
Running the following version of Hive: hive-0.12.0
Running the following version of Presto Server: presto-server-0.54
Running the following version of Discovery Server: discovery-server-1.16
Using the following Presto CLI: presto-cli-0.54-executable.jar

The Hive instance works well when browsing directly or via Thrift:

[localhost:10001] hive> show tables;
test
[localhost:10001] hive> describe test;
foo int None
bar string None
[localhost:10001] hive> select * from test limit 1;
1 First String

I added a Presto server. I am able to connect and browse the metastore:
presto:default> show tables;

Table

test
(1 row)

Query 20131204_201247_00006_mdebb, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:00 [1 rows, 21B] [2 rows/s, 61B/s]

Query 20131204_201250_00007_mdebb, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:00 [2 rows, 129B] [4 rows/s, 320B/s]

However, when I attempt to select data from the table I get the following error:

Client Output:
presto:default> select * from test;

Query 20131204_201334_00009_mdebb, FAILED, 1 node
Splits: 1 total, 0 done (0.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20131204_201334_00009_mdebb failed: java.io.IOException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4

My guess is that Presto is using a jar from an older version of Hadoop. Is there a way to correct this? I know HBASE requests that you replace their common hadoop jar with the jar in use by the HDFS instance.

Server Stack Dump:
java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-15.0.jar:na]
at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:433) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:392) ~[na:na]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-15.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-15.0.jar:na]
at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:463) [presto-main-0.54.jar:0.54]
at com.facebook.presto.execution.SqlStageExecution.access$300(SqlStageExecution.java:80) [presto-main-0.54.jar:0.54]
at com.facebook.presto.execution.SqlStageExecution$5.run(SqlStageExecution.java:435) [presto-main-0.54.jar:0.54]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:71) ~[na:na]
at com.facebook.presto.hive.ForwardingPath.getFileSystem(ForwardingPath.java:47) ~[na:na]
at com.facebook.presto.hive.FileSystemWrapper$1.getFileSystem(FileSystemWrapper.java:78) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable.loadPartitionSplits(HiveSplitIterable.java:181) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable.access$100(HiveSplitIterable.java:73) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable$2.call(HiveSplitIterable.java:154) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable$2.call(HiveSplitIterable.java:149) ~[na:na]
... 4 common frames omitted
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113) ~[na:na]
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) ~[na:na]
at com.sun.proxy.$Proxy150.getProtocolVersion(Unknown Source) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) ~[na:na]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) ~[na:na]
at com.sun.proxy.$Proxy150.getProtocolVersion(Unknown Source) ~[na:na]
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:281) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:245) ~[na:na]
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100) ~[na:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446) ~[na:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67) ~[na:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464) ~[na:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263) ~[na:na]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) ~[na:na]
at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:92) ~[na:na]
at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:87) ~[na:na]
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721) ~[guava-15.0.jar:na]
at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:68) ~[na:na]
... 10 common frames omitted

Presto With CDH5 HA

Hi,facebook team:
When I executed sql querying with presto-cli ,the console shown below:
Query 20140120_071822_00005_sihwm failed: java.net.UnknownHostException: sunrise-cluster
The sunrise-cluster is dfs.nameservices of CDH5 HA
Please help me.
Thank you.

Whether the presto has supported to SQL Querying on hbase currently?

Hi Presto Team:
I would like to use Presto for SQL Querying on a table that has been exist in my hbase.So far,I find Presto that has supported to query data from datasources as Hive/HDFS with SQL,but i am not sure whether it has supported to query data from a exist table of hbase as well as SQL Querying. Please give me some helps,Thank you very much.I will wait your replies online.

Presto JDK setting

So far, Presto looks in /usr/bin/java for JDK runtime (CentOS 6.3). Is there any way to change it?
CDH 4.3 use JDK specified in /usr/java/default (CentOS).

Support of Custom or Third Party SerDE

I have some of the previously created Hive External Tables which are created with
ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' .
Now while I was using Hive to query on those tables I placed the CSVSerde library inside Hive lib and as Hive uses Map Reduce Job to query tables I also had to put jar inside Hadoop.
Currently I want to query those table with Presto. But it giving me a :
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe com.bizo.hive.serde.csv.CSVSerde does not exist) exception.
I already tried by putting the CSVSerde.jar & HiveSerDe.jar inside the presto server lib. But still getting the same error. Thank you in advance.

Add support for ORC

Trying to read an ORC table fails with an exception. At a minimum, we probably need to upgrade to Hive 0.12.

See https://groups.google.com/d/msg/presto-users/iKmsQuwuFMs/NJVukDHZFQAJ

publishing released artifacts to maven central repository

would be nice to have

how can I config the discovery to build a cluster with 4 hosts?

I have 4 server
master need to run presto-server/bin/launcher or discovery/bin/launcher or both?
node1-node3 what launcher should to run.

Drop table in Hive is not reflecting in Presto CLI in real time

Hi,
Drop table in Hive is not refleting in Presto CLI. When I create one table in Hive it is reflected in Presto CLI within few sec. but when I drop a table in Hive it was not refleted in Presto after several min. Then I "quit" from that instance and reconnect it. Then it is reflected there that the table is dropped in Hive.

Extend json path syntax to support field names containing reserved characters

There's currently no way to extract values for fields in json structures whose names contain any of the special json path characters ($, [, ], ., etc.).

The syntax for json path expressions (for json_extract and friends) should be extended. One possible option is to allow bracket notation for field references (see http://goessner.net/articles/JsonPath/).

For example, to wanted to get the value for key "1.0" from the following structure:

{ "1.0": "apple", "2.0": "banana" }

The corresponding json path expression would be $["1.0"]. Note that the expression uses " instead of ' to make it fit more naturally within SQL strings.

Add support for implicit cross joins

presto:default> select p.foo, i.bar from pokes p, invites i;

Query 20131115_215838_00012_89y4z failed: Cross joins not yet supported

Any plan to support this important SQL feature?

Thanks

IN predicates with value lists don't match between coerced types

presto> select 1.0 in (1, 2, 3);
 _col0
-------
 false
(1 row)

This is incorrect.

UDF Support

Here're my thought about supporting UDF on Presto and I want to implement and send a pull request if the following approach is acceptable.

Presto UDF

The core logic for the Presto functions is at FunctionRegistry.java which defineds window, aggregate and scalar types of functions.

We might want to add functions without changing Presto code, then we can implement & use experimental functions easily. But Presto doesn't have a room for UDF, UDAF and UDTF yet.

So here're an idea of how Presto could support UDF and UDAF in a othogonal way. We might not consider UDTF at this time to make thinghs simple but it would not be much different. Also supporting existing Hive UDF is out of scope of this topic and should be an issue of the HiveConnector.

Presto already has well-defined plugin arhitecture. And sets of UDF and UDAF functions could be thought as a plugin having a FunctionFactory.class services
- So UD(A)F functions can be easily a part of existing connectors or we can build a separate functions-only plugin.
FunctionFactory should be able to return List<FunctionInfoHandle>

+++ b/presto-spi/src/main/java/com/facebook/presto/spi/FunctionInfoHandle.java
@@ -0,0 +1,9 @@
+package com.facebook.presto.spi;
+
]+
+public interface FunctionInfoHandle
+{
+    // TODO: Define
+}

+++ b/presto-spi/src/main/java/com/facebook/presto/spi/FunctionFactory.java
@@ -0,0 +1,9 @@
+package com.facebook.presto.spi;
+
+import java.util.List;
+
+public interface FunctionFactory
+{
+    List<FunctionInfoHandle> listFunctions();
+}

diff --git a/presto-server/src/main/java/com/facebook/presto/server/PluginManager.java b/presto-serv
index f2bc881..cfefef8 100644
--- a/presto-server/src/main/java/com/facebook/presto/server/PluginManager.java
+++ b/presto-server/src/main/java/com/facebook/presto/server/PluginManager.java
@@ -153,6 +153,10 @@ public class PluginManager
             for (SystemTable systemTable : plugin.getServices(SystemTable.class)) {
                 systemTablesManager.addTable(systemTable);
             }
+
+            for (FunctionFactory functionFactory : plugin.getServices(FunctionFactory.class)) {
+                metadataManager.addCustomFunctions(functionFactory);
+            }
         }
     }

MetadataManager pass the returned list of FunctionInfoHandle to FunctionRegistry as it can merge/append the list
FunctionRegistry should have an ability to build FunctionInfo list from FunctionInfoHandle by using FunctionRegistry.FunctionListBuilder. The constructor of FunctionRegistry has very good example of how to build the list.

diff --git a/presto-main/src/main/java/com/facebook/presto/metadata/MetadataManager.java b/presto-ma
index 6287c99..ed91cf0 100644
--- a/presto-main/src/main/java/com/facebook/presto/metadata/MetadataManager.java
+++ b/presto-main/src/main/java/com/facebook/presto/metadata/MetadataManager.java
@@ -75,6 +75,10 @@ public class MetadataManager
         internalSchemas.add(new ConnectorMetadataEntry(connectorId, connectorMetadata));
     }

+    public void addCusttomFunctions(FunctionFactory functionFactory) {
+        functions.addCustomFunctions(functionFactory.listFunctions());
+    }
+
     @Override
     public FunctionInfo getFunction(QualifiedName name, List<Type> parameterTypes)
     {

diff --git a/presto-main/src/main/java/com/facebook/presto/metadata/FunctionRegistry.java b/presto-m
index 3ea97bd..07770a9 100644
--- a/presto-main/src/main/java/com/facebook/presto/metadata/FunctionRegistry.java
+++ b/presto-main/src/main/java/com/facebook/presto/metadata/FunctionRegistry.java
@@ -230,6 +230,10 @@ public class FunctionRegistry
         return functionsByHandle.get(handle);
     }

+    public void addCustomFunctions(List<FunctionInfoHandle> functions) {
+        // TODO: Implement
+    }
+
     private static List<Type> types(MethodHandle handle)
     {
         ImmutableList.Builder<Type> types = ImmutableList.builder();

How to test the plugged functions

At first, we can create usual unit tests by refering the function call tests at TestExpressionCompiler.java, even if it would still not be easy to create unit tests for window and aggregate functions.

In the future, we could invent a descriptive testing framework or development kit like the Hive PDT (Plugin Developer Kit): https://cwiki.apache.org/confluence/display/Hive/PluginDeveloperKit

Maven Dependency Missing

I have download the source and opened the project using root level pom.xml. The required libraries are automatically downloaded into my maven folder. But When I tried to build the project it turn out that 3 Libraries are missing :

1. Maven: com.facebook.presto:presto-main:test-jar:tests:0.54-SNAPSHOT
2. Maven: com.facebook.presto:presto-parser:test-jar:tests:0.54-SNAPSHOT
3. Maven: com.facebook.presto:presto-server:test-jar:tests:0.54-SNAPSHOT

I have re-downloaded and re-opened the project.
Screenshot:

Please help me out. Thank you in advance.

Error Starting Presto Server on Worker

I have an HDFS/Hive instance set up using MySQL as a metastore.

I have a Presto coordinator setup on a system:

[root@viper etc]# cat config.properties
coordinator=true
datasources=jmx
http-server.http.port=8081
presto-metastore.db.type=h2
presto-metastore.db.filename=/hadoop/presto/db
task.max-memory=1GB
discovery-server.enabled=true
discovery.uri=http://viper:8081

[root@viper etc]# cat node.properties
node.id=c2a9dd40-43f6-4088-9cea-ebd6b9ee923c
node.environment=production
node.data-dir=/hadoop/presto/data

[root@viper etc]# ping -c 1 viper
PING viper (172.16.100.19) 56(84) bytes of data.
64 bytes from viper (172.16.100.19): icmp_seq=1 ttl=64 time=0.057 ms

--- viper ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.057/0.057/0.057/0.000 ms

The server starts fine, and (if there is a datanode configured) runs well.

I try to start the Presto server on my worker, with the following configuration:

[root@defiance etc]# cat config.properties
coordinator=false
datasources=jmx,hive
http-server.http.port=8081
presto-metastore.db.type=h2
presto-metastore.db.filename=/hadoop/presto/db
task.max-memory=1GB
discovery.uri=http://viper:8081

[root@defiance etc]# cat node.properties
node.id=b948a821-f4e6-49c8-92f2-94dfcd5d407d
node.environment=production
node.data-dir=/hadoop/presto/data

[root@defiance etc]# ping -c 1 viper
PING viper (172.16.100.19) 56(84) bytes of data.
64 bytes from viper (172.16.100.19): icmp_seq=1 ttl=64 time=0.178 ms

--- viper ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.178/0.178/0.178/0.000 ms

I notice the following in the log output:

2013-12-10T15:45:57.509+0000     INFO   main    org.eclipse.jetty.server.AbstractConnector      Started [email protected]:8081

I get a bunch of errors (see stack trace below). It appears that the Presto server is attempting to connect and failing. Any idea why?

Stack Trace:

2013-12-10T15:45:59.563+0000    ERROR   main    com.facebook.presto.server.PrestoServer Guice creation errors:

1) Error in custom provider, org.jboss.netty.channel.ChannelException: Failed to create a selector.
  while locating io.airlift.http.client.AsyncHttpClientModule$SharedNettyIoPoolProvider
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:75)
  while locating io.airlift.http.client.netty.NettyIoPool
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:78)
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:78)
  while locating io.airlift.http.client.AsyncHttpClient annotated with @io.airlift.discovery.client.ForDiscoveryClient()
    for parameter 3 at io.airlift.discovery.client.HttpDiscoveryLookupClient.<init>(HttpDiscoveryLookupClient.java:63)
  while locating io.airlift.discovery.client.HttpDiscoveryLookupClient
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:51)
  while locating io.airlift.discovery.client.DiscoveryLookupClient
    for parameter 0 at io.airlift.discovery.client.CachingServiceSelectorFactory.<init>(CachingServiceSelectorFactory.java:32)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:65)
  while locating io.airlift.discovery.client.CachingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:117)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:117)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:66)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for parameter 0 at io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:49)
  at io.airlift.discovery.client.DiscoveryBinder.bindSelector(DiscoveryBinder.java:64)

1 error
com.google.inject.CreationException: Guice creation errors:

1) Error in custom provider, org.jboss.netty.channel.ChannelException: Failed to create a selector.
  while locating io.airlift.http.client.AsyncHttpClientModule$SharedNettyIoPoolProvider
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:75)
  while locating io.airlift.http.client.netty.NettyIoPool
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:78)
  at io.airlift.http.client.AsyncHttpClientModule.configure(AsyncHttpClientModule.java:78)
  while locating io.airlift.http.client.AsyncHttpClient annotated with @io.airlift.discovery.client.ForDiscoveryClient()
    for parameter 3 at io.airlift.discovery.client.HttpDiscoveryLookupClient.<init>(HttpDiscoveryLookupClient.java:63)
  while locating io.airlift.discovery.client.HttpDiscoveryLookupClient
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:51)
  while locating io.airlift.discovery.client.DiscoveryLookupClient
    for parameter 0 at io.airlift.discovery.client.CachingServiceSelectorFactory.<init>(CachingServiceSelectorFactory.java:32)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:65)
  while locating io.airlift.discovery.client.CachingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:117)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:117)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:66)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for parameter 0 at io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:49)
  at io.airlift.discovery.client.DiscoveryBinder.bindSelector(DiscoveryBinder.java:64)

1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:175) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) ~[guice-3.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:95) ~[guice-3.0.jar:na]
        at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:277) ~[bootstrap-0.84.jar:0.84]
        at com.facebook.presto.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:91) ~[na:na]
        at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:92) ~[presto-main-0.54.jar:0.54]
        at com.facebook.presto.metadata.CatalogManager.loadCatalog(CatalogManager.java:79) ~[presto-main-0.54.jar:0.54]
        at com.facebook.presto.metadata.CatalogManager.loadCatalogs(CatalogManager.java:64) ~[presto-main-0.54.jar:0.54]
        at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:87) [presto-server-0.54.jar:0.54]
        at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:47) [presto-server-0.54.jar:0.54]
Caused by: org.jboss.netty.channel.ChannelException: Failed to create a selector.
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:51) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69) ~[netty-3.6.2.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39) ~[netty-3.6.2.Final.jar:na]
        at io.airlift.http.client.netty.NettyIoPool.<init>(NettyIoPool.java:65) ~[http-client-0.84.jar:0.84]
        at io.airlift.http.client.AsyncHttpClientModule$NettyIoPoolProvider.get(AsyncHttpClientModule.java:189) ~[http-client-0.84.jar:0.84]
        at io.airlift.http.client.AsyncHttpClientModule$NettyIoPoolProvider.get(AsyncHttpClientModule.java:157) ~[http-client-0.84.jar:0.84]
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013) ~[guice-3.0.jar:na]
        at io.airlift.http.client.AsyncHttpClientModule$HttpClientProvider.get(AsyncHttpClientModule.java:139) ~[http-client-0.84.jar:0.84]
        at io.airlift.http.client.AsyncHttpClientModule$HttpClientProvider.get(AsyncHttpClientModule.java:97) ~[http-client-0.84.jar:0.84]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ~[guice-3.0.jar:na]
        at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:89) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:98) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) ~[guice-3.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
        at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:83) ~[guice-3.0.jar:na]
        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
        at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:75) ~[guice-3.0.jar:na]
        at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
        at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
        at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:147) ~[guice-3.0.jar:na]
        at com.google.inject.internal.Initializer.injectAll(Initializer.java:92) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:173) ~[guice-3.0.jar:na]
        ... 9 common frames omitted
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method) ~[na:1.7.0_45]
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) ~[na:1.7.0_45]
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[na:1.7.0_45]
        at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.7.0_45]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335) ~[netty-3.6.2.Final.jar:na]
        ... 85 common frames omitted
2013-12-10T15:45:59.565+0000     INFO   Thread-48       io.airlift.bootstrap.LifeCycleManager   Life cycle stopping...

Query on table partitioned by timestamp and timestamp contains only date (no time) leads to error

I added a fix to this minor issue (+ one more unit test) here:

https://github.com/ShacharZehavi/presto

Query 20140131_162200_00045_nm339 failed: Invalid format: "2012-12-09" is too short
java.lang.IllegalArgumentException: Invalid format: "2012-12-09" is too short
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at com.facebook.presto.hive.HiveUtil.parseHiveTimestamp(HiveUtil.java:117)
at com.facebook.presto.hive.HiveClient$2.apply(HiveClient.java:829)
at com.facebook.presto.hive.HiveClient$2.apply(HiveClient.java:797)
at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:646)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:268)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:226)
at com.google.common.collect.FluentIterable.toList(FluentIterable.java:334)
at com.facebook.presto.hive.HiveClient.getPartitions(HiveClient.java:619)
at com.facebook.presto.spi.classloader.ClassLoaderSafeConnectorSplitManager.getPartitions(ClassLoaderSafeConnectorSplitManager.java:59)
at com.facebook.presto.split.SplitManager.getPartitions(SplitManager.java:51)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteTableScan(PredicatePushDown.java:782)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteTableScan(PredicatePushDown.java:135)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitTableScan(PlanRewriter.java:269)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitTableScan(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:176)
at com.facebook.presto.sql.planner.plan.PlanRewriter.rewrite(PlanRewriter.java:42)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitLimit(PlanRewriter.java:256)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitLimit(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:72)
at com.facebook.presto.sql.planner.plan.PlanRewriter.defaultRewrite(PlanRewriter.java:50)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteNode(PredicatePushDown.java:156)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteNode(PredicatePushDown.java:135)
at com.facebook.presto.sql.planner.plan.PlanNodeRewriter.rewriteLimit(PlanNodeRewriter.java:25)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitLimit(PlanRewriter.java:250)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitLimit(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:72)
at com.facebook.presto.sql.planner.plan.PlanRewriter.rewrite(PlanRewriter.java:42)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitProject(PlanRewriter.java:199)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitProject(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:76)
at com.facebook.presto.sql.planner.plan.PlanRewriter.defaultRewrite(PlanRewriter.java:50)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteProject(PredicatePushDown.java:168)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteProject(PredicatePushDown.java:135)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitProject(PlanRewriter.java:193)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitProject(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:76)
at com.facebook.presto.sql.planner.plan.PlanRewriter.rewrite(PlanRewriter.java:42)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitOutput(PlanRewriter.java:237)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitOutput(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:78)
at com.facebook.presto.sql.planner.plan.PlanRewriter.defaultRewrite(PlanRewriter.java:50)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteNode(PredicatePushDown.java:156)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown$Rewriter.rewriteNode(PredicatePushDown.java:135)
at com.facebook.presto.sql.planner.plan.PlanNodeRewriter.rewriteOutput(PlanNodeRewriter.java:85)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitOutput(PlanRewriter.java:231)
at com.facebook.presto.sql.planner.plan.PlanRewriter$RewritingVisitor.visitOutput(PlanRewriter.java:65)
at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:78)
at com.facebook.presto.sql.planner.plan.PlanRewriter.rewrite(PlanRewriter.java:42)
at com.facebook.presto.sql.planner.plan.PlanRewriter.rewriteWith(PlanRewriter.java:31)
at com.facebook.presto.sql.planner.optimizations.PredicatePushDown.optimize(PredicatePushDown.java:132)
at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:113)
at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:213)
at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:194)
at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:155)
at com.facebook.presto.execution.SqlQueryManager$QueryStarter.run(SqlQueryManager.java:388)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Add support for third-party compression codecs like LZO

We need a non-hacky way to support Hadoop compression codecs that can't be shipped with Presto (for example, LZO which is GPL). Having a very basic plugin interface (inside the Hive plugin, yay) using ServiceLoader that can add the codec to the Hadoop configuration will probably work. The plugins can bundle the native libraries like the Hadoop code already does.

Additionally, it would be good to allow compressed files to be splittable, so we would need to change split creation to use the methods from the InputFormat.

See https://groups.google.com/forum/#!topic/presto-users/_Fe9YrZ3gYg

Presto 5.6 doesn't work with CDH 4.3 HA

I used -D parameter specified in README, but I still got the same exception java.net.UnknownHostException.
$ ps -ef | grep wzhu00
..
wzhu00 31902 1 9 10:53 ? 00:00:44 java -cp /tmp/presto-server-0.56/lib/* -server -Xmx2G -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -XX:+CMSClassUnloadingEnabled -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -XX:PermSize=150M -XX:MaxPermSize=150M -XX:ReservedCodeCacheSize=150M -Xbootclasspath/p:/tmp/presto-server-0.56/lib/floatingdecimal-0.1.jar -Dlog.output-file=/tmp/presto-server-054/data/var/log/server.log -Dnode.data-dir=/tmp/presto-server-054/data -Dnode.id=ffffffff-ffff-ffff-ffff-ffffffffffff -Dnode.environment=production -Dhive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml -Dlog.enable-console=false -Dconfig=/tmp/presto-server-0.56/etc/config.properties com.facebook.presto.server.PrestoServer
$ file /etc/hadoop/conf/core-site.xml
/etc/hadoop/conf/core-site.xml: XML document text
$ file /etc/hadoop/conf/hdfs-site.xml
/etc/hadoop/conf/hdfs-site.xml: XML document text

Here is what is in server.log:
2014-01-10T11:01:16.008-0600 ERROR Stage-20140110_170115_00004_i9dzy.1-180 com.facebook.presto.execution.SqlStageExecution Error while starting stage 20140110_170115_00004_i9dzy.1
java.lang.RuntimeException: java.io.IOException: java.lang.IllegalArgumentException: java.net.UnknownHostException: obuhd-dev
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-15.0.jar:na]
at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:475) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable$HiveSplitQueue.computeNext(HiveSplitIterable.java:434) ~[na:na]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-15.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-15.0.jar:na]
at com.facebook.presto.execution.SqlStageExecution.scheduleSourcePartitionedNodes(SqlStageExecution.java:599) [presto-main-0.56.jar:0.56]
at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:534) [presto-main-0.56.jar:0.56]
at com.facebook.presto.execution.SqlStageExecution.access$300(SqlStageExecution.java:92) [presto-main-0.56.jar:0.56]
at com.facebook.presto.execution.SqlStageExecution$5.run(SqlStageExecution.java:506) [presto-main-0.56.jar:0.56]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: java.lang.IllegalArgumentException: java.net.UnknownHostException: obuhd-dev
at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:71) ~[na:na]
at com.facebook.presto.hive.ForwardingPath.getFileSystem(ForwardingPath.java:47) ~[na:na]
at com.facebook.presto.hive.FileSystemWrapper$1.getFileSystem(FileSystemWrapper.java:78) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable.loadPartitionSplits(HiveSplitIterable.java:188) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable.access$100(HiveSplitIterable.java:77) ~[na:na]
at com.facebook.presto.hive.HiveSplitIterable$2.run(HiveSplitIterable.java:158) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.executeOrMerge(BoundedExecutor.java:82) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor.access$000(BoundedExecutor.java:41) ~[na:na]
at com.facebook.presto.hive.util.BoundedExecutor$1.run(BoundedExecutor.java:53) ~[na:na]
... 3 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: obuhd-dev
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414) ~[na:na]
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164) ~[na:na]
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:448) ~[na:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:410) ~[na:na]
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:128) ~[na:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308) ~[na:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87) ~[na:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342) ~[na:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324) ~[na:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351) ~[na:na]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194) ~[na:na]
at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:92) ~[na:na]
at com.facebook.presto.hive.FileSystemCache$2.call(FileSystemCache.java:87) ~[na:na]
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721) ~[guava-15.0.jar:na]
at com.facebook.presto.hive.FileSystemCache$1$1.getFileSystem(FileSystemCache.java:68) ~[na:na]
... 11 common frames omitted
Caused by: java.net.UnknownHostException: obuhd-dev
... 33 common frames omitted

Querying INFORMATION_SCHEMA.TABLES fails with table_name filter

Example query:

select * from hive.INFORMATION_SCHEMA.TABLES where table_name = 'task';

This fails with with "schemaName is absent!" due to MetadataUtil.checkTableName(). Besides being the wrong exception type (should be IAE not ISE) that check is very confusing. It would be much simpler like this:

if (tableName.isPresent() && !schemaName.isPresent()) {
    throw new IllegalArgumentException("tableName specified but schemaName is missing");
}

But the root problem is that InformationSchemaDataStreamProvider.extractQualifiedTablePrefix() should be checking if the schema exists.

No support for __HIVE_DEFAULT_PARTITION__

Presto 0.54 lacks support for dynamically partitioned tables that have a __HIVE_DEFAULT_PARTITION__.

query fails with:

presto:tpcds_rcfile> select * from store_sales limit 100;
Query 20131213_161730_00003_eaeq3 failed: For input string: "__HIVE_DEFAULT_PARTITION__"

approx_percentile broken when second parameter is a constant bigint

This query causes an error:

  select approx_percentile(42, 1);

The problem is the second argument is not properly widened to a double. Here is a stacktrace:

java.lang.UnsupportedOperationException
    at com.facebook.presto.block.uncompressed.UncompressedLongBlockCursor.getDouble(UncompressedLongBlockCursor.java:163)
    at com.facebook.presto.operator.aggregation.LongApproximatePercentileAggregation.addInput(LongApproximatePercentileAggregation.java:51)
    at com.facebook.presto.operator.aggregation.LongApproximatePercentileAggregation.addInput(LongApproximatePercentileAggregation.java:25)
    at com.facebook.presto.operator.AggregationOperator$VariableWidthAggregator.addValue(AggregationOperator.java:339)
    at com.facebook.presto.operator.AggregationOperator.addInput(AggregationOperator.java:157)
    at com.facebook.presto.operator.Driver.process(Driver.java:220)
    at com.facebook.presto.operator.Driver.processFor(Driver.java:243)
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:609)
    at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:424)
    at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:544)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Prestodb JAVA_HOME configuration.

Since presto only run on jdk1.7 and it's does not have any env conf file, I wanted to set JAVA_HOME explicitly in configuration file so that it can use jdk1.7 and should not affect other jdk apps.

Is there any workaround for that?

Thanks

IllegalArgumentException when all splits are pruned

java.lang.IllegalArgumentException: tasks is empty
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
    at com.facebook.presto.execution.SqlStageExecution.<init>(SqlStageExecution.java:68)
    at com.facebook.presto.execution.SqlStageManager.createStage(SqlStageManager.java:52)
    at com.facebook.presto.execution.SqlQueryExecution.createStage(SqlQueryExecution.java:289)
    at com.facebook.presto.execution.SqlQueryExecution.access$000(SqlQueryExecution.java:55)
    at com.facebook.presto.execution.SqlQueryExecution$2.apply(SqlQueryExecution.java:368)
    at com.facebook.presto.execution.SqlQueryExecution$2.apply(SqlQueryExecution.java:364)
    at com.google.common.collect.Maps$2.transformEntry(Maps.java:937)
    at com.google.common.collect.Maps$TransformedEntriesMap$1$1$1.getValue(Maps.java:1214)
    at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:286)
    at com.facebook.presto.util.MapTransformer.immutableMap(MapTransformer.java:47)
    at com.facebook.presto.execution.SqlQueryExecution.createStage(SqlQueryExecution.java:261)
    at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:233)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:162)
    at com.facebook.presto.execution.SqlQueryManager$QueryStarter.run(SqlQueryManager.java:298)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

prestodb / presto Goto Github PK

presto's People

Contributors

Stargazers

Watchers

Forkers

presto's Issues

Background

Problem

other comments

Table

Presto UDF

How to test the plugged functions

Recommend Projects

Recommend Topics

Recommend Org