tagomoris / shib Goto Github PK

Hi.

I try to use shib(version commit cb59598).

I click 'show table schemas' link at top page, but multiple db select box does not appear.

config.js

  executer: {
    name: 'hiveserver', // or 'hiveserver2' (not implemented) or 'huahinmanager' (not implemented)
    host: 'myhost',
    port: 10000,
    support_database: true,
    default_database: 'default'
  },

Because app.js refers the default_database variable of client object, I doubt of _default_database variable(not default_database) of lib/shib/client.js.

By the way, I use the old shib with KT(version commit e50c5f3), multiple db select box appears in this version.

Thank you.

Hi, first of all, thank you so much for open sourcing such an awesome software!

I found a typo in README.md and since it was in sample configuration, it got me for a while, so I created p-r.

https://github.com/achiku/shib/pull/1/files

Hello,

project appears dead - still, i tried and received the following:

> [email protected] start
> NODE_PATH=lib node app.js

fs.js:45
} = primordials;
    ^

ReferenceError: primordials is not defined

on

root@29f8859cdde7:/shib# node -v
v15.0.1
root@29f8859cdde7:/shib# gulp -v
CLI version: 2.3.0
Local version: Unknown
root@29f8859cdde7:/shib#

Hi ,

I am Trying to run this windows environment .
I'm getting this issue in windows .
C:\Users\user>node -v
v14.15.4

C:\Users\user>npm -v
6.14.10

There is already a ticket . as per the comment i have added . but it didnt worked for me
Can you explain me where exacttly we need to change . ?

C:\Users\user\Desktop\ProjectNewCastle\shib\shib>npm start

[email protected] start C:\Users\user\Desktop\ProjectNewCastle\shib\shib
NODE_PATH=config

'NODE_PATH' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: NODE_PATH=config
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\user\AppData\Roaming\npm-cache_logs\2021-01-27T12_07_52_517Z-debug.log

C:\Users\user\Desktop\ProjectNewCastle\shib\shib>

When i click the table want to see the patition informations I got a message: 'access denied' after check the code I found there is no headers: {'X-Shib-AuthInfo': authInfo} info in the request. So can not get the username. The code is in shib.js and below is the issue part.

function show_tables_dialog() {
$('#tables')
.dynatree('destroy')
.empty()
.hide();
$('#tablesdiag').dialog({modal:false, resizable:true, height:400, width:400, maxHeight:650, maxWidth:950});
$('#tablesdiag .loadingimg').show();

var selected = $('#table_pairs option:selected');
var engine = selected.data('engine');
var dbname = selected.data('database');

var get_path = '/tables?engine=' + encodeURIComponent(engine) + '&db=' + encodeURIComponent(dbname);
authGet(get_path, function(data){
$('#tablesdiag .loadingimg').hide();
$('#tables')
.show()
.dynatree({
children: data.map(function(v){return {title: v, key: v, isFolder: true, isLazy: true};}),
autoFocus: false,
autoCollapse: true,
clickFolderMode: 2,
activeVisible: false,
onLazyRead: function(node){

      node.appendAjax({
        url: '/partitions',
        data: { key: node.data.key, engine: engine, db: dbname },
        cache: false
      });
    }
  });

});
};

• Configuration based
  • Database access permit/prohibit list
  • Table access permit/prohibit list

For tables with many (100k or more) partitions, "show partitions" requires 1 or so minutes.
Almost of wait is from FetchResults call of thrift library.

Hive and Presto occur it .
My project work well and it can get the result.
But the console show the the username is undefined.
Why?
My config.js -->

var servers = exports.servers = {
listen: 3000,
fetch_lines: 1000, // lines per fetch in shib
query_timeout: null, // shib waits queries forever
setup_queries: [],
storage: {
datadir: './var'
},
engines: [
{
label: 'hive2',
executer: {
name: 'hiveserver2',
host: '172.16.210.190',
port: 10000,
username: 'hadoop',
support_database: true
},
monitor: null
},
{
label: 'presto-hive2',
executer: {
name: 'presto',
host: '172.16.210.123',
port: 8999,
user: 'shib',
catalog: 'hive2',
support_database: true
},
monitor: null
},{
label: 'presto-hive1',
executer: {
name: 'presto',
host: '172.16.210.123',
port: 8999,
user: 'shib',
catalog: 'hive',
support_database: true
},
monitor: null
}

],
};

Although I set query_timeout at config.js, shib doesn't kill yarn job.

I want shib to kill yarn job after timeout because such job is harmful and give a negative effect on other jobs.

Query Status:error
Engine:hadoop, Database:default
Result:Failed to execute statement:Error while processing statement: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime

I don't know how to change job name of tez...

error log is the following. but I can't reproduce

Wed Jun 08 2016 18:39:08 GMT+0900 (JST) [WARN] : fetch error, { err: 
   { message: 'Query was canceled',
     errorCode: 3,
     errorName: 'USER_CANCELED',
     errorType: 'USER_ERROR',
     failureInfo: 
      { type: 'com.facebook.presto.spi.PrestoException',
        message: 'Query was canceled',
        suppressed: [],
        stack: [Object] } } }

/path/to/shib/lib/shib/engine.js:341
            if (rows === null || rows.length < 1 || (rows.length == 1 && rows[
                                     ^
TypeError: Cannot read property 'length' of undefined
    at /path/to/shib/lib/shib/engine.js:341:38
    at fill [as _onTimeout] (/path/to/shib/lib/shib/engines/presto/index.js:224:29)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)

Fix js

Shib seems to mess up DOUBLE numbers. When running a query summing a FLOAT column from HIVE in the presto command line I get 30.427878061784213 as result. With shib it shows {"s":1,"e":1,"c":[3,0,4,2,7,8,7,8,0,6,1,7,8,4,2,1,3]}. The number is in there but it for some reason shows it as JSON?

npm thrift-0.8.0 dose not exist in npm registry. Where can I download this from?

https://npmjs.org/package/thrift

Instead of this, using thrift-dev, working well.

https://npmjs.org/package/thrift-dev

• Tag list?
  • may be not useful...

Hi, thanks for making an awesome query tool!

I wasn't able to get the /show/full/:result_id route to work under node 4.2.2 - it basically pauses after sending the chunk on L542 of app.js. It doesn't look like the drain event is being emitted from the res, which would normally trigger the resume on L549.

I was able to get it working by adding another rStream.resume(); after the res.write(chunk); on L542, but this seems kludgy.

Out of curiosity, it seems like this route is just sending the result file from the filesystem, why not use res.sendfile?

I am running this code against an Impala HiveServer2 instance (CDH 4.5). In the Impala service I have turned off sasl auth for the HiveServer2 service.

When I run the code, I get this error:
[cloudera@cloudera-vm-1 search]$ NODE_PATH=/usr/local/node_modules/thrift/lib/thrift node hive_beeline_client.js
connected
{ seq: 'fetchOperationResult',
op: 'GetTables',
err: null,
res:
{ status:
{ statusCode: 3,
infoMessages: null,
sqlState: 'HY000',
errorCode: null,
errorMessage: 'Malformed THandleIdentifier (guid size: 30, expected 16, secret size: 26, expected 16)' },
operationHandle:
{ operationId: [Object],
operationType: 4,
hasResultSet: false,
modifiedRowCount: null } } }
[ { err: 'Failed to operate "GetTables"',
res: { status: [Object], operationHandle: [Object] },
status: 3 },
undefined ]

/home/cloudera/node_modules/node-thrift/lib/thrift/connection.js:94
throw e;
^
TypeError: Cannot read property 'rows' of undefined
at /home/cloudera/search/hive_beeline_client.js:74:35
at fetchOperationResult (/home/cloudera/search/hive_beeline_client.js:12:5)
at /home/cloudera/search/hive_beeline_client.js:72:41
at client._reqs.(anonymous function) (/home/cloudera/node_modules/node-thrift/lib/thrift/connection.js:84:11)
at Object.TCLIServiceClient.recv_GetTables (/home/cloudera/search/gen-nodejs/TCLIService.js:2122:12)
at /home/cloudera/node_modules/node-thrift/lib/thrift/connection.js:87:37
at Socket. (/usr/local/node_modules/thrift/lib/thrift/transport.js:185:5)
at Socket.EventEmitter.emit (events.js:95:17)
at Socket. (_stream_readable.js:746:14)
at Socket.EventEmitter.emit (events.js:92:17)

At the Impala server end, I see this in the logs:
I0116 15:36:33.833312 11034 impala-hs2-server.cc:252] OpenSession(): request=TOpenSessionReq {
01: client_protocol (i32) = 0,
02: username (string) = "",
03: password (string) = "",
}
I0116 15:36:33.839167 11034 impala-hs2-server.cc:476] GetTables(): request=TGetTablesReq {
01: sessionHandle (struct) = TSessionHandle {
01: sessionId (struct) = THandleIdentifier {
01: guid (string) = "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\x11\xef\xbf\xbd\xef\xbf\xbdN\xe4\xa8\xb9\x14\xef\xbf\xbd \xef\xbf\xbdy\b",
02: secret (string) = "\xc8\x9a\xef\xbf\xbdW}\xef\xbf\xbdG\xdf\xa7\n\xef\xbf\xbd\xef\xbf\xbdJ\tf\xef\xbf\xbd",
},
},
}
I0116 15:36:33.883034 11034 status.cc:44] Malformed THandleIdentifier (guid size: 30, expected 16, secret size: 26, expected 16)
@ 0x6dd5ad (unknown)
@ 0x79ff0b (unknown)
@ 0x798011 (unknown)
@ 0x799a38 (unknown)
@ 0x906a5f (unknown)
@ 0x8fa5d4 (unknown)
@ 0x767c9e (unknown)
@ 0x13363e9 (unknown)
@ 0x13227ff (unknown)
@ 0x13245a4 (unknown)
@ 0x7070eb (unknown)
@ 0x708a1e (unknown)
@ 0x81e34c (unknown)
@ 0x81f490 (unknown)
@ 0xa6a804 (unknown)
@ 0x7fa9326359d1 start_thread
@ 0x7fa931977b6d clone
I0116 15:36:33.883095 11034 impala-hs2-server.cc:489] GetTables(): return_val=TGetTablesResp {
01: status (struct) = TStatus {
01: statusCode (i32) = 3,
03: sqlState (string) = "HY000",
05: errorMessage (string) = "Malformed THandleIdentifier (guid size: 30, expected 16, secret size: 26, expected 16)",
},
02: operationHandle (struct) = TOperationHandle {
01: operationId (struct) = THandleIdentifier {
01: guid (string) = "",
02: secret (string) = "",
},
02: operationType (i32) = 4,
03: hasResultSet (bool) = false,
},
}
I0116 15:36:33.897066 11034 impala-server.cc:2164] Connection from client 127.0.0.1:34982 closed, closing 1 associated session(s)
I0116 15:36:33.899447 11034 status.cc:44] Session closed
@ 0x6dd5ad (unknown)
@ 0x761577 (unknown)
@ 0x7618c3 (unknown)
@ 0x761b94 (unknown)
@ 0x6fc277 (unknown)
@ 0x1336533 (unknown)
@ 0x13227ff (unknown)
@ 0x13245a4 (unknown)
@ 0x7070eb (unknown)
@ 0x708a1e (unknown)
@ 0x81e34c (unknown)
@ 0x81f490 (unknown)
@ 0xa6a804 (unknown)
@ 0x7fa9326359d1 start_thread
@ 0x7fa931977b6d clone

For maintenance.

• use <a href=...> tag for tsv/csv downloads

When I attempt to execute a query from localhost:3000, I get:

Fri Jan 08 2016 12:52:53 GMT-0800 (PST) [ERROR] : ServerError, [TypeError: Cannot read property 'pairs' of null]
Fri Jan 08 2016 12:52:53 GMT-0800 (PST) [ERROR] : TypeError: Cannot read property 'pairs' of null

Any tips on how to debug?

shibを使っています。

データの中に入っている日本語が文字化けるようです。Versionは0.2.0です。
hiveの中に入っているデータが日本語の場合、文字化けるようです。
結果を見ると、下記の写真のようになります。

http://img.twitrpix.com/byabl

一部抜粋すると
nm16334500 1322909970 ã�«ã�¼ã��ã�£ã�®æ�±æ�¹ç�©èª�ã��第ï¼�話 ã�¯ã��ã��ã�©ã��ã��ã�°ã��ã�¸ã�§ã��kiiã�§ã��ã��ä»�å��ã�¯å¹½ã��å­�æ§�ã��å��ç�»å ´ã�§ã��ã��ã��ä»�å��ã�®ã�µã� ã��ã�®çµµã�¯ç§�ã��çµ�æ§�æ°�ã�«å�¥ã�£ã�¦ã��ã��ã��ã�¾ã��ã��ã��ã��ã��ã��ã��ã��ã��ã��ã��

のように文字化けます。

engine.jsの fetcher.fetch(fetchnum, function(err, rows){の段階で既に文字化けている様です。

hiveserverは1を使っています。

var servers = exports.servers = {
listen: 3000,
fetch_lines: 1000,
query_timeout: null, // seconds. (null:shib will wait query response infinitel
y).
setup_queries: [],
storage: {
datadir: './var'
},
executer: {
name: 'hiveserver', // or 'hiveserver2' (not implemented) or 'huahinmanager'
(not implemented)
host: '192.168.1.8',
port: 10000,
support_database: true,
default_database: 'default'
},
//monitor: null
monitor: {
name : 'huahin_mrv1', // or 'huahin_yarn'
host: '192.168.1.8',
port: 9020
}

As I was preparing for the p-r (#28), some tests failed, so this is roughly I checked. Any ideas?

Summary

• shib/result.js
  • This module dosen't exist in the repository, but tests still exist.
  • Maybe forgot to delete?
• instanciate
  • Assertion errors, which seems like extected result is [], but Query.result is set to undefined if it's instanciated without arg.result.
  • https://github.com/tagomoris/shib/blob/master/lib/shib/query.js#L45-L48
  • https://github.com/tagomoris/shib/blob/master/lib/shib/query.js#L51-L78
• serialized
  • queryid is different from expected value. I did't get into too deep to check this out.

$ node -v
v0.12.0
$ npm install
$ npm test

stdout

> [email protected] test /Users/achiku/work/github/shib
> NODE_PATH=lib node_modules/nodeunit/bin/nodeunit test/*.js


t_access_control.js
✔ test1
✔ default_rule
✔ default_allowed
✔ default_denied

t_query.js
✔ test1
✔ removeNewLines
✔ removeNewLinesAndComments
✔ checkQueryString
✔ generateQueryId
✖ instanciate

AssertionError
    at Object.deepEqual (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.instanciate (/Users/achiku/work/github/shib/test/t_query.js:71:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError
    at Object.deepEqual (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.instanciate (/Users/achiku/work/github/shib/test/t_query.js:76:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError
    at Object.deepEqual (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.instanciate (/Users/achiku/work/github/shib/test/t_query.js:81:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

✖ serialized

AssertionError:
[ 'c148cdb90a70ef34dab88d8e1af967a6',
  '2015-03-07T08:45:27.252Z',
  null,
  undefined,
  undefined,
  'select f1,f2 from hoge_table',
  'running',
  '0996cea4fbff7e7caedab52e5f9d6e20',
  '{"error":"","lines":null,"bytes":null,"completed_at":null,"completed_msec":null,"schema":[]}' ]
deepEqual
'{"querystring":"select f1,f2 from hoge_table","results":[],"queryid":"c148cdb90a70ef34dab88d8e1af967a6"}'
    at Object.deepEqual (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:87:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError: '88db57a36ce1d24d1f5739a0c63f73dd' == 'c148cdb90a70ef34dab88d8e1af967a6'
    at Object.equals (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:90:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError
    at Object.equals (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:91:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError:
[ '2734efa50f0dff08129e3485b523f782',
  '2015-03-07T08:45:27.256Z',
  null,
  undefined,
  undefined,
  'select f1,f2 from hoge_table where service="news"',
  'running',
  '7decc7f94d9bdce8842211ad4c1ef7ca',
  '{"error":"","lines":null,"bytes":null,"completed_at":null,"completed_msec":null,"schema":[]}' ]
deepEqual
'{"querystring":"select f1,f2 from hoge_table where service=\\"news\\"","results":[],"queryid":"2734efa50f0dff08129e3485b523f782"}'
    at Object.deepEqual (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:95:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError: '88db57a36ce1d24d1f5739a0c63f73dd' == '2734efa50f0dff08129e3485b523f782'
    at Object.equals (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:98:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

AssertionError
    at Object.equals (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.module.exports.testCase.serialized (/Users/achiku/work/github/shib/test/t_query.js:99:10)
    at Object.<anonymous> (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/lib/core.js:118:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:134:25
    at /Users/achiku/work/github/shib/node_modules/nodeunit/deps/async.js:515:17

✔ composed
✔ parseTableNames

stderr

module.js:338
    throw err;
          ^
Error: Cannot find module 'shib/result'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/Users/achiku/work/github/shib/test/t_result.js:2:14)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)
npm ERR! Test failed.  See above for more details.

The following error sometimes occurs.

shib log.

/.../shib/app.js:274
  q['queryid'] = query.queryid;
                      ^
TypeError: Cannot read property 'queryid' of null
    at pseudo_query_data (/.../shib/app.js:274:23)
    at null.<anonymous> (/.../shib/app.js:364:14)
    at /.../shib/lib/shib/client.js:193:14
    at Statement.<anonymous> (/.../shib/lib/shib/localdiskstore.js:153:17)
Mon Feb 23 2015 19:01:09 GMT+0900 (JST) [INFO] : Starting shib.

I am trying to start shib using presto. On executing "npm start" I get the following error log

Environment:
1-node presto-cluster (version: 0.157)
nodejs version 0.10.48

My config.js file

var servers = exports.servers = { listen: 3000, fetch_lines: 1000, query_timeout: 30, setup_queries: [], storage: { datadir: './var' }, engines: [ { label: 'prestocluster1', executer: { name: 'presto', host: '10.113.59.74', port: 9999, user: 'shib', catalog: 'hive', support_database: true }, monitor: null } ], };

ERROR LOG

Thu Nov 24 2016 17:03:27 GMT+0530 (IST) [INFO] : Starting shib.
Thu Nov 24 2016 17:03:27 GMT+0530 (IST) [ERROR] : databases, { message: 'execution error:java.lang.NullPointerException: timeZoneId is null\n\tat java.util.Objects.requireNonNull(Objects.java:228)\n\tat com.facebook.presto.client.ClientSession.(ClientSession.java:144)\n\tat com.facebook.presto.server.ResourceUtil.createClientSessionForRequest(ResourceUtil.java:199)\n\tat com.facebook.presto.server.ExecuteResource.createQuery(ExecuteResource.java:77)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)\n\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)\n\tat org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:315)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:297)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:267)\n\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)\n\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)\n\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)\n\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)\n\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)\n\tat io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)\n\tat io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)\n\tat org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:523)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:745)\n',
error: null,
code: 500 }

PS: I did not get any build errors while trying to build shib

This issue needs js code fix.

Just tried to setup shib on a windows machine...didn't get past npm start--hoping someone can shed some light on what this error is?

npm-debug.txt

C:\Users\sandbox\NodeHDFS\shib>npm start

[email protected] start C:\Users\sandbox\NodeHDFS\shib
NODE_PATH=lib node app.js

'NODE_PATH' is not recognized as an internal or external command,
operable program or batch file.

npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "start"
npm ERR! node v4.2.3
npm ERR! npm v2.14.7
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: NODE_PATH=lib node app.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'NODE_PATH=lib node app.js'.
npm ERR! This is most likely a problem with the shib package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! NODE_PATH=lib node app.js
npm ERR! You can get their info via:
npm ERR! npm owner ls shib
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\sandbox\NodeHDFS\shib\npm-debug.log

C:\Users\sandbox\NodeHDFS\shib>node --version
v4.2.3

Is it ok to loosen query validation a bit to allow WITH clause?

We are extensively using shib to query Presto, and as our query gets more and more complex, we would want to use WITH clause to simplify how query looks.

WITH clasuse for Presto

If you have any concern, please let me know. If you are ok with this idea, I will send p-r for this.

Hi ,

I am Trying to run this windows environment .
I'm getting this issue in windows .
C:\Users\user>node -v
v14.15.4

C:\Users\user>npm -v
6.14.10

There is already a ticket . as per the comment i have added . but it didnt worked for me
Can you explain me where exacttly we need to change . ?

C:\Users\user\Desktop\ProjectNewCastle\shib\shib>npm start

[email protected] start C:\Users\user\Desktop\ProjectNewCastle\shib\shib
NODE_PATH=config

'NODE_PATH' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: NODE_PATH=config
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\user\AppData\Roaming\npm-cache_logs\2021-01-27T12_07_52_517Z-debug.log

C:\Users\user\Desktop\ProjectNewCastle\shib\shib>

Currently disabled databases in engine 'hiveserver2', because of partition listing.

• HiveServer2 rejects queries like use dbname
  • used for show partitions tablename
  • problem details are not investigated yet
• show partitions doesn't support expression like dbname.tablename
  • Hive 0.13 will support it

More than one queries are issued at the same time, SQLITE_BUSY error occurs.

And, TypeError occurs.

• shib log

Mon Mar 02 2015 09:30:03 GMT+0900 (JST) [WARN] : Invalid query submitted, { query: '...',
  err: 'SQLITE_BUSY: database is locked' }
Mon Mar 02 2015 09:30:03 GMT+0900 (JST) [ERROR] : Error in app, { [Error: SQLITE_BUSY: database is locked] errno: 5, code: 'SQLITE_BUSY' }
Mon Mar 02 2015 09:30:03 GMT+0900 (JST) [ERROR] : Error: SQLITE_BUSY: database is locked

/.../shib/app.js:439
    res.send(pseudo_status(query.state));
                                ^
TypeError: Cannot read property 'state' of null
    at null.<anonymous> (/.../shib/app.js:439:33)
    at /.../shib/lib/shib/client.js:193:14
    at Statement.<anonymous> (/.../shib/lib/shib/localdiskstore.js:153:17)
Mon Mar 02 2015 09:35:02 GMT+0900 (JST) [INFO] : Starting shib.

If you access shib API with queryid which does not exists in DB, TypeError occurs.

curl -s -X POST -F 'queryid=aaaaaaa' http://shib.server.local:3000/delete

/.../shib/lib/shib/client.js:244
    var resultid = query.resultid;
                        ^
TypeError: Cannot read property 'resultid' of null
    at null.<anonymous> (/.../shib/lib/shib/client.js:244:25)
    at /.../shib/lib/shib/client.js:193:14
    at Statement.<anonymous> (/.../shib/lib/shib/localdiskstore.js:153:17)