mradamlacey / elasticsearch-tableau-connector Goto Github PK

This should also include the Elasticsearch document's _id value

KO is not preserving the selected value if the array that a select's options are bound to changes.

This shouldn't happen when using the optionsValue binding - but still seeing the behavior. Workaround of saving off the state, changing the options array, and reassigning the selected value.

I am using Elasticsearch service managed by AWS so we dont have access to elasticsearch.yml or anyway we cannot enable CORS support in Elasticsearch server.
Please suggest a way where we do not require the use of CORS and get the required data

Is it possible to have multiple indexes. Similar to how elasticsearch api handles multiple indices separated by commas

Support using results from aggregation query as data in Tableau.

Support the following bucket aggregations:

• terms
• range
• date range
• date histogram

Special characters in field names cause issues in Tableau that are valid in Elasticsearch mappings.

Example is the common field @timestamp found in part of the default Logstash mappings.

Support client-side formatting for date formatting to remove the requirement to enable server-side scripting.

Consider using moment (or other client side library) to parse the date as is returned from Javascript.

Consider providing optional UI where the user can provide a date format for specific fields for a type, or define '*' to use the same format for all fields.

Given that Elasticsearch is often used to capture high volumes of streaming log data, it would be very useful to support incremental refresh, as described here: https://tableau.github.io/webdataconnector/docs/wdc_incremental_refresh

I got a fresh node of CentOS loaded with nginx, grunt, and grunt-cli load. Used grunt-cli to install the web connector in the default location of /usr/nginx/html/public/elasticsearch-connector.html. When I browse to this url I get please open this URL from within Tableau but when I load it in Tableau I get the same message. I started it with nodejs on port 30000 but get the same message in both places.
I haven't been able to get it to pull up the configuration page.
What am I doing wrong here?

• error handling on invalid query - show error message in connector
• data gather - make sure abort is called when error
• support child objects (industry type field example)
• date range - validation broken for 2 or more

Need to update to the 2.1 release of the Tableau WDC SDK.

I know the readme says these are ignored, but I figure this isn't desirable and I didn't see an existing issue for it, so here we are :)

Hi, I have tried to connect to the latest Elastic Version 5.0.0 and I can see on Tableau only the numeric fields; no string field is displayed. Tx in advance.

Hi,

We are trying to use elasticsearch-tableau-connector on local desktop and getting "CORS request was denied" error while connecting to elasticsearch. Our elasticsearch url is https://xxxx:9200

I have already added following section in elasticsearch.yml file of all elasticsearch servers, still we are getting this error.
Please provide us some pointers to troubleshoot it.
#Cors enable for Tableau
http.cors.enabled : true
script.inline : true
script.indexed : true
http.cors.allow-origin : "*"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length
http.jsonp.enable : true
cors.enabled : true
http.cors.allow-credentials : true
http.detailed_errors.enabled : true

Tested connectivity and I am able to reach to elasticsearch from local desktop. Output below:
"name" : "node335M",
"cluster_name" : "cd_elk_cluster-dev",
"version" : {
"number" : "2.2.0",
"build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
"build_timestamp" : "2016-01-27T13:32:39Z",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}

Thanks,
Rakesh

Hi, Thanks for this connector , we were looking for similar implementation. I have few questions on how to install and run this.

• Do I need a tableau server to install this?
• Can this connector run as a webapp in a tomcat server ?
• My office nw doesn't allow me to run grunt builds so how do I build it.

I am seeing this issue when connecting to my elastic search server. This happens when I select (Search Result Mode). However, Aggregation mode works fine.

I copied dist folder to Tableau Server, and ran following commands within dist folder
tabadmin import_webdataconnector elasticsearch-connector.html
tabadmin import_webdataconnector elasticsearch-connector.min.css
tabadmin import_webdataconnector elasticsearch-connector.min.js

And from Tableau go to 'Web Data Connector' and enter the URL of the connector:
http://.../webdataconnectors/elasticsearch-connector.html

It showed the connector html in Tableau Desktop, but had this error:
ReferenceError: Unable to process binding "aceEditor: function () {return undefined; }" Message: Can't find variable: ace file: http://.../webdataconnectors/elasticsearch-connector.min.js line: 24

Hello- this is awesome but I'm having some issues using it and connecting to multiple Indexes on my ES cluster. I can create 2 but then clicking on one just overwrites the other one.

I've tried even running 2 instances (changing the port) and it still only keeps one as the data source-- overwrite the other as I click and load it.

Is there something I'm missing to use this plugin to pull in 2 distinct indexes from the same ES cluster?

Add a 'Preview' option in the connector UI to allow users to see a tabular view of the data to be returned to Tableau.

After clicking a 'Preview' button, a query should be issues to Elasticsearch and a data table should be populated with results.

Reuse the existing methods to retrieve the Elasticsearch results.

This also provides an ability for users to debug the queries sent to Elasticsearch as error message(s) should be displayed if there's an errors.

When the total search hit total is less than the specific limit (e.g. total number of documents is 9, and the specified limit is 10) - when in 'Data gather' mode - the connector will hang indefinitely.

I have some date/time values in my ES data that look like this (UTC-7):

But when this value is brought into Tableau via the ES Connector, they are reinterpreted to UTC time:

Is it possible to get the connector to read the field exactly as-is rather than change it to UTC?

Hi,

Thanks for your great work on the elasticsearch connector first!

I meet an issue at the end of Tableau Executing query part. in attachment, there is a video recorded the progress.

ENV: Macbook pro, elasticsearch 2.3.3 within Docker 1.11.2, tableau desktop 9.3
Data: only 1 piece:
curl -XPUT 'http://localhost:9200/db/user/1' -d ' { "id": 1, "name": "hugo" }'
I run a elasticsearch service on my Mac, and use the connector to connect it.
It's fine on the connector webpage(see attachment "connector_fine_page" ), index, type showed itself.
But after click "submit" button, it turns to tableau loading popup, the steps are: connecting to database, Executing query, Loading metadata, Creating extract database, Executing query... (then stops here, will run about 10 mins to pop a timeout page, see "Executing query forever", "finall error").

Much appreciate if you could look into this.

video.zip

Need to update the connector source to support the latest version of the Tableau Web Data Connector API - 1.1.1

Refer to: https://github.com/tableau/webdataconnector for the latest version

The index name field shows me all correct indexes but the type fails.
Imgur Link

When I enter the type "log" manually and try to get a preview this error shows up
Unable to get Elasticsearch types, status code: 400; No handler found for uri [//productivity/log/_mapping] and method [GET] Bad Request

When I run "get /productivity/log/_mapping" in kibana it show me the mapping of the data correctly.

I added following commands to elasticsearch.yml:
http.cors.enabled: true
http.cors.allow-origin: "*"

When trying to create an extract of an ES index of ~ 65 million documents, the following error (eventually) appears:

TypeError: undefined is not an object (evaluating 'scrollResult.results.length') file: http://$TABLEAU_SERVER/webdataconnectors/elasticsearch-connector.min.js line: 38
Unable to create extract

Hello again!

Not sure if this is really an issue. The connector works fine but all the timestamp fields are coming as null. Is this something known or something I may have done wrong?

Thanks

Add support to add Tableau fields for child objects in search results.

This does NOT include support for child arrays, or 'nested' type mappings. This is simply child objects (arbitrarily deep object graphs).

In addition to 'Relative', 'Absolute' and 'Custom', add a 'Now' option that will use the current time (the Elasticsearch 'now' value) from the value in a date range.

Connector UI should still be available to use the 'Preview' feature even when the connector is loaded outside of Tableau Desktop of the Tableau SDK simulator.

This will allow users to preview what their data extracts will look like.

When loaded from Tableau Desktop or the SDK simulator, the 'Submit' button should be enabled, otherwise disabled.

Hi there!

Great to see a connector for ElasticSearch (this should be something native from Tableau, of course but well..).

Sorry to bother but...

I'm having issues with the connector. I enter a valid cluster node URL, a valid index name and a valid type but I always get a Bad request response. Even using a custom query. The query works fine in the same URL via cURL. The error is the following, without custom query (big, so it's on pastebin): http://pastebin.com/fMDvCBx0
Is there anything I can do here to get more information on how to proceed?

My ES version: 1.6.0
Tableau Desktop is on latest.

Thanks

Hi,

Thank you for the connector!

I tried to load 1 record from my Elasticsearch endpoint, so I set the "Batch size of per request to Elasticsearch" and "Total limit on number of rows to sync" to 1.

It took me more than 25 minutes, but nothing was loaded. The size of each record is very small.

How long does it usually take to load the metadata?

My Elasticsearch URL is http://lgb-es-01:9200. When I populate the "Elasticsearch URL" with this, I can see and select my index and types correctly. But hitting "Preview" or "Submit" causes an "Elasticsearch URL is not a valid URL" error.

Running ES 5.4, Tableau 10.2.0.

Suppose I have indices twitter1, twitter2, twitter3 and twitter4, each of which contains 1 document:

PUT twitter1/tweet/1
{
    "message" : "This is tweet 1",
    "@timestamp" : "1499369761",
    "user" : "twitter1"
}
PUT twitter2/tweet/2
{
    "message" : "This is tweet 2",
    "@timestamp" : "1499369763"
}
PUT twitter3/tweet/3
{
    "message" : "This is tweet 3",
    "@timestamp" : "1499369763",
    "user" : "twitter3",
    "data" : "this is twitter 3"
}
PUT twitter4/tweet/4
{
    "@timestamp" : "1499369764",
    "user" : "twitter4",
    "data" : "this is twitter 4"
}

When I use an alias to combine these, as so:

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "twitter1", "alias" : "twitter-alias" } },
        { "add" : { "index" : "twitter2", "alias" : "twitter-alias" } },
        { "add" : { "index" : "twitter3", "alias" : "twitter-alias" } },
        { "add" : { "index" : "twitter4", "alias" : "twitter-alias" } }
    ]
}

then most of the fields does not appear in the resulting Tableau table:

But if I were to instead combine those indices into a "super-index" of sorts:

POST _reindex
{
    "source": { "index": "twitter1"},
    "dest": {"index": "test-twitter-combined"},
}
POST _reindex
{
    "source": { "index": "twitter2"},
    "dest": {"index": "test-twitter-combined"},
}
POST _reindex
{
    "source": { "index": "twitter3"},
    "dest": {"index": "test-twitter-combined"},
}
POST _reindex
{
    "source": { "index": "twitter4"},
    "dest": {"index": "test-twitter-combined"},
}

then they all exist correctly:

It seems as though in the alias case, the connector looks at the fields from the first document it sucks in (based on _sequence, looks on the only document in twitter2 and uses that as all the fields.

The sequence I used to generate the indicies and perform reindexing/aliasing is here:
tableau-es-commands.txt

After connecting to the Web data connector, I failed to connect to the data that I've published
on Elasticsearch every time I write in the "Type" area, the error information is "unable to get elasticsearch type", as is shown, I've set the _type as 'user' in the config file in "input" area.
Tried every way, and it doesn't work, don't know where goes wrong.
Thanks!!

I am using the incremental refresh on a date field but when the connector passes dates to Tableau, it strips out the milliseconds accuracy.
So when you make a incremental refresh, it might include duplicate records if there are multiple on the same second. Example:

1) Existing records:
2017-07-07T01:01:01.200Z Record 1
Incremental refresh from an earlier date will include this record correctly.

2) Existing records, a new record (Record 2) is added:
2017-07-07T01:01:01.200Z Record 1
2017-07-07T01:01:01.300Z Record 2
Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z
But it will exclude the first record (see here) so this is expected behaviour and only Record 2 will be added.

3) Existing records, a new record (Record 3) is added:
2017-07-07T01:01:01.200Z Record 1
2017-07-07T01:01:01.300Z Record 2
2017-07-07T01:01:01.400Z Record 3
Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z because the milliseconds are not saved.
It will exclude the first record but not the second one, so record 2 will be duplicated in the Tableau extract.

My workaround for now is to force all dates to be integers and making the ElasticSearch query with a timestamp instead of a date. So I don't have a proper pull request to offer because this removes the Date parsing from the connector.

The Connection Name does not seem to be a required field when initially creating the data extract. But leaving it blank causes Tableau to hang when querying the data.

The connector does not function when the Elasticsearch data source is configured with authentication in the Tableau Online Sync Client.

The sync client logs indicate a 'password not found' error.

To fix this, a change is needed to pass the username/password credentials for the datasource as normal Tableau connection data. This puts these credentials in cleartext in log files and wherever the connector metadata is stored (Tableau Online, users with Tableau Desktop).

Hi,
I was in charge to try your connector for my company but I couldn't make it work.

My elasticsearch is working but when i start the connector, the web interface (localhost:3000) get stuck in a loading screen.
I followed your guide to install the connector without any errors and I modified the File elasticsearch.yml

:
http.cors.enabled: true

http.cors.allow-origin: "*"

The following commands worked well :

npm config set proxy http://ProxyOfTheCompagny:8080
npm config set https-proxy http:// ProxyOfTheCompagny:8080
npm install -g grunt
npm install -g bower
npm install
bower install
grunt build:dist
grunt

Do you know this issue ? How to solve it ?

Thanks very much for your time.

In Elasticsearch 5.x - add support for text and keyword fields to map as string Tableau columns

Scan/Scroll API is more performant for large data extracts (and it looks like Elasticsearch 2.0+ doesn't allow from/size queries > 10000).

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

Hi

I clone the repository run the build and start the connector by

node index,js

In the Tableau designer web data connector,
if I put the URL http://localhost:3000 then I get error: "Cannot GET /"
if I put the URL http://localhost:3000/elasticsearch-connector.html, In the error detail shows:
TypeError: undefined is not an object (evaluating '$.fn') file: http://localhost:3000/bootstrap3-typeahead.js line: 459

Do I set it up wrong or it is a bug?

Thanks.

When I establish the connection to Elasticsearch and bring in fields, the result is inconsistent with what would happen for other Tableau connections:

• Field names are not capitalized
• Non-leading underscores are not replaced with spaces
• The connection name is appended to the field

Consider these fields pulled in from a MySQL database and compare to pulling in fields from ES via the connector:

I am planning a migration of some workbooks from MySQL to Elasticsearch, and these differences require me to continually hand-adjust all imported fields from ES before I can "Replace Data Source". Is it possible to make the import of fields more Tableau-like?

see: http://onlinehelp.tableau.com/current/pro/desktop/en-us/help.htm#data_clean_adm.html

Hi, I am using ES 5.2.0., I downloaded the elasticsearch-tableau-connector release-1.0 from this link and built the package. When I launch WDC from my Tableau 9.3 Desktop instance,

http://my-elasticsearch-tableau-connector

Only the numeric fields are from my Elasticsearch index is visible. It is not showing any string fields.
Then i found this forum post, I was hoping that would fix the problem. But it didn't, when i change the JS file as shown here, the grunt build:dist would just work fine, but when launch the WDC, it is throwing error on tableau.dataTypeEnum.string

Reference: https://community.tableau.com/message/581909

`31. RE: ELASTICSEARCH CONNECTOR
Abed Arnaout
Vizitor
ABED ARNAOUT JAN 25, 2017 11:37 AM (IN RESPONSE TO ANDREA CAMPAGNA)
For those having issues with text fields in Elasticsearch 5.x, you can do the following fix (I'll also submit a merge request to github):

• In "elasticsearch-connector.js" file, make sure you have keyword & text fields in the elasticsearchTableauDataTypeMap as follow:

var elasticsearchTableauDataTypeMap = {
string: tableau.dataTypeEnum.string,
keyword: tableau.dataTypeEnum.string,
text: tableau.dataTypeEnum.string,
float: tableau.dataTypeEnum.float,
long: tableau.dataTypeEnum.int,
integer: tableau.dataTypeEnum.int,
double: tableau.dataTypeEnum.float,
date: tableau.dataTypeEnum.datetime,
boolean: tableau.dataTypeEnum.bool,
geo_point: tableau.dataTypeEnum.string
},

When you create a new connection, then right click and select Edit Connection, the resulting form does not populate with your previously entered settings. So it's like starting new every time rather than editing.

As alternative to simply providing a custom query that contains aggregations, provide UI that lets the user build an aggregation query by selecting metrics and buckets