dashaub / espandas

Reading and writing pandas DataFrames in Elasticsearch

espandas

Reading and writing pandas DataFrames in ElasticSearch

Requirements.

This package should work on both python 2(>=2.7) and 3(>=3.4) but has primarily been tested on python 2.7. ElasticSearch is of course required and should be version 2.x.

Installation

The package is hosted on PyPi and can be installed with pip:

pip install espandas

Alternatively, the development version from Github can be installed:

pip install git+git://github.com/dashaub/espandas.git

Tests

Unit tests can be run with pytest or nosetests. Code coverage can be established with pytest-cov from PyPi:

py.test --cov=espandas

Usage

This example assumes ElasticSearch is running on localhost on the standard port. If different connection infromation needs to be specified, it can be passed to the Espandas() constructor as keyward arguments. The DataFrame to insert must have a column that will be used for the unique identifier _id in ElasticSearch: the default value is uid_name = 'indexId'.

import pandas as pd
import numpy as np
from espandas import Espandas

# Example data frame
df = (100 * pd.DataFrame(np.round(np.random.rand(100, 5), 2))).astype(int)
df.columns = ['A', 'B', 'C', 'D', 'E']
df['indexId'] = (df.index + 100).astype(str)

# Create a client and write the DataFrame. If necessary, connection
# information to the ES cluster can be passed in the espandas constructor
# as keyword arguments.
INDEX = 'foo_index'
TYPE = 'bar_type'
esp = Espandas()
esp.es_write(df, INDEX, TYPE)


# Query for the first ten rows and see that they match the original
k = df.indexId[0:10]
res = esp.es_read(k, INDEX, TYPE)
res == df.iloc[0:10].astype('str')

License

(c) 2017 David Shaub

This package is free software released under the GPL-3 license.

espandas's Issues

Numeric fields load as str

Hello,
I have an issue when I tried to visualize my data into kibana after upload it in ElasticSearch through this package. My numeric fields are converted in str because of line 83 espandas.py file.
I have implemented a quick solution : delete this line and replace na by a string : 'Unknown'
I am working on a better solution : create a mapping and pass it as arg when index is created.

What do you think of these solutions? Are you interested by my contribution to your project?

Kevin

Connect with a no localhost remote host with user and password

Hello,

i would like to know how to create a connection with espandas like this:

elk = Elasticsearch([{
    'host': '192.168.0.100',
    'port': '9200'
    }], http_auth=('elastic', 'changeme'))

Thanks

AttributeError: 'Espandas' object has no attribute 'es_write'

Pandas deprecated functions should be reviewed

Greetings, I was testing your software. Congratulations it looks very functional, However there are problems with the most recent versions of pandas since reindex_axis is no longer avaible in pandas 1.1.3 I will try to make a correction if I get sometime. In the mean time I report this bug and people that may use this code should consider runing a specific version of pandas.

Incompatible with pandas > 0.21.0: AttributeError: 'DataFrame' object has no attribute 'reindex_axis'

Using a pandas 0.25.3 dataframe the following error occurs:

File "/home/x/anaconda3/envs/x/lib/python3.7/site-packages/espandas/espandas.py", line 82, in es_write
df = df.reindex_axis(sorted(df.columns), axis=1)
File "/home/x/anaconda3/envs/x/lib/python3.7/site-packages/pandas/core/generic.py", line 5179, in getattr
return object.getattribute(self, name)
AttributeError: 'DataFrame' object has no attribute 'reindex_axis'

Deprecated since version 0.21.0: Use reindex instead.

see:
https://pandas.pydata.org/pandas-docs/version/0.24/reference/api/pandas.DataFrame.reindex_axis.html