CSV River strange behavior about elasticsearch-river-csv

[vtajzich]

Hi,

thank you for your detailed info. I would like to reproduce the issue and profile our csv river to find root cause.

• are you able provide us w/ test data set for testing?
• did you tried it on more powerful server? Just to see if the issue is not caused by indexing in ES itself

Regards,

Vitek

[proxylab]

Hi Vitek,

Thanx for the reponse. Well right now I haven't test it to a stronger server although I will soon. Previously I was doing bulk insert with bulk API and didn't face any similar behaviour. As for the sample data I guess we must agree to push it into a public directory since is about a couple of thousands csv files.

Alex

[vtajzich]

Hi,

If you don't mind you can share the data just w/ us in order to fix river. Don't have to share them publicly.

In the mean time I'll look for some big data set and will try to profile river.

[proxylab]

Hello Vitek, I have just found out that I apart from 80-90% I/O(program total time spending on I/O) using iotop I get values around 50%wa with top command which is a strong indicator that my CPU waits for the disk too long. It seems that I have an issue with the disk although I have send you some sample data at your e-mail.

Thank you,
Alex

[vtajzich]

I'm going to look on it. However I'm too busy these days. Just wanted to let you know I think of it.

[dagr9782]

Hello proxylab, I am having the same situation here, the first thing I thought was to separate the input files (csv) and the ES database into different disks for better I/O, but that does not sounds promising. Have you come to any conclusions on that issue?

[vtajzich]

@dagr9782 I'll try to reproduce the issue to fix it. How big (columns & rows) csv file is needed?

[vtajzich]

I just tried to index some test data. 23M of lines needs ~ 10.5GB of RAM.

Could you, please, try it once more again on more powerful HW?

Also try to disable all plugins during such a heavy load (except csv river and head ... )

[vtajzich]

@proxylab do you need further help?

[dagr9782]

Hello Vitek,

thanks a lot for you attention.

I loaded 3 indices:

1. size: 18.0Gi / docs: 199.680.808 / 3 rows (took 4 hours)
2. size: 27.0Gi / docs: 182.834.166 / 3 rows (took 5 hours)
3. size: 115Gi /docs: 548.358.452 / 9 rows

I believe 2 things impacted most on the performance:

1. increasing csv-river bulk_size to 20000 (more than that crashed the JVM
   for Heap Space - limited HW)
2. ordering and removing duplicates on the input file

As I suspected the network impacted the load speed (at night it was much
faster), I also disabled the refresh_interval and the replicas during the
load. After successful load, I enabled those again "on the fly".

So I believe it was a matter of adjusting things.

Best Regards,

Diego

2015-10-02 4:21 GMT-03:00 Vitek Tajzich [email protected]:

@proxylab https://github.com/proxylab do you need further help?

—
Reply to this email directly or view it on GitHub
#45 (comment)
.

[vtajzich]

@dagr9782 thank you for your reply. As the river works correctly I going to close the issue.