pabloem / beam_utils Goto Github PK

Hi, and thanks very much for this library.

I noticed that when trying to feed the CsvFileSource a gzip file, I got this error:

TypeError: argument 1 must be an iterator

After some digging I found out that csv.reader() expects an iterator, which, unfortunately, is not what self.open_file() returns (see FileBasedSource).

This test exposes the problem, and this addition to sources.py fixed my problem.

Hi,

Do you have any sample code of how to use this utility ?

Many thanks

Have you run into the problem where the csv reader will not read more than one record? Any csv I try only reads one line and converts it into a dict.

Thank you for providing the package.

The following import is failing:

from apache_beam.io import fileio

You are only using fileio for the CompressionType.

The correct import would be:

from apache_beam.io.filesystem import CompressionTypes

(are you still actively developing this package? I'd be happy to add more struff from my local beam_utils if you do)

Hi,

thanks a lot for writing this little library. It really helped me handling CSV Files.

But since the delimiter in my files wasn't the default ",", I had problems getting the library to work.
It seems that in CsvFileSource you included an option for different CSV delimiters, but the options are actually never used.

reader = csv.reader(self._file)

should be:

reader = csv.reader(self._file, delimiter=self.delimiter)

Thanks for your help and best regards,
Tobias

I had some trouble using the JSONLinesFileSource; it was hanging when trying to process a large file and processing smaller files unpredictably.

After having a look at the source, I realised that JSONLinesFileSource is designed to process streaming JSON where all of the objects are on one line, using a buffer. JSONLines is a format where each object is on a separate line (http://jsonlines.org).

It might be a good idea to rename the Source in order to avoid this confusion?

https://stackoverflow.com/questions/59861800/unable-to-use-csvfilesource-from-beam-utils-sources-in-apache-beam-program-on-cl