azavea / django-queryset-csv Goto Github PK

I found when writing utf-8 output both keys and values had to be encoded in the _sanitize_unicode_record function. Here's the implementation i used, which may be slightly redundant:

def _sanitize_unicode_record(record):
  return {unicode(k): unicode(v).encode('utf-8') for k, v in record.items()}

Hi, I export a file containing ñáéíóú characters pretty common in other languages, but it's rendering Ñ instead of ñ, as well as other encoding for the accents.

I don't know what is generating the issue, I saw you gave csv_kwargs = {'encoding': 'utf-8'}

pip freeze gave me django-queryset-csv==1.0.1

Can you guys provide a example on Class based views

Where does it generate the CSV file with the model data?

These parameters are currently undocumented and have caused some confusion.

This should be covered by annotating a queryset and then exporting it, but opening an issue to officially support this and add test coverage for it.

This issue was raised to me from a user by email.

This is likely to be a better performing default for most people than a standard HttpResponse, but since StreamingHttpResponse is not a subclass of HttpResponse and has a different API, making it the default would be a breaking change (and we would add a non_streaming option to restore the old behaviour).

We could instead make it an optional feature by adding a render_to_streaming_csv_response function or adding an optional streaming option to render_csv_to_response.

See the Django docs for a general overview of StreamingHttpResponse and how to use it with CSVs

Ok, now I'm just bikeshedding, but the package name is terrible

qs_csv or something would be much more easier to stomach :)

I'd like to generate a CSV file not only with attributes, but also properties. The model could look like this:

class WorkingTime(models.Model):
    """\
    Used for time tracking on projects.
    """
    employee = models.ForeignKey(Employee)
    start_time = models.DateTimeField(default=datetime.datetime.now())
    end_time = models.DateTimeField(default=datetime.datetime.now(), blank=True, null=True)

    def _delta(self):
        """\
        Returns the difference between start and end time of the logged work.
        """
        return self.end_time - self.start_time

    delta = property(_delta)

Now I'd like to generate the CSV like this:

# make query
csv_qs = csv_qs.values(                        
    'start_time',                              
    'end_time',                                
    'employee__last_name',                     
    'employee__first_name',                    
    'delta'                                    
)                                              
return render_to_csv_response(                 
    csv_qs,                                    
    field_header_map={                         
        'start_time': _(u'Begin'),
        'end_time': _(u'End'),                
        'employee__last_name': _(u'Nachname'), 
        'employee__first_name': _(u'Vorname'), 
        'delta': _(u'Arbeitszeit'),            
    },                                         
    append_datestamp=True,                     
    filename="Arbeitszeiten",                  
    delimiter=';')                             

Is that somehow possible? Thanks!

It looks like Following example at Usage section in README causes error on python3.5 environment.

Target Code

If you need to write the CSV to a file you can use write_csv instead:

from djqscsv import write_csv

qs = Foo.objects.filter(bar=True).values('id', 'bar')
with open('foo.csv', 'w') as csv_file:
  write_csv(qs, csv_file)

RESULT

TypeError: write() argument must be str, not bytes

write_csv method trying to write byte (BOM) to file, so We need to open file as binary mode.

Correct Example

with open('foo.csv', 'wb') as csv_file:
  write_csv(qs, csv_file)

I'm glad if you test those examples in your environment and confirm whether correction works.
Kind Regards,
Asayu123

Problem with:

í  237 u’\xed’ \xc3\xad    \xed    Latin small i with acute

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/utils/decorators.py", line 29, in _wrapper
    return bound_func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in bound_func
    return func(self, *args2, **kwargs2)
  File "/Users/xpalacin/Sites/django_guiaterapeutica/main/views.py", line 148, in get
    text = render_to_csv_response(queryset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/djqscsv/djqscsv.py", line 49, in render_to_csv_response
    write_csv(queryset, response, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/djqscsv/djqscsv.py", line 122, in write_csv
    writer.writerow(merged_header_map)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 148, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 1: ordinal not in range(128)

There seems to be no obvious way to give a format like %Y-%m-%d or %Y-%m-%d - %H:%M for a date or datetime field.

I am using djqscsv.render_to_csv_response() function to export my django querset to *.csv file. when I canceled downloading the *.csv file, I got error like that:

Exception happened during processing of request from ('127.0.0.1', 61041)
[12/Oct/2018 17:54:05] "POST /accounts/data_extraction/ HTTP/1.1" 200 2375804
Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 138, in run
    self.finish_response()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
    self.write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 279, in write
    self._write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 453, in _write
    result = self.stdout.write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socket.py", line 594, in write
    return self._sock.send(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
[12/Oct/2018 17:54:05] "POST /accounts/data_extraction/ HTTP/1.1" 500 59
Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 138, in run
    self.finish_response()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
    self.write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 279, in write
    self._write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 453, in _write
    result = self.stdout.write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socket.py", line 594, in write
    return self._sock.send(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 141, in run
    self.handle_error()
  File "D:\Company projects\websystem\truckcaliber\lib\site-packages\django\core\servers\basehttp.py", line 88, in handle_error
    super(ServerHandler, self).handle_error()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 368, in handle_error
    self.finish_response()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
    self.write(data)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 274, in write
    self.send_headers()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 331, in send_headers
    if not self.origin_server or self.client_is_modern():
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 344, in client_is_modern
    return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 625, in process_request_thread
    self.finish_request(request, client_address)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 354, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 681, in __init__
    self.handle()
  File "D:\Company projects\websystem\truckcaliber\lib\site-packages\django\core\servers\basehttp.py", line 155, in handle
    handler.run(self.server.get_app())
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 144, in run
    self.close()
  File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\simple_server.py", line 36, in close
    self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'

What is wrong there?

Excel can't read unicode csvs unless there's a BOM in front of them. See http://stackoverflow.com/questions/5838605/python-dictwriter-writing-utf-8-encoded-csv-files

2c86170 broke code coverage for some reason

It would be great if the library would support writing lists to csv.

Something like this:

header = [...]
data = [[], [], []]

render_to_csv_response(data, header=header)

Hey guys, thank you for the py3 support.
May I ask for the new pip package.

Thanks

It looks terrible

https://pypi.python.org/pypi?:action=display&name=django-queryset-csv&version=0.2.0

Downloading/unpacking django-queryset-csv>=0.2.4 (from -r ../../env/requirements.txt
(line 25))
Running setup.py egg_info for package django-queryset-csv
Traceback (most recent call last):
File "", line 16, in
File "/usr/local/otm/env/build/django-queryset-csv/setup.py", line 10, in
long_description=open('README.rst').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 16, in

File "/usr/local/otm/env/build/django-queryset-csv/setup.py", line 10, in

long_description=open('README.rst').read(),

IOError: [Errno 2] No such file or directory: 'README.rst'

Resource interpreted as Document but transferred with MIME type text/csv: <path_to_file.csv>

There's a bit if code duplication in the unit tests. It's a little hard to reason about what's under test. 100% test coverage is great, but quality is more important.

Is there a way to use the Model.get_FOO_display() method to use the human-readable values for a choices field in the CSV?

This method is available in Django templates, as documented here: https://docs.djangoproject.com/en/1.11/ref/models/instances/#django.db.models.Model.get_FOO_display

If this doesn't already exist, if you point me in the right direction, I can try my hand at a PR.

I setup a toy view per the documentation. It works great except for one thing.

In the CSV file the ID column has some strange characters in it. See screenshot.

It seems like the issue is related to Microsoft Excel, because if i open the file in a text editor everything looks fine

This blog post is referenced on PIP and in the README, and is referenced as the source of documentation.

https://www.azavea.com/blog/2014/03/11/exporting-django-querysets-to-csv/

A sample of the broken snippets:

When pushing tags, Travis CI automatically pushes a new release of the library to PyPI.
This is working correctly, but we are trying to push artifacts for every combination in our build matrix, which causes all builds in the matrix after the first to fail, since the release is already on PyPI.

We should restrict our deploy step to only run once, if possible. Travis CI build stages might be helpful here: https://docs.travis-ci.com/user/build-stages

This isn't actively harmful, since pushing artifacts to PyPI is the last step in the Travis CI test script, and we will separately test the master branch at the same time, but it would be nice to fix this.

If there is a parameter in function render_to_csv_response(), which I can modify the filename of CSV generated ?

import djqscsv
data=...
djqcsv.render_to_csv_response(data)

Thanks!

Hi, this library works great for me with one exception: when I save the download and look in Explorer to open the file, it just shows up as a generic file (not a csv). The only file type option to save the file is "all files". This is confirmed by examining the file's properties. If I manually add the file extension by editing the file name, then the file opens just fine.

This error occurs whether I specify a file name or not.

On the initial download pop up (where it asks you to either save or open), it shows file type as "application/vnd.ms-excel".

Any help with this would be greatly appreciated. Thanks.

OTM2 uses a copy+pasted version of _sanitize_unicode_record to export JSON values. Consider unit testing this function and publicly exporting it.

Hi,

Is it possible to render a csv with post data from a form? It looks like the view is using a GET response, but is there a way to combine the two?

This is a public function and part of the API - it needs documentation and an example of how to use it.

export csv has  on the first column, looks like a pretty common issue

It only happened inside the first colomn description

The filename should be able to be called anything.

Related, the _append_datestamp function should be updated to use os.splitext instead of just truncating a fixed length from the end of the filename.

Models:
class Model():
field1...
field2...
...

class ModelB():
fk(Model)...
field3....
field4....
...
class ModelC():
fk(Model)
field5
field6
...

For example:
data = Model.objects.all()

for obj in data:
ModelReverse = obj.modelb_set.all()
OtherModel = obj.modelc_set.all()
etc .....

is posible output in only single file???
similar this:

field1,field2,field3,field4,field5,field6

Hii, there I believe this is a really nicely done work and it is going to help in my work but don't you think we can also have a reverse of this which imports the data from CSV into the database. If you have some thoughts on it let me know, I am more than happy to contribute.

I pass in the name of an event object to the filename keyword in render_to_csv_response(). If the title attribute of the event contains a . then I get the error that the file extension must be .csv.

I can just do a event.title.replace('.', '') before passing it in, but I am wondering why not include this line in the _validate_and_clean_filename() function to sanitize the input?

I can submit a PR for it if you are ok with the idea

Can i choose delimiter?

I want you to take title and get Título

Anexo.csv:

ID,Título,fichero,orden
1,Maniobras de reanimación. Soporte vital básico con y sin desfibrilador,anexos/a1.pdf,0

views.py:

class Anexo(models.Model):
    # Anexo Fields
    title = models.CharField(_(u'Título'), max_length=255)
    fichero = models.FileField(upload_to="anexos")
    orden = models.PositiveSmallIntegerField(default=0, blank=False, null=False)

    def __unicode__(self):
        return strip_tags(self.title)

The documentation has lagged behind the code. New features should be documented.

Django 1.9 got rid of ValuesQuerySet which causes errors when djqscsv attempts to import it.

Hi azavea:

First of all, thanks for the work, it really made my life easier! However, I still wonder if you guys have any plans to support m2m fields. I know it's tricky because django's Model.objects.values() doesn't play very well with m2m fields, and it's also tricky to handle which field in the m2m model you want to use for display. If you guys have some thoughts about it, like putting together the unicode for each m2m entry, etc, please let me know. Thanks!

Leaning too heavily on StringIO to mock a csv file. There needs to be an easy way to produce CSVs from this repo, so that they can be poked at in various CSV editors. This can either be done in the unit tests, or with views in the test_app.

https://github.com/azavea/django-queryset-csv/blob/master/djqscsv/__init__.py#L74 should be:

response['Content-Disposition'] = 'attachment; filename="%s";' % filename_param

My queryset has fields in a .extra clause. I want the exported CSV to use a different column name for those fields. But if I specify that renaming via field_header_map it doesn't work--the column header uses the original field name rather than the name I specified.