Coder Social home page Coder Social logo

pyexcel / pyexcel Goto Github PK

View Code? Open in Web Editor NEW
1.2K 46.0 166.0 4.01 MB

Single API for reading, manipulating and writing data in csv, ods, xls, xlsx and xlsm files

Home Page: http://docs.pyexcel.org

License: Other

Makefile 0.04% Shell 0.16% Python 99.70% Batchfile 0.10%
python excel csv xls ods xlsx xlsm fods

pyexcel's Introduction

pyexcel - Let you focus on data, instead of file formats

image

image

image

image

image

image

image

image

image

image

image

image

Support the project

If your company has embedded pyexcel and its components into a revenue generating product, please support me on github, patreon or bounty source to maintain the project and develop it further.

If you are an individual, you are welcome to support me too and for however long you feel like. As my backer, you will receive early access to pyexcel related contents.

And your issues will get prioritized if you would like to become my patreon as pyexcel pro user.

With your financial support, I will be able to invest a little bit more time in coding, documentation and writing interesting posts.

Known constraints

Fonts, colors and charts are not supported.

Nor to read password protected xls, xlsx and ods files.

Introduction

Feature Highlights

A list of supported file formats
file format definition
csv comma separated values
tsv tab separated values
csvz a zip file that contains one or many csv files
tsvz a zip file that contains one or many tsv files

xls

a spreadsheet file format created by MS-Excel 97-2003

xlsx

MS-Excel Extensions to the Office Open XML SpreadsheetML File Format.

xlsm an MS-Excel Macro-Enabled Workbook file
ods open document spreadsheet
fods flat open document spreadsheet
json java script object notation
html html table of the data structure
simple simple presentation
rst rStructured Text presentation of the data
mediawiki media wiki table

image

  1. One application programming interface(API) to handle multiple data sources:
    • physical file
    • memory file
    • SQLAlchemy table
    • Django Model
    • Python data structures: dictionary, records and array
  2. One API to read and write data in various excel file formats.
  3. For large data sets, data streaming are supported. A genenerator can be returned to you. Checkout iget_records, iget_array, isave_as and isave_book_as.

Installation

You can install pyexcel via pip:

$ pip install pyexcel

or clone it and install it:

$ git clone https://github.com/pyexcel/pyexcel.git
$ cd pyexcel
$ python setup.py install

One liners

This section shows you how to get data from your excel files and how to export data to excel files in one line

Read from the excel files

Get a list of dictionaries

Suppose you want to process History of Classical Music:

History of Classical Music:

Name Period Representative Composers
Medieval c.1150-c.1400 Machaut, Landini
Renaissance c.1400-c.1600 Gibbons, Frescobaldi
Baroque c.1600-c.1750 JS Bach, Vivaldi
Classical c.1750-c.1830 Joseph Haydn, Wolfgan Amadeus Mozart
Early Romantic c.1830-c.1860 Chopin, Mendelssohn, Schumann, Liszt
Late Romantic c.1860-c.1920 Wagner,Verdi
Modernist 20th century Sergei Rachmaninoff,Calude Debussy

Let's get a list of dictionary out from the xls file:

>>> records = p.get_records(file_name="your_file.xls")

And let's check what do we have:

>>> for row in records:
...     print(f"{row['Representative Composers']} are from {row['Name']} period ({row['Period']})")
Machaut, Landini are from Medieval period (c.1150-c.1400)
Gibbons, Frescobaldi are from Renaissance period (c.1400-c.1600)
JS Bach, Vivaldi are from Baroque period (c.1600-c.1750)
Joseph Haydn, Wolfgan Amadeus Mozart are from Classical period (c.1750-c.1830)
Chopin, Mendelssohn, Schumann, Liszt are from Early Romantic period (c.1830-c.1860)
Wagner,Verdi are from Late Romantic period (c.1860-c.1920)
Sergei Rachmaninoff,Calude Debussy are from Modernist period (20th century)

Get two dimensional array

Instead, what if you have to use pyexcel.get_array to do the same:

>>> for row in p.get_array(file_name="your_file.xls", start_row=1):
...     print(f"{row[2]} are from {row[0]} period ({row[1]})")
Machaut, Landini are from Medieval period (c.1150-c.1400)
Gibbons, Frescobaldi are from Renaissance period (c.1400-c.1600)
JS Bach, Vivaldi are from Baroque period (c.1600-c.1750)
Joseph Haydn, Wolfgan Amadeus Mozart are from Classical period (c.1750-c.1830)
Chopin, Mendelssohn, Schumann, Liszt are from Early Romantic period (c.1830-c.1860)
Wagner,Verdi are from Late Romantic period (c.1860-c.1920)
Sergei Rachmaninoff,Calude Debussy are from Modernist period (20th century)

where start_row skips the header row.

Get a dictionary

You can get a dictionary too:

>>> my_dict = p.get_dict(file_name="your_file.xls", name_columns_by_row=0)

And let's have a look inside:

>>> from pyexcel._compact import OrderedDict
>>> isinstance(my_dict, OrderedDict)
True
>>> for key, values in my_dict.items():
...     print(key + " : " + ','.join([str(item) for item in values]))
Name : Medieval,Renaissance,Baroque,Classical,Early Romantic,Late Romantic,Modernist
Period : c.1150-c.1400,c.1400-c.1600,c.1600-c.1750,c.1750-c.1830,c.1830-c.1860,c.1860-c.1920,20th century
Representative Composers : Machaut, Landini,Gibbons, Frescobaldi,JS Bach, Vivaldi,Joseph Haydn, Wolfgan Amadeus Mozart,Chopin, Mendelssohn, Schumann, Liszt,Wagner,Verdi,Sergei Rachmaninoff,Calude Debussy

Please note that my_dict is an OrderedDict.

Get a dictionary of two dimensional array

Suppose you have a multiple sheet book as the following:

Top Violinist:

Name Period Nationality
Antonio Vivaldi 1678-1741 Italian
Niccolo Paganini 1782-1840 Italian
Pablo de Sarasate 1852-1904 Spainish
Eugene Ysaye 1858-1931 Belgian
Fritz Kreisler 1875-1962 Astria-American
Jascha Heifetz 1901-1987 Russian-American
David Oistrakh 1908-1974 Russian
Yehundi Menuhin 1916-1999 American
Itzhak Perlman 1945- Israeli-American
Hilary Hahn 1979- American

Noteable Violin Makers:

Maker Period Country
Antonio Stradivari 1644-1737 Cremona, Italy
Giovanni Paolo Maggini 1580-1630 Botticino, Italy
Amati Family 1500-1740 Cremona, Italy
Guarneri Family 1626-1744 Cremona, Italy
Rugeri Family 1628-1719 Cremona, Italy
Carlo Bergonzi 1683-1747 Cremona, Italy
Jacob Stainer 1617-1683 Austria

Most Expensive Violins:

Name Estimated Value Location
Messiah Stradivarious $ 20,000,000 Ashmolean Museum in Oxford, England
Vieuxtemps Guarneri $ 16,000,000 On loan to Anne Akiko Meyers
Lady Blunt $ 15,900,000 Anonymous bidder

Here is the code to obtain those sheets as a single dictionary:

>>> book_dict = p.get_book_dict(file_name="book.xls")

And check:

>>> isinstance(book_dict, OrderedDict)
True
>>> import json
>>> for key, item in book_dict.items():
...     print(json.dumps({key: item}))
{"Most Expensive Violins": [["Name", "Estimated Value", "Location"], ["Messiah Stradivarious", "$ 20,000,000", "Ashmolean Museum in Oxford, England"], ["Vieuxtemps Guarneri", "$ 16,000,000", "On loan to Anne Akiko Meyers"], ["Lady Blunt", "$ 15,900,000", "Anonymous bidder"]]}
{"Noteable Violin Makers": [["Maker", "Period", "Country"], ["Antonio Stradivari", "1644-1737", "Cremona, Italy"], ["Giovanni Paolo Maggini", "1580-1630", "Botticino, Italy"], ["Amati Family", "1500-1740", "Cremona, Italy"], ["Guarneri Family", "1626-1744", "Cremona, Italy"], ["Rugeri Family", "1628-1719", "Cremona, Italy"], ["Carlo Bergonzi", "1683-1747", "Cremona, Italy"], ["Jacob Stainer", "1617-1683", "Austria"]]}
{"Top Violinist": [["Name", "Period", "Nationality"], ["Antonio Vivaldi", "1678-1741", "Italian"], ["Niccolo Paganini", "1782-1840", "Italian"], ["Pablo de Sarasate", "1852-1904", "Spainish"], ["Eugene Ysaye", "1858-1931", "Belgian"], ["Fritz Kreisler", "1875-1962", "Astria-American"], ["Jascha Heifetz", "1901-1987", "Russian-American"], ["David Oistrakh", "1908-1974", "Russian"], ["Yehundi Menuhin", "1916-1999", "American"], ["Itzhak Perlman", "1945-", "Israeli-American"], ["Hilary Hahn", "1979-", "American"]]}

Write data

Export an array

Suppose you have the following array:

>>> data = [['G', 'D', 'A', 'E'], ['Thomastik-Infield Domaints', 'Thomastik-Infield Domaints', 'Thomastik-Infield Domaints', 'Pirastro'], ['Silver wound', '', 'Aluminum wound', 'Gold Label Steel']]

And here is the code to save it as an excel file :

>>> p.save_as(array=data, dest_file_name="example.xls")

Let's verify it:

>>> p.get_sheet(file_name="example.xls")
pyexcel_sheet1:
+----------------------------+----------------------------+----------------------------+------------------+
| G                          | D                          | A                          | E                |
+----------------------------+----------------------------+----------------------------+------------------+
| Thomastik-Infield Domaints | Thomastik-Infield Domaints | Thomastik-Infield Domaints | Pirastro         |
+----------------------------+----------------------------+----------------------------+------------------+
| Silver wound               |                            | Aluminum wound             | Gold Label Steel |
+----------------------------+----------------------------+----------------------------+------------------+

And here is the code to save it as a csv file :

>>> p.save_as(array=data,
...           dest_file_name="example.csv",
...           dest_delimiter=':')

Let's verify it:

>>> with open("example.csv") as f:
...     for line in f.readlines():
...         print(line.rstrip())
...
G:D:A:E
Thomastik-Infield Domaints:Thomastik-Infield Domaints:Thomastik-Infield Domaints:Pirastro
Silver wound::Aluminum wound:Gold Label Steel

Export a list of dictionaries

>>> records = [
...     {"year": 1903, "country": "Germany", "speed": "206.7km/h"},
...     {"year": 1964, "country": "Japan", "speed": "210km/h"},
...     {"year": 2008, "country": "China", "speed": "350km/h"}
... ]
>>> p.save_as(records=records, dest_file_name='high_speed_rail.xls')

Export a dictionary of single key value pair

>>> henley_on_thames_facts = {
...     "area": "5.58 square meters",
...     "population": "11,619",
...     "civial parish": "Henley-on-Thames",
...     "latitude": "51.536",
...     "longitude": "-0.898"
... }
>>> p.save_as(adict=henley_on_thames_facts, dest_file_name='henley.xlsx')

Export a dictionary of single dimensonal array

>>> ccs_insights = {
...     "year": ["2017", "2018", "2019", "2020", "2021"],
...     "smart phones": [1.53, 1.64, 1.74, 1.82, 1.90],
...     "feature phones": [0.46, 0.38, 0.30, 0.23, 0.17]
... }
>>> p.save_as(adict=ccs_insights, dest_file_name='ccs.csv')

Export a dictionary of two dimensional array as a book

Suppose you want to save the below dictionary to an excel file :

>>> a_dictionary_of_two_dimensional_arrays = {
...      'Sheet 1':
...          [
...              [1.0, 2.0, 3.0],
...              [4.0, 5.0, 6.0],
...              [7.0, 8.0, 9.0]
...          ],
...      'Sheet 2':
...          [
...              ['X', 'Y', 'Z'],
...              [1.0, 2.0, 3.0],
...              [4.0, 5.0, 6.0]
...          ],
...      'Sheet 3':
...          [
...              ['O', 'P', 'Q'],
...              [3.0, 2.0, 1.0],
...              [4.0, 3.0, 2.0]
...          ]
...  }

Here is the code:

>>> p.save_book_as(
...    bookdict=a_dictionary_of_two_dimensional_arrays,
...    dest_file_name="book.xls"
... )

If you want to preserve the order of sheets in your dictionary, you have to pass on an ordered dictionary to the function itself. For example:

>>> data = OrderedDict()
>>> data.update({"Sheet 2": a_dictionary_of_two_dimensional_arrays['Sheet 2']})
>>> data.update({"Sheet 1": a_dictionary_of_two_dimensional_arrays['Sheet 1']})
>>> data.update({"Sheet 3": a_dictionary_of_two_dimensional_arrays['Sheet 3']})
>>> p.save_book_as(bookdict=data, dest_file_name="book.xls")

Let's verify its order:

>>> book_dict = p.get_book_dict(file_name="book.xls")
>>> for key, item in book_dict.items():
...     print(json.dumps({key: item}))
{"Sheet 2": [["X", "Y", "Z"], [1, 2, 3], [4, 5, 6]]}
{"Sheet 1": [[1, 2, 3], [4, 5, 6], [7, 8, 9]]}
{"Sheet 3": [["O", "P", "Q"], [3, 2, 1], [4, 3, 2]]}

Please notice that "Sheet 2" is the first item in the book_dict, meaning the order of sheets are preserved.

Transcoding

Note

Please note that pyexcel-cli can perform file transcoding at command line. No need to open your editor, save the problem, then python run.

The following code does a simple file format transcoding from xls to csv:

>>> p.save_as(file_name="birth.xls", dest_file_name="birth.csv")

Again it is really simple. Let's verify what we have gotten:

>>> sheet = p.get_sheet(file_name="birth.csv")
>>> sheet
birth.csv:
+-------+--------+----------+
| name  | weight | birth    |
+-------+--------+----------+
| Adam  | 3.4    | 03/02/15 |
+-------+--------+----------+
| Smith | 4.2    | 12/11/14 |
+-------+--------+----------+

Note

Please note that csv(comma separate value) file is pure text file. Formula, charts, images and formatting in xls file will disappear no matter which transcoding tool you use. Hence, pyexcel is a quick alternative for this transcoding job.

Let use previous example and save it as xlsx instead

>>> p.save_as(file_name="birth.xls",
...           dest_file_name="birth.xlsx") # change the file extension

Again let's verify what we have gotten:

>>> sheet = p.get_sheet(file_name="birth.xlsx")
>>> sheet
pyexcel_sheet1:
+-------+--------+----------+
| name  | weight | birth    |
+-------+--------+----------+
| Adam  | 3.4    | 03/02/15 |
+-------+--------+----------+
| Smith | 4.2    | 12/11/14 |
+-------+--------+----------+

Excel book merge and split operation in one line

Merge all excel files in directory into a book where each file become a sheet

The following code will merge every excel files into one file, say "output.xls":

from pyexcel.cookbook import merge_all_to_a_book
import glob


merge_all_to_a_book(glob.glob("your_csv_directory\*.csv"), "output.xls")

You can mix and match with other excel formats: xls, xlsm and ods. For example, if you are sure you have only xls, xlsm, xlsx, ods and csv files in your_excel_file_directory, you can do the following:

from pyexcel.cookbook import merge_all_to_a_book
import glob


merge_all_to_a_book(glob.glob("your_excel_file_directory\*.*"), "output.xls")

Split a book into single sheet files

Suppose you have many sheets in a work book and you would like to separate each into a single sheet excel file. You can easily do this:

>>> from pyexcel.cookbook import split_a_book
>>> split_a_book("megabook.xls", "output.xls")
>>> import glob
>>> outputfiles = glob.glob("*_output.xls")
>>> for file in sorted(outputfiles):
...     print(file)
...
Sheet 1_output.xls
Sheet 2_output.xls
Sheet 3_output.xls

for the output file, you can specify any of the supported formats

Extract just one sheet from a book

Suppose you just want to extract one sheet from many sheets that exists in a work book and you would like to separate it into a single sheet excel file. You can easily do this:

>>> from pyexcel.cookbook import extract_a_sheet_from_a_book
>>> extract_a_sheet_from_a_book("megabook.xls", "Sheet 1", "output.xls")
>>> if os.path.exists("Sheet 1_output.xls"):
...     print("Sheet 1_output.xls exists")
...
Sheet 1_output.xls exists

for the output file, you can specify any of the supported formats

Hidden feature: partial read

Most pyexcel users do not know, but other library users were requesting partial read

When you are dealing with huge amount of data, e.g. 64GB, obviously you would not like to fill up your memory with those data. What you may want to do is, record data from Nth line, take M records and stop. And you only want to use your memory for the M records, not for beginning part nor for the tail part.

Hence partial read feature is developed to read partial data into memory for processing.

You can paginate by row, by column and by both, hence you dictate what portion of the data to read back. But remember only row limit features help you save memory. Let's you use this feature to record data from Nth column, take M number of columns and skip the rest. You are not going to reduce your memory footprint.

Why did not I see above benefit?

This feature depends heavily on the implementation details.

pyexcel-xls (xlrd), pyexcel-xlsx (openpyxl), pyexcel-ods (odfpy) and pyexcel-ods3 (pyexcel-ezodf) will read all data into memory. Because xls, xlsx and ods file are effective a zipped folder, all four will unzip the folder and read the content in xml format in full, so as to make sense of all details.

Hence, during the partial data is been returned, the memory consumption won't differ from reading the whole data back. Only after the partial data is returned, the memory comsumption curve shall jump the cliff. So pagination code here only limits the data returned to your program.

With that said, pyexcel-xlsxr, pyexcel-odsr and pyexcel-htmlr DOES read partial data into memory. Those three are implemented in such a way that they consume the xml(html) when needed. When they have read designated portion of the data, they stop, even if they are half way through.

In addition, pyexcel's csv readers can read partial data into memory too.

Let's assume the following file is a huge csv file:

>>> import datetime
>>> import pyexcel as pe
>>> data = [
...     [1, 21, 31],
...     [2, 22, 32],
...     [3, 23, 33],
...     [4, 24, 34],
...     [5, 25, 35],
...     [6, 26, 36]
... ]
>>> pe.save_as(array=data, dest_file_name="your_file.csv")

And let's pretend to read partial data:

>>> pe.get_sheet(file_name="your_file.csv", start_row=2, row_limit=3)
your_file.csv:
+---+----+----+
| 3 | 23 | 33 |
+---+----+----+
| 4 | 24 | 34 |
+---+----+----+
| 5 | 25 | 35 |
+---+----+----+

And you could as well do the same for columns:

>>> pe.get_sheet(file_name="your_file.csv", start_column=1, column_limit=2)
your_file.csv:
+----+----+
| 21 | 31 |
+----+----+
| 22 | 32 |
+----+----+
| 23 | 33 |
+----+----+
| 24 | 34 |
+----+----+
| 25 | 35 |
+----+----+
| 26 | 36 |
+----+----+

Obvious, you could do both at the same time:

>>> pe.get_sheet(file_name="your_file.csv",
...     start_row=2, row_limit=3,
...     start_column=1, column_limit=2)
your_file.csv:
+----+----+
| 23 | 33 |
+----+----+
| 24 | 34 |
+----+----+
| 25 | 35 |
+----+----+

The pagination support is available across all pyexcel plugins.

Note

No column pagination support for query sets as data source.

Formatting while transcoding a big data file

If you are transcoding a big data set, conventional formatting method would not help unless a on-demand free RAM is available. However, there is a way to minimize the memory footprint of pyexcel while the formatting is performed.

Let's continue from previous example. Suppose we want to transcode "your_file.csv" to "your_file.xls" but increase each element by 1.

What we can do is to define a row renderer function as the following:

>>> def increment_by_one(row):
...     for element in row:
...         yield element + 1

Then pass it onto save_as function using row_renderer:

>>> pe.isave_as(file_name="your_file.csv",
...             row_renderer=increment_by_one,
...             dest_file_name="your_file.xlsx")

Note

If the data content is from a generator, isave_as has to be used.

We can verify if it was done correctly:

>>> pe.get_sheet(file_name="your_file.xlsx")
your_file.csv:
+---+----+----+
| 2 | 22 | 32 |
+---+----+----+
| 3 | 23 | 33 |
+---+----+----+
| 4 | 24 | 34 |
+---+----+----+
| 5 | 25 | 35 |
+---+----+----+
| 6 | 26 | 36 |
+---+----+----+
| 7 | 27 | 37 |
+---+----+----+

Stream APIs for big file : A set of two liners

When you are dealing with BIG excel files, you will want pyexcel to use constant memory.

This section shows you how to get data from your BIG excel files and how to export data to excel files in two lines at most, without eating all your computer memory.

Two liners for get data from big excel files

Get a list of dictionaries

Suppose you want to process the following coffee data again:

Top 5 coffeine drinks:

Coffees Serving Size Caffeine (mg)
Starbucks Coffee Blonde Roast venti(20 oz) 475
Dunkin' Donuts Coffee with Turbo Shot large(20 oz.) 398
Starbucks Coffee Pike Place Roast grande(16 oz.) 310
Panera Coffee Light Roast regular(16 oz.) 300

Let's get a list of dictionary out from the xls file:

>>> records = p.iget_records(file_name="your_file.xls")

And let's check what do we have:

>>> for r in records:
...     print(f"{r['Serving Size']} of {r['Coffees']} has {r['Caffeine (mg)']} mg")
venti(20 oz) of Starbucks Coffee Blonde Roast has 475 mg
large(20 oz.) of Dunkin' Donuts Coffee with Turbo Shot has 398 mg
grande(16 oz.) of Starbucks Coffee Pike Place Roast has 310 mg
regular(16 oz.) of Panera Coffee Light Roast has 300 mg

Please do not forgot the second line to close the opened file handle:

>>> p.free_resources()

Get two dimensional array

Instead, what if you have to use pyexcel.get_array to do the same:

>>> for row in p.iget_array(file_name="your_file.xls", start_row=1):
...     print(f"{row[1]} of {row[0]} has {row[2]} mg")
venti(20 oz) of Starbucks Coffee Blonde Roast has 475 mg
large(20 oz.) of Dunkin' Donuts Coffee with Turbo Shot has 398 mg
grande(16 oz.) of Starbucks Coffee Pike Place Roast has 310 mg
regular(16 oz.) of Panera Coffee Light Roast has 300 mg

Again, do not forgot the second line:

>>> p.free_resources()

where start_row skips the header row.

Data export in one liners

Export an array

Suppose you have the following array:

>>> data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

And here is the code to save it as an excel file :

>>> p.isave_as(array=data, dest_file_name="example.xls")

But the following line is not required because the data source are not file sources:

>>> # p.free_resources()

Let's verify it:

>>> p.get_sheet(file_name="example.xls")
pyexcel_sheet1:
+---+---+---+
| 1 | 2 | 3 |
+---+---+---+
| 4 | 5 | 6 |
+---+---+---+
| 7 | 8 | 9 |
+---+---+---+

And here is the code to save it as a csv file :

>>> p.isave_as(array=data,
...            dest_file_name="example.csv",
...            dest_delimiter=':')

Let's verify it:

>>> with open("example.csv") as f:
...     for line in f.readlines():
...         print(line.rstrip())
...
1:2:3
4:5:6
7:8:9

Export a list of dictionaries

>>> records = [
...     {"year": 1903, "country": "Germany", "speed": "206.7km/h"},
...     {"year": 1964, "country": "Japan", "speed": "210km/h"},
...     {"year": 2008, "country": "China", "speed": "350km/h"}
... ]
>>> p.isave_as(records=records, dest_file_name='high_speed_rail.xls')

Export a dictionary of single key value pair

>>> henley_on_thames_facts = {
...     "area": "5.58 square meters",
...     "population": "11,619",
...     "civial parish": "Henley-on-Thames",
...     "latitude": "51.536",
...     "longitude": "-0.898"
... }
>>> p.isave_as(adict=henley_on_thames_facts, dest_file_name='henley.xlsx')

Export a dictionary of single dimensonal array

>>> ccs_insights = {
...     "year": ["2017", "2018", "2019", "2020", "2021"],
...     "smart phones": [1.53, 1.64, 1.74, 1.82, 1.90],
...     "feature phones": [0.46, 0.38, 0.30, 0.23, 0.17]
... }
>>> p.isave_as(adict=ccs_insights, dest_file_name='ccs.csv')
>>> p.free_resources()

Export a dictionary of two dimensional array as a book

Suppose you want to save the below dictionary to an excel file :

>>> a_dictionary_of_two_dimensional_arrays = {
...      'Sheet 1':
...          [
...              [1.0, 2.0, 3.0],
...              [4.0, 5.0, 6.0],
...              [7.0, 8.0, 9.0]
...          ],
...      'Sheet 2':
...          [
...              ['X', 'Y', 'Z'],
...              [1.0, 2.0, 3.0],
...              [4.0, 5.0, 6.0]
...          ],
...      'Sheet 3':
...          [
...              ['O', 'P', 'Q'],
...              [3.0, 2.0, 1.0],
...              [4.0, 3.0, 2.0]
...          ]
...  }

Here is the code:

>>> p.isave_book_as(
...    bookdict=a_dictionary_of_two_dimensional_arrays,
...    dest_file_name="book.xls"
... )

If you want to preserve the order of sheets in your dictionary, you have to pass on an ordered dictionary to the function itself. For example:

>>> from pyexcel._compact import OrderedDict
>>> data = OrderedDict()
>>> data.update({"Sheet 2": a_dictionary_of_two_dimensional_arrays['Sheet 2']})
>>> data.update({"Sheet 1": a_dictionary_of_two_dimensional_arrays['Sheet 1']})
>>> data.update({"Sheet 3": a_dictionary_of_two_dimensional_arrays['Sheet 3']})
>>> p.isave_book_as(bookdict=data, dest_file_name="book.xls")
>>> p.free_resources()

Let's verify its order:

>>> import json
>>> book_dict = p.get_book_dict(file_name="book.xls")
>>> for key, item in book_dict.items():
...     print(json.dumps({key: item}))
{"Sheet 2": [["X", "Y", "Z"], [1, 2, 3], [4, 5, 6]]}
{"Sheet 1": [[1, 2, 3], [4, 5, 6], [7, 8, 9]]}
{"Sheet 3": [["O", "P", "Q"], [3, 2, 1], [4, 3, 2]]}

Please notice that "Sheet 2" is the first item in the book_dict, meaning the order of sheets are preserved.

File format transcoding on one line

Note

Please note that the following file transcoding could be with zero line. Please install pyexcel-cli and you will do the transcode in one command. No need to open your editor, save the problem, then python run.

The following code does a simple file format transcoding from xls to csv:

>>> import pyexcel
>>> p.save_as(file_name="birth.xls", dest_file_name="birth.csv")

Again it is really simple. Let's verify what we have gotten:

>>> sheet = p.get_sheet(file_name="birth.csv")
>>> sheet
birth.csv:
+-------+--------+----------+
| name  | weight | birth    |
+-------+--------+----------+
| Adam  | 3.4    | 03/02/15 |
+-------+--------+----------+
| Smith | 4.2    | 12/11/14 |
+-------+--------+----------+

Note

Please note that csv(comma separate value) file is pure text file. Formula, charts, images and formatting in xls file will disappear no matter which transcoding tool you use. Hence, pyexcel is a quick alternative for this transcoding job.

Let use previous example and save it as xlsx instead

>>> import pyexcel
>>> p.isave_as(file_name="birth.xls",
...            dest_file_name="birth.xlsx") # change the file extension

Again let's verify what we have gotten:

>>> sheet = p.get_sheet(file_name="birth.xlsx")
>>> sheet
pyexcel_sheet1:
+-------+--------+----------+
| name  | weight | birth    |
+-------+--------+----------+
| Adam  | 3.4    | 03/02/15 |
+-------+--------+----------+
| Smith | 4.2    | 12/11/14 |
+-------+--------+----------+

Available Plugins

A list of file formats supported by external plugins
Package name Supported file formats Dependencies

pyexcel-io

csv, csvz1, tsv, tsvz2

pyexcel-xls

xls, xlsx(read only), xlsm(read only)

xlrd, xlwt

pyexcel-xlsx xlsx openpyxl

pyexcel-ods3

ods

pyexcel-ezodf, lxml

pyexcel-ods ods odfpy
Dedicated file reader and writers
Package name Supported file formats Dependencies
pyexcel-xlsxw xlsx(write only) XlsxWriter
pyexcel-libxlsxw xlsx(write only) libxlsxwriter
pyexcel-xlsxr xlsx(read only) lxml
pyexcel-xlsbr xlsb(read only) pyxlsb
pyexcel-odsr read only for ods, fods lxml
pyexcel-odsw write only for ods loxun
pyexcel-htmlr html(read only) lxml,html5lib
pyexcel-pdfr pdf(read only) camelot

Plugin shopping guide

Since 2020, all pyexcel-io plugins have dropped the support for python versions which are lower than 3.6. If you want to use any of those Python versions, please use pyexcel-io and its plugins versions that are lower than 0.6.0.

Except csv files, xls, xlsx and ods files are a zip of a folder containing a lot of xml files

The dedicated readers for excel files can stream read

In order to manage the list of plugins installed, you need to use pip to add or remove a plugin. When you use virtualenv, you can have different plugins per virtual environment. In the situation where you have multiple plugins that does the same thing in your environment, you need to tell pyexcel which plugin to use per function call. For example, pyexcel-ods and pyexcel-odsr, and you want to get_array to use pyexcel-odsr. You need to append get_array(..., library='pyexcel-odsr').

Other data renderers
Package name Supported file formats Dependencies Python versions

pyexcel-text

write only:rst, mediawiki, html, latex, grid, pipe, orgtbl, plain simple read only: ndjson r/w: json

tabulate

2.6, 2.7, 3.3, 3.4 3.5, 3.6, pypy

pyexcel-handsontable handsontable in html handsontable same as above

pyexcel-pygal

svg chart

pygal

2.7, 3.3, 3.4, 3.5 3.6, pypy

pyexcel-sortable sortable table in html csvtotable same as above

pyexcel-gantt

gantt chart in html

frappe-gantt

except pypy, same as above

Footnotes

Acknowledgement

All great work have been done by odf, ezodf, xlrd, xlwt, tabulate and other individual developers. This library unites only the data access code.

License

New BSD License


  1. zipped csv file

  2. zipped tsv file

pyexcel's People

Contributors

akshayakrsh avatar andrealmar avatar arunkumar-ra avatar ayan-b avatar chfw avatar craiga avatar darylyu avatar garfunkel avatar hiaselhans avatar jayvdb avatar jnothman avatar julzhk avatar marksmayo avatar mathsyouth avatar nikolas avatar quis avatar rmzelle avatar svisser avatar timgates42 avatar wesleyacheng avatar williamjamir avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyexcel's Issues

Fail to save a book as a xls file after two sheets of the same name are added the book

Here's the example code:

import pyexcel as pe
import pyexcel.ext.xls
import glob
import os

merged = pe.Book()
for file in glob.glob(os.path.join("scattered-csv-files","*.csv")):
    sheet = pe.load(file)
    import pdb; pdb.set_trace()
    merged += sheet
merged.save_as("merged.xls")

Here is exception:
...
File "build/bdist.linux-i686/egg/xlwt/Workbook.py", line 348, in add_sheet
Exception: invalid worksheet name u'csv_a89107b6b1d911e4b5700019bb49856f'

texttable-0.8.1

Currently the minimum version of texttable is >=0.8.2 , but I cant find a bug in 0.8.1.

See two commits in https://github.com/jayvdb/pyexcel/commits/texttable-0.8.1 , and successful build at https://travis-ci.org/jayvdb/pyexcel/builds/126812754 .

If there are serious bugs in 0.8.1, we should add a test that fails on 0.8.1.

Debian sid and ubuntu trusty-yakkety all provide 0.8.1 as OS packages, so it would be good if we could use 0.8.1, or request that debian/ubuntu bump their version if there is a serious bug.

https://packages.debian.org/search?suite=sid&searchon=names&keywords=python-texttable
http://packages.ubuntu.com/trusty/python-texttable
http://packages.ubuntu.com/yakkety/python-texttable

Convert ODS to CSV: One empty column is added at the right

I tried the following code to convert a ods with python 2.7:

python
import pyexcel as pe
import pyexcel.ext.ods3
pyexcel.save_as(file_name=args.source, dest_file_name=args.destination)

The result has the following problem:

One empty column is added at the right.

This problem can be worked around as follows:

sheet = pe.get_sheet(file_name=args.source)
sheet.name_columns_by_row(0)
del sheet.column[""]
sheet.save_as(args.destination)

But why was this column there?

moban fails with IOError due to missing directory commons

It appears a directory commons, specified in .moban.yml, is missing.

pyexcel]$ moban 
Traceback (most recent call last):
  File "/usr/bin/moban", line 9, in <module>
    load_entry_point('moban==0.0.2', 'console_scripts', 'moban')()
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/main.py", line 36, in main
    handle_moban_file(options)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/main.py", line 93, in handle_moban_file
    handle_moban_file_v1(moban_file_configurations, options)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/mobanfile.py", line 23, in handle_moban_file_v1
    engine.render_to_files(list_of_templating_parameters)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/engine.py", line 42, in render_to_files
    self._render_with_finding_data_first(sta.data_file_index)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/engine.py", line 56, in _render_with_finding_data_first
    data = self.context.get_data(data_file)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/engine.py", line 68, in get_data
    return open_yaml(self.context_dirs, file_name)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/utils.py", line 45, in open_yaml
    data.pop(LABEL_OVERRIDES))
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/utils.py", line 37, in open_yaml
    the_yaml_file = search_file(base_dir, file_name)
  File "/usr/lib/python2.7/site-packages/moban-0.0.2-py2.7.egg/moban/utils.py", line 62, in search_file
    the_file))
IOError: Both pyexcel.yaml and commons/config/pyexcel.yaml does not exist

formula lost

i have posted this issue on bitbucket/openpyxl but the post was deemed invalid.

In pyexcel, there seems no data_only option in load_book(), so I can't read cell value with formula in some cells(not all the cells.) In my case. the cell in first row with formula can be read but following rows are not.

save_to_memory closes stream so you can't use it later

...
output = StringIO()
book = pe.Book({'hoja1':[['datos', 'de', 'prueba'],[1, 2, 3]], })
book.save_to_memory('csv', output)
logger.debug(output.getvalue())
This will raise an exception "I/O operation on closed file"

Changes Date Format Incorrectly

from pyexcel.ext import xls
from pyexcel.ext import xlsx

pyexcel.get_book(file_name='Data Issue.xls').save_as(folderpath + "CSV Input Files" + "Processing_" + newFileName + ".csv")

This is a bit of the code I use to convert a bunch of excel files of various formats to csvs. The attached file is the xls file that I try to convert and contains a column with dates in a date format. The date is 06/12/16 and this library changes it to 06/11/2012 for some reason.

Github won't allow me to upload a .xls so I have attached the xlsx file. To replicate the issue, save the file as a .xls. The problem will still occur.
Date Issue.xlsx

Any idea why this could be happening? Potentially it is that I import the xlsx after the xls library.

Chinese character not supported.

In [18]: array = [['这', '是', '中', '文'], ['这', '是', '中', '文']]

In [19]: pyexcel.Sheet(array)

Out[19]: ---------------------------------------------------------------------------
UnicodeEncodeError Traceback (most recent call last)
...
UnicodeEncodeError: 'ascii' codec can't encode characters in position 58-61: ordinal not in range(128)

save_as from xlsx to csv: error array index out of range

#!/usr/bin/env python
import pyexcel
import os
from pyexcel.ext import xlsx
from pyexcel.ext import xls

pyexcel.save_as(file_name="Test2.xlsx", dest_file_name="Test2.csv")

When I tried to use the code above to save the attached file as a csv file in python 3, the following error occurred:
IndexError: array index out of range
Test2.xlsx

Support for hidden sheets

I was just wondering if support for hidden sheets was included or planned in pyexcel. I did a search on the source code for "hidden" and found nothing. Thanks!

Import error pyexcel_chart

I got this exception, pyexcel installed via pip. Thanks.

Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import pyexcel
Failed to import pyexcel_chart due to %s
Traceback (most recent call last):
File "C:\Users\myuser\Envs\myvirtualenv\lib\site-packages\pyexcel\renderers__init__.py", line 19, in
import pyexcel_chart
ImportError: No module named 'pyexcel_chart'

no sheet can have more than 26 columns in a xls file

if one xls file has more than 26 columns in a sheet, the data from column 'AA' will be lost when I use function load() and save_as()

sample code:

import pyexcel as pe
import pyexcel.ext.xls # import it to be able handle xls file
import pyexcel.ext.xlsx # xlsx fi

book=pe.load('test1.xls')
book.save_as("test1.xlsx")

book2=pe.load('test2.xls')
book2.save_as("test2.xlsx")

tests fail on master

There are now 25 tests failing on master, using the same commit (b6fd56e) as previously passed.

I'm not sure what the cause is, as the new builds have the same frozen list as the old list; which is also identical on py27 and py33:

codecov==2.0.3
cov-core==1.15.0
coverage==4.0.3
ezodf==0.3.2
Flask==0.10.1
future==0.15.2
glob2==0.4.1
itsdangerous==0.24
jdcal==1.2
Jinja2==2.8
lxml==3.6.0
MarkupSafe==0.23
mock==1.0.1
nose==1.3.4
nose-cov==1.6
numpy==1.9.1
openpyxl==2.2.2
py==1.4.26
pyexcel==0.2.1
pyexcel-io==0.2.0
pyexcel-ods3==0.2.0
pyexcel-text==0.2.0
pyexcel-xls==0.2.0
pyexcel-xlsx==0.2.0
pytest==2.6.4
requests==2.10.0
SQLAlchemy==1.0.12
tabulate==0.7.5
texttable==0.8.4
Werkzeug==0.11.9
wheel==0.24.0
xlrd==0.9.4
xlwt-future==0.8.0

The "Build system information" details looks the same, including LANG variables, etc.

Unicode in column headers

I encountered the following error when opening an xlsx file, containing 'Prénom' and 'Numéro National' as column headers:

>>> import pyexcel as pe
>>> records = pe.get_records(file_name='Afganistan.xlsx')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pyexcel/core.py", line 317, in get_records
    **keywords)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel/core.py", line 71, in get_sheet
    sheet = Sheet(named_content.payload, named_content.name, **sheet_params)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel/sheets/nominablesheet.py", line 469, in __init__
    self.name_columns_by_row(name_columns_by_row)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel/sheets/nominablesheet.py", line 538, in name_columns_by_row
    self._column_names = make_names_unique(self.row_at(row_index))
  File "/usr/local/lib/python2.7/dist-packages/pyexcel/sheets/nominablesheet.py", line 55, in make_names_unique
    new_names.append(str(item))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 2: ordinal not in range(128)

Obviously, line 55 there chokes on any non-ascii characters. Since I only need the library for this particular excel file at the moment, I simply removed the str() call since I knew it would not cause issues, but a more solid solution could be needed (I don't know all the types item could be at that point).

Sheet name not honored when saved

Hello,

If a single sheet is created and then saved with save_as method, the name of the sheet is not saved in the final file.

import pyexcel as pe
import pyexcel.ext.xls
import pyexcel.ext.xlsx
import pyexcel.ext.ods

content = [
    ['Col1', 'Col2', 'Col3'],
    [1, 2, 3],
    [3, 2, 1]
]

sheet = pe.Sheet(content, name="Myname")
sheet.save_as("mysheet.xls")
sheet.save_as("mysheet.xlsx")
sheet.save_as("mysheet.ods")
sheet.save_as("mysheet.csv")

Quick fix could be to pass the sheet name as a parameter when creating the Writer in save_as method. I have a commit ready with this change if you agree with the solution.
It includes the following changes:

diff --git a/pyexcel/sheets/sheet.py b/pyexcel/sheets/sheet.py
index bd655c3..33a3ebd 100644
--- a/pyexcel/sheets/sheet.py
+++ b/pyexcel/sheets/sheet.py
@@ -100,7 +100,7 @@ class Sheet(NominableSheet):
     def save_as(self, filename, **keywords):
         """Save the content to a named file"""
         from ..writers import Writer
-        w = Writer(filename, **keywords)
+        w = Writer(filename, sheet_name=self.name, **keywords)
         w.write_reader(self)
         w.close()

diff --git a/pyexcel/writers.py b/pyexcel/writers.py
index 3c4ee7f..83bcc91 100644
--- a/pyexcel/writers.py
+++ b/pyexcel/writers.py
@@ -174,13 +174,13 @@ class Writer(SheetWriter):
     of the data files
     """

-    def __init__(self, file, **keywords):
+    def __init__(self, file, sheet_name=None, **keywords):
         """Constructor for single sheet writer

         This class creates only one sheet writer and stick with it
         """
         self.bookwriter = BookWriter(file, **keywords)
-        self.writer = self.bookwriter.create_sheet(None).writer
+        self.writer = self.bookwriter.create_sheet(sheet_name).writer

     def close(self):
         """

Best regards,
Andreas

Use OrderedDict to maintain roder of keys

This is not so much an issue as it is a question for the Devs. Im new to Github and could not see another way to ask these questions. Please forgive me if this is improper use.

Use OrderedDict to maintain oder of keys when saving sheet. Is it possible?

As a side question I believe the license of your software Pyexcel is New BSD. What if anything would I have to do to satisfy those requirements to use your product in my software? Is I understand it, I would have to include the New BSD license in the application and the ReadMe file? Is their anything else that I need to do to fill any requirements you might have.

Thank you

Different displays between when opening the file and directly in v0.2.1

I remarked something strange in this version 0.2.1.

When we put a number as an int, if we print the table, it displays the number as an int.

But, if we save the file then reopenning it, displays it as a float. It is strange.

Look at the following code:

>>> import pyexcel as pe
>>> import pyexcel.ext.ods3
>>> a=[[60, 50.0, '120']]
>>> s=pyexcel.get_sheet(array=a)
>>> s
Sheet Name: pyexcel_sheet1
+----+------+-----+
| 60 | 50.0 | 120 |
+----+------+-----+
>>> s.save_as("sr.ods")
>>> o = pe.get_sheet(file_name="sr.ods")
>>> o
Sheet Name: pyexcel_sheet1
+------+------+-----+
| 60.0 | 50.0 | 120 |
+------+------+-----+
>>> 

Before saving into a file 60 is displayed as an int, and after saving into a file it is displayed as a float (60.0).

Even if the format is a string it is displayed as a float

This a bug of display and very annoying, since even if a number (like serial number for example) is wirtten as a text into a table it is still displayed as a float,

Example:

'>>> ReportWorkbook = pe.get_book(file_name=rep_name)
'>>> ReportWorkbook['Infos']
Sheet Name: Infos
+-----------------+------------------+
| Date | 2016-03-31 10:59 |
+-----------------+------------------+
| Model name | XS360_EU |
+-----------------+------------------+
| Time by X | 60 |
+-----------------+------------------+
| Product A | 123 |
+-----------------+------------------+
| Product B | 4.567e+15 |
+-----------------+------------------+
| Bandwidth (UDP) | N/A |
+-----------------+------------------+
| Other infos | N/A |
+-----------------+------------------+
'>>> ReportWorkbook['Infos'].column[1]
['2016-03-31 10:59', 'XS360_EU', 60.0, '0123', '04566651561653122', 'N/A', 'N/A']

As seen above the serial numbers of products A and B are 0123 and 04566651561653122 are stocked as a text (string), however. Whend displayed with print they are displayed as a float so we se the first '0' disappear and 4.567e+15 insetead of 04566651561653122

numeric data with some user formated style in a xls file will lost

ie:
a cell has a numeric value with formated stype "yyyy/m/d h:mm" in a xls file

like:
image

with following code and the data is lost:

import pyexcel as pe
import pyexcel.ext.xls # import it to be able handle xls file
import pyexcel.ext.xlsx # xlsx fi

book=pe.load('test3.xls')
book.save_as("test3.xlsx")

Line termination for CSV output incorrect/can not be set

I tried the following code to convert a ods with python 2.7:

import pyexcel as pe
import pyexcel.ext.ods3
pyexcel.save_as(file_name=args.source, dest_file_name=args.destination)

The result has the following problem:

The result has the windows style line termination CR LF. I would expect that the result follows the system default. I understand that this is a problem with the csv module, but there you can set the "lineterminator" - it would be good to allow that here, too, and document this feature.

Problems saving UTF-8 characters

We are trying to make this code work using special characters:

from pyexcel_ods import save_data
data = {}
data.update({"Sheet 1": [[1, 2, 3], [4, 5, 6]]})
data.update({"Sheet 2": [[u"row 1", u"Héllô!", u"HolÁ!"]]})
save_data("your_file.ods", data)

Unfortunately, this throws the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_ods/__init__.py", line 296, in save_data
    write_data(afile, data, file_type=file_type, **keywords)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_io/__init__.py", line 256, in store_data
    writer.write(data)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_io/base.py", line 258, in write
    sheet.write_array(sheet_dicts[name])
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_io/base.py", line 222, in write_array
    self.write_row(r)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_ods/__init__.py", line 257, in write_row
    self.write_cell(tr, x)
  File "/usr/local/lib/python2.7/dist-packages/pyexcel_ods/__init__.py", line 246, in write_cell
    tc.setAttrNS(OFFICENS, x_odf_value_token, x)
  File "/usr/local/lib/python2.7/dist-packages/odf/element.py", line 440, in setAttrNS
    self.attributes[(namespace, localpart)] = c.convert((namespace, localpart), value, self)
  File "/usr/local/lib/python2.7/dist-packages/odf/attrconverters.py", line 1488, in convert
    return conversion(attribute, value, element)
  File "/usr/local/lib/python2.7/dist-packages/odf/attrconverters.py", line 77, in cnv_double
    return str(arg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)

Is there something we are missing when writing down Unicode characters? This is important in lots of languages which have characters that go beyond basic ASCII.

CSV UTF-8 encoding

Hi!

First of great package, thank you!

I encountered a problem with csv UTF-8 encoding when converting a xlsx file to csv.

If I follow the recommended way from the docs:

import pyexcel
import pyexcel.ext.xlsx


pyexcel.save_as(file_name="example.xlsx", dest_file_name="exmaple.csv")

and the xlsx file is UTF-8, the dest_file is not. Even if I add encoding="UTF-8" nothing changes.

The only workaround I found is to use the CSV standard library:

import csv
import pyexcel
import pyexcel.ext.xlsx


sheet = pe.get_sheet(file_name="example.xlsx")
with open("example.csv", "w", newline="", encoding="UTF-8") as csvfile:
    writer = csv.writer(csvfile)
    for row in sheet:
        writer.writerow(row)
    csvfile.close()

Do not interpret numbers as float?

I tried the following code to convert a ods with python 2.7:

import pyexcel as pe
import pyexcel.ext.ods3
pyexcel.save_as(file_name=args.source, dest_file_name=args.destination)

The result has the following problem:

If a cell contains "1", then it is converted into "1.0". There is a nice example about formatters at https://pyexcel.readthedocs.org/en/latest/tutorial03.html#convert-a-column-of-numbers-to-strings, but it does not explain how to prevent the cell to become float. Maybe this is simply a documentation issue?

Remark: Yes, my spreadsheet contains cells with a number 1, which should in fact be strings - but entering a string "1" in a spreadsheet is way slower than just entering a number...

xls does not save itself as ods files do

Let's consider thoose two codes :

>>> import os, sys
>>> import pyexcel as pe
>>> import pyexcel.ext.ods3
>>> from pyexcel_ods3 import get_data, save_data
>>> al = pe.get_book(file_name="wb.ods")
>>> al
Sheet Name: Feuil1
+---+-------+
| A | B     |
+---+-------+
| 1 | alpha |
+---+-------+
| 2 | beta  |
+---+-------+
| 3 | gamma |
+---+-------+
>>> al.save_as("tst.ods")
>>> 

With the ods file everything is OK

And now this one:

>>> import os
>>> import sys
>>> import pyexcel as pe
>>> import pyexcel.ext.xls
>>> al = pyexcel.get_book(file_name="wb.xls")
>>> al
Sheet Name: Feuil1
+---+-------+
| A | B     |
+---+-------+
| 1 | alpha |
+---+-------+
| 2 | beta  |
+---+-------+
| 3 | gamma |
+---+-------+
>>> al.save_as("tst.xls")
Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    al.save_as("tst.xls")
etc, etc, etc....
AttributeError: 'dict_items' object has no attribute 'sort'

Why by using the same coding we does not have the same success, we have an error with xls

Issue installing from pip on Windows with 0.2.2

If this is not the right place to report this issue, let me know the right place. I am seeing issues installing the newest version of pyexcel on Windows 64bit and on OSX 64bit:

Collecting pyexcel
  Using cached pyexcel-0.2.2.zip
    Complete output from command python setup.py egg_info:
    error in pyexcel setup command: Invalid environment marker: platform_python_implementation=="PyPy"

I am not seeing this issue with 0.2.1 so my workaround is to stay on the previous version:

pip install pyexcel==0.2.1

only one sheet left when i open and then save a multi-sheet file

I have a 2 sheets excel file(no matter xls or xlsx). When I open it and then save it as a new file, only sheet 1 left in the new file.

Here is the code i use(i can provide the excel file if you need):

import pyexcel as pe
import pyexcel.ext.xls
import pyexcel.ext.xlsx

book=pe.load('test6.xls')
for sheet in book:
pass

book.save_as('test6rr.xls')

book=pe.load('test6.xlsx')
for sheet in book:
pass

book.save_as('test6rr.xlsx')

Encoding

When I used make_response_from_query_sets from a database, if some columns have encoding like iso-8859-1 is broken. There any way for set the enconding type?

The log is


Traceback (most recent call last):
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/enquesta/app.py", line 57, in operation_export
    file_name="operation_data")
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_webio/__init__.py", line 322, in make_response_from_query_sets
    dest_file_type=file_type, **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/core.py", line 161, in save_as
    return sources.save_sheet(sheet, **dest_keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/__init__.py", line 33, in save_sheet
    source.write_data(sheet)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/file_source_output.py", line 77, in write_data
    sheet, **self.keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/renderers/_excel.py", line 36, in render_sheet_to_stream
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 70, in save_data
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 92, in store_data
    writer.close()
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_xls/xls.py", line 243, in close
    self.work_book.save(self.file_alike_object)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 696, in save
    doc.save(filename_or_stream, self.get_biff_data())
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 660, in get_biff_data
    shared_str_table   = self.__sst_rec()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 622, in __sst_rec
    return self.__sst.get_biff_record()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 77, in get_biff_record
    self._add_to_sst(s)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 92, in _add_to_sst
    u_str = upack2(s, self.encoding)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/UnicodeUtils.py", line 50, in upack2
    us = unicode(s, encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 8: ordinal not in range(128)

Enhancement suggestion: The table could be displayed without sheet name

Hi,

The pyexcel display module is aboslutely sublime and useful not only for manipulating excel/ods/csv workbooks, but also to display a result in the console. However, the fact that we have sheet name keyword and its name every time near the table that we have to display is sometimes annoying, when we don't need it.

So I suggest to have an option or a new functionality, so that instead of having this (*)
2016-04-27_101732

We'll have this
2016-04-27_101733

And possibily an option or functionality to get this kind of display (instead of (*)):

2016-04-27_101732

IndexError: list index out of range for file with combined cells

The attached file test.ods.tar.gz contains a group of combined cells, as seen:

screenshot from 2016-10-14 21 28 15

Attempting to read it with

import pyexcel_ods as pe
pe.get_data('test.ods')

results in:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    pe.get_data('test.ods')
  File "/home/gabriel/.pyenv/versions/catalog-env/lib/python2.7/site-packages/pyexcel_ods/__init__.py", line 29, in get_data
    return read_data(afile, file_type=file_type, **keywords)
  File "/home/gabriel/.pyenv/versions/catalog-env/lib/python2.7/site-packages/pyexcel_io/io.py", line 34, in get_data
    data[key] = list(data[key])
  File "/home/gabriel/.pyenv/versions/catalog-env/lib/python2.7/site-packages/pyexcel_io/sheet.py", line 83, in to_array
    cell_value = self._cell_value(row, column)
  File "/home/gabriel/.pyenv/versions/catalog-env/lib/python2.7/site-packages/pyexcel_ods/ods.py", line 64, in _cell_value
    cell_value = row_cache[column]
IndexError: list index out of range

No moduled named pyexcel.ext.*

Hi, I can't import pyexcel.ext.plugin.
my pip freeze:
decorator==4.0.9
Django==1.8.12
django-constance==1.1.2
et-xmlfile==1.0.1
ipython==4.1.2
ipython-genutils==0.1.0
jdcal==1.2
openpyxl==2.3.4
path.py==8.1.2
pexpect==4.0.1
pickleshare==0.6
ptyprocess==0.5.1
pyexcel==0.2.0
pyexcel-io==0.1.0
redis==2.10.5
simplegeneric==0.8.1
texttable==0.8.4
traitlets==4.2.1
xlrd==0.9.4

`ImportError Traceback (most recent call last)
in ()
----> 1 import pyexcel.ext.plugin

/home/gf/.virtualenvs/me-app-env/local/lib/python2.7/site-packages/pyexcel/exthook.pyc in load_module(self, fullname)
115 setattr(sys.modules[self.wrapper_module], modname, module)
116 return module
--> 117 raise ImportError('No module named %s' % fullname)
118
119 def is_important_traceback(self, important_module, tb):

ImportError: No module named pyexcel.ext.plugin
`

Travis is testing pyexcel-0.2.1

Before 0.2.1 was cut, Travis tests on master had the following results in pip freeze

pyexcel-io==0.2.0
pyexcel-ods3==0.2.0
pyexcel-xls==0.2.0
pyexcel-xlsx==0.2.0

https://travis-ci.org/pyexcel/pyexcel/jobs/126440816#L311 (87ec8c7)

After 0.2.1 was cut, Travis tests on master had the following results in pip freeze

pyexcel==0.2.1
pyexcel-io==0.2.0
pyexcel-ods3==0.2.0
pyexcel-text==0.2.0
pyexcel-xls==0.2.0
pyexcel-xlsx==0.2.0

https://travis-ci.org/pyexcel/pyexcel/jobs/126588417#L319 (badecbf)

The problem is that b014629 (Apr 29 2016) added pyexcel-text>=0.2.0 to tests/requirements.txt , and pyexcel-text 0.2.0 depends on pyexcel 2.1.0, so pip installs it from pypi before running the tests.

The very simple solution is for travis to re-install pyexcel[master] before running the tests.

However there is a larger problem of dependency management.

If we change the dependencies of pyexcel , pyexcel installed pyexcel-text[master], which
installed pyexcel>=0.2.0 (from pypi), the old dependencies of pyexcel>=0.2.0 (from pypi) are installed, which means we cant test on Travis with reduced/alternative dependencies, unless we forcibly uninstall them after rnd_requirements.txt and tests/requirements.txt.

An example of this is #38, where I want to downgrade texttable on Python 2. pyexcel>=0.2.0 (from pypi) will install texttable 0.8.2, whereas I want to install texttable>=0.8.1 . When I install texttable 0.8.1 using apt, pip upgrades it to texttable 0.8.2, so it wont be tested.

Another related problem is that master is not incompatible with pyexcel-io 0.1.0, yet the requirements still show pyexcel-io>=0.1.0 .

And there is one solution for all of those.
Firstly, to install pyexcel-text[master], we need to also install . at the same time. However bugs in pip/setuptools means that alone doesnt work. First we need to install everything in rnd_requirements.txt without dependencies. That installs the master of -text, -io, etc.
Then reprocess it with dependencies. Patch coming shortly...

csv columns lost

With the latest pypi versions, (0.2.0, io 0.1.0), the following CSV input looses all columns after "ISSN 1"

ERAID,Title,"Foreign Title","FoR 1","FoR 1 Name","FoR 2","FoR 2 Name","FoR 3","FoR 3 Name","ISSN 1","ISSN 2","ISSN 3","ISSN 4","ISSN 5","ISSN 6","ISSN 7"
125026,306090,,1201,Architecture,,,,,1536-1519,,,,,,,,

e.g. converting to html using text plugin

Sheet Name: test.csv
<table>
<tbody>
<tr><td>ERAID </td><td>Title </td><td>Foreign Title</td><td>FoR 1</td><td>FoR 1 Name  </td><td>FoR 2</td><td>FoR 2 Name</td><td>FoR 3</td><td>FoR 3 Name</td><td>ISSN 1   </td></tr>
<tr><td>125026</td><td>306090</td><td>             </td><td>1201 </td><td>Architecture</td><td>     </td><td>          </td><td>     </td><td>          </td><td>1536-1519</td></tr>
</tbody>
</table>

This bug doesnt affect the latest code in pyexcel and pyexcel-io , however when using master on these repos, the text plugin breaks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.