ho-nl / magento1-ho_import Goto Github PK

At this moment there is an array passed to all the methods, there isn't any actual raw XML passed.

I'd like to add some xpath-like, maybe css-style array parsing to it becomes very easy to select the element you need.

We for example have a feed of products. When products get removed from the feed it should delete/disable the products in Magento.

• The idea is that an import-source is responsible for the products it imports. We create a product/customer/category attribute ho_import_profile and a datetime field that marks the last import date ho_import_updated_at.
• We add an option to the import that allows you to delete products or disable products when they aren't available in the feed. Maybe this can be easily implemented using an event.
• Implement the delete functionality in a fast mannor (just run a DELETE query on the entity tables and let the foreign key contraints do the rest or is there another fast way to do this?)
• Run the delete indexer?

Hi

When trying to import media images during a product import, and using the dryrun option:
➜ php shell/hoimport.php -action import -profile product_import -dryrun 1
or
➜ php shell/hoimport.php -action csv -profile product_import -dryrun 1

We get this error:

09:59:41 - 16MB Dry run 16799 rows from temp csv file (var/import/product_import.csv)
PHP Fatal error:  Call to undefined method Mage_ImportExport_Model_Import_Adapter_Csv::unsetValue() in app/code/community/AvS/FastSimpleImport/Model/Import/Entity/Product.php on line 348
PHP Stack trace:
PHP   1. {main}() shell/hoimport.php:0
PHP   2. Ho_Import_Shell_Productimport->run() shell/hoimport.php:197
PHP   3. Ho_Import_Shell_Productimport->importAction() shell/hoimport.php:45
PHP   4. Ho_Import_Model_Import->process() shell/hoimport.php:101
PHP   5. Ho_Import_Model_Import->_dryRun() app/code/community/Ho/Import/Model/Import.php:96
PHP   6. AvS_FastSimpleImport_Model_Import->dryrunProductImport() app/code/community/Ho/Import/Model/Import.php:690
PHP   7. AvS_FastSimpleImport_Model_Import->validateSource() app/code/community/AvS/FastSimpleImport/Model/Import.php:154
PHP   8. Mage_ImportExport_Model_Import_Entity_Abstract->isDataValid() app/code/community/AvS/FastSimpleImport/Model/Import.php:506
PHP   9. AvS_FastSimpleImport_Model_Import_Entity_Product->validateData() app/code/core/Mage/ImportExport/Model/Import/Entity/Abstract.php:597
PHP  10. AvS_FastSimpleImport_Model_Import_Entity_Product->_preprocessImageData() app/code/community/AvS/FastSimpleImport/Model/Import/Entity/Product.php:159

This only happens when using dryrun, if you don't use it, the import works as expected.
I'm not sure if this is a bug in Ho_Import or in Avs_FastSimpleImport or in Mage_ImportExport though ...

Btw: I first got an error about the undefined method setValue for the _media_is_disabled and _media_position attributes on line 313 and 316 of the same file as the above, but I got around those by explicitly defining a value in the import profile for them.

The import always looks for fields in first tag in root element.

I have an XML like this;

<?xml version="1.0" encoding="UTF-8"?>
<channel xmlns:g="http://base.google.com/ns/1.0">
<title><![CDATA[Pricelist]]></title>
<description/>
<link>http://www.webshop.com</link>
<item>
... fields ...
</item>
<item>
... fields ...
</item>
<item>
... fields ...
</item>
<item>
... fields ...
</item>
</channel>

When I start the import, it gives me;
"Column names is empty or is not an array"

This is because it tries to look for the product data fields in the first tag it encounters, which is <title> and not <item>. Maybe adding a field that sets the name of the item that contains all the product data is an idea?

So te source tag becomes something like this;

<source model="ho_import/source_adapter_xml">
          <file>var/import/gxml-feed.xml</file>
          <rootNode>channel</rootNode>
          <productNode>item</productNode>
</source>

Hi

I just tried using Ho_Import v1.6.1 for a new import we are starting. I also use AvS_FastSimpleImport v0.6.4
But when trying to run an import, I get this error:

PHP Fatal error:  Call to undefined method AvS_FastSimpleImport_Model_Import_Entity_Product::getEntityBySku() in app/code/community/Ho/Import/Model/Observer.php on line 53

Now, I can see the method getEntityBySku was added in the develop branch of AvS_FastSimpleImport.
So what's the best thing to do right now?

1. Wait until AvS_FastSimpleImport gets a new stable release. How long will this take?
2. Use the develop branch of AvS_FastSimpleImport. Is it stable enough right now?
3. Go back to Ho_Import v1.5?
4. Something else?

Please advice :)

It's a follow-up to the last issue I created (#65) since it's been closed.

@paales I won't bother you too much with it since a manual copy of the orignal file to var/import/ is enough to make the script work, but from what I see there's no CSV file created automatically. No error logged in any log file or from the shell script output. Can you point me to the part of the script taking care of that? I then could make a little debug session to see what's wrong.

Cheers

Hi,

I'm currently using Ho_Import to import customers into a Magento instance and it works great! But there's something I don't understand.

I configured a profile named "import_customers", setting a CSV file as source :

<source model="ho_import/source_adapter_csv">
    <file>var/import/customers/customers.csv</file>
    <delimiter>,</delimiter>
    <enclosure>"</enclosure>
</source>

Then, when I run the profile from command line, here's what coming up :

$ php ../../shell/hoimport.php -action csv -profile import_customers
13MB Mapping source fields and saving to temp csv file (/home/user/httpdocs/var/import/import_customers.csv)
13MB Getting source adapter ho_import/source_adapter_csv
13MB Loading source file /home/user/httpdocs/var/import/customers/customers.csv
13MB Processing  rows from temp csv file (/home/user/httpdocs/var/import/import_customers.csv)
13MB Start profile /home/user/httpdocs/var/import/import_customers.csv

So the module wants me to have the CSV file at :

• /home/user/httpdocs/var/import/customers/customers.csv
• /home/user/httpdocs/var/import/import_customers.csv

Could you tell me why the module asks me to put the same file in two different locations? Is there a way to only keep the first path?

Anyway, thanks for all the hard work. Beside that little interrogation, the module is perfect, full of documentation, useful reports and highly flexible. To think that a few hours ago I was still banging my head with Dataflow...

I've been trying to get this working for the past couple days, and almost have it functioning properly, but have an issue. I am now getting the following error when I run the -dryrun:

Warning: array_combine(): Both parameters should have an equal number of elements in /Users/Graphics/Sites/Magento/app/code/community/Ho/Import/Model/Source/Adapter/Csv.php on line 220

I've tried everything I can think of with no luck. It seems as though it's not finding the first line in my CSV file after the header line. When I run a line -search command it cannot find the SKU in the first row, but it can find the SKU of the second row, third row, etc. When I run the -action line -profile profile_name command it shows the second row of data, not the first. I'm stumped.

I'm importing my products from my erp system and was wondering if it's possible to import multiple tier prices for one product? I'm using an CSV-file with this value as tier_price:
. It represents the quantity and discount percentage. I also can calculate the discount myself on generating the CSV-file.

But here is my question. How can I map this in my config xml?

Split on ***

10:-25% (or calculate price)
20:-30% (or calculate price)
50:-35% (or calculate price)

to this? 

<_tier_price_website/>
<_tier_price_customer_group/>
<_tier_price_qty/>
<_tier_price_price/>

I have the following code in the XML

                        <_media_attribute_id helper="ho_import/import::getAttributeId">
                            <attribute value="media_gallery"/>
                        </_media_attribute_id>

                        <_media_image helper="ho_import/import::getFieldSplit">
                            <field field="afbeeldingen"/>
                            <split>,</split>
                        </_media_image>

                        <_media_lable use="name"/> <!-- the spelling mistake is in the core, so we use it here -->
                        <_media_position value="0"/>
                        <_media_is_disabled value="0"/>

                        <image field="hoofdafbeelding"/>
                        <image_label use="name"/>

                        <thumbnail use="image"/>
                        <thumbnail_label use="name"/>

                        <small_image use="image"/>
                        <small_image_label use="name"/>

But i can't import more then one image. The Media attribute ID is not working for the second image.

Observer of Ho_Import prevents default Magento import to work in adminpanel:

PHP Fatal error:  Call to undefined method Mage_ImportExport_Model_Import_Entity_Product::getEntityBySku() in .../Ho_Import/app/code/community/Ho/Import/Model/Observer.php on line 53

Reason: $adapter in Ho_Import_Model_Observer::linkProductsToProfile() is supposed to be instance of AvS_FastSimpleImport_Model_Import_Entity_Product

Fix: Check if ($adapter instanceof AvS_FastSimpleImport_Model_Import_Entity_Product)

AOEpeople/Aoe_Scheduler#112

I'm using PHP 5.5 which throws that error/warning:
Deprecated functionality: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /Users/xxx/Sites/zookal-site/htdocs/lib/Zend/Filter/PregReplace.php on line 171

In class Ho_Import_Model_Import your are using class Zend_Filter_Word_UnderscoreToCamelCase which generates that error in class Zend_Filter_Word_SeparatorToCamelCase

Create a custom method without using the /e modifier?

I'm trying to get my product importer working again by using your module. My experience with the module is that the mapping is really easy to use, but really powerful as well.

Apparently I seem to have duplicate products in my feed, producing duplicate SKUs. I tried to use the getFieldCombine helper like this:

<sku helper="ho_import/import::getFieldCombine">
    <fields>
        <prefix field="sku"/>
        <ordercode field="bestelcode"/>
    </fields>
    <glue>-</glue>
</sku>

The sku's it generates are fine, but still there appear duplicates.

Now here's my actual question: What's the best way to deal with duplicate products in the feed within this module? Do I write my own helper to prevent it?

I'm not sure if this is a bug. Magento give this error after a successful Category-product relation import.

exception 'Mage_Core_Exception' with message 'Invalid entity_type specified: catalog_category_product' in /home/user/domains/domain.com/public_html/berto/app/Mage.php:595
#0 /home/user/domains/domain.com/public_html/berto/app/code/core/Mage/Eav/Model/Config.php(328): Mage::throwException('Invalid entity_...')
#1 /home/user/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Import.php(1005): Mage_Eav_Model_Config->getEntityType('catalog_categor...')
#2 /home/user/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Import.php(1091): Ho_Import_Model_Import->_getEntityTypeId()

From Ho Import function, row 1005

    protected function _getEntityTypeId()
    {
        /** @var Mage_Eav_Model_Entity_Type $entityType */
        $entityType = Mage::getSingleton('eav/config')->getEntityType((string) $this->_getEntityType());

        return $entityType->getId();
    }

Profile code in config.xml:

<!-- CATEGORY PRODUCT RELATION IMPORT -->
            <category_products>
                <entity_type>catalog_category_product</entity_type>
                <source model="ho_import/source_adapter_xml">
                    <file>var/import/source.xml</file>
                </source>
                <import_options>
                    <partial_indexing>1</partial_indexing>
                </import_options>
                <fieldmap>
                    <_root value="Default Category" />
                    <_category helper="ho_import/import::formatField">
                        <format>%s/%s</format>
                        <fields>
                            <father field="Categoria_x0020_Descrizione" />
                            <child field="Tipologia_x0020_Descrizione" />
                        </fields>
                    </_category>
                    <_sku helper="ho_import/import::formatField">
                        <format>%s-%s</format>
                        <fields>
                            <sku field="barcode" />
                            <size field="tg" />
                        </fields>
                    </_sku>
                    <position value="0" />
                </fieldmap>
            </category_products>

ho_import.log

2015-07-07T09:08:03+00:00 INFO (6): Processing 53 rows from temp csv file (/home/user/domains/domain.com/public_html/berto/var/import/category_products.csv)
2015-07-07T09:08:03+00:00 INFO (6): Start profile /home/user/domains/domain.com/public_html/berto/var/import/category_products.csv
2015-07-07T09:08:03+00:00 INFO (6): Profile category_products done in 0.27 seconds, 196.3 entities/s, 196.3 rows/s.
2015-07-07T09:08:03+00:00 INFO (6): Clean entity link table

Hi!
I have an error when trying the example:

I try to map some fields like mentioned in the example:

                    <gender helper="ho_import/import::getFieldMap">
                        <field field="Geschlecht"/>
                        <mapping>
                            <male from="M" to="male"/>
                            <female from="F" to="female"/>
                        </mapping>
                    </gender>

But now I get following error

Notice: Undefined index: @  in .../app/code/community/Ho/Import/Helper/Import.php on line 375

That is because Ho_Import_Model_Mapper::mapItem does not return the correct mapping. That happens in the Method mapItem in line 150:

$attributes = isset($fieldConfig['@']) ? $fieldConfig['@'] : array();

At this moment, fieldConfig has not the correct format. I appended an image how the array $fieldConfig looks like. So $fieldConfig['@'] does not exist.

The method finally returns null and so these values cannot get mapped. How to solve this?

Kind regards, Rokko

Hi,
I try to just use "store,websites,sku,qty" these fields to update the existing products stock.
When using it
And try "php hoimport.php -action line -profile products_profile " it told
| Invalid value for Attribute Set column (set does not exists?) | ERROR |
and
| Required attribute 'name' has an empty value | ERROR |
| Required attribute 'price' has an empty value | ERROR |
| Required attribute 'short_description' has an empty value | ERROR |
| Required attribute 'tax_class_id' has an empty value | ERROR |
+-----------------------------------------------------------+------------------+

I think it should not needed when update the stock only.
Or does Ho_Import only support create ,not support update products?

Would you please remove the dev-master entry in the required magento-hackathon/magento-composer-installer directive to allow also other versions of the magento-hackathon/magento-composer-installer besides the dev-master version :-)

I know you don't use composer ;-)

new composer.json:

   "require": {
      "magento-hackathon/magento-composer-installer": "*"
   }

Hi,

is there a way to import more than the 3 basic media (thumb,small, base) ?

Hi,

I was wondering if it's possible to import translations of a category or a product by "_store"? I'm building my csv dynamically like this:

when I use the fastsimpleimport/import it will scream if you don't have media_gallery set, which it is currently doing now. It would be handy that if it's not set and that it defaults with

Mage::getSingleton('catalog/product')->getResource()->getAttribute('media_gallery') ->getAttributeId();

Hi,

The function is_subscribed with a value of 0 or 1 is not working. Al customers that are imported get 0 als value for the is_subscribed.

            <fieldmap>
                <_website helper="ho_import/import::getAllWebsites"><limit>1</limit></_website>
                <group_id value="1"/>
                <email field="email"/>
                <firstname field="Voornaam" defaultvalue="ONBEKEND"/>
                <lastname field="Achternaam" required="1"/>
                <password field="wachtwoord" />

                <is_subscribed field="Nieuwsbrief"/>

            </fieldmap>

Regards,
Joost

When i use the following: command php hoimport.php -action line -profile profile_name
I get the following error: Warning: min(): Array must contain at least one element in /home/admin/domains/**/public_html/app/code/community/Ho/Import/Model/Source/Adapter/Xml.php on line 376

I'm looking into the field mapping output, there was the value _store not filled in. This was in the config file: <_store value="admin" required="1”/>

config file:

<?xml version="1.0"?>
<config>
    <modules>
        <Trinos_Import>
            <version>0.1.0</version>
        </Trinos_Import>
    </modules>
    <global>
        <helpers>
            <import>
                <class>Xml_Import_Trinos</class>
            </import>
        </helpers>

        <ho_import>

            <profile_name>

                <entity_type>catalog_product</entity_type>          

                <downloader model="ho_import/downloader_http">
                    <url>http://www.*****.xml</url>
                </downloader>

                <source model="ho_import/source_adapter_xml">
                    <file>/home/admin/domains/****/public_html/var/test.xml</file>
                </source>

                <import_options>
            <!--<continue_after_errors>1</continue_after_errors>-->
            <!--<ignore_duplicates>1</ignore_duplicates>-->
            <partial_indexing>1</partial_indexing>
            </import_options>


                <fieldmap>

                    <!-- GENERAL PRODUCT INFORMATION -->

                        <_store value="admin" required="1"/>
                        <_type value="simple"/>
                        <_attribute_set value="Default"/>
                        <_product_websites value="base"/>

                        <name field="name"/>


                        <description value="findReplaceniks"/>


                        <short_description field="description_short"/>
                        <status value="1"/>
                        <visibility value="4"/>
                        <sku field="productId" required="1"/>

                        <price helper="ho_import/import::parsePrice">
                            <pricefield field="price"/>
                        </price>

                        <tax_class_id value="2"/>


                    <!-- USER DEFINED FIELDS -->

                        <merk field="brand"/>

                    <!-- MEDIA FIELDS -->



                    <!-- INVENTORY -->
                         <qty field="stock"/>
                        <!--<min_qty value="0"/>-->
                        <!--<use_config_min_qty value="1"/>-->
                        <!--<is_qty_decimal value="0"/>-->
                        <!--<backorders value="1" />-->
                        <!--<use_config_backorders value="1"/>-->
                        <!--<min_sale_qty value="1"/>-->
                        <!--<use_config_min_sale_qty value="1"/>-->
                        <!--<max_sale_qty value="10000"/>-->
                        <!--<use_config_max_sale_qty value="1"/>-->
                        <is_in_stock value="yes"/>
                        <!--<notify_stock_qty value=""/>-->
                        <!--<use_config_notify_stock_qty value="1"/>-->
                        <!--<manage_stock value="1"/>-->
                        <!--<use_config_manage_stock value="1"/>-->
                        <!--<stock_status_changed_auto value="0"/>-->
                        <!--<qty_increments value="1"/>-->
                        <!--<use_config_qty_increments value="1"/>-->
                        <!--<enable_qty_increments value="0"/>-->
                        <!--<use_config_enable_qty_inc value="1" />-->
                        <!--<is_decimal_divided value="0"/>-->



                </fieldmap>

            </profile_name>

        </ho_import>

    </global>

</config>

I'm trying to import a configurable product.

I created config.xml similar to this.

CSV file has 3 lines: two for simple products and one for configurable product that is associated with these two products.

All products are imported properly, but they are not associated.

CSV that I used:

"sku","_type","color","short_description","description","price","name","tax_class","visibility","weight","category_ids","_super_attribute_code","_super_attribute_option","_super_attribute_price_corr"
"test1","simple","Black","Test 1","Test 1 Product",10,"Test 1 Product Name",0,4,,3,"color","Black",
"test2","simple","White","Test 2","Test 2 Product","123,99","Test 2 Product Name",0,,1,"3,4","color","White",
"test0","configurable",,"Test Conf","Test Conf",0,"Test Conf",0,4,0,3,"color",,

Is it possible to import configurable products with this extension? Is any field missing in CSV file that I used? Could you provide any sample configuration for import of configurable products?

I'm having an issue when running the shell. While running a -action line test I get tout: command not found AFTER Mapping 'profile':1.

Also, when running a -dryrun and actual import I get the following message:

Recoverable Error: Argument 1 passed to Ho_Import_Model_Mapper::setItem() must be an array, boolean given, called in /app/code/community/Ho/Import/Model/Import.php on line 457 and defined in /app/code/community/Ho/Import/Model/Mapper.php on line 46

Any pointers would be greatly appreciated.

I have a problem with importing around 8000 products. Get the following error:

SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`speel_db2`.`cataloginventory_stock_item`, CONSTRAINT `FK_CATINV_STOCK_ITEM_PRD_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`product_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE )

The first what i did was a clean install of a new Magento shop. But problem get back. This is the config.xml

<config>
  <modules>
    <Company_Import>
      <version>0.1.0</version>
    </Company_Import>
  </modules>
  <global>
    <helpers>
      <company_import>
        <class>Company_Import_Helper</class>
      </company_import>
    </helpers>
    <ho_import>
            <speel_toys>
                <entity_type>catalog_product</entity_type>

                <downloader model="ho_import/downloader_ftp">
                    <host>XXX</host>
                    <username>XXXX</username>
                    <password>XXXX</password>
                    <file>XXX.csv</file> 
                    <target>var/import</target>
                    <timeout>10</timeout> 
                    <passive>1</passive> 
                    <ssl>0</ssl>
                    <file_mode>1</file_mode>
                </downloader>


                <source model="ho_import/source_adapter_csv">
                     <file>var/import/XXX.csv</file>
                     <delimiter>;</delimiter>
                </source>

                <import_options>
                    <error_limit>50</error_limit>
                    <continue_after_errors>1</continue_after_errors>
                    <ignore_duplicates>1</ignore_duplicates>
                    <dropdown_attributes>
                        <country>merk</country>
                    </dropdown_attributes>
                </import_options>


                <fieldmap>

                    <_type value="simple"/>
                    <_attribute_set value="Default"/>
                    <_product_websites helper="ho_import/import::getAllWebsites"/>

                    <name field="9 Pocket pages"/>
                    <description field="9 Pocket pages"/>
                    <short_description field="9 Pocket pages"/>         
                    <sku field="9POC170100" required="1"/>

                    <price helper="ho_import/import::parsePrice">
                        <pricefield field="0,10"/>
                    </price>

                    <special_price helper="ho_import/import::parsePrice">
                        <special_pricefield field="0,25"/>
                    </special_price>

                    <image helper="ho_import/import::getMediaImage">
                        <imagefield field="9POC170100"/>
                        <limit>1</limit>
                        <filename field="9POC170100"/> 
                        <extension value="jpg"/>
                    </image>
                    <image_label field="9 Pocket pages"/>

                    <tax_class_id helper="ho_import/import::getFieldMap">
                       <value field="21%"/>
                       <mapping>
                           <hoogbtw from="21%" to="2"/>
                           <laagbtw from="6%" to="2"/>
                        </mapping>
                    </tax_class_id>

                    <_category value="Overige"/>

                    <merk field="Trading cards"/>

                    <qty value="100"/>
                    <is_in_stock helper="ho_import/import::getFieldBoolean">
                        <value field="100"/>
                    </is_in_stock>


                </fieldmap>

            </speel_toys>

        </ho_import>
  </global>
</config>

Hello,
It was not clear at all that I needed to set the default _attribute_set. I would have assumed since it was not doc'd that it would have just been set to the default attribute set. I did figured it out that you do need to add <_attribute_set value="Default"/> in the end.

It's also worth noting that the instructions should say that you must have all the required fields that AvS_FastSimpleImport has and that this doesn't provide a default if the node is not in the config, like the <_attribute_set value="Default"/> example. I ended up with

| Required attribute 'description' has an empty value       | ERROR     |
| Required attribute 'short_description' has an empty value | ERROR     |
| Required attribute 'media_gallery' has an empty value     | ERROR     |
| Required attribute 'tax_class_id' has an empty value      | ERROR     |

so it would have need good to have a note about this. Cheers and good stuff here.

In this pull requests it becomes possible to ignore fields while importing. This makes it possible to not empty

Explanation about the functionality here: avstudnitz/AvS_FastSimpleImport#118 (comment)

• Add an attribute to a field ignoreifempty.

In theory you could already use this by making use of the defaultvalue attribute, but that wont retrieve the actual ignore field value dynamically.

I'm working on an import that contains a URL to download the image from. I couldn't find a clear example of this, so I've tried a few different things. The closest I've gotten is the images download to the import folder correctly, but they are never imported into Magento. Here's my latest attempt:

<!-- MEDIA FIELDS -->
<_media_image helper="ho_import/import_product::getMediaImage"/>
<image helper="ho_import/import_product::getMediaImage"><image field="image"/></image>
<image_label use="name"/>
<small_image helper="ho_import/import_product::getMediaImage"><image field="small_image"/></small_image>
<small_image_label use="name"/>
<thumbnail helper="ho_import/import_product::getMediaImage"><image field="thumbnail"/></thumbnail>
 <thumbnail_label use="name"/>
<gallery/>
<media_gallery value="none"/>

The images appear as a URL in a column named 'image' in a CSV file. The rest of the CSV file imports fine. The URL is a valid URL - I'm able to download the images manually, and I did have at least one version of this XML that grabbed the images and then populated the image field with '/.jpg' as expected, but the image never actually appeared on the product.

Workaround for now is to change the chunkSize manually, need to make it configurable.

Currently we use an intermediate file when importing. It saves to CSV and that CSV is imported. If we could import without this immediate step we do not need to 'prepare' the import, we can just run it.

Advantages:

• Debug the actual source line in addition to the current imported line.
• Progress indicators
• No intermediate file, less possible encoding errors
• No writing to disk while importing

I get an error while trying to import a profile: Recoverable Error: Argument 1 passed to Ho_Import_Model_Source_Adapter_Spreadsheet::__construct() must be of the type array, string given

The problem lies in the combination of two things:

• Ho_Import_Model_Import::getSourceAdapter() passes a string with only the filename to the source adapter if there's only one argument
• Ho_Import_Model_Source_Adapter_Spreadsheet has a type hint on it's constructor: __construct(array $config). In the first line of the constructor, the type gets checked though and if it's a string it gets converted to an array with a 'file' key.

Currently I work around it by defining another dummy option to the <source> tag so there are multiple arguments to pass to the source adapter but that's not a very constructive solution :)

<source model="ho_import/source_adapter_spreadsheet">
    <file>var/import/products_language.xlsx</file>
    <foo>bar</foo>
</source>

I think the best and easiest way is to just remove the type hint from the constructor of Ho_Import_Model_Source_Adapter_Spreadsheet.

Class Ho_Import_Model_Source_Adapter_Db has to many unsetted parameters in the $config array which will be used for creating the DSN.

I can only connect to a PDO PGSQL source when using that modified constructor:

    /**
     * Constructor.
     * @param array $config
     */
    public function __construct(array $config)
    {
        $logHelper = Mage::helper('ho_import/log');
        $model = $config['model'];
        $query = $config['query'];
        //$config['readOnly'] = true;

        unset($config['model']);
        unset($config['query']);
        $config2 = $config;
        unset($config2['type']);
        unset($config2['pdoType']);

        /** @var Zend_Db_Adapter_Abstract $db */
        $db = new $model($config2);

        if (isset($config['limit']) || isset($condig['offset'])) {
            $limit  = (int) isset($config['limit']) ? $config['limit'] : 0;
            $offset = (int) isset($config['offset']) ? $config['offset'] : 0;
            $logHelper->log($logHelper->__('Setting limit to %s and offset to %s', $limit, $offset), Zend_Log::NOTICE);
            $query = $db->limit($query, $config['limit'], $offset);
            $logHelper->log($query, Zend_Log::DEBUG);
        }

        $logHelper->log('Fetching data...');
        $result = $db->fetchAll($query);
        $logHelper->log('Done');
        $config['data'] = &$result;

        return parent::__construct($config);
    }

So type, pdotype and readonly needs to be removed for connection to a DSN source.

I'm currently unsure why readOnly=true is at the top and where it is needed. Same with type and pdoType ... so therefore I haven't attached a PR.
Maybe you could have a look and fix that.

Cyrill

When the import is done by cronjob the base image, thumbnail and small image is not set. See screenshot:

When we do the import with SSH; the base image, thumbnail and small image is wel set right.

This is the part of the config for images.

                <_media_attribute_id helper="ho_import/import::getMediaAttributeId"/>
                <_media_image helper="trinos_import/data::getMediaImage">
                    <imagefield field="image_path"/>
                </_media_image>

                <_media_lable/>
                <_media_position value="0"/>
                <_media_is_disabled value="1"/>

                <image use="_media_image"/> 
                <image_lable use="name"/>

                <thumbnail use="_media_image"/>
                <thumbnail_lable use="name"/>

                <small_image use="_media_image"/>
                <small_image_lable use="name"/>

                <_media_lable use="name"/>
                <_media_position value="0"/>
                <_media_is_disabled value="1"/>

When we do a import from a XML we get the following error.

Warning: json_encode(): Invalid UTF-8 sequence in argument  in /home/admin/domains/speeltoys.nl/public_html/lib/Zend/Json.php on line 146

The CSV that the tool has made is Western (Windows Latin 1) encoding.

I see you use your own fork of the spreadsheet-reader. Not a bad idea. But could you update that repository? There's a bug in the currently forked version.

I got an exception while trying to load an Excel document: Warning: include(SpreadsheetReader/XLSX.php): failed to open stream: No such file or directory in /var/www/magentics/dlight.magentics.net/lib/Varien/Autoload.php.

This is caused by SpreadsheetReader (in /lib/spreadsheet-reader):

if (!class_exists('SpreadsheetReader_'.$Type))
{
    require(dirname(__FILE__).DIRECTORY_SEPARATOR.'SpreadsheetReader_'.$Type.'.php');
}

class_exists() by default tries to load the class if it doesn't exist. Because the file can't be autoloaded by Varien_Autoload, it generates a warning. This bug was fixed in the upstream version: nuovo/spreadsheet-reader@d311aee

Hi!

I'm not sure if I should see it as a bug or feature. It may be coded by design, but not keeping all usecases in mind.

$flatRow['_store'] = $store == 'admin' ? '' : $store;

It looks like the condition was meant for product imports, where you like to force it to admin and map storeview-data via <store_view>.
On the other hand, if you import customers, you may force whole records to go into store-views. Therefore you specify a "_store" column in you CSV-datasource. This it not possible, because it gets overriden by the mentioned statement above.

See Model/Import.php:593

                if ($flatRow) {
                    //if a column is required we add it here.
                    foreach ($allFieldConfig[$store] as $key => $column) {
                        if (!isset($flatRow[$key]) && isset($column['@']) && isset($column['@']['required'])) {
                            $flatRow[$key] = '';
                        }
                    }

                    $flatRow['_store'] = $store == 'admin' ? '' : $store;
                    $flattenedRows[]   = $flatRow;
                }

In my opinion it should read like:

                    if(!isset($flatRow['_store'])){
                        $flatRow['_store'] = $store == 'admin' ? '' : $store;
                    }
                    $flattenedRows[]   = $flatRow;

The only drawback I see, you can "override" the store for product imports in your datasource and this could mess things up.

Hi

We have an issue when trying to map the a category name where the to name contains an &

For example, we have this in our import definition:

<_category helper="ho_import/import::getFieldMap">
    <field field="CAT"/>
    <mapping>
        <TOPS from="TOPS" to="Default Category/Tops &amp; T-shirts/Tops"/>
    </mapping>
</_category>

But the & isn't converted to & which makes the importer complain, as it can't find the category:

➜ php hoimport.php -action line -profile product_import
11:45:50 - 22MB Getting source adapter ho_import/source_adapter_db
11:45:50 - 22MB Fetching data...
11:45:50 - 40MB Done
...
11:45:52 - 55MB
+--------------------------+-------------------------------------------+
| key                      | 1:1                                       |
+--------------------------+-------------------------------------------+
| _type                    | simple                                    |
| _attribute_set           | Default                                   |
| _category                | Default Category/Tops & T-shirts/Tops |
| ...                      |                                           |
| Category does not exists | ERROR                                     |
+--------------------------+-------------------------------------------+

Hi again

I believe there is a bug in Ho_Import_Model_Import::_createImportCsv on line 403:
https://github.com/ho-nl/Ho_Import/blob/bd1d407e1054568baf51b55a5f9b8e79bff07a9d/app/code/community/Ho/Import/Model/Import.php#L403

I think the $rowCount++; shouldn't be there.

We ran into an issue where we skipped all rows of a source, so no data should be imported, but since the rowCount was still saying there were more then 300 lines to be imported, it continued with the import and ran into exceptions further down the line.

If I remove this particular line, the problem seems to be fixed.

When an XML file is too small it isn't able to read it because it reaches the end of the file. Therefor it won't scan for the rootNode:

https://github.com/ho-nl/Ho_Import/blob/master/app/code/community/Ho/Import/Model/Source/Adapter/Xml.php#L430

I'm trying to add some translations to a dropdown attribute but each translation is added as an option. I'm I doing something wrong?

<import_options>
    <dropdown_attributes>
        <manufacturer>manufacturer</manufacturer>
        <gender>gender</gender>
        <product_type>product_type</product_type>
        <product_line>product_line</product_line>
        <size>size</size>
    </dropdown_attributes>
    <multiselect_attributes>
        <colors>colors</colors>
        <extra_colors>extra_colors</extra_colors>
    </multiselect_attributes>
</import_options>

<gender field="gender">
    <store_view>
        <nl_website_be field="gender_nl" />
        <fr_website_be field="gender_fr" />
        <website_nl field="gender_nl" />
        <website_fr field="gender_fr" />
        <website_com field="gender_en" />
        <website_de field="gender_de" />
    </store_view>
</gender>


<colors helper="ho_import/import::getFieldSplit">
    <field field="colors"/>
    <split>,</split>
    <store_view>
        <nl_website_be helper="ho_import/import::getFieldSplit">
            <field field="colors_nl"/>
            <split>,</split>
        </nl_website_be>
        <fr_website_be helper="ho_import/import::getFieldSplit">
            <field field="colors_fr"/>
            <split>,</split>
        </fr_website_be>
        <website_nl helper="ho_import/import::getFieldSplit">
            <field field="colors_nl"/>
            <split>,</split>
        </website_nl>
        <website_fr helper="ho_import/import::getFieldSplit">
            <field field="colors_fr"/>
            <split>,</split>
        </website_fr>
        <website_com helper="ho_import/import::getFieldSplit">
            <field field="colors_en"/>
            <split>,</split>
        </website_com>
        <website_de helper="ho_import/import::getFieldSplit">
            <field field="colors_de"/>
            <split>,</split>
        </website_de>
    </store_view>
</colors>

Did you consider adding Ho_Import to packages.firegento.com (https://github.com/magento-hackathon/composer-repository on GitHub)?

I created a simple import and ran this command in the shell:

 php hoimport.php -action import -profile catalog_product_example01 -dryRun 1

The output was

2014-01-31 11:47:27 - 8.5MB Mapping source fields and saving to temp csv file (/path/to/magento/var/import/catalog_product_example01.csv)
2014-01-31 11:47:27 - 8.5MB Getting source adapter ho_import/source_adapter_csv
2014-01-31 11:47:27 - 8.5MB Fieldmapping catalog_product_example01 with 5 rows (done in 0 seconds, 5 rows/s)
2014-01-31 11:47:27 - 8.5MB Processing 5 rows from temp csv file (/path/to/magento/var/import/catalog_product_example01.csv)
2014-01-31 11:47:27 - 8.5MB Setting option action to import
2014-01-31 11:47:27 - 8.5MB Setting option profile to catalog_product_example01
2014-01-31 11:47:27 - 8.5MB Setting option dryRun to 1
2014-01-31 11:47:27 - 8.5MB Start import catalog_product
2014-01-31 11:47:27 - 16.75MB Import catalog_product_example01 done in 0.4 seconds, 12.5 rows/s, 12.5 items/s.
2014-01-31 11:47:27 - 16.75MB No errors found in 5 rows
2014-01-31 11:47:27 - 16.75MB Done

The data was imported. Did I miss something? I tested on a vanilla CE 1.8.0.0 installation.

The XML has the following URL structure for images: http://www.cartridge.net/handler/showimage.ashx?brandId=25&size=300&cid=274

But in the field mapping the use this als name: /showimage.ashx?brandId=25&size=300&cid=274

How can i fix this? So i can import the images also

while I realize that method was deprecated in favor of getAttributeId, the existing code may need to be cleaned up with a cleaner outcome:

lines #404-406:

public function getMediaAttributeId($line) {
    return $this->getMediaAttributeId('media_gallery');
}

causes Segmentation Fault 11

Hi,

i get errors when i try to uninstall (modman remove) and later again install the module modman clone & (modman depoy).
The Setup-Script is executed again then. That implicates that the "import-profile" EAV-Attribute ist trying to be created again. Throws an Duplicate-Entry Error in MySQL. Result: Shop doesn't work any more.

So what would be a clean way to uninstall Ho_Import including the EAV-Attribute it creates ?

Hi

I was wondering if it is somehow possible to define multiple helpers for one field in the xml import profile?

For example: we want to use both the ho_import/import::getFieldSplit and ho_import/import::getMediaImage for one field.
Right now we solve this by writing our own helper which extends from the ho_import/import helper and then calls those two functions, but it is a bit tedious to do it like this.

So, it would be great if you could add a feature where you can add multiple helper calls, in a specific order, to one field in the xml import profile I believe.

With Magento 1.9.1.1 after correct Category-product relation import, critical error occurs:

2015-07-08T19:06:50+00:00 INFO (6): Setting option ignore_duplicates to 1
2015-07-08T19:06:50+00:00 INFO (6): Setting option partial_indexing to 1
2015-07-08T19:06:50+00:00 INFO (6): Mapping source fields and saving to temp csv file (/home/username/domains/domain.com/public_html/berto/var/import/category_products.csv)
2015-07-08T19:06:50+00:00 INFO (6): Getting source adapter ho_import/source_adapter_xml
2015-07-08T19:06:50+00:00 INFO (6): Loading source file var/import/accord/ArtModa.xml
2015-07-08T19:06:50+00:00 INFO (6): Fieldmapping category_products with 53 rows, 53 entities (done in 0.03 seconds, 1766.67 rows/s)
2015-07-08T19:06:50+00:00 INFO (6): Processing 53 rows from temp csv file (/home/username/domains/domain.com/public_html/berto/var/import/category_products.csv)
2015-07-08T19:06:50+00:00 INFO (6): Start profile /home/username/domains/domain.com/public_html/berto/var/import/category_products.csv
2015-07-08T19:06:52+00:00 INFO (6): Profile category_products done in 1.75 seconds, 30.29 entities/s, 30.29 rows/s.
2015-07-08T19:06:52+00:00 INFO (6): Clean entity link table
2015-07-08T19:06:52+00:00 CRIT (2): #0 /home/username/domains/domain.com/public_html/berto/app/code/core/Mage/Eav/Model/Config.php(328): Mage->throwException('entity_type specificato non valido: catalog_category_product')
#1 /home/username/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Import.php(1005): Mage_Eav_Model_Config->getEntityType('catalog_category_product')
#2 /home/username/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Import.php(1091): Ho_Import_Model_Import->_getEntityTypeId()
#3 /home/username/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Import.php(126): Ho_Import_Model_Import->_cleanEntityLinkTable()
#4 /home/username/domains/domain.com/public_html/berto/app/code/community/Ho/Import/Model/Observer.php(107): Ho_Import_Model_Import->process()
#5 (): Ho_Import_Model_Observer->process(Mage_Cron_Model_Schedule)
#6 /home/username/domains/domain.com/public_html/berto/app/code/core/Mage/Cron/Model/Observer.php(325): call_user_func_array(Array, Array)
#7 /home/username/domains/domain.com/public_html/berto/app/code/core/Mage/Cron/Model/Observer.php(72): Mage_Cron_Model_Observer->_processJob(Mage_Cron_Model_Schedule, Mage_Core_Model_Config_Element)
#8 /home/username/domains/domain.com/public_html/berto/app/code/core/Mage/Core/Model/App.php(1338): Mage_Cron_Model_Observer->dispatch(Varien_Event_Observer)
#9 /home/username/domains/domain.com/public_html/berto/app/code/core/Mage/Core/Model/App.php(1317): Mage_Core_Model_App->_callObserverMethod(Mage_Cron_Model_Observer, 'dispatch', Varien_Event_Observer)
#10 /home/username/domains/domain.com/public_html/berto/app/Mage.php(448): Mage_Core_Model_App->dispatchEvent('default', Array)
#11 /home/username/domains/domain.com/public_html/berto/cron.php(76): Mage->dispatchEvent('default')

After this import, Magento admin shows "One or more of the Indexes are not up to date". Is there an error with partial indexing?

Magento 1.9.1.1, Ho_import (master branch), AvS_FastSimpleImport (develop branch)