M2M2: Migration

Published on 19 Mar 2019 , edited on 04 Apr 2019

Ah the migration. Oh the imports and exports. Woe is me. Countless hours spending crafting the biggest spreadsheets and CSV files. Error after error, so much frustration you eventually go beyond rock-bottom and reach a state of true nirvana.

Let’s hope Magento’s Migration tool handles this for me this time around.

What this tool does is connect to a M1 database, and it reads out the data, creates a mapping to the new M2 structure, my guess is at this point it allows me to check and/or update it, and then copy all the data over. I had some prior experience with the initial version of this tool and that was far from pretty.. Let’s jump in.

Installation

First up, the installation. And as I wrote in the server-setup, server side user management has always been a bit of a mystery to me. No different this time around, but I figured it out! Get to a terminal, logon to your M2 site and navigate to it’s root directory. I ran into a issue where composer wouldn’t be found and I got some permission errors. SO check that you have set your correct paths/aliases to run composer and the Magento CLI first. Then we switch to the Magento user, add a repository for the data migration tool and pull it in. Keep your Magento Marketplace keys at the ready, you’ll be prompted for them;

sudo su - magento
composer config repositories.magento composer https://repo.magento.com
composer require magento/data-migration-tool:<version>

Configuration

First, move to the installed migration tool directory (match your M1 installation version in the path) as the Magento user, and create a config file from the template;

sudo su - magento
cd /opt/magento/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.4.0
cp config.xml.dist config.xml
sudo nano config.xml

Now that is a big file innit. I’m expecting a bunch of connection issues and whatnot, so I’m just giving this a first go and we’ll fix it as we go along. We start with the bare minimum of input; find these nodes and fill in your connection data (where user/password is for the MySQL database, and the crypt_key is found in your M1 app/etc/local.xml);

<source>
    <database host="127.0.0.1" name="magento1" user="root" password="pass"/>
</source>
<destination>
    <database host="127.0.0.1" name="magento2" user="root" password="pass"/>
</destination>
<options>
    <source_prefix>magento1</source_prefix>
    <crypt_key>f3e25abe619dae2387df9fs594f01985</crypt_key>
</options>

At this point we could set up custom mapping to transfer (also) custom tables from M1. I’m gonna skip this too for now, to see what it does and doesn’t do out of the box. If we even get that far.

For now, to be sure, dump your current (clean) database before anything, this way we can always revert back to this moment and try again.

Running the migration

As the Magento user, run magento migrate:settings with the full path to the previously made config.xml, like so (the -r flag resets the migration to start from the beginning each time, useful for testing);

magento migrate:settings -r /opt/magento/public_html/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.4.0/config.xml

And what’s this? Huh? No errors? IT WORKS?! JUST LIKE THAT? Can’t believe it. But if we are on a lucky streak, let’s see if we can get the data too!

magento migrate:data -r -a /opt/magento/public_html/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.4.0/config.xml

Intermission update from the future: I found out that I had duplicate installs of the migration tool (one inside the public_html and one above that). Causing all sorts of trouble. So I removed both and installed a fresh copy inside the public_html folder. From here on out, the below part doesn’t play out as it did, so I scrapped it.


And there are the dragons in the distance. Turns out it flips out over our bucket of custom attributes and it crashes on a few modules who added tables to the indexer, so it seems. Let’s get to reading about this and how to fix it.

Mapping

So it looks like we can skip most of the missing attribute warnings by using the -a flag. But I do get some errors from modules who tied themselves to products and attributes;

[2019-03-20 12:44:32][ERROR]: Class mana_filters/source_filterable does not exist but mentioned in: eav_attribute.source_model for attribute_id=550
[2019-03-20 12:44:32][ERROR]: Class mana_filters/source_filterable does not exist but mentioned in: eav_attribute.source_model for attribute_id=551
[2019-03-20 12:44:32][ERROR]: Class mana_filters/source_display does not exist but mentioned in: eav_attribute.source_model for attribute_id=553
[2019-03-20 12:44:32][ERROR]: Class postnl_deliveryoptions/product_attribute_source_shippingDuration does not exist but mentioned in: eav_attribute.source_model for attribute_id=557
[2019-03-20 12:44:32][ERROR]: Class postnl_deliveryoptions/product_attribute_source_productType does not exist but mentioned in: eav_attribute.source_model for attribute_id=565
[2019-03-20 12:44:32][ERROR]: Class mana_filters/filter does not exist but mentioned in: eav_entity_type.entity_model for entity_type_id=9
[2019-03-20 12:44:32][ERROR]: Class mana_filters/filter_attribute_collection does not exist but mentioned in: eav_entity_type.entity_attribute_collection for entity_type_id=9

Now Magento states “A class from Magento 1 codebase could not be found in Magento 2 codebase during the EAV migration step. In most cases, the missing class belongs to an extension .”, and to solve it among other options “add the attribute to the ignore group in the eav-attribute-groups.xml.dist file.”. So let’s do that. It doesn’t state what the exact markup should be, so this is another trial-and-error run.

This answer seems pretty straightforward so let’s start there. Going trough the same motions as mentioned;

  • in eav_attribute table, id 550, 551 and 553 all have entity_type_id 9
  • the codes in that table are is_enabled, is_enabled_in_search and display
  • the eav_entity_type table shows entity_type_code m_filter for id 9

So that would result in the following code

<group name="ignore">
    ...
    <attribute type="m_filter">is_enabled</attribute>
    <attribute type="m_filter">is_enabled_in_search</attribute>
    <attribute type="m_filter">display</attribute>
</group>

But so far no luck, running the migration again gives me the same errors. It felt like the file wasn’t read or something, so I copied it without the .dist extension and updated the path to the file in config.xml and now I get a new error (yay, progress..!)

Invalid groups filename: /opt/magento/public_html/vendor/magento/data-migration-tool/etc/opensource-to-opensource/eav-attribute-groups.xml

So I’m pretty sure it has something to do with the .dist filenames and their mentions in the config.xml. If I remove the .dist extension from both the file and the config-path to it, I get above error and the whole migration doesn’t even start. If I revert it (put .dist back everywhere) it does run, but it doesn’t “see” my additions. There is logic to this, not sure which yet.

For the PostNL thing there is a guide here. Possible other source; here.



So after realizing I had two separate installations of the migration tool, I cleared everything out and got a fresh copy. Set up the config.xml again and ran the imports. This time, it appears to just start migrating but eventually it spouts;

SQLSTATE[08S01]: Communication link failure: 1153 Got a packet bigger than 'max_allowed_packet' bytes  

Which would indicate that yes, the migration was underway. I set the max_allowed_packets to beyond its limit (1000M) in two files, and that seems to actually update it

sudo nano /etc/mysql/conf.d/mysql.cnf 
sudo nano /etc/mysql/my.cnf
sudo service mysql restart

After this I set bulk size in the config.xml to 100;

<bulk_size>100</bulk_size>

And ran the migration again. This time it takes far longer because of this bulk size option, but it passes without max_allowed_packet errors. Eventually I got a new error, and fixed it by just commenting it:

[2019-03-28 15:38:26][ERROR]: Mismatch of entities in the documents: rating, rating_store
In Data.php line 145: Volume Check failed  

And at this point, migration completes! Ofcourse with a bunch of skipped warnings and errors, but after running magento setup:upgrade I got a functional frontend and I see that my categories are showing. No products yet, but I didn’t expect to be done by any means.

Epilogue

So I set out to restart everything to have the migration run without any errors. But when I combed through said errors I realised all of them were from (old) modules and we have no need for that data to be migrated. So change of plans; we start testing the store, probably tweaking the migration anyway to make it cleaner, but that’s a story for another time.

Migration done! Pretty much without too many headaches. Wow.

M2M2

Moving To Magento Two; A big move from multiple Magento 1.9.x systems to Magento 2.x. Documented every step chronologically (currently ongoing!).

All categories

All tags

All posts