Integration tests fail massively

[Vinai]

Currently I'm unable to run the integration tests.
Steps to reproduce:

$ cd dev/tools && php tests.php --type=integration

I have to assign 1G memory to PHP to make them run, but even then pretty much every test gives me a stacktrace.
The same happens if I call phpunit directly without the tests.php script.

And its not always the same stacktrace. Some are due to Magento\RecurringPayment missing, but some others seem completely unrelated to the payment methods.
See below at the bottom of this issue for a few examples.

What I don't get is why the integration tests on travis run.

cd dev/tests/integration/; ./../../../vendor/bin/phpunit -c phpunit.xml.dist;

Any idea what I am doing wrong? Was something changed recently in how to run the integration tests?

Here a couple examples of from the exceptions and errors:

Fatal error: Call to a member function getId() on a non-object in m2.dev/htdocs/dev/tests/integration/testsuite/Magento/Catalog/Model/AbstractTest.php on line 125

m2.dev/htdocs/dev/tests/integration/framework/Magento/TestFramework/Annotation/DataFixture.php(159): Magento\TestFramework\Annotation\DataFixture->_applyOneFixtPHP Fatal error:  Call to a member function getId() on a non-object in m2.dev/htdocs/dev/tests/integration/testsuite/Magento/Catalog/Model/AbstractTest.php on line 125

Error in fixture: "m2.dev\/htdocs\/dev\/tests\/integration\/testsuite\/Magento\/Catalog\/_files\/products.php"
exception 'ReflectionException' with message 'Class Magento\RecurringPayment\Model\Product\Attribute\Backend\Recurring does not exist' in m2.dev/htdocs/lib/internal/Magento/Framework/Code/Reader/ClassReader.php:18

Error in fixture: "m2.dev\/htdocs\/dev\/tests\/integration\/testsuite\/Magento\/Catalog\/_files\/products_rollback.php"
exception 'Magento\Eav\Model\Entity\Attribute\Exception' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'main_table.website_id' in 'where clause', query was: SELECT `main_table`.*, `cp_table`.`type_id` FROM `cataloginventory_stock_item` AS `main_table`
 INNER JOIN `catalog_product_entity` AS `cp_table` ON main_table.product_id = cp_table.entity_id WHERE (`main_table`.`product_id` IN('')) AND (`main_table`.`website_id` = '1')' in /Volumesm2.dev/htdocs/lib/internal/Magento/Framework/ObjectManager/Factory/AbstractFactory.php:98

[Vinai]

Ups, I got it. Refreshing the test database to match the current m2 version helps.

[Vinai]

Ups, too early. That didn't help after all :(

[buskamuza]

Try to cleanup everything before running tests:

1. empty DB
2. cleanup var directory
3. cleanup dev/tests/integration/tmp directory

Maybe DB cleanup doesn't work properly or/and you have old generated files and they are not removed automatically.

[Vinai]

Thanks, one of those 3 almost did the trick. Now 29% of the integration tests execute without errors, even though many fail.

% ./../../../vendor/bin/phpunit -c phpunit.xml.dist
PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/phpunit.xml

............SF.F..S..........................................   61 / 3057 (  1%)
....IIIIIII.......FFFFFFFFF.F.F.FFFFFFFFFFFF....FFFFFF...F...  122 / 3057 (  3%)
...I.........................................................  183 / 3057 (  5%)
................SSS.....F.........................F.F.FF.....  244 / 3057 (  7%)
.........................................................SFFF  305 / 3057 (  9%)
EFFFFFFFFFFFFFFFF.F..FFFFFFFF...................S............  366 / 3057 ( 11%)
.............................................................  427 / 3057 ( 13%)
..........................S..................................  488 / 3057 ( 15%)
................................S............................  549 / 3057 ( 17%)
........I....................................................  610 / 3057 ( 19%)
.......................I.....................................  671 / 3057 ( 21%)
.............................................................  732 / 3057 ( 23%)
...................................................S.........  793 / 3057 ( 25%)
.............................................................  854 / 3057 ( 27%)
..........................I..............F....I..............  915 / 3057 ( 29%)
...........PHP Fatal error:  Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32
PHP Stack trace:
PHP   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
PHP   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
PHP   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
PHP   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
PHP   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
PHP   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
PHP  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
PHP  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
PHP  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951

Again I'm wondering how this is different from how travis is executing the integration tests.

[buskamuza]

Hm. I wonder, why PHPUnit says that it reads information from phpunit.xml, while you specify phpunit.xml.dist for it?

[Vinai]

Yes, interesting. I have added a phpunit.xml with the only difference to the phpunit.xml.dist being 1G available memory for running the tests.

% diff -u phpunit.xml.dist phpunit.xml                                                                                                                                    ✭
--- phpunit.xml.dist    2015-01-05 15:44:51.000000000 +0100
+++ phpunit.xml 2015-01-05 23:39:34.000000000 +0100
@@ -38,6 +38,7 @@
         <includePath>testsuite</includePath>
         <ini name="date.timezone" value="America/Los_Angeles"/>
         <ini name="xdebug.max_nesting_level" value="200"/>
+        <ini name="memory_limit" value="1024M"/>
         <!-- Local XML configuration file ('.dist' extension will be added, if the specified file doesn't exist) -->
         <const name="TESTS_INSTALL_CONFIG_FILE" value="etc/install-config-mysql.php"/>
         <!-- Semicolon-separated 'glob' patterns, that match global XML configuration files -->

[maksek]

Hey @Vinai, any luck with tests? They are straight forward, did you specify DB connection, last errors looks like no DB connected.

[Vinai]

Thanks for checking in. Yes, I have the DB connection configured. WIthout it the integration tests abort before the first test is executed.
No change from my last post above.

[maksek]

If you clean all required folders, drop DB and create new empty one. Then launching tests - what last error do you have.

[Vinai]

Here is what I get

% php --version
PHP 5.5.20 (cli) (built: Dec 27 2014 17:36:56)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with uopz v2.0.6, Copyright (c) 2014, by Joe Watkins <[email protected]>
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans
% pwd
/Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration
% rm -rf ../../../var/cache ../../../var/generation tmp/*
% echo "DROP DATABASE magento_integration_tests; CREATE DATABASE magento_integration_tests;" | mysql
% ../../../vendor/bin/phpunit -c phpunit.xml
PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/phpunit.xml

................SF.F..S......................................   61 / 3059 (  1%)
........IIIIIII.......FFFFFFFFF.F.F.FFFFFFFFFFFF....FFFFFF...  122 / 3059 (  3%)
F......I.....................................................  183 / 3059 (  5%)
....................SSS.....F.........................F.F.FF.  244 / 3059 (  7%)
.............................................................  305 / 3059 (  9%)
SFFFEFFFFFFFFFFFFFFFF.F..FFFFFFFF...................S........  366 / 3059 ( 11%)
.............................................................  427 / 3059 ( 13%)
..............................S..............................  488 / 3059 ( 15%)
....................................S........................  549 / 3059 ( 17%)
............I................................................  610 / 3059 ( 19%)
...........................I.................................  671 / 3059 ( 21%)
.............................................................  732 / 3059 ( 23%)
.......................................................S.....  793 / 3059 ( 25%)
.............................................................  854 / 3059 ( 27%)
..............................I..............F....I..........  915 / 3059 ( 29%)
...............PHP Fatal error:  Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32
PHP Stack trace:
PHP   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
PHP   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
PHP   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
PHP   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
PHP   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
PHP   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
PHP  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
PHP  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
PHP  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951

Fatal error: Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32

Call Stack:
    0.0014     982168   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
    0.0096    1235232   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
    0.0097    1235864   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
   27.4045   98097192   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
   27.4103   98184288   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
  811.7143  454923376   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
  811.7826  454137224   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
  811.7826  454137632   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
  812.8949  459494536   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
  813.0525  456755376  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
  813.0526  456756256  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
  813.0526  456756464  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951


=== Memory Usage System Stats ===
Memory usage (OS):  766.51M (164.66% of 465.50M reported by PHP)
Estimated memory leak:  301.01M (39.27% of used memory)

%

And thats it.

[maksek]

Did you try to skip the test and run without it? Wonder if it is specific test issue. Trying to reproduce.

[maksek]

@Vinai, i have fresh cloned magneto2, cleaned up everything in var/*, and and tests passed well.
Can you try to clean not only var/cache and var/generate, but all children folders in var/ and try again. Seems like environment glitch.

[Vinai]

I've tried the following:

1. Remove all folders in var/, but same result.
2. Fresh clone of magento 2, just ran composer install && composer update, and configured the DB in dev/tests/integration/etc/install-config-mysql.php file.
   No regular magento setup, no custom code.
   This time it is in the develop branch as in the previous tests it was the master branch.
   However, even here I get exactly the same result.

Here is my install-config-mysql.php file:

<?php

return [
    'db_host' => 'localhost',
    'db_user' => 'm2-test-db-user',
    'db_pass' => 'not-my-password',
    'db_name' => 'magento_integration_tests',
    'db_prefix' => '',
    'backend_frontname' => 'backend',
    'admin_username' => \Magento\TestFramework\Bootstrap::ADMIN_NAME,
    'admin_password' => \Magento\TestFramework\Bootstrap::ADMIN_PASSWORD,
    'admin_email' => \Magento\TestFramework\Bootstrap::ADMIN_EMAIL,
    'admin_firstname' => \Magento\TestFramework\Bootstrap::ADMIN_FIRSTNAME,
    'admin_lastname' => \Magento\TestFramework\Bootstrap::ADMIN_LASTNAME,
];

I'll try skipping the test that kills the process tomorrow. But still, so many other tests are already marked as failed that seem to run smoothly in travis.

[buskamuza]

with uopz v2.0.6, Copyright (c) 2014, by Joe Watkins <[email protected]>
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans

@Vinai , could you, please, try to disable these extensions?

And if it doesn't help, could you run the tests with --log-junit or any other kind of log and provide the log here. Maybe failed tests will help to understand the issue deeper.

[Vinai]

Progress, even though I can't explain some of it.
Here is what I did, just in case it might help someone else.

1. Disable all zend extensions
2. Get a clean clone of m2, checked out develop branch
3. Configure dev/tests/integration/etc/install-config-mysql.php file
4. Run integration tests

Result: a few errors and failed tests, but much fewer.

Next I enabled only the xdebug zend extension and repeated steps 2-4
Result: same as the first time, that is a few errors and failed tests, but much fewer.

For the next step I enabled also the opcache zend extension and repeated steps 2-4 again.
Result: still the same as the first time, that is a few errors and failed tests, but much fewer.

Finally I also enabled the uopz extension again and ran the tests again.
Result: still the same as the first time, that is a few errors and failed tests, but much fewer!

So now I have effectively the same set up as last night, but a different outcome.

The remaining number of errors and failures was small enough that I decided to have a detailed look why they where failing.
This resulted in the issue #939

The remaining tests I was able to fix by using a custom phpunit.xml with 1GB of memory.

Going back to my "main" m2 dev instance, the integration tests now also run. I found additional failures due to invalid themes I had created, which I was able to fix without much trouble.
The only remaining issue now is the one I reported (see above).

I'm puzzled and not sattisfied by not knowing what was causing the issue, but it obviously must have been due to my local setup. However, now that it is resolved I can focus on more important things.
Thank you for your help, it is much appreciated.