Batch Processing using toIterable()

[stlrnz]

Hi all!

I've updated my application to Doctrine ORM 2.8 some days ago.

Before that, my Repository code to iterate through a large result set looked like the following and worked perfectly even on >500.000 rows without leaking any memory.

    public function iterateAll(): Generator
    {
        $iterator = $this->createQueryBuilder('e')->getQuery()->iterate(null, Query::HYDRATE_SIMPLEOBJECT);

        foreach ($iterator as $equipment) {
            // index 0 is always the object
            yield $equipment[0];
        }
    }

Since Query::iterate() is deprecated now, I tried to use Query::toIterable() as suggested.

    public function iterateAll(): Generator
    {
        $iterator = $this->createQueryBuilder('e')->getQuery()->toIterable([], Query::HYDRATE_SIMPLEOBJECT);

        foreach ($iterator as $equipment) {
            yield $equipment;
        }
    }

With this implementation I stumbled over a massive memory leak in my application. I debugged this and found out that the AbstractHydrator is never releasing the Objects in AbstractHydrator::toIterable(). The called method AbstractHydrator::hydrateRowData() (or in my case SimpleObjectHydrator::hydrateRowData) just adds new entries to $result.

orm/lib/Doctrine/ORM/Internal/Hydration/AbstractHydrator.php

Line 177 in 378944d

$this->hydrateRowData($row, $result);

orm/lib/Doctrine/ORM/Internal/Hydration/SimpleObjectHydrator.php

Line 157 in 378944d

$result[] = $entity;

Is that the intended behaviour? Wouldn't it be better to clear $result in every loop cycle to free memory?

If that's intended, is there a better way to loop through large result sets?

[beberlei]

This is a bug

[simPod]

@beberlei this looks like in order to fix #3238 we need to keep hydrated objects in memory so ObjectHydrator can look them up. Any idea? Seems like we can't have both to me 🤔