InMemory DbContext InvalidOperationException With Two DbContext Instances

[Cooksauce]

Scenario

Integration testing using InMemory context

If I have two DbContext instances and move an entity from one to the other by calling DbContext.Attach(..) will the first instance still be tracking the entity? I'm getting an exception about lazy loading after disposal so I'm wondering if the second context (which gets disposed) is still hanging on to the reference?

If not, any other insight would be greatly appreciated.

Example

public class Author
{
    public int Id { get; set; }
    public virtual Book Book { get; set; }
}

public class Book
{
   public int Id { get; set; }
   public string Words { get; set; }
}


// Usage
public async Task SomeMethodAsync()
{
    var dbRoot = new InMemoryDatabaseRoot();
    using var dbOne = NewInMemoryDbContext(dbRoot);
    var author = dbOne.Authors.Find(1);

    using (var dbTwo = NewInMemoryDbContext(dbRoot))
    {
        dbTwo.Attach(author);
        author.Book.Words = "abc";
        dbTwo.SaveChanges();
    }

    author = dbOne.Authors.Find(1);
    var book = author.Book; // throws InvalidOperationException below
}

Exception Message

Error generated for warning 'Microsoft.EntityFrameworkCore.Infrastructure.LazyLoadOnDisposedContextWarning: An attempt was made to lazy-load navigation property 'Book' on entity type 'AuthorProxy' after the associated DbContext was disposed.'...

Relevant Stacktrace

   at Microsoft.EntityFrameworkCore.Diagnostics.EventDefinition`2.Log[TLoggerCategory](IDiagnosticsLogger`1 logger, WarningBehavior warningBehavior, TParam1 arg1, TParam2 arg2)
   at Microsoft.EntityFrameworkCore.Diagnostics.CoreLoggerExtensions.LazyLoadOnDisposedContextWarning(IDiagnosticsLogger`1 diagnostics, DbContext context, Object entityType, String navigationName)
   at Microsoft.EntityFrameworkCore.Internal.LazyLoader.ShouldLoad(Object entity, String navigationName, NavigationEntry& navigationEntry)
   at Microsoft.EntityFrameworkCore.Internal.LazyLoader.Load(Object entity, String navigationName)
   at Microsoft.EntityFrameworkCore.Proxies.Internal.LazyLoadingInterceptor.Intercept(IInvocation invocation)
   at Castle.DynamicProxy.AbstractInvocation.Proceed()
   at Castle.Proxies.AuthorProxy.get_Book()
   at ...SomeMethodAsync() 

Further technical details

EF Core version: 3.0.0
Database provider: InMemory
Target framework: .NET Core 3.0

[ajcvickers]

@Cooksauce

will the first instance still be tracking the entity?

Yes. When non using proxies there is no alternative, since the entity itself has no idea if it is being tracked or not. However, for proxies it might make sense to do this--we did something similar in EF6.

[Cooksauce]

@ajcvickers
That explains why the last line would throw, correct?

Just to note: I am using lazy loading proxies on the InMemory context here.

[ajcvickers]

@Cooksauce Yes. If we make the change I suggested then the second call to Find will return a new instance because the original entity instance will no longer being tracked by the first context.

[Cooksauce]

@ajcvickers I should be able to get around this by looping through the change tracker and detaching all of the tracked entities before disposing the context, yes?

When trying that, I run into another issue. When I try to detach an entity, I get exceptions about severing relationships as if I'm deleting them?

Why is EF treating an entity detachment as if it's being deleted? Is that by design?

Method here:

public static void ClearCache<TEntity>(this MyContext context, IEnumerable<TEntity> selector)
{
    if (selector is null)
        throw new ArgumentNullException(nameof(selector));

    var entries = context.ChangeTracker.Entries()
        .Where(entry => selector
            .Any(entity => entry.Entity.Equals(entity)));

    foreach (var entry in entries)
    {
         entry.State = EntityState.Detached;
    }
}

public static void ClearCache(this MyContext context)
{
    ClearCache(context, context.ChangeTracker.Entries().Select(entry => entry.Entity));
}

Stacktrace

System.InvalidOperationException : The association between entities 'Book' and 'Author' with the key value '{AuthorId: 1}' has been severed but the relationship is either marked as 'Required' or is implicitly required because the foreign key is not nullable. If the dependent/child entity should be deleted when a required relationship is severed, then setup the relationship to use cascade deletes.
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry.HandleConceptualNulls(Boolean sensitiveLoggingEnabled, Boolean force, Boolean isCascadeDelete)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.CascadeDelete(InternalEntityEntry entry, Boolean force, IEnumerable`1 foreignKeys)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry.SetEntityState(EntityState entityState, Boolean acceptChanges, Boolean modifyProperties, Nullable`1 forceStateWhenUnknownKey)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.CascadeDelete(InternalEntityEntry entry, Boolean force, IEnumerable`1 foreignKeys)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry.SetEntityState(EntityState entityState, Boolean acceptChanges, Boolean modifyProperties, Nullable`1 forceStateWhenUnknownKey)
   at Microsoft.EntityFrameworkCore.ChangeTracking.EntityEntry.set_State(EntityState value)
   at ClearCache[TEntity](MyContext context, IEnumerable`1 selector)
   at ClearCache(MyContext context)

[ajcvickers]

@Cooksauce Why are you trying to reuse the first context instance?

[Cooksauce]

@ajcvickers Main reason is because this is all in an IDbContextTransaction created by that first context.

[Cooksauce]

Maybe more context will help: I have a bulk import service which performs bulk imports outside of EF (for performance reasons), but is able to utilize an EF transaction & DbConnection. This allows us use EF to retrieve/modify/save an owning entity, then bulk insert a huge list of child entities, then update that owning entity, all in one database transaction. So we can get all of the benefits EF brings while dropping down directly to Npgsql for high performance inserts.

I'm trying to write some integration tests which are all in memory; making use of the InMemory provider.
To do this, I replace the production IBulkImportService with an implementation which writes to an InMemory DbContext. This is where the issues start. I think we have enough abstraction here to be able to run this with a real database provider (Npgsql) or the EF InMemory provider, but it seems the InMemory context is behaving differently than one would expect.

This works and runs as expected when using as production with Npgsql provider

Obviously, this situation isn't explicitly supported by the EF Core team. But it's very close to working very well for it. I think the InMemory provider could be easily improved in some areas to more closely mimic a real database provider. If I had an Npgsql DbContext which was tracking entities, and I looped through the change tracker and detached all of those entities, would it delete them from the database? If not, then the InMemory provider should also not treat that as if I'm deleting them.
(you can see in the stacktrace from the previous comment EF is calling deletion methods CascadeDelete(..))

Reduced Interface (for brevity)

public interface IBulkImportService
{
    /// <summary>
    /// Performs an async bulk insert of an entity stream on the provided transaction which can be canceled.
    /// If canceled in the middle of import, a <see cref="TaskCanceledException"/> will be thrown. It is up to the caller to Commit or Rollback the transaction.
    /// </summary>
    /// <typeparam name="TEntity"></typeparam>
    /// <param name="entities"></param>
    /// <param name="transaction"></param>
    /// <param name="cancellationToken"></param>
    /// <returns></returns>
    Task<int> BulkInsertAsync<TEntity>(IAsyncEnumerable<TEntity> entities, IDbContextTransaction transaction, CancellationToken cancellationToken) where TEntity : class;
}

In Memory Implementation

public class InMemoryBulkImportService : IBulkImportService
{
    public async Task<int> BulkInsertAsync<TEntity>(IAsyncEnumerable<TEntity> entities, IDbContextTransaction transaction, CancellationToken cancellationToken) where TEntity : class
    {
        using (var db = NewInMemoryContext()) //Uses a shared InMemoryDatabaseRoot, resolved from DI container
        {
            await foreach (var entity in entities.WithCancellation(cancellationToken))
                db.Attach(entity);

            var changes = await db.SaveChangesAsync(cancellationToken);
            return changes;
        }
    }
}

Calling Service Under Test

IBulkImportService _bulkImportService;

using (var dbOne = GetContext()) / / resolved from the DI container
{
    using var transaction = await db1.Database.BeginTransactionAsync();

    var author = await db1.Authors.FindAsync(1);
    author.Book = new Book();
    await db1.SaveChangesAsync();

    IAsyncEnumerable<Word> wordsToAdd = CreateWords().Select(x =>
    {
        x.Book = author.Book;
        x.BookId = author.Book.Id;
    });
    var count = await _bulkImportService.BulkInsertAsync(wordsToAdd, transaction, token);

    author.Book.WordCount = count;
    await db1.SaveChangesAsync();

    await transaction.CommitAsync();
}

Model

public class Author
{
    public int Id { get; set; }
    public virtual Book Book { get; set; }
}

public class Book
{
   public int Id { get; set; }
   public int WordCount { get; set; }
   public virtual ICollection<Word> Words { get; set; }
}

public class Word
{
    public int Id { get; set; }
    public string Text { get; set; }
    public virtual Book Book { get; set; }
    public int BookId { get; set; }
}

Solution

Don't explicitly dispose InMemory context, let IServiceProvider dispose them. Avoids the lazy load on disposed context exception.

[ajcvickers]

@Cooksauce Thanks for the additional info. I'm not sure that the in-memory database is the correct choice for this. See the discussion in #18457. Transactions are one thing not supported by the in-memory database.

[ajcvickers]

Also, as it says in the exception message, you can disable the warning:

optionsBuilder.ConfigureWarnings(e => e.Ignore(CoreEventId.DetachedLazyLoadingWarning))

This will turn attempting to load on on a disposed context into a no-op.

[Cooksauce]

@ajcvickers I'd rather avoid setting a broad ignore like that, since it could ignore real problems.

Would you mind clarifying whether the InMemory provider should be treating entity detachments as a delete?

(you can see in the stacktrace above it's is calling deletion methods CascadeDelete(..))

[Cooksauce]

@ajcvickers Do you know the answer to this? I just need to know whether this is by design or a bug.

[ajcvickers]

@Cooksauce You may just need to change the cascade timing. Or you may be hitting something like this: #18982

If it isn't covered by these things, then please attach a small, runnable project or post a complete code listing that reproduces the behavior your are seeing.

[Cooksauce]

Thanks!
I think I missed that in the breaking changes.