Subquery.Any(predicateParameter) throws exception

[smitpatel]

Stackoverflow question: http://stackoverflow.com/questions/43057695/cant-extract-this-function-from-an-expression

Earlier discussion on #3722
from #3722 (comment)
@mikebridge wrote

I have subquery that is generating that same error, but only when the Func is passed into the method containing the query.

    System.NotSupportedException : Could not parse expression 's.Programs.Where(__StartsWith_0)': The given arguments did not match the expected arguments: Object of type 'System.Linq.Expressions.TypedParameterExpression' cannot be converted to type 'System.Linq.Expressions.LambdaExpression'.

I have a "School" entity which contains "Program" entities, and I'm doing a subquery like the following---and this works fine:

public IQueryable<SchoolResult> SearchShoolPrograms(
            Expression<Func<School, bool>> schoolExpression)
{
    return _dbContext.School
        .Where(schoolExpression)
        // ... 
        .Select(s => new SchoolResult
         {
             Name = s.Name,
             Programs = s.Programs.Where(p => p.Name == "Test")
                   .Select(p => new ProgramResult
                   {
                         Id = p.Id,
                         Name = p.Name,
                   });
         }
}

SearchShoolPrograms(school => school.Active);

However, I can't extract the "Where" function without generating an error. I think this should work the same:

public IQueryable<SchoolResult> SearchShoolPrograms(
            Expression<Func<School, bool>> schoolExpression,
            Func<Program,bool> programExpression)
{
    return _dbContext.School
        .Where(schoolExpression)
        // ... 
        .Select(s => new SchoolResult
         {
             Name = s.Name,
             Programs = s.Programs.Where(programExpression)
                   .Select(p => new ProgramResult
                   {
                         Id = p.Id,
                         Name = p.Name,
                   });
         }
}

SearchSchoolPrograms (s=> s.Active, p => p.Name == "Test");

Any idea why these are different? This is with EntityFrameworkCore 1.1.1.

#3722 (comment)

I'm sure I'm missing something obvious here. But it looks like I get the same error if I try to extract the Func from the Expression. This works when I pass it into my SearchShoolPrograms method:

Expression<Func<School, bool>> expr = school => 
    school.Active && school.Programs.Any(program => program.Name.StartsWith("U"));

This doesn't:

Func<Program, bool> p = program => program.Name.StartsWith("U");
Expression<Func<School, bool>> expr = school => school.Active && school.Programs.Any(p);

I think they should be equivalent, but I'm still getting System.NotSupportedException : Could not parse expression 'school.Programs.Any(__p_0)': The given arguments did not match the expected arguments: Object of type 'System.Linq.Expressions.TypedParameterExpression' cannot be converted to type 'System.Linq.Expressions.LambdaExpression'.

#3722 (comment)

@smitpatel sorry, yes, here it is:

System.NotSupportedException : Could not parse expression 'school.Programs.Any(__p_0)': The given arguments did not match the expected arguments: Object of type 'System.Linq.Expressions.TypedParameterExpression' cannot be converted to type 'System.Linq.Expressions.LambdaExpression'.
   at Remotion.Linq.Parsing.Structure.MethodCallExpressionParser.CreateExpressionNode(Type nodeType, MethodCallExpressionParseInfo parseInfo, Object[] additionalConstructorParameters)
   at Remotion.Linq.Parsing.Structure.MethodCallExpressionParser.Parse(String associatedIdentifier, IExpressionNode source, IEnumerable`1 arguments, MethodCallExpression expressionToParse)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression(MethodCallExpression methodCallExpression, String associatedIdentifier)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseTree(Expression expressionTree)
   at Remotion.Linq.Parsing.Structure.QueryParser.GetParsedQuery(Expression expressionTreeRoot)
   at Remotion.Linq.Parsing.ExpressionVisitors.SubQueryFindingExpressionVisitor.Visit(Expression expression)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at Remotion.Linq.Parsing.ExpressionVisitors.SubQueryFindingExpressionVisitor.Visit(Expression expression)
   at System.Linq.Expressions.ExpressionVisitor.VisitLambda[T](Expression`1 node)
   at System.Linq.Expressions.Expression`1.Accept(ExpressionVisitor visitor)
   at Remotion.Linq.Parsing.ExpressionVisitors.SubQueryFindingExpressionVisitor.Visit(Expression expression)
   at System.Linq.Enumerable.SelectListPartitionIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Remotion.Linq.Parsing.Structure.MethodCallExpressionParser.Parse(String associatedIdentifier, IExpressionNode source, IEnumerable`1 arguments, MethodCallExpression expressionToParse)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression(MethodCallExpression methodCallExpression, String associatedIdentifier)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression(MethodCallExpression methodCallExpression, String associatedIdentifier)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression(MethodCallExpression methodCallExpression, String associatedIdentifier)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression(MethodCallExpression methodCallExpression, String associatedIdentifier)
   at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseTree(Expression expressionTree)
   at Remotion.Linq.Parsing.Structure.QueryParser.GetParsedQuery(Expression expressionTreeRoot)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQueryCore[TResult](Expression query, INodeTypeProvider nodeTypeProvider, IDatabase database)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.Generic.IAsyncEnumerable<TResult>.GetEnumerator()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.<ToListAsync>d__129`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ProgramManager.Services.ProgramMatchingService.<SearchSchoolPrograms>d__4.MoveNext() in C:\Visual Studio 2017\Projects\MyProject\src\ProgramManager\Services\ProgramMatchingService.cs:line 89
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ProgramManager.Tests.Integration.Services.ProgramMatchingServiceTests.<It_Should_Return>d__4.MoveNext() in C:\Visual Studio 2017\Projects\MyProject\test\ProgramManager.Tests.Integration\Services\ProgramMatchingServiceTests.cs:line 48
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

@smitpatel If it helps, I created a test project that generates the error with an in-memory db.

[smitpatel]

Error here is, we get Func in expression tree and we convert the closure into a parameter. Therefore we end up with TypedParameterExpression instead of LambdaExpression as expected for Any()

[mikebridge]

@smitpatel Sorry, just to clarify, are you saying that EF should do reflection on the Func that's being passed as a parameter and it is supposed to create an Expression from it (in other words, it is a bug in EF), or are you saying that a client must pass a Lambda Expression, not a Func (i.e. the bug is in the client code)?

[smitpatel]

@mikebridge - EF will not do reflection and look inside if you pass Func.
If the client wants expression to be translated to server, Expression must be passed.
When EF receives a Func, it cannot translate it to server but it can do evaluation of it in memory after fetching data from server. That is what EF is supposed to do. At present it throws exception. that is the ef bug.

[mikebridge]

@smitpatel thanks for the clarification!

[pawepaw]

@smitpatel
But when querying navigation properties EF allows only to pass Func<> because it treats ICollection<> as IEnumerable. If what you said is true it would mean that all sub queries on navigation properties are executed in memory and that would be a false statement. Am I right?

Edit:
Ok i understand now. Inlined Func<> can be treated as Expression ...

[Luis-Palacios]

Any status update on this @smitpatel ? I'm hating having to repeat so much code because I can't create and reuse expressions for Where or nested Select

[smitpatel]

I have wonderful update for this.
Following works in current nightly builds

public async Task<IEnumerable<QueryResult>> SearchWithExtracted()
{
    Expression<Func<Child, bool>> p = program => program.Name.StartsWith("U");
    Expression<Func<Parent, bool>> expr = parent => parent.Active
                                                    && parent.Children.AsQueryable().Any(p);
    return await CreateQueryable(expr).ToListAsync();

}

The issue here was when Subquery was Enumerable, user could not pass Expression<Func<>> to Any operator. Using Subquery.AsQueryable() allows us to pass such parameter. Since we are getting Expression we translate it correctly.
Support for AsQueryable operator added in #6132
Any subquery which disallows passing in Expression<Func<>> can now use AsQueryable call to remove that restriction.

[smitpatel]

Regression test, which is same as above was added as part of #6132

[Luis-Palacios]

That is great news @smitpatel I hope it will be released soon!