-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ExecuteUpdate/Delete (AKA bulk update, without loading data into memory) #795
Comments
This is something we do want to do in EF, but not for the initial RTM of EF Core. Moving to backlog. |
Bulk is currently available in EntityFramework.Extended by loresoft (source https://github.com/loresoft/EntityFramework.Extended, Nuget https://www.nuget.org/packages/EntityFramework.Extended/). Maybe if license allows, you could think about grabbing code from that project |
Unfortunately EF.Extended isn't testable at all and author isn't going to address that. |
I mostly agree. EF.Extended is indeed difficult to test, but not actually "not testable at all", as after several attempts I made my PR run on AppVeyor. I did my own fork and implemented my own methods for MySQL (only because the original project didn't work with it) because the author is not maintaining the project any more. What about porting that code, and unit-testing it, in the original EF package? 😭 😭 |
Hi, I'm starting to investigate how complex would be to add support for set-based operation, and I'm actually only investigating Delete operation, I really would like to start a discussion with the team and the community members to start a discussion on how this feature can be designed. Hoping that this will be achieved by EF Core, I'm posting my initial thoughts/questions on this subjects.
My initial proposal is the following: Delete
usage: or
usage: Update
usage: or (as suggested above)
usage: Merge Any suggestion? This is just a quick list of initial thoughts, I will update this one with new findings (if any 😄 ) during this initial investigation, but would like to get the discussion started. |
This makes sense. We likely need to introduce UpdateExpression etc. and then port the existing impl. to use it, too. |
Looks good to me. |
Replying to @ilmax
I think the most difficult part is to guess whether some modifications propagate to entities already within the session. This is not currently addressed by EntityFramework.Extended in my knowledge, and is supposed to cause troubles. Example:
The above differs whether The above was a simple scenario because EF knows we are working on the cats, so affected entities are of
The update is supposed to run on the Cats, but, in practice, modifications apply to This scenario is really over-complicated, even though easy to run in-memory. I am speaking of it only to illustrate possible misuses of the bulk statement. Final point of discussion for the update, I want to ask if multi-column updates are supported and how. I mean
Cheers, |
Another point of discussion is that I like is the overload I have used in my examples: so
Normally databases return the number of affected rows, and it is helpful to the application. So here are 2 separate proposals:
|
@djechelon Good points, IMO at first we can reject set-based query if the context is already tracking object of the same type, unless it's proven to be very simple to propagate changes to tracked objects. About the second example Multi-column update IMO should be supported, I don't think that support it should be much different than what's required for single column ones.
Is this change on your radar? Would you like me to open an issue to track this request? |
@ilmax Currently we use Raw SQL for these batch operations e.g. I wouldn't reject the query because some of the affected rows might already being tracked . I would love the see this feature and liked the methods signature. |
When to implement this feature? |
Seems like https://github.com/zzzprojects/EntityFramework-Plus provides batch delete etc., and has support for EF Core. Anyone looked into this? |
Are there any plans to implement this? |
As of today there is extension library EFCore.BulkExtensions with Bulk Operations (Insert, Update, Delete) and Batch (Delete, Update)
More info here: https://github.com/borisdj/EFCore.BulkExtensions |
I have tried EntityFramework-Plus in my own project, but had to discard it. It dynamically tries to load the SQLServer EF provider at runtime, because it uses some of its functionality to generate SQL for the batch queries. If the actual provider is not SQLServer, it then applies a few manual translations (e.g. changing field quoting style). However, in our deployment we will not have the SQLServer EF provider available. |
There are three community effors to support bulk operations: Has anyone had experience with them? I have a large database where I only do inserts of around 50 records at a time and EF Core goes into a death spiral, eating all memory on the machine. I'm trying to evaluate which of these would solve my problem. |
Dear @RehanSaeed, I have tried successfully the EntityFramework.Extended to perform bulk deletes, but I needed to compile my own version of it. Your question is unclear. What operation are you performing so that EF core goes into out of memory? Are you massively inserting? Or are you loading all your results into memory at a certain time? Why not discussing on stackoverflow? |
Just inserting. EntityFramework.Extended does not seem to support bulk inserts. |
No doubt. Bulk inserts can be handled by EF.BulkInserts, which I failed to use. Eventually I used the SqlBulkLoader "manually" |
@roji @smitpatel Sorry to tag you guys but I wanted to ask a question before this is released: I noticed that the .SetProperty(p => p.Name, _ => "Shay") Instead of just: .SetProperty(p => p.Name, "Shay") Is there a particular reason behind this?! Why not just allow simple values? 🤔 |
I hope it's simpler |
@zhuangState The first syntax isn't a valid C# expression (there are plans to improve expressions so it could become reality one day) and the latter syntax wouldn't be feasible due to the introduction of the |
@aradalvand it use anonymous object, there no |
I hope C# will have "Cascade notation" like Dart language, so we can write:
|
Does the current implementation allow limiting the number of rows affected, eg |
@crozone which database supports the SQLs you posted above? AFAIK most databases do not allow placing LIMIT and ORDER BY clauses on DELETE or UPDATE as you're showing. However, when a LINQ operator is used that cannot be directly translated to SQL, EF uses a subquery to achieve the same effect. For example, the following LINQ operation: ctx.Blogs
.OrderBy(b => b.Id)
.Take(10)
.ExecuteUpdate(sp => sp.SetProperty(b => b.Name, "Foo")); ... translates into the following SQL: UPDATE [b]
SET [b].[Name] = N'Foo'
FROM [Blogs] AS [b]
INNER JOIN (
SELECT TOP(@__p_0) [b0].[Id], [b0].[Name]
FROM [Blogs] AS [b0]
ORDER BY [b0].[Id]
) AS [t] ON [b].[Id] = [t].[Id] |
@aradalvand Thank you for your reply |
@roji Hmm perhaps this is only a MySQL/MariaDB feature? Neither Postgres nor MSSQL appear to support it after all. SQLite supports it, but apparently only if compiled with SQLITE_ENABLE_UPDATE_DELETE_LIMIT ... Very glad to see that EF will join against a subquery to get the same result regardless, so it doesn't even matter. The ability to |
@crozone right, I remember now for both these databases. Note that this feature has been implemented in such a way that providers may implement such "non-standard" operators if they so wish. That means that the default EF logic will create a subquery, but the MySQL provider can implement support for OrderBy/Take to produce the tighter SQL above. /cc @lauxjpn |
EF does not provide a batch update mechanism. A proposal is below.
Context.Customers.Update().Where.( c => c.CustType ==“New”).Set( x => x.CreditLimit=0)
Will you consider this feature?
More details here: https://entityframework.codeplex.com/workitem/52
Design summary: #795 (comment)
The text was updated successfully, but these errors were encountered: