Add new overloads for Select and other LINQ methods

[AjayKMehta]

Currently, if you want to use a partially applied version of a function of arity greater than one (e.g. Func<T, A, R>) in a call to Select<T, R>(IEnumerable<T>, Func<T,R>) where the argument of type A is a local variable, you have 2 options, neither of which is great.

To illustrate these 2 options, consider the contrived example where you have the function defined below and want to use this in a call to Select for an IEnumerable<int>:

string MakeString(int len, bool toLower) => new string(toLower ? 'a' : 'A', len);

1. Create a closure:

bool toLower = userInput == "Lower";
var results = Enumerable.Range(1, 10).Select(i => MakeString(i, toLower));

2. To avoid creating a closure (and allocating), wrap the function (since C# doesn't support partial application like functional languages):

Func<int, string> MakeStringWrapper(bool toLower) => x => MakeString(x, toLower)
var results2 = Enumerable.Range(1, 10).Select(MakeStringWrapper(toLower));

This is not too bad since C# now supports local functions so you no longer have to pollute your code with private helper methods but this is still not ideal.

I propose adding an overload as an improvement over both these options:

public static IEnumerable<TResult> Select<TSource,TArg,TResult> (this IEnumerable<TSource> source, Func<TSource,TArg,TResult> selector, TArg arg1);

Simplified implementation (I get that actual implementation is more complex and specialized based on runtime types) would look like:

public static IEnumerable<TResult> Select<TSource,TArg,TResult> (this IEnumerable<TSource> source, Func<TSource,TArg,TResult> selector, TArg arg1)
{
    foreach(var item in source)
    {
        yield return selector(item, arg1);
    }
}

This is not a fringe issue as I have seen this a lot in LINQ code in the wild. Similar overloads could be added for other LINQ methods like GroupBy, OrderBy.

More overloads could possibly be added for the case where parameter order is switched for selector, i.e. Func<TArg,TSource,TResult> instead of Func<TSource,TArg,TResult> as shown above and for more arities.

The idea for this came from thinking of how adding *args and/or **kwargs to function signatures in Python addresses this situation 😀

def add(a, b):
      return a + b


def map_vals(iterable, f, *args, **kwargs):
	for item in iterable:
		yield f(item, *args, **kwargs)


list(map_vals(range(10), add, 2))
# [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

list(mapVals(range(10), add, b=-1))
# [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8]

[ghost]

Tagging @eiriktsarpalis as an area owner

[Joe4evr]

This is what I like to call "the TState pattern" (since the extra type parameter is usually called TState), and I suggested the same thing on Twitter some time ago, except I phrased to be more general, as opposed to only LINQ.

[Suchiman]

@AjayKMehta

2. To avoid creating a closure (and allocating), wrap the function (since C# doesn't support partial application like functional languages):

Func<int, string> MakeStringWrapper(bool toLower) => x => MakeString(x, toLower)
var results2 = Enumerable.Range(1, 10).Select(MakeStringWrapper(toLower));

This is no better than 1. in regards to allocations and worse in readability, this just moves the closure allocation to the expression x => MakeString(x, toLower) which closes over toLower which isn't an argument to that lambda but to the outer function, see sharplab.
You can also see it in VS if you hover the fat arrow:

The only true allocation free solution is the explicit TState version you're proposing, which has prior art in

runtime/src/libraries/System.Private.CoreLib/src/System/String.cs

Line 360 in 6d395de

public static string Create<TState>(int length, TState state, SpanAction<char, TState> action)

but doing this on a large scale will cause so much code duplication...

[AjayKMehta]

@Suchiman Yep, you are right. I somehow overlooked that. I quickly wrote the example code in LINQPad 6 (free edition) so I didn't see any warnings. Next time I'll use VS or VS Code 😄

IMHO having a few overloads would be great. Covering all the cases up to arity of 3 means 5 more overloads.

[svick]

Considering that LINQ is inherently fairly allocation-heavy, would this actually significantly decrease the allocations?

[eiriktsarpalis]

My feeling is that if allocations are a source of concern, then you shouldn't be using LINQ in the first place. For the particular issue you bring up, I believe it is almost always the case that you could refactor the code in a way such that only one closure is allocated, the one being passed to the Select method.

Given that we'd probably also need to support arities > 2, and the fact that it can be implemented trivially as an extension method like you demonstrated, I don't believe there would be a lot of value in adding this to the main API, though happy to be corrected.

[AjayKMehta]

@eiriktsarpalis Yes, I could write the code so that I have only 1 closure allocated.

This is not pressing. Thanks for looking into this.

[adamsitnik]

@AjayKMehta thank you for your proposal

I agree with all the points that @eiriktsarpalis has made and having said that I am going to close the issue.