Alternative pipeline strategies and operators

[dead-claudia]

Bringing over a summary of some of the insane amounts of discussion that occurred in the bind operator proposal's repo regarding function pipelining, since it's far more relevant here than there now (since this is a TC39 repo now dedicated to the topic), and I think we could use a historical record and starting point for this. Apologies for the long length.

---

Pipeline strategies

There are three primary strategies you could use to pipeline function calls:

_{(For expository purposes, I'm using @ for all three variants to make them less visibly different while still making it seem like a fitting operator. Note that I'm not actually proposing it, since it conflicts with decorators.)}

1. foo@bar(...) ↔ bar.call(foo, ...)

   This was what was initially proposed in the bind operator proposal. It is fairly object-oriented in that it views callees more like extension methods than just functions, and calls them accordingly – the context (foo in this case) is passed as this, and the arguments as you would expect.

2. foo@bar(...) ↔ bar(foo, ...)

   This was proposed by @gajus in this particular comment. It is more procedural, and can be viewed as threading a value through several functions much like Clojure's thread-first macro.

3. foo@bar(...) ↔ bar(...)(foo)

   This was proposed by @gilbert in this repo. It is more functional in nature, since it involves piping through a series of single-arg functions, usually returned from calling another function (in this case, from bar(...)).

Pros and Cons

Of course, these are not all equal, and they each have their strengths and weaknesses. None of these are magic silver bullets, and they all have their shortcomings. I did go into some detail on this previously, but I thought I'd relay it here, too.

1. foo@bar(...) ↔ bar.call(foo, ...)

   Pros:

   • It plays well with the philosophy that it's an extension method of sorts. (This could also be considered a con by some.)
   • It lends itself well to borrowing generic native prototype methods like Array.prototype.join and Object.prototype.toString.
   • It reduces the need for func.call, func.apply, and Reflect.apply.

   Cons:

   • Existing third-party library support is fairly low. (This could change depending on if this variant is accepted.)
   • The heavily object-oriented appearance is likely to be contentious, for similar reasons to why ES6 classes have met some sharp criticism.
   • It naturally begets a binding variant like foo@bar being equivalent to bar.bind(foo). This kind of behavior is much more contentious, and feel free to explore the bind operator proposal's issues to see this in action.
   • Due to its reliance on this, it's impossible to define such utilities using arrow functions, and inline utilities are pretty verbose (which could be rectified):

     foo()
     @function () { return this + 1 }()
     @bar()

2. foo@bar(...) ↔ bar(foo, ...)

   Pros:

   • It lends itself well to concise helpers defined via arrow functions.
   • It avoids most of the issues with this, including much of the contention.
   • It encourages function-based APIs without becoming functional enough to be contentious.
   • It works well with many existing libraries such as Three.js, Lodash, and the like. (This is a pretty weak plus, since libraries can adapt to any of these.)

   Cons:

   • Use of native prototype methods have to be wrapped somehow. (This is a pretty weak minus, since var dethisify = Function.bind.bind(Function.call) can trivially wrap them.)
   • It looks somewhat object-oriented even though it avoids this. (This could turn off some functional programming fans.)
   • It may surprise newcomers already familiar with object-oriented languages, due to it not using this.
   • Defining inline helpers is slightly awkward:

     foo()
     @(x => x + 1)()
     @bar()

3. foo@bar(...) ↔ bar(...)(foo)

   Pros:

   • It lends itself well to functional idioms. (This could be considered a con to some.)
   • It avoids this and related problems entirely.
   • It allows easy use of inline helpers:

     foo()
     @(x => x + 1)
     @bar()

   Cons:

   • The functional nature of it is likely to be contentious.
   • Existing library support exists, but is limited. (This could change depending on if this variant is selected.)
   • It requires currying for calls to not depend on this specific form for ease of use. (This is a similar issue to the this dependence of the first option.)
   • It's substantially harder for engines to optimize, since it enforces an observable closure allocation at least, and currying is hard to optimize without native support. (Engines can adapt for the former, and a language-level curry proposal would solve the latter.)

   In addition, this could be simulated by using a native composition operator/method and immediately calling it with the value to pipe through, although it would cause the functions to all create their closures before any of them could be evaluated. (This does make optimization a bit more difficult without inlining first.)

Alternate operators

As expected, there have been numerous other operators already suggested for signaling pipelining. Here's a list of many of them:

• :: – proposed with the original bind proposal
• ~> – proposed in this issue
• -> – proposed in this comment
• |> – proposed originally in this repo
• .. – proposed in this issue
• & – proposed in this issue, although it's obviously a no-go

I've noted that ~> is a bit awkward to type, and a few others pointed out that .. runs the risk of confusion with spread, a potential future range syntax. The remaining 3 (::, ->, and |>) are likely more viable, although I'll point out that |> is itself a little awkward to type in my personal experience (small hands 👐 😄).

[gilbert]

Can you explain how option 3 is substantially harder to optimize? Or are you commenting on the performance of curried functions in general?

[flying-sheep]

For prior art I’d also like to mention R’s magrittr, cornerstone to the very popular “tidyverse” programming style. It uses a mixture of 2 and 3, with a few extensions:

1. str %>% tolower and str %>% tolower() are the same thing: tolower(str)
2. lst %>% sort(TRUE) works like variant 2: sort(lst, TRUE)
3. str %>% grep('pat', .) is a way to use the LHS as a specific parameter instead of the first one: grep('pat', str)
4. num %>% { . + 1 } is a way to use expressions in the pipe: num + 1 (doesn’t fit JS IMHO)
5. . %>% foo is an anonymous function (useless for us because we have =>)

I’m not proposing to adapt this. I think it’s too complicated, and especially confusing with higher-order functions:

const adder = rhs => lhs => lhs + rhs
const plus1 = adder(1)

// work as expected
2@plus1      //→ plus1(2)
3@plus1()    //→ plus1(3)
// the first one is a bug bound to happen
4@adder(2)   //→ adder(2, 4)
4@(adder(2)) //→ adder(2)(4)

Other people might think that the pragmatic usefulness outweighs its complexity, though.

[flying-sheep]

Regarding syntax:

• -> already has many meanings: it’s about pointers in many languages, creates a function in coffeescript, and assigns things in R, scala, and definitely more. I haven’t seen it anywhere with piping semantics, so we’d probably add yet another meaning to the pile.
• :: IMHO really only makes sense for variant 1, as it’s for binding and scope access in some languages.
• |> is good to type for germans* 😜 and is used in some functional languages. My feeling is that of the three it’s used most for piping in other languages.

*The key in the bottom left is used to type <>| on german keyboards. In R, %>% is awkward to type, so editors usually have CtrlShiftM bound to inserting it.

[dead-claudia]

@gilbert

Can you explain how option 3 is substantially harder to optimize? Or are you commenting on the performance of curried functions in general?

Yes and no.

For the operator itself, closure elimination is a necessity, but in order to do that, you have to first inline the function to eliminate the closure, and then come back to inline the closure itself. JS engines do optimize IIFEs, but that level of inlining is usually a bit deeper than what they tend to look for (and this is why heavily functional JS tends to be a bit slow compared to the procedural equivalent). It's not out of the realm of what they can do (consider native Promise performance and recent V8 optimizations for Array.prototype.map and friends), it's just significantly harder for them to do it. It's a pretty weak minus, but I've included other similarly weak arguments in the list, too.

For curried functions, it's something that's exceptionally difficult to do performantly at the language level (as opposed to natively). I'll save the gory details for another time, though.

[dead-claudia]

@flying-sheep

-> already has many meanings: it’s about pointers in many languages, creates a function in coffeescript, and assigns things in R, scala, and definitely more. I haven’t seen it anywhere with piping semantics, so we’d probably add yet another meaning to the pile.

Not really, considering Clojure's -> (thread-first) macro does literally the same exact thing as option 1/2. (Method calls and function calls use identical syntax in that language.)

|> [...] is used in some functional languages. My feeling is that of the three it’s used most for piping in other languages.

I'd agree, but it doesn't look very JS-y. (Most languages with that operator also coincidentally use f x y or f x, y instead of f(x, y), so the incredibly distinct, tall bar of separation actually helps readability quite a bit.)

|> is good to type for germans* 😜 [...]

*The key in the bottom left is used to type <>| on german keyboards. In R, %>% is awkward to type, so editors usually have CtrlShiftM bound to inserting it.

Consider yourself lucky. 😄 QWERTY keyboards have ?// there, but | is located between the Backspace and Enter keys, which is hugely awkward IMHO.

[gilbert]

For the operator itself, closure elimination is a necessity

If you mean an operator call with a function like this:

var y = 10
var result = get() |> x => x + y

then yes, closure elimination will yield better performance. But as I understand it, the transformation is so straightforward that I wouldn't think to call it substantially harder for engines to optimize.

[dead-claudia]

@gilbert Don't forget this common case (as well as less contrived ones):

var add = x => y => x + y
var result = get() |> add(10)

Also, consider curried functions - those are when things get funky. (Ramda and lodash-fp both come to mind here, where you'll need a lot more than simple closure elimination, and the use of curried functions is quite idiomatic in those circles.)

[gilbert]

The performance of that example is no different than what would happen today:

var add = x => y => x + y
var result = add(10)( get() )

This is orthogonal to the pipeline operator. Whether you use the operator or not, you're still working with a closure created by add's curry.

[littledan]

For the surface syntax, I like sticking with |> because of its presence in other languages.

For the semantics, I wrote about my thoughts in a slide deck that I intend to present to TC39 soon. Personally, I like the track that this repository is on.

This three-way semantic question is really important, and I hope to get committee feedback on it when the proposal is presented.

[Volune]

There are subtleties in the third case that I think haven't been addressed:

If x @ a.b is do {let tmp = x, f = a.b; f(x)}
Where the right part is evaluated, then used as a callable object.
This example makes sense if the pipeline operator is just a new operator with the right precedence.

Or x @ a.b is do {let tmp = x; a.b(x)}
Where the value of the left part is magically appended as an argument of the right part, then the whole is evaluated.
In this example, I think it's misleading to call the new syntax an operator. This behavior is closer to what I would expect with partial application.

I am concerned about these differences because of the impact on object methods, or await/yield keywords. I think the second example is more powerful, but as it is, less readable.

[littledan]

@Volune In my draft spec text, it's more like the second one (in that the receiver is preserved). See #52 . IMO this kind of "power" is pretty modest and the bare minimum of what you'd expect from something built-in to the language. Feedback appreciated!

[MeirionHughes]

In relation to the operator: is it out of the question to simply use >> and define it as a pipeline if rhs is function?

I mean:

console.log(
  10 >> function (a) {
    console.log(a);
  }
);

is currently parsed by nodejs at least, but is seemingly all ignored and the output is still 10.

[littledan]

>> is not really available as an operator. It already has a meaning, as signed right shift.

[grofit]

Personally I prefer the :: (or ~> at a push) operator, as it is less moving round the keyboard and they don't look as abrupt when used in syntax, I do like the idea of >> but I appreciate its already got one meaning/context and adding another could be confusing.

[MeirionHughes]

@littledan yes, but number >> function and function >> function is currently undefined, right?

[littledan]

@MeirionHughes Well, it's defined--it will cast the function to a number in a particular way, getting 0, which you're calling "seemingly all ignored". For example, "hello" >> (x => console.log(x)) evaluates to 0 rather than printing "hello" as you might hope.

[MeirionHughes]

okay yeah, it is defined what will happen to lhs and rhs; but from the looks of it its just coercing both sides to an int32: https://www.ecma-international.org/ecma-262/8.0/index.html#sec-signed-right-shift-operator

I don't think that explicitly rules out the possibility of checking if the rhs is a function, and if so then pipe lhs into rhs - unless, of course, there is a use-case for wanting to coerce function to 0?

ToUint32(function) -> ToNumber(function) doesn't seem to be defined? (I delved deeper - function is just Object) https://www.ecma-international.org/ecma-262/8.0/index.html#sec-tonumber so presumably you end up with ToUint32(function): NaN -> +0

[littledan]

@MeirionHughes Everything here is defined. It does go through NaN as you say, deterministically.

[MeirionHughes]

sorry @littledan, I should have delved deeper. You're right, its probably a bad idea because you could technically do this:

let func = function () {}
func[Symbol.toPrimitive] = function(){return 2};
console.log(16 >> func);

4

oh well :(

[aikeru]

As a developer who is invested in the future of pipelining in JavaScript (I, and teams I've worked on, have heavily used the compelling :: operator as implemented in Babel and understood by WebStorm in production code), I'd like to make an appeal for a simple proposal that gets us, the users of the language, the bare minimum that is easily understood by developers today and already serves us well.

Simply that, from a basic POV, foo::bar transpiles to bar.bind(foo) and foo::bar() becomes either bar.call(foo) or bar.apply(foo), which enables chaining functions and extension methods in JavaScript. This is extremely powerful. I can use it both with objects and to enable functional composition. The transpilation is simple and the macro-performance concerns seem to be negligible. It seems like a natural fit for the flexible, multi-paradigm language that JavaScript is.

I love JavaScript and I'm very passionate about the language. I would hate for anything to be introduced that would cause damage or grief. I'd also hate to go back to the dark ages when I had to write functions "backwards" or compose them with chain() or similar.

If you took a moment to read this, thank you. I hope I've contributed something positive to this discussion. I'm very happy to see it continue to evolve, and I'll be excited for whichever pipeline strategy becomes part of the language.

[littledan]

@aikeru Thanks for your contribution. There's two cases in your desugaring, so I want to ask for details about the two parts separately.

For the call semantics, I'm wondering, what does it enable to have the left argument to the chaining operator be passed as the receiver (as you are suggesting, and as in the previous :: proposal) vs the sole argument to the function (as in this proposal)? Do you find it to be "more object oriented"? Do function parameters not work well for your code base?

For the bind semantics, I'm wondering what you think of the partial application proposal. You can do things like bar(foo, ...) to tersely get a function which is bar, but always with foo added in as the first argument, for example. Would this meet your use cases?

[ljharb]

One use case I find particularly compelling is borrowing built-in prototype methods - eg, someArrayLike::Array.prototype.forEach(callback) - the pipeline operator (as commonly specced) is not a substitute for the bind operator (as commonly specced) for this use case.

[ljharb]

An additional use case is caching builtin methods for robust scripts; eg:

const { map, filter } = Array.prototype; // assuming everything is untouched at execution time

export default (x) => {
  x.map(…).filter(…); // breaks if `x` is not an array or if `delete Array.prototype.map`
  x::map(…)::filter(…); // 100% robust against later modification of Array.prototype
};

[aikeru]

@littledan Thanks for your response, and your question. I'll try to thoughtfully answer it from my personal experience.

For the call semantics, I'm wondering, what does it enable to have the left argument to the chaining operator be passed as the receiver (as you are suggesting, and as in the previous :: proposal) vs the sole argument to the function (as in this proposal)?

I hope I understand -- if you mean that instead of being passed as the this context, it would be passed as the first argument to the function, that sounds great. Any functions written against the current implementation (as well as prototype built-ins) could be used like so:

// before (current Babel)
foo::bar("baz") ---> bar.call(foo, "baz")
// after
foo::bar.call("baz") ---> bar.call(foo, "baz")

I suspect this would make FP style a bit more natural/convenient, which is great.

Do you find it to be "more object oriented"? Do function parameters not work well for your code base?

I personally do think that the current implementation works more naturally with the new classes, etc. However, function parameters would work fine as well.

For the bind semantics, I'm wondering what you think of the partial application proposal. You can do things like bar(foo, ...) to tersely get a function which is bar, but always with foo added in as the first argument, for example. Would this meet your use cases?

There are some really great things in that proposal, and I had not read it before. One thing that is really nice about the current :: implementation is, once you understand how the this binding works, whatever arguments you pass, if any, map exactly as you'd expect for any function invocation.
I would hope that whichever way it goes, I would get the following behavior, as I find it very convenient:

foo |> bar() ---> bar(foo)
foo |> bar("baz") ---> bar(foo, "baz")

It's been a long day and though I skimmed over the examples I did not see one that illustrates what happens if you do not pass a placeholder, and instead only pass one argument (I apologize if I missed it) after a pipe operator. If this proposal means I would be forced to pass a placeholder in order to get the above, that would be a bummer, but if it becomes baked into the standard, it would still be a great way to chain / pipeline functions together.
Here's what I mean:

// What happens?
const foo = "hello"
function bar(a,b) { console.log(a,b) }
foo |> bar("baz")

Thanks again for your time and thoughtful consideration.

[aikeru]

@littledan
PS One of the greatly simplified things about the :: operator as implemented today is that no new scope is created, nor anything that looks like a scope. It's also very simple/small in - uh - "scope" - which may or may not be a good thing, but seems to me like small changes are usually easier, both to implement, to teach and to fully vet. It makes me wonder, is it possible to ship the |> operator by itself, without placeholders or the new ... behavior?

Anyway, I've said a lot, and I'm sure there are people thinking about this very hard and I appreciate their efforts! Thanks for letting me participate.

[dead-claudia]

Just thought I'd drop in and say that I found this ensuing discussion very interesting to read and follow. 😄

_{(Mission accomplished! :-))}

[MeirionHughes]

I'd just like to also point out that it would be useful if a thought could be given to how the pipeline operator works with Iterables and Generators; presumably, it should just-work™:

function* where(source, predicate){
  for(let item of source){
    if(predicate(item){
      yield item;
    }
  }
}

[1,2,3,4] |> _ => where(_, x => x > 2);
[1,2,3,4] |> where(x => x > 2) ; // needs partial application proposal? 

[grofit]

This is another problem I find with the |> operator, it looks odd unless you put a space in there, so using the below as an example:

[1,2,3,4] |> where(x => x > 2);

If I were to use either of the other operators they don't need the space, are easier to type* and look more readable without spaces.

* = for me on a UK keyboard anyway, due to most dev symbols being right hand side

[1,2,3,4]::where(x=> x> 2); // looks fine without spaces
[1,2,3,4]~>where(x=> x> 2); // looks ok without spaces, although less succinct as the previous example
[1,2,3,4]>>where(x=> x> 2); // In some ways I do prefer this, but all of them look easy enough to read

[1,2,3,4]|>where(x => x > 2); // looks odd because of the | symbol being too difficult to read there

[flying-sheep]

:: certainly looks best without spaces. the others look too busy.

[littledan]

@aikeru "adding as the first argument" is a little vague; it can correspond to pipeline strategy 2 or 3 above, as defined by @isiahmeadows . Your examples are strategy 2, but this proposal is currently going for 3. Now is the time to think through these tradeoffs!

In option 3, the function is called with the left operand as the sole argument, not added to the arguments list. You can combine with the partial application proposal to get the same effect:

x |> f(?, y);

// is similar to

f(x, y);

The advantage of this version is that the first argument isn't as privileged--partial application can put the argument anywhere.

What do you mean by a "placeholder"?

[littledan]

@MeirionHughes I believe all that should just work, modulo the decision about option 2 vs option 3. If we go with option 3, and have partial application, the code example would look like

[1, 2, 3, 4] |> where(?, x => x > 2)

[aikeru]

@littledan please read "placeholder" to refer to the ? symbol.
I think I understand now. Under the current Babel implementation I can write...

function map(fn) { return fn(this) }
2::map(x => x +1) // 3

I suppose under option #3 with a |> the same functionality would be achieved with...

const map = (fn) => (val) => fn(val)
2 |> map(x => x + 1)

Please correct me if I'm wrong!

I'd like to freely share my opinion on option 3, which you mention is the current path of the proposal.

It feels like option 3 is awkward in JavaScript and the ? placeholder argument and the ... partial application, while possibly great features, are trying to compensate. It seems to be inspired by languages like F#, where function currying is the default and very natural. At one time, I thought that option #3 or similar (I had read the es pipeline operator proposal) was the better of the two ideas (as contrasted against option #1, which I understand is the current Babel implementation), but having worked with option #1 extensively, it feels like option #1 is natural in JS, whereas option #3 is trying very hard to bring pipelining from other languages in the same style that they do it.

If I'm trying to explain the current bind operator implementation to a newbie, I start with teaching .call(), .apply() and .bind() and then it's a natural and elegant extension of those features.
For the |> operator in option #3, along with the overloaded ? symbol, it seems like something new and different.
There seems to be a larger number of concept to cover, and option #3 doesn't seem complete without the additional proposal for partial application, etc.

I hope I haven't offended anyone, and whichever proposal or option makes it, I'll be cheering for it and helping my fellow developers master it.

PS I wrote this from my cell so my apologies for any mistakes. I'll review it again and edit it when I can.

[littledan]

@aikeru I agree that 3 only makes sense with the partial application proposal based on ?. If that proposal seems unnatural or too complicated to many programmers, it might be better for us to go with 1 or 2. What do you think about 1 vs 2?

[MeirionHughes]

might be a hard sell on 1: foo@bar(...) ↔ bar.call(foo, ...) because ts39 disliked the 'abusing' of this.

2: foo@bar(...) ↔ bar(foo, ...) is nice, simple, and solves the non-trivial chaining issue we currently have in understandable fashion.

Unless backward pipeline support #3 is desired, I'd personally vote for :: as the operator too, on grounds of simplicity and readability.

result = await [1, 2, 3, 4]::where(x => x > 2)::select(async x => x * 2)::first();
result = await [1, 2, 3, 4]|>where(x => x > 2)|>select(async x => x * 2)|>first();

[aikeru]

@MeirionHughes
#2 also makes it easy to get the previous functionality or work with built-in prototypes, like:

// Today I might do...
foo::Array.prototype.slice()
// With option #2 it's just
foo::Array.prototype.slice.call()

I could easily refactor existing code that uses :: today to option #2, by either adding .call(...) or adding a new argument at the beginning and changing this to that argument.

[MeirionHughes]

@aikeru aye, I have a sneaky suspicion that the babel binding-operator transform is being used in the wild quite a lot.

Jeeewiz.... 1.5 million downloads last month. I would think that that download count is a testament to the fact that chaining is a problem people need fixing...

So its useful to consider your refactor is a simple fix.

[ljharb]

Refactoring to add .call isn't acceptable to me, because I want my code to be robust against delete Function.prototype.call.

[aikeru]

@ljharb
Option #1 is probably my favorite but it seems like you are making it robust by aliasing the function from the prototype (ie slice). Couldn't you just alias the "call" off of that function?

const callSlice = Array.prototype.slice.call
foo::callSlice()

[ljharb]

@aikeru no, because in your example, callSlice === Function.prototype.call - the slice isn't preserved anywhere. You'd need Function.call.bind(Array.prototype.slice), or the proposed ::Array.prototype.slice.call.

[gilbert]

Another point to consider: private fields are already in stage 3. Ironically, this seems to make option 1 less useful. Take the following example:

obj::foo()

Here foo is made to look like an extension to f, but it is actually not; foo, as I understand it, will not have access to obj's private fields.

[ljharb]

@gilbert neither would obj.foo = function () {}; obj.foo() - the dot (or bracket) doesn't guarantee the function has access to private fields, only the lexical position of the declaration does.

[tvald]

In other languages (C#), extension methods don't have access to private fields.

[aikeru]

With respect to the bind operator, babel-plugin-transform-function-bind now shows over 1.6million downloads (up from 1.5million when @MeirionHughes mentioned it 11 days ago) over the last month on npm. I'm guessing this doesn't equate 1:1 to usage, but it does mean the tooling is there for developers to use it.

My new current struggle is that while the :: operator has done amazing things for myself and my colleagues up to this point, it complicates the adoption of other tooling (ie: it has limited adoption by IDEs, neither Flow or TypeScript will adopt it as a stage 0 feature) without sending it through a transpiler first.
As a technology leader for my current employer, I'm seriously having to consider abandoning :: for something lodash/etc-based, because we would like to adopt a type annotation system. This makes me sad (lodash is great, losing :: is not).

I'm still very hopeful that some variant of this will find its way into stage-1-and-beyond status. I know we wont be the only ones celebrating this event, when it finally happens. I'm certain it will bring joy to many JS developers. It wont be the operator everybody wants, but it'll empower everyone to write more elegant code.

Thanks.

[fahad19]

I mentioned about Option 2 in #59:

function double(x) {
  return x * 2;
}

function add(x, y) {
  return x + y;
}

const score = 10;

const newScore = score
  |> double // output of `score` is first argument: `double(10)`
  |> add(7) // output of `double(10)` is first argument: `add(20, 7)`

console.log(newScore); // 27

[littledan]

I presented on this question at the September 2017 TC39 meeting. My understanding of the committee's feelings was roughly support for the current strategy, plus a number of of additional feature requests.