Update to 1.2.0-RC release

[nomisRev]

Hey all,

Here is a WIP PR that updates the repo to 1.2.0-alpha.63 which is planned to be released end of next week. This is the last minor release before 2.0.0 later this year, and thus includes all deprecations and new APIs to guarantee smooth transition period and a binary compatible release supporting all new APIs.

It has been extremely hard to gather feedback on the changes listed and discussed below, so any and all your thoughts are very welcome. We want to guarantee the success of functional programming in Kotlin by faciliting a strong core for functional programming in Kotlin, and thus we want to embrace common use-cases in the community and work together with everyone who's interessted in functional programming.

So I'd love to hear your thoughts on everything mentioned below, so we can work together to facilitate some of the deprecated APIs in Quiver for those that like this fluent style APIs. And/or, we can discuss some APIs that you consider core and critical functionality and we can discuss to keep them in Arrow.

If we decide to move combinators from Arrow to Quiver, like traverse or zip we could even provide automatic migration through OpenRewrite. Their Kotlin support is evolving quite rapidly, and is looking very promising for automating these kind-of changes.

Thank you for the great efforts by building this library, and exposing it to the rest of the community!

Raise builder

I've added a Raise based builder, I wrote a low-level implementation to avoid unnecessary allocations. This makes the DSL more performant then flatMap, and can thus be used in many places instead of fluent APIs. We've found this to be much more approachable, and its the rationale about some removals of the API.

Currently I've added a OutcomeRaise type, but this can be completely removed once context receivers make into the language.

Since the DSL is built using Raise<E> rather than, Either<Failure<E>, Absent> it can seamlessly interop with any Either (or Raise<E>) based code. The OutcomeRaise DSL itself adds functionality to interop with Option, and nullable types. This allows leveraging monadic style DSLs in combination with suspend everywhere. I also added a recover DSL based error-handler, that can also interopt with Option and Either.

In the future this can be replaced with context(Raise<None>, Raise<E>):

context(Raise<None>, Raise<E>)
fun <E, A> Outcome<E, A>.bind(): A = when(this) {
  Absent -> option.raise(None)
  is Failure -> raise(error)
  is Present -> value
}

With as builder:

inline fun <E, A> outcome(block: context(Raise<None>, Raise<E>) () -> A): Outcome<E, A> =
 either {
   option { block() }
 }.toOutcome()

Similarly, we could currently also build a DSL based on None and E, but I choose to write a slightly more optimised version in this PR. Here is a gist of how that looks.

Constraining E : Any is also possible, but I think less desirable since it requires all dependent code to also be constrained E : Any. Here is a gist of how that looks.

Duplication in API

I found some APIs to be duplicated, and so far I just alias them removing the internal implementations.

• tapLef/tapRight are called onLeft and onRight in Arrow to be in line with Result.onSuccess and Result.onFailure. These can also be used for forEach, but inside the DSL we often just use also.

• Map.getOption is called getOrNone in Arrow to be in line with getOrNull. Also it contained a nested null bug in Quiver. (Fix can be backported to current version).

Deprecations

Traverse

Traverse combinators have been deprecated in Arrow, and is planned to be removed. It seems however that Quiver is relying on it heavily, or the downstream projects. I am mostly interested in feedback here, and am wondering what your opinions are on the removal in Arrow. Do you think it's something we should keep in Arrow, or perhaps it would benefit from being moved to Quiver.

Our rationale for removing Traverse is that it's almost exclusively used with Iterable in the shape of Iterable<A>.traverse(transform: (A) -> M<B>): M<List<B>>. This pattern can now be achieved with map and bind.

There is however not a convenient alternative for traversing nested data types like Option, Either, etc. We found this non-blocking at the time for the removal. Since we haven't relied on it a lot ourselves working mostly with Either/Raise<E>, ? and suspend. Nor have we gotten feedback that this methods are critical to users. If this is not the case for you, I'd love to hear about it.

Zip

Zip is being deprecated in Arrow in favor of the DSL, zip suffers from the arity-n problem and is therefore limited to only 9 arguments. The DSL does not suffer from this problem, and provide the same functionality.

Since the new DSL is inline it naturally allows suspend when needed, and removes the use-case of zip in to avoid .eager { } usages.

Validated

Validated is being removed, and it's functionality is being projected over Raise. See mapOrAccumulate, and zipOrAccumulate.

Review notes

I noticed that some APIs are suspend, but don't use any suspend. These can be simply inline, and I also noticed that not all functions that can be inline are marked inline and this hinders allowing suspend inside the lambdas.

Since I wasn't sure how strict binary compatibility is being dealed with atm, I change this code atm.
Following methods are using suspend instead of inline:

• Either.tapLeft
• Either.forEach
• Either.leftForEach
• Option.leftForEach

Following methods are missing inline

• Either<E, Option<A>>.mapOption
• B.toEither
• Outcome.catch
• Outcome.catchOption
• Outcome.orThrow
• Outcome.traverse

I created a PR addressing these changes on main, but it results in an binary breaking changes. Albeit source-compatible.

[cwmyers]

@nomisRev We deeply appreciate you giving us so much of your time and energy both to reviewing quiver and the ongoing work in Arrow!

[cwmyers]

👍

[millyrowboat]

Hi @nomisRev!

Firstly, thanks so much for the engagement, encouragement and contribution here. It's a bit of a fangirl/boy moment for us to be able to give back to the FP in Kotlin community, and receive such a quick response from one of the maintainers of Arrow. These are the sorts of conversations we’ve been having internally for a long time now, and we’d love to share our thoughts externally here, on Arrow, FP in Kotlin and the purpose of Quiver.

Story time

First off to set the scene, it might be worth reiterating in more detail why Quiver was created and how it’s evolved. When we first adopted Arrow (~2020), it was immediately familiar as a lot of us came from Scala backgrounds, and were comfortable with Haskell. We used a lot of the types in the library and wrote Kotlin like we would Scala (IO, State, Validated etc – we did a couple of conference talks about this journey if you’re interested). As you know, Kotlin isn’t really set up for writing code this way, as it lacks a lot of advanced features of other languages (Typeclasses, HKT etc), which made the early Arrow API very awkward to use (.fix() anyone?).

This wasn’t very scalable as we onboarded new people, and also handed over some systems to other teams that found the code inscrutable.

The choice

Eventually, when Arrow v0.13 was released (the precursor to 1.0), we found that a lot of the types we were using were deprecated (IO, State etc), and that we had a choice to make.

Fork Arrow, or migrate?

Forking the library was never really an option. But this move did signal intent from the Arrow maintainers and 47 Degrees (Xebia) that the future of Arrow lay further away from trying to replicate more native FP languages, and become more idiomatic Kotlin. During that journey, we found that more and more APIs and methods we were using were being deprecated from Arrow.

Quiver began as a common library for a grab bag of types and extensions we were duplicating across all our services. Eventually, it evolved to plug the gap between the direction Arrow was going, which seemed to be a more streamlined and idiomatic library, and how we wanted to write FP in kotlin. It also doubled as a teaching tool for us to share FP knowledge with new starters in the team, and more broadly. For example, If you have an Either<E, Option<A>> and wanted to convert it to an Option<Either<E,A>> what function would you call and how would you find it? Without sequence or traverse you could just find the right map / flatMap combination but that takes significant mental gymnastics (esp for newbies!). Having traverse and friends in the library allows for quick solving of problems as the types clearly show what the function does.

While we think there’s room for improvement, in a way we’re still inclined to keep Quiver as a place we can add or bring back the methods and APIs that (personally) we found useful in Arrow, and have it available to (a minority maybe of) people who are reaching for those niche tools in Arrow without an answer.

Changes

So with that preamble in mind, regarding your changes:

• We would like to keep zip and traverse methods. We would support Arrow moving these to Quiver as optional methods (feel free to open a PR for these)
• We’re happy to take or leave Validated, as we’ve personally moved past it (Applicatives can be hard to explain and sustain in a Kotlin environment long term) but it could still be helpful to others in this space.
• Let’s keep discussing what might make sense to port over to Quiver more long term if Arrow continues down the streamlining path, we really hope to have an open dialogue with you all!

[nomisRev]

Hey @millyrowboat,

Thank you for providing feedback, and providing your background story. Really amazing you have been using Arrow since before 0.13.0, thank you for your continued support and trust in Arrow 🙌

This wasn’t very scalable as we onboarded new people, and also handed over some systems to other teams that found the code inscrutable.

This is exactly the problem we faced, and is why we set out to improve these things. Our goal has however always been to continue serving all use-cases, and functionality we had before. Which is also why we're so excited for context receivers in combination with Raise. They offer a natural solution for nested monads, or monad transformers. Some examples discussed at the bottom. This solves the use-case we originally had with final tagless or type class hierarchies.

(.fix() anyone?).

😂 I don't miss explaining this at all.

Fork Arrow, or migrate?

To be honest, I am glad you didn't fork as this would've been a big failure for Arrow IMO. Since we want to be a common ground for FP in Kotlin, and serve all use-cases. Preventing fragmentation of libraries as we've seen and experienced in other languages, that typically results in an even higher learning curve and even more complex migrations or interoperability.

we’re still inclined to keep Quiver as a place we can add or bring back the methods and APIs that (personally) we found useful in Arrow
people who are reaching for those niche tools in Arrow without an answer.

Yes, I absolutely agree and is why I am super excited to see Quiver come to live. It serves as a perfect middle ground for these kind of combinators. Arrow Core has a couple main goals two important ones are it should be easy to learn and lightweight. We've found that easy to learn and small API surface goes hand-in-hand.

It should be a core library to decrease the threshold to add it as a base dependency if you're building a functional library in Kotlin. For this reason it should also cover all common use-cases though.

With that in mind, I am on the fence about traverse, but I think zip is non-essential for Arrow Core since it's covered by the DSL without the arity-n problem. I've had to frequently explain why it's not available for arity-10 or higher. We've also found that having duplicated functionality is often also more confusing than helpful.

I will ignite a discussion with some of the other Arrow maintainers, and get back to you on traverse.

As mention on Slack, but not yet here. We're releasing 1.2.0-RC end of this week as a kind-of preview version of 1.2.0 (2.0.0 + 1.x.x deprecations). We're encouraging everyone to explore 1.2.0-RC, and let us know if any critical APIs were deprecated. This will also gives us a better idea on which APIs are critical for the Arrow community, and which ones can be perhaps be ported/moved to Quiver.

We're committed to serving everyone in the community, and therefore I am working with OpenRewrite with to provide automatic migrations/refactoring. And we'll stay on 1.2.x for some time so that everyone has ample time to provide feedback, and remove @Deprecated usage. This also gives us the opportunity to move some code to Quiver, before we actually make any breaking changes in Arrow itself and break downstream projects.

We’re happy to take or leave Validated, as we’ve personally moved past it (Applicatives can be hard to explain and sustain in a Kotlin environment long term) but it could still be helpful to others in this space.

Exactly, and I am happy to hear that ☺️ We've worked very hard to cover all use-cases of Validated within Either and this should serve all use-cases for Validated. I personally think it's only useful in cases where you write code over F, and need to obtain an Applicative + Parallel instance to summon parZip for Either to accumulate errors.

Some PR's as reference:

• mapOrAccumulate
• zipOrAccumulate
• bindNel inside xOrAccumulate
• Raise DSL xxOrAccumulate

For example, If you have an Either<E, Option> and wanted to convert it to an Option<Either<E,A>> what function would you call and how would you find it? Without sequence or traverse you could just find the right map / flatMap combination but that takes significant mental gymnastics (esp for newbies!). Having traverse and friends in the library allows for quick solving of problems as the types clearly show what the function does.

To come back to this, currently are answer to this is context receivers. Sadly they're not available yet, and will probably not be for at least a year 😞 Context receivers in contrast to monads are composable 🥳 It's also shown in the PR description where I replace Outcome with context(Raise<None>, Raise<E>).

Traditionally you have to specify the order of the monad: Option<Either<E, A>> vs Either<E, Option<A>> and then typically we use traverse & co to compose functions. With Context receivers this is no longer the case. A function constrained by context(Raise<None>, Raise<E>) can result in both Option<Either<E, A>> and Either<E, Option<A>>. Furthermore you can handle None or E in an ad-hoc basis, and you can compose other Raise, Option or Either values in an ad-hoc style.

context(Raise<None>, Raise<String>)
fun example(): Int = raise("failure")

val a: Option<Either<String, Int>> = option { either { example() } }
val b: Either<String, Option<Int>> = either { option { example() } }

context(Raise<None>)
fun option(): Int =
  either { example() }.getOrElse { it.length }

context(Raise<String>)
fun either(): Int =
  option { example() }.getOrElse { raise("The optional value was not found") }

We've however not entirely sure yet how people will react to context receivers, or how the hand-off to other teams would be as you've described as an issue above. The initial reactions and feedback we've received seems to be extremely positive, albeit that is very likely skewed impression of how new people might react when being handed such code.

I think this'll cover most use-cases where people using traverse today, replace it with simple invoke (or bind), but it's a new style of FP and is a bit foreign if you're more familiar with the style from Scala or Haskell.

An example:

fun a(): Either<String, Int> = "failure".left()
fun b(int: Int): Option<Int> = Some(int + 1)

context(Raise<None>, Raise<String>)
fun traverse(): Int {
  val x = a().bind()
  b(x).bind()
} // lift into `Either<String, Option<Int>>`, `Option<Either<String, Int>>`, or resolve effects independently.

Let’s keep discussing what might make sense to port over to Quiver more long term if Arrow continues down the streamlining path, we really hope to have an open dialogue with you all!

Let's do that! We're open for dialogue with all of you (and everyone in the community) ☺️

As a side-note: 47 Degrees | Xebia Functional is a sponsor of Arrow, and does not own any part of the OSS organisation or the library. In fact I've worked on Arrow for almost 3 years before joining 47 Degrees. Arrow is an OSS project, and the Arrow contributors are fully committed to the Kotlin/Arrow community and everyone in it. Their sponsorship takes form in building the website, offering work hours to maintain the library, write documentation, interact with the community and organise Kotlin (Arrow) related meetups / events, etc.

PS: Is anyone from the Quiver team coming to KotlinConf? I'd love to meet up and have a chat.

[millyrowboat]

@nomisRev

To be honest, I am glad you didn't fork as this would've been a big failure for Arrow IMO.

Us too!

With that in mind, I am on the fence about traverse, but I think zip is non-essential for Arrow Core since it's covered by the DSL without the arity-n problem. I've had to frequently explain why it's not available for arity-10 or higher. We've also found that having duplicated functionality is often also more confusing than helpful.

That makes total sense. TBH we have not hit the arity-10 problem yet 😅 but it makes sense for Arrow to support the broader version. Let us know how you go with traverse.

Exactly, and I am happy to hear that ☺️ We've worked very hard to cover all use-cases of Validated within Either and this should serve all use-cases for Validated. I personally think it's only useful in cases where you write code over F, and need to obtain an Applicative + Parallel instance to summon parZip for Either to accumulate errors.

The only time we've thought about using Validated over Either, is for more specific error types. We like the philosophy of very communicative type signatures (very Haskell type thinking 😆 ), where you should be able to understand what a function does without having to read its name. It's also a good start point to teaching Applicatives if people are really interesting in learning Functional Programming more broadly (beyond Kotlin).

Context receivers seem very exciting! We'll try them out and let you know how we go. Again, we may choose to maintain the older APIs as a familiarity thing but I remember saying I would never stop using IO when suspend came around and now I'm super happy with it 😄 So we'll see.

PS: Is anyone from the Quiver team coming to KotlinConf? I'd love to meet up and have a chat.

Unfortunately, we're all based in Australia and Amsterdam is very far away! I don't think we'll be making that conference 😅 but if you're ever down under please hit us up!

[nomisRev]

Hey @millyrowboat,

Unfortunately, we're all based in Australia and Amsterdam is very far away! I don't think we'll be making that conference 😅 but if you're ever down under please hit us up!

Ah, that is too bad. I'd love to come down under, but I sadly don't see it happening soon 😅 Unless someone wants to fly me over for a conference or something 😁

That makes total sense. TBH we have not hit the arity-10 problem yet 😅 but it makes sense for Arrow to support the broader version. Let us know how you go with traverse.

We discussed it, and we're planning to move forward with removing traverse but we're very excited to contribute it to Quiver. I actually have a couple of PRs ready, but I've also hit some roadblocks. I'll open the PRs, and we can discuss it there. (I'll open them tomorrow, so I can take the time to properly fill in the descriptions).

To that end I was curious. Could I convince you to support more Kotlin targets? We're going to follow the recommendation of Kotlin targets from 1.9.0, could I entice you to do the same? Nothing special should be done, since all the code in this library is vanilla Kotlin so just updating the Gradle setup a bit is sufficient. I'd be happy to help out in this regard too.

It's also a good start point to teaching Applicatives if people are really interesting in learning Functional Programming more broadly (beyond Kotlin).

That's definitely true, but we feel it's unnecessary in Kotlin. When I need to teach it it's typically when working with Scala or Haskell and so I only deal with it when I have too.

Context receivers seem very exciting! We'll try them out and let you know how we go. Again, we may choose to maintain the older APIs as a familiarity thing but I remember saying I would never stop using IO when suspend came around and now I'm super happy with it 😄 So we'll see.

I went through this journey as well. I can very highly recommend context receivers, I was very pleasantly surprised. For me it was a much bigger win than IO to suspend. All the pieces seem to fall together with context receivers even more. I'd be happy to discuss it more deeply if you're interested, I can also share some resources / examples. Feel free to DM me on Slack, or reach out in the #arrow channel ☺️

[dave08]

Any ETA on this now that Arrow's 1.2.0-RC is out? Thanks 😄!

[nomisRev]

Now that KotlinConf is behind us, I can polish up this PR and we can merge it ☺️ There are a couple other @Deprecated things I still need to move over to Quiver but I can also work on that this week 🥳

When they want to cut a release is up to the Quiver team, of course. I am also hoping to provide OpenRewrite scripts so people can automatically include quiver and migrate imports for 2.0.

[nomisRev]

@cwmyers @millyrowboat @hugomd @Synesso this PR is ready for review now ☺️

I will open some more PRs for traverse, zip, etc that are currently missing.

I am curious how you feel about making the project KMP, I would suggest doing that when bumping to 1.9.0 when it's released shortly and following the recommendation for targets. I would be more than happy to make the relevant changes in Gradle, but would need some help from your side to verify the publishing. I can test it locally with publishToMavenLocal.

[millyrowboat]

Thanks Simon! We're happy to merge this PR in as well.

We discussed as a team, and we're happy to support KMP! Feel free to raise the PR with the gradle changes; we'll test that publishing still works. If you have any issues, we can tap @swankjesse on the shoulder to help 😉

We will also await the traverse and zip additions! Looking forward to it. We're thinking about SLAs for PR reviews, and release schedules. We're going to try and align with Arrow as much as possible, but it will still be best effort (as most of open source is!). Thank you so much for helping and contributing here, it really is much appreciated.

[nomisRev]

🥳

Feel free to raise the PR with the gradle changes;
We will also await the traverse and zip additions!

I will raise the PRs in the coming week 🙌

It's been my pleasure contributing to Quiver ☺️