Quotes: generate arbitrary trait or class implementations

[ghik]

The problem

Using Scala 2 macros, we have built an RPC framework that can automatically generate implementations of arbitrary traits. These implementations are effectively network proxies that translate every method call into a network request. When the request is received by the RPC server, a reverse translation is performed and a real implementation of this trait is invoked.

A very simplified example of this mechanism:

trait RawRpc {
  def invoke(methodName: String, parameters: Map[String, Json]): Future[Json]
}
object RawRpc {
  def asReal[RealRpc](rawRpc: RawRpc): RealRpc = macro ...
  def asRaw[RealRpc](realRpc: RealRpc): RawRpc = macro ...
}

The asReal macro needs to do the following:

• enumerate all (abstract) methods of RealRpc
• for every parameter of every method, find a typeclass instance that would allow it to serialize it into Json
• for every method's result type, find a typeclass instance that would allow it to deserialize it from Json (assuming that the result type is a Future)
• generate an implementation of every method that forwards it to rawRpc
• inspect annotations on methods and parameters in order to treat them in some special way

The asRaw macro needs similar capabilities in order to perform a reverse translation.

Situation in Scala 3

It's unclear to me whether something like this is possible with Scala 3. Looking at the Quotes API, it should in principle be possible if I was able to generate arbitrary trees, starting with ClassDef. However, it seems that currently ClassDef.apply is commented out and marked as TODO.

So my questions are:

• First of all, is this a valid use case for Scala 3 metaprogramming?
• If yes, can this be done with Quotes API?
• Are there any serious obstacles preventing methods like ClassDef.apply from being exposed?

(Note: I asked this question briefly on Gitter and it was recommended to me to create an issue.)

[joroKr21]

We also need to be able to implement traits in order to port cats-tagless to Scala 3. This is blackbox macro territory so it should be a perfectly reasonable use case for Scala 3 macros.

[nicolasstucki]

What are the classes that you are generating? Could you give a small example of the code you would have and what code is generated?

[joroKr21]

Example from cats-tagless (where ~> is FunctionK from cats and Stream is from fs2):

trait InvariantK[Alg[_[_]]] extends Serializable {
  def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G]
}

trait FunctorK[Alg[_[_]]] extends InvariantK[Alg] {
  def mapK[F[_], G[_]](af: Alg[F])(fk: F ~> G): Alg[G]
  override def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G] = mapK(af)(fk)
}

trait ContravariantK[Alg[_[_]]] extends InvariantK[Alg] {
  def contramapK[F[_], G[_]](af: Alg[F])(fk: G ~> F): Alg[G]
  override def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G] = contramapK(af)(gk)
}

final case class User(name: String, age: Int)

trait UserRepo[F[_]] {
    type Id

    def create(user: User): F[Id]
    def findAll(olderThan: Int): Stream[F, User]
    def pickOne(ids: Id*)(f: List ~> F): F[User]
  }

  val functorK: FunctorK[UserRepo] = Derive.functorK
  // macro generated
  new FunctorK[UserRepo] {
    override def mapK[F[_], G[_]](af: UserRepo[F])(fk: F ~> G): UserRepo[G] = new UserRepo[G] {
      type Id = af.Id

      override def create(user: User): F[Id] =
        fk(af.create(user))

      override def findAll(olderThan: Int): Stream[F, User] =
        FunctorK[Stream[*[_], User]].mapK(af.findAll(olderThan))(fk)

      override def pickOne(ids: Id*)(f: FunctionK[List, G]): F[User] =
        fk(af.pickOne(ids: _*)(ContravariantK[FunctionK[List, *[_]]].contramapK(f)(fk)))
    }
  }

Sorry, I think I messed up ContravariantK[FunctionK[List, *[_]]] but I think the idea is clear - arguments are in contravariant position so we can use ContravariantK[[F[_]] => ArgType[F]] to contramapK them.

[nicolasstucki]

Good, it seems that you can create that class in a much simpler way. As you seem to know that you want to implement an instance of FunctorK/RawRpc you can just use quoted code for that part, the rest will need some reflection.

Here is the basic idea

def asRaw[RealRpc](realRpc: RealRpc): RawRpc = ${ asRawExpr('realRpc) } 
private def asRaw[RealRpc: Type](realRpc: Expr[RealRpc])(using Quotes): Expr[RawRpc] = 
  '{ 
    new RawRpc {
      def invoke(methodName: String, parameters: Map[String, Json]): Future[Json] = 
        ${invokeExpr(realRpc, 'methodName, 'parameters) }
    } 
  }

private def invokeExpr[RealRpc: Type](realRpc: Expr[RealRpc], methodName: Expr[String], parameters: Expr[Map[String, Json]])(using Quotes): Expr[RawRpc] = 
  import quotes.reflect._
  ...

[joroKr21]

Note: we need to implement both an instance of FunctorK and UserRepo - and UserRepo is provided by the user.

[nicolasstucki]

Then the second might be problematic. I will have to have a deeper look.

[ghik]

Coming back to my example with RawRpc, let's assume a typeclass for Json serialization:

trait JsonCodec[T] {
  def encode(value: T): Json
  def decode(json: Json): T
}
object JsonCodec {
  def apply[T](implicit codec: JsonCodec[T]): JsonCodec[T] = codec
}

Then let's assume an example user RPC trait:

trait MyRpc {
  def handleStuff(foo: String, bar: Int): Future[Boolean]
  def doSomethingElse(baz: Double): Future[Unit]
}

An invocation of RawRpc.asReal for MyRpc would generate something like this:

new MyRpc {
  def handleStuff(foo: String, bar: Int): Future[Boolean] =
    rawRpc.invoke("handleStuff", Map(
      "foo" -> JsonCodec[String].encode(foo), 
      "bar" -> JsonCodec[Int].encode(bar)
    )).map(JsonCodec[Boolean].decode)
  
  def doSomethingElse(baz: Double): Future[Unit] =
    rawRpc.invoke("doSomethingElse", Map(
      "baz" -> JsonCodec[Double].encode(baz)
    )).map(JsonCodec[Unit].decode)
}

Conversely, an invocation of RawRpc.asRaw for MyRpc would generate something like this:

new RawRpc {
  def invoke(methodName: String, parameters: Map[String, Json]): Future[Json] = methodName match {
    case "handleStuff" => realRpc.handleStuff(
      JsonCodec[String].decode(parameters("foo")), 
      JsonCodec[Int].decode(parameters("bar"))
    ).map(JsonCodec[Boolean].encode)
    
    case "doSomethingElse" => realRpc.doSomethingElse(
      JsonCodec[Double].decode(parameters("baz"))
    ).map(JsonCodec[Unit].encode)
  }
}

Of course, the MyRpc is an arbitrary user trait. Also, our framework is more generic than the example above, because:

• RawRpc may take multiple forms so the macro may need to do reflection on RawRpc too (not just on user's RealRpc)
• some methods and parameters of RealRpc may be treated specially based on their signature, annotations, etc.

[nicolasstucki]

Unfortunately, it is not something we can realistically have by 3.0. We aim to have this in 3.1.

Meanwhile, we could see if there is a possible workaround. It looks like we could use a java.lang.reflect.Proxy to handle the calls to handleStuff/doSomethingElse. The code of this proxy would be the code you would generate for the bodies of those methods with some extra wrapping logic.

[ghik]

I see, thank you.
Unfortunately java.lang.reflect.Proxy isn't a solution for us because we need this to be working in Scala.js. The best thing now seems to compile these traits with Scala 2 (macro invocations will be limited to their companion objects) and consume (invoke & implement) them in Scala 3.

[nafg]

Javascript has a similar concept to Java's Proxy

[nicolasstucki]

Could that work in this instance?

[ghik]

In my case I will most likely wait for this issue to be resolved rather than use dynamic proxies of any kind.

[joroKr21]

Same here, proxy doesn't cut it because we want to modify the parameters / result type

[ghik]

Out of curiosity - why is it difficult to implement ClassDef.apply when it already has copy that looks almost exactly the same?

def copy(original: Tree)(name: String, constr: DefDef, parents: List[Tree /* Term | TypeTree */], derived: List[TypeTree], selfOpt: Option[ValDef], body: List[Statement]): ClassDef

I was wondering if I could emulate apply by creating some fake ClassDef tree e.g. '{new Object {}} and then calling copy on it.

[ScalaWilliam]

👍 for having this, only learned just now that I can't upgrade to Scala 3 because cats-tagless is not available

[cornerman]

I was wondering if I could emulate apply by creating some fake ClassDef tree e.g. '{new Object {}} and then calling copy on it.

I am in need for a solution too and ended up trying this copy workaround. It partly works, as you can then create the ClassDef, but I have an issue referencing its Symbol.

First, I am creating/copying a classDef:

val Inlined(_, _, Block((otherClass: ClassDef) :: _, _)) = '{class Foo}.asTerm
val classDef = ClassDef.copy(otherClass)("Bar", otherClass.constructor, List(TypeTree.of[T]), None, Nil) // I also tried using "Foo" as a name here as well

Then, I define this class and try to instantiate it with new:

Block(
  List(classDef),
  Apply(Select.unique(New(TypeIdent(classDef.symbol)), "<init>"), Nil)
).asExpr

That does nearly create what I want, except that classDef.symbol still refers to the old class Foo (even though it is not inside the tree) and not to the new class Bar. Not sure, what ClassDef.copy should actually do but it somehow does not create a new Symbol.

The generated code looks like this and it compiles - just instantiating the wrong class:

{
  class Bar() extends test.Command
  new Foo()
}

I have also tried to create Symbol.classSymbol("Bar") myself, but that does not resolve - the compiler gives an error on the call-site: Bad symbolic reference. A signature refers to Bar/T in package <empty> which is not available.

Is there any way to construct a Symbol for this macro-defined class Bar or should we just wait? :)

[cornerman]

I would like to help, implementing this feature. Are there any pointers or things that I should look out for?

[nafg]

The autowire approach should be more doable, because you don't have you actually create instances of the trait. The macro rewrites calls to methods of the trait.

The downside is that it's more verbose -- you have to tack on .call everywhere, and the macro is invoked everywhere, so the de/serializers need to be in scope at all the call sites.

[nafg]

Another approach, which is verbose but mechanical, is you provide a trait with a macro method called impl. Then the user has to extend the trait explicitly, but the right hand side of every method is simply the word impl. The macro sees what method it's called from and generates the appropriate implementation.

It's very quick for users to create the boilerplate - they extend their trait and the impl trait (or perhaps the impl macro is imported not mixed in), then either the IDE generates method stubs with a right hand side of ??? or you copy paste the compiler output which gives you those stubs, then you find/replace ??? with impl.

It's straightforward to codegen that too of course.

Definitely much less pleasant for the user but I think it's better than being blocked from upgrading.

[nafg]

A different idea for library design that's also suboptimal but possible with Scala 3 is instead of using a trait with methods and inheritance, you use a case class whose fields are functions. Unfortunately function parameters can't have names. Anyway that replaces the trait with abstract methods, then the server implementation is an instance of the case class with function values filled out rather than a subclass of the trait with the methods implemented, and then the client macro doesn't have to subclass anything, it just has to build an instance of the case class.

[LPTK]

Unfortunately function parameters can't have names.

Actually they can, unless I'm misunderstanding what you mean.

Example:

scala> val f: [A] => (x: A, y: A) => (A, A) = [A] => (_: A) -> (_: A)
val f: [A] => (A, A) => (A, A) = <function2>

scala> f(y = 1, x = 2)
val res0: (Int, Int) = (2,1)

[nafg]

Oh, great.

Here's another approach: https://github.com/propensive/polyvinyl

[jdegoes]

My company is interested in sponsoring a production-worthy and well-thought out implementation of this feature.

[ScalaWilliam]

amazing, thank you @nicolasstucki !

when do you expect this will be released?

[smarter]

when do you expect this will be released?

We try to not rush stabilizing APIs since we're bound by binary compatibility constraints afterwards, but one thing that would help would be to get feedback from users on whether the APIs implemented in #14124 actually solve their problems (see the PR changes for documentation and test cases).

To try it out, just set your scalaVersion to a recent nightly to have access to experimental APIs (scroll to the bottom of https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/ to find the version number), e.g.

scalaVersion := "3.2.0-RC1-bin-20220419-ef16034-NIGHTLY"

[cornerman]

@smarter I am currently trying to use this feature. So far, it generally works :)

But I have one issue right now. I want to create an instance that implements a given trait. So I collect all applicable members of the trait and implement them through some arranged protocol. Right now, I am just not able to programmatically create a new MethodSymbol for another class from a given MethodSymbol. Example:

    val tree = TypeTree.of[T]

    val methods = tree.symbol.memberMethods

    def decls(cls: Symbol): List[Symbol] = methods.map { method =>
        //  just reusing the method here does not work, I guess, because the owner is wrong. So try to create a new MethodSymbol.
        Symbol.newMethod(cls, method.name, ???) // How to reuse the type definition of the given method? I failed to build `MethodType` myself from the information available. My tries have all ended up in assertion errors that were difficult to understand.
    }

I would then create the Symbol.newClass with the given information and a body implementing those methods.

How can I do this correctly? Any help would be appreciated.

[nicolasstucki]

@cornerman could you share a self contained minimal example that shows the failure you are getting. The assertion failure on its own might also be helpful insight.

One thing that you might be missing is the override flag.

[cornerman]

@nicolasstucki Sure, I can. Actually I think, I have fixed the main issue by using the changeOwner function on the MethodSymbol. It works now. You can see the code here: https://github.com/cornerman/scala3-macro-testing/blob/master/src/main/scala/Test.scala#L76

Is this generally the correct approach?

There is still some TODOs in the code:

• How can I create a nullary method symbol? Because commenting this in does fail. https://github.com/cornerman/scala3-macro-testing/blob/master/src/main/scala/Main.scala#L6
• How to copy the flags of the method? Which ones are safe to copy? Because taking all flags does fail. https://github.com/cornerman/scala3-macro-testing/blob/master/src/main/scala/Test.scala#L76

[nicolasstucki]

It seems we are missing the Symbol.info that allow us to get the type of the definition. Nice workaround.

I created #15024 adding this new method as Symbol.tpe. You can see in the test case your code using this method.

Also, to create a nullary method you have to give it a ByNameType.

[cornerman]

Awesome. Thank you! And the ByNameType works :)

[ScalaWilliam]

@ghik was this issue solved for you? I notice it is closed but was not quite sure.