Casting self to a trait inside a default method is unsound

[yorickpeterse]

Please describe the bug

When defining a default method in a trait, we currently allow one to cast or move self into a trait. For example:

trait TraitA {}

trait TraitB: TraitA {
  fn invalid1 {
    self as ref TraitA
  }

  fn invalid2 {
    let a: ref TraitA = self
  }
}

class A {}

impl TraitA for A {}

impl TraitB for A {}

This is unsound if the trait is implemented for a type which can't be cast to a trait, such as Int or an inline type as introduced per 8ca46bf and #783.

The mentioned PR introduces a self_type field on the TraitInstance structure, which we can use to detect if some type is self or not. This in turn can be used to disallow such casts. For example:

diff --git a/std/fixtures/diagnostics/cast_self_to_trait.inko b/std/fixtures/diagnostics/cast_self_to_trait.inko
new file mode 100644
index 00000000..0686be5e
--- /dev/null
+++ b/std/fixtures/diagnostics/cast_self_to_trait.inko
@@ -0,0 +1,23 @@
+fn example[T: TraitA](value: ref T) {}
+
+trait TraitA {}
+
+trait TraitB: TraitA {
+  fn invalid1 {
+    self as ref TraitA
+  }
+
+  fn invalid2 {
+    let a: ref TraitA = self
+  }
+
+  fn valid {
+    example(self)
+  }
+}
+
+class A {}
+
+impl TraitA for A {}
+
+impl TraitB for A {}
diff --git a/types/src/check.rs b/types/src/check.rs
index ecf75e04..f858a090 100644
--- a/types/src/check.rs
+++ b/types/src/check.rs
@@ -747,14 +747,22 @@ impl<'a> TypeChecker<'a> {
             },
             TypeId::TraitInstance(lhs) => match right_id {
                 TypeId::TraitInstance(rhs) => {
+                    if lhs.self_type {
+                        return false;
+                    }
+
                     self.check_traits(lhs, rhs, env, rules)
                 }
                 TypeId::TypeParameter(_) if rules.kind.is_cast() => false,
                 TypeId::TypeParameter(rhs) if rhs.is_copy(self.db) => false,
-                TypeId::TypeParameter(rhs) => rhs
-                    .requirements(self.db)
-                    .into_iter()
-                    .all(|req| self.check_traits(lhs, req, env, rules)),
+                TypeId::TypeParameter(rhs) => {
+                    let mut lhs = lhs;
+
+                    lhs.self_type = false;
+                    rhs.requirements(self.db)
+                        .into_iter()
+                        .all(|req| self.check_traits(lhs, req, env, rules))
+                }
                 _ => false,
             },
             TypeId::TypeParameter(lhs) => match right_id {

Applying the above diff introduces another problem: std.iter.Peekable is implemented such that it expects a boxed Iter value as its input. This means that std.iter.Iter.peekable becomes invalid with the above diff applied, with no suitable workaround. To resolve that, we also need to re-introduce Self as a type such that peekable() can be implemented as fn pub move peekable -> Peekable[T, Self] with the Peekable signature changed to Peekable[T, I: mut + Iter[T]].

Blocked by

• Find a way to reintroduce Self types without the complexity #643

Operating system

Fedora

Inko version

main