Language Integrated Definition of Rules

[AdamSobieski]

I would like to propose a C# language idea: the language integrated definition of various kinds of rules.

Rule engine software appear to contain domain-specific languages with which to define interoperable rules. NRules contains the Fluent Rules DSL, for example, and the Microsoft Rules Engine utilizes a text version of lambda expressions.

What do you think about extending the C# language to simplify the definition of various kinds of rules? Might C# language designers be able to generalize over kinds of rules and their definitions as they did with relational algebra for LINQ?

New language features could better equip software engineers for scenarios including business rules engines, automated decision support, machine learning, and automated decision-making.

What do you think?

[smoothdeveloper]

To me, it sounds a bit like F# computation expressions, where you can write down an algorithm using normal constructs, like if, for, try/with, etc. intermixed with special operations (function calls, that are type checked and may affect the type of the remaining computation).

But there is another aspect to it, in those examples of use case areas you give (rules engine, automata, etc.), that rules are, somehow stored as data (an abstract syntax tree), and those can be modelled as data structures, as exemplified in Roslyn codebase, and C# is gearing towards enabling expressivity with OO & value initialisation constructs, so, a C# developer is never too far to sketch own DSL baked in data as C# OO constructs, and an interpreter for it.

@AdamSobieski could you expand on code samples that would show as "LINQ for rules for C# v++", that may help narrow down a bit, implication on C# language itself.

Thanks!

[AdamSobieski]

To your question, brainstorming:

var rule3 = [Priority(10)]
            from c in customers
            when c.Name == "John Doe"
            do
            {
              DoSomething(1);
            }
            else when c.Name == "Jane Doe"
            do
            {
              DoSomething(2);
            }
            else
            do
            {
              DoSomethingElse();
            };

What do you think?

A using keyword for rule definitions, intended for context-related variables, might be unnecessary when the rule definitions are language-integrated.

Also, I am thinking about {} scopes with a do keyword:

var rule1 = [Priority(10)]
            from c in customers
            when c.Name == "John Doe"
            do
            {
              DoSomething();
            };

var rule2 = [Priority(10)]
            from c in customers
            from a in accounts
            when c.IsPreferred
            when a.IsActive
            when a.Owner == c
            do
            {
              c.DoSomething();
            };

That's a good question about the entirety of keywords needed for expressiveness. I am looking at existential rules, universal quantification, and, or, not, and other features showcased in the NRules documentation. I can take a closer look at NRules and the Microsoft Rules Engine upcoming and write a follow-up with a fuller set of keywords.

[HaloFour]

What does that compile down to? All LINQ clauses compile directly to C# method invocations, either directly or through extension methods. It's also not obvious where that attribute would apply, the closest lambda is passed to SelectMany for the second from clause.

Also curious how something like else would work since that would be a significant change to how LINQ works as it doesn't track alternate streams. do would also seemingly require support for statement/block lambda expressions.

[AdamSobieski]

@HaloFour, as envisioned, there would be a new static class for extension methods: Rule. Rule would, in a manner resembling System.Linq.Queryable, provide most of the system's extension methods.

Brainstorming, resembling IQueryable and IQueryProvider, there could be IRule and IRuleProvider interfaces. Then, extension methods in Rule would call, on a rule argument, rule.Provider.CreateRule(...) after having first created expressions. In theory, all of the functionality would work with a fluent interface system based on IRule. As envisioned, the system would use lambda expressions from System.Linq.Expressions.

As for how something like else might work, I agree that it would be a significant change from LINQ. Perhaps it would make use of a Conditional extension method somehow?

The do having {} does appear to require support for statement/block lambda expressions.

[HaloFour]

Brainstorming, resembling IQueryable and IQueryProvider, there could be IRule and IRuleProvider interfaces.

That's kind of an implementation detail, one the compiler wouldn't care about. As long as the compiler can lower the LINQ clauses to method invocations on the target instance it'll be happy. I doubt that the compiler knows that IQueryable<T> or IQueryProvider exist at all.

It's how any new clauses like else or do would be translated into method invocations that are the more important piece to resolve. It might be hard to sell else as a LINQ clause if it doesn't make sense in the context of general purpose LINQ providers or use cases outside of your rules engine.

There is a separate championed proposal for do here: #101
There hasn't been a lot of progress and I think there was debate as to whether do would imply foreach or would allow invoking a general purpose terminal operation on the underlying type.

[AdamSobieski]

Thanks for the information about #101 . If it has not already been considered, I would like to add language-integrated rule definitions to the list of uses for a do keyword.

[CyrusNajmabadi]

I'm not really understanding waht is being asked for at a language level here. What syntactic and semantic changes are you interested in? And how do you envision them being compiled/encoded at a metadata/il level? Thanks!

[AdamSobieski]

@CyrusNajmabadi, language-integrated rule definitions, and rule serialization and deserialization functionalities, would better equip software engineers for scenarios including advanced database usage, knowledgebase usage, business rules engine usage, automated decision support, machine learning, and automated decision-making.

Specific syntactic features considered, thus far, include adding metadata to language-integrated queries and/or rules. This might entail that an IRule interface would extend ICustomAttributeProvider and provide other features for adding metadata at runtime.

Syntactically, this could resemble:

var rule1 = [Name("MyRule")]
            [Description("Test rule that demonstrates metadata usage")]
            [Tag("Test")]
            [Tag("Metadata")]
            [Priority(10)]
            [Repeatability(RuleRepeatability.Repeatable)]
            from c in customers
            when c.Name == "John Doe"
            do
            {
               DoSomething();
            }

Syntactic features considered also include potentially expanding on the set of LINQ keywords. That is, there could be a need for more keywords for rules, in particular if we desire to provide support for many of or all of the types of rules used throughout software engineering.

At least one of these new keywords has been discussed previously (#101), the do keyword.

With respect to what the desired syntactic and semantic features would compile to, the extension methods utilized could be placed in a static class, Rule, which utilizes IRule and IRuleProvider in a manner based on the Queryable, IQueryable, and IQueryProvider pattern.

[CyrusNajmabadi]

i'm really not certain waht you're asking for. What do any of the above constructs in your example compile down to?

[AdamSobieski]

@CyrusNajmabadi, the above example:

var rule1 = [Name("MyRule")]
            [Description("Test rule that demonstrates metadata usage")]
            [Tag("Test")]
            [Tag("Metadata")]
            [Priority(10)]
            [Repeatability(RuleRepeatability.Repeatable)]
            from c in customers
            when c.Name == "John Doe"
            do
            {
               DoSomething();
            }

might compile to something resembling:

var tmp = System.Rules.Rule.Do(System.Rules.Rule.When(customers, lambda_expr_1), lambda_expr_2);
tmp.SetCustomAttribute(typeof(NameAttribute), "MyRule");
tmp.SetCustomAttribute(typeof(DescriptionAttribute), "Test rule that demonstrates metadata usage");
tmp.SetCustomAttribute(typeof(TagAttribute), "Test");
tmp.SetCustomAttribute(typeof(TagAttribute), "Metadata");
tmp.SetCustomAttribute(typeof(PriorityAttribute), 10);
tmp.SetCustomAttribute(typeof(RepeatabilityAttribute), RuleRepeatability.Repeatable);
var rule1 = tmp;

[HaloFour]

Building on existing LINQ functionality I would suggest that it would be more likely to compile to something like the following:

customers.Where(c => c.Name == "John Doe")
    .ForEach(c => c.DoSomething())
    .SetCustomAttribute(new NameAttribute("MyRule"))
    .SetCustomAttribute(new DescriptionAttribute("Test rule that demonstrates metadata usage"))
    .SetCustomAttribute(new TagAttribute("Test"))
    .SetCustomAttribute(new TagAttribute("Metadata"))
    .SetCustomAttribute(new PriorityAttribute(10))
    .SetCustomAttribute(new Repeatability(RuleRepeatability.Repeatable));

This keeps to the LINQ paradigm of being ignorant of how the query is interpreted, the compiler only cares that the methods that the clauses lower to exist and whether they accept a lambda or an expression of a lambda. What that does depends on what customers is and how those instance or extension methods are implemented.

In this case the compiler would only need to support lowering do to a ForEach method or similar, and it would need to do something with those attributes. The latter I'm less sure about as that is a pretty radical departure from how attributes work in the language and runtime. Attributes on lambdas today are emitted as proper attributes on the emitted method, and they don't work with lambda expressions as the expressions API doesn't support them. In this case the attributes aren't even associated with any of the lambdas or expression lambdas so there really isn't anywhere obvious for them to do, even if using them within LINQ expressions was to be supported.

[AdamSobieski]

Thank you. That looks solid and I think that you are right that we should try to reuse as much existing LINQ functionality as possible.

It is interesting to me that we can express rules by reusing Where, ForEach, and SelectMany.

I am thinking about use cases which involve knowledgebases where we could construct a rule from a kb instance and find that it, the kb instance, also appears in the ForEach lambda expression. As envisioned, a ForEach lambda expression could update the knowledgebase or add content to it. So, do could have both foreach and update semantics.

We can view a knowledgebase as being, amongst other things, an IQueryable collection of predicate calculus expressions.

While a more complex example might also use SelectMany, to show the gist of the idea:

var rule1 = kb.Where(x => x.Predicate=="Name" && x.Arguments[1]=="John Doe")
              .ForEach(x => kb.Add("Likes", x.Arguments[0], "Chocolate"));

So, rules can be treated as first-class objects and could be added to kbs which subsequently perform described reasoning and inferencing using them:

kb.AddRule(rule1);

[GSPP]

@AdamSobieski it seems that you have given rule engines some thought and that you have used them. I have always wondered what the use case is. It seems that you can write the same logic simpler in normal C#. Would you care to explain? I'd be interested.

For example, when we take your code snippet:

[Name("MyRule"), Description("Test rule that demonstrates metadata usage")]
[Tag("Test"), Tag("Metadata")]
[Priority(10)]
[Repeatability(RuleRepeatability.Repeatable)]
public class RuleWithMetadata : Rule
{
    public override void Define()
    {
        When()
            .Match<Customer>(c => c.Name == "John Doe");
        Then()
            .Do(ctx => DoSomething());
    }
}

How would this rule definition be used?

[AdamSobieski]

@GSPP, that example from NRules showcases that rules can be expressed with C# to be stored in and loaded from .NET assemblies. This is similar to how formal-language grammars can be stored in and loaded from .NET assemblies with Irony. Developers could instead choose to utilize external files or databases/knowledgebases and have programs otherwise load and utilize rules, grammars, or other data.

As interesting, software like NRules utilize the Rete algorithm. "The Rete algorithm provides a generalized logical description of an implementation of functionality responsible for matching data tuples ("facts") against productions ("rules") in a pattern-matching production system (a category of rule engine). A production consists of one or more conditions and a set of actions which may be undertaken for each complete set of facts that match the conditions."

I have considered scenarios like LINQ to predicate calculus, and LINQ to knowledgebases, where knowledgebases could interface as IQueryable<> collections of sentences and as collections of rules. I've thought about how simpler, more elegant, more readable syntaxes for these scenarios might convenience integrating these functionalities into applications, and how they might convenience teaching knowledge-based AI, logic, and mathematics courses. With first-class rules, perhaps there could be, as there is today a System.Data namespace, a System.Knowledge namespace.

Also on the topic of AI and .NET, I have more recently been looking into SciSharp.

To language-integration, I remember how exciting it was when LINQ appeared back in 2007, how useful it was, and how numerous projects sprung up using it or derived from it, e.g., bindable LINQ, parallel LINQ, and so forth. Occasionally, I get an idea towards enhancing software development and/or computer science education with a new language feature or some "syntactic sugar".

[AdamSobieski]

Here are some sketches of using knowledgebases in C# with LINQ. These sketches involve type-strong predicate calculus. In these sketches, one can query on sentences and typed elements (e.g., things) simultaneously.

In the following sketch, the knowledgebase kb implements IQueryable<Sentence> and provides means of querying the elements in the knowledgebase on its Elements property. Here, Person and Clothes could be classes or interfaces.

from s in kb
from x in kb.Elements.OfType<Person>()
from y in kb.Elements.OfType<Clothes>()
where s.Predicate.Name == "wearing"
where s[0] == x
where s[1] == y
select s;

To be able to deliver syntax like f(x, y) in a LINQ query, I've looked at possibilities including delegates and operator overloading of ():

from f in kb.Predicates
from x in kb.Elements.OfType<Person>()
from y in kb.Elements.OfType<Clothes>()
where kb.Contains( f(x, y) )
select x;

if predicates are delegates which receive typed arguments and produce Sentences, then one could:

from f in kb.Predicates.OfType<Func<Person, Clothes, Sentence>>()
from x in kb.Elements.OfType<Person>()
from y in kb.Elements.OfType<Clothes>()
where kb.Contains( f(x, y) )
select x;

one could also create some OfSignature extension methods for IQueryable<Predicate>s:

from f in kb.Predicates.OfSignature<Person, Clothes>()
from x in kb.Elements.OfType<Person>()
from y in kb.Elements.OfType<Clothes>()
where kb.Contains( f(x, y) )
select x;

I'm interested in making using knowledgebases and type-strong predicate calculus as simple and elegant as possible for developers. There are a number of ways to approach delivering these features and the above sketches intend to show but a few.

The gist of the language-integrated rules topic is that adding rules to knowledgebases could be as simple as making queries with a do {...} part which invokes kb.Add(Sentence item) and then adding that query/rule to knowledgebases using kb.AddRule(...). So, in these approaches, the syntaxes for formulating queries and formulating rules are interrelated.

Are these sketches simple and readable? Does anyone have any other ideas regarding language-integrated knowledgebase use?

[CyrusNajmabadi]

I'm not really certain what in the above requires any changes to the the language. Can you clarify? All of hte above looks like what you can already write in C# today. Thanks!

[AdamSobieski]

@CyrusNajmabadi, brainstorming, there are at least two relevant ideas for the language and LINQ that come to mind, and, if others should have more ideas for knowledgebase queries and rules, there could be more.

1. The do { ... } construct which can be of use for making rules from LINQ syntax.
2. Drawing variables from IQueryables, possibly with something like an IQueryable2 interface.

public interface IQueryable2 : IQueryable
{
  public IQueryable Variables { get; } // or Elements ?
}

So:

from s in kb
from x in kb.Variables.OfType<Person>()
from y in kb.Variables.OfType<Clothes>()
where s.Predicate.Name == "wearing"
where s[0] == x
where s[1] == y
select s;

could become (something like):

from s in kb
let Person x in kb
let Clothes y in kb
where s.Predicate.Name == "wearing"
where s[0] == x
where s[1] == y
select s;

and:

from f in kb.Predicates
from x in kb.Variables.OfType<Person>()
from y in kb.Variables.OfType<Clothes>()
where kb.Contains( f(x, y) )
select x;

could become (something like):

from f in kb.Predicates
let Person x in kb
let Clothes y in kb
where kb.Contains( f(x, y) )
select x;

or (something like):

from s in kb
let Predicate f in kb
let Person x in kb
let Clothes y in kb
where s == f(x, y)
select x;

or (something like):

from s in kb
let Predicate<Person, Clothes> f in kb
let Person x in kb
let Clothes y in kb
where s == f(x, y)
select x;

As we construct more examples, and consider the often-to-be-typed parts and the could-be-clearer parts, we might arrive at yet more ideas for the language and LINQ.