[Refactor] Split transformer.py into StmtBuilder and ExprBuilder (Stage 1)

[xumingkuan]

Related issue = #2193

This PR replaces ASTTransformerPreprocess with StmtBuilder and ExprBuilder.

This PR doesn't modify the behavior (and the code) of the other 2 passes in transformer.py.

Detailed changeset:

• Move all members of the class ASTTransformerPreprocess to BuilderContext. These members should be context instead of members of a pass.
• Move ScopeGuard, parse_stmt, parse_expr to ast_builder_utils.py.
• Rename all functions in ASTTransformerPreprocess from visit_Xxx to build_Xxx and move them to expr_builder.py and stmt_builder.py.
• Remove all generic_visit in ASTTransformerPreprocess in transformer.py. Since we need to know if it's a statement or an expr before building the subtree of AST, we cannot use a general generic_visit. The invocations of this function is rewritten as build_expr, build_exprs, build_stmt, build_stmts.
• Note that build_exprs and build_stmts create a variable_scope. Also note that I do not use a list comprehension in these two functions because with will not work properly otherwise.
• Note that build_Module does not use build_stmts, just like its not using generic_visit before this PR. This is because if we create a variable_scope associated with the statement list (Python list here), the parameters passed into the Taichi kernel will be deleted when destructing the variable scope.
• Fix some missing fields required by the Python AST but not required when executing it (compiling Taichi). (Otherwise the AST may not be printed properly.)
• Throw exception if we encounter an AST node without corresponding builder (e.g., if build_Name is not implemented in ExprBuilder and we invoke build_expr with a Name node as the parameter).
• Throw TaichiSyntaxError if the Python keyword global/nonlocal is used or when a Python set (including set comprehension) is used.
• Add many builder functions which look trivial and in fact unnecessary if we use visit_Xxx before this PR. I think there's a benefit of writing these functions explicitly -- they show which features in Python are supported in Taichi, and throw exceptions when users use features in Python that are not considered by Taichi developers, which may or may not be supported.

Note:

• Not supported both before and after this PR:
  • a = f'aaa{123}' (ast.JoinedStr, together with ast.FormattedValue)
  • b = a[1:2] (ast.Slice)
  • b = (n**2 for n in aa if n>5 if n<10) (ast.GeneratorExp)
  • a = lambda x, y: x + y (ast.Lambda)
• (Probably incorrectly) supported before this PR but unsupported after this PR:
  • (b): ti.i32 = 1 (ast.AnnAssign)
  • yield a and yield from a (ast.Yield and ast.YieldFrom)
  • {1, 2, 3} and {x for x in numbers} (ast.Set and ast.SetComp)
  • class Foo: ... (ast.ClassDef)
  • def foo() (ast.FunctionDef) in a kernel
• Not supported but did not throw syntax error before this PR:
  • ast.AsyncFunctionDef, ast.Await, ast.AsyncFor, ast.AsyncWith do not throw syntax error before this PR. They are not considered in this PR.
  • global and nonlocal throw TaichiSyntaxError after this PR. They may crash later before this PR.

[CLAassistant]

All committers have signed the CLA.

[k-ye]

Thanks for this huge refactoring!

1. How many stages do you expect there will be?
2. Could you write some comments to guide the review process?

[k-ye]

I wonder what's the situation with maybe_transform_ti_func_call_to_stmt after this PR? Also does this method generate a ExprStmt?

[xumingkuan]

No. This PR should not modify the behavior of the Python frontend (the whole transformer.py together with new files introduced in this PR).

[xumingkuan]

Thanks for this huge refactoring!

1. How many stages do you expect there will be?
2. Could you write some comments to guide the review process?

1. I would expect that there will be only 2 stages -- the other one is to remove maybe_transform_ti_func_call_to_stmt and merge the multiple-return check into the new ast builders. I think there will be only one pass in Python after this refactoring.
2. Sure!

[xumingkuan]

Do we support Python 3.9 now?

[k-ye]

Do we support Python 3.9 now?

Yep

[xumingkuan]

Is NamedExpr unsupported in Python 3.6?

[k-ye]

LGTM!!

[ljcc0930]

LGTM!

[k-ye]

FYI you don't have to [skip ci] now.... New commit will automatically cancel the previous CI 😆