Using pretty print from sympy for OutputForm

[mmatera]

In WMA, OutputForm produces a "2D" text representation of the expression. I guess that this could be accomplished by using the low-level interface of pretty-print in Sympy. The tricky part here would be to convert a box structure into something that pretty print can handle...

[rocky]

There is an Expression.to_sympy() method. I imagine that can be called and when if it succeeds, then Sympy's pretty print can be called.

But I wouldn't call this OutputForm but rather SympyForm since that is what it is. A proper output form would do largely the steps SymPy uses, but adapted when WMA and SymPy differ.

As a prototype, this works. But keep in mind that in the long term this increases pressure on an existing problem that we do too many conversions into and out of Sympy. In cases where we are slow, not saving the intermediate Sympy form which causes recalculation of the same SymPy expression (possibly different values bound to symbols, but from SymPy's standpoint the SymPy expression with respect to symbols is the same).

[rocky]

Filling in more detail above. I made this patch:

--- a/mathics/builtin/arithmetic.py
+++ b/mathics/builtin/arithmetic.py
@@ -103,6 +103,8 @@ class _MPMathFunction(SympyFunction):
         # if no arguments are inexact attempt to use sympy
         if all(not x.is_inexact() for x in args):
             result = Expression(self.get_name(), *args).to_sympy()
+            from sympy import pprint
+            pprint(result)
             result = self.prepare_mathics(result)
             result = from_sympy(result)
             # evaluate leaves to convert e.g. Plus[2, I] -> Complex[2, 1]

And then ran mathics:

$ mathics
1
1/4
1/2
1
1/2
1
1
1/2
    ⎛_Mathics_User_System`z⎞
tanh⎜──────────────────────⎟
    ⎝          2           ⎠
    ⎛    ⎛_Mathics_User_System`z⎞⎞
atan⎜tanh⎜──────────────────────⎟⎟
    ⎝    ⎝          2           ⎠⎠
    ⎛_Mathics_User_System`z⎞
tanh⎜──────────────────────⎟
    ⎝          2           ⎠

Mathics 4.1.0.dev0
on CPython 3.8.13 (default, Mar 18 2022, 07:00:29) 
using SymPy 1.10.1, mpmath 1.2.1, numpy 1.22.3
...

I should note that to change the symbols back into their simple form, e.g. z instead of _Mathics_User_System`z the right way to do this is to make the substitution on the sympy object, not the resulting string. (This kind of mistake we've seen in the past, and the code still has I think hacks like this.)

The above points out the kind of thing I alluded to earlier in that we have this sympy object result, but it is not saved for future use say inside a SymPy pprint, or to more advantage, a future evaluation.

[mmatera]

@rocky, actually, I had the hope to use something at a lower level, in a way to convert directly a box expression into a "2D" form. In any case, the idea was not to fix this right now but to take it into account during the Makebox refactory.

[rocky]

I don't fully understand what is meant by "use something at a lower level", but if it is possible, I think we should avoid conversion from one too-constrained format into some other format instead of formatting to the desired format initially.

In the past there have been these hacks that convert formatted strings (in the wrong form) to other strings. Thus doing work to undo work that had previously been done.

Here, working with strings suffers the problem of not understanding the structure of the object and as a result it makes a substitution outside of the region of the string (or part of the structure) that it is supposed to work inside.

Conceptually, here is how I think of the process. Let me know how this is wrong or how I am being too simple minded.

The user inputs something which is a parse Expression. To use a simple example, let's say: x^2. That could be wrapped inside a Format function like StandardForm. Recall also the front-end also declares the kind of printing capabilities it has. For example, can it handle MathML? Unicode? So "StandardForm" should be thought of as having these options attached in the function call.

I chose x^2 specifically because evaluating that basically does not much in the way of expression rewriting, finding the method to apply, figuring out how to bind arguments, and it doesn't do much in evaluation either. So you get back pretty much what was entered after conversion to FullForm which happens implicitly.

So now StandardForm has to process a Mathics Expression. If it knows that MathML isn't supported but text is (which is always the case) and if it sees that unicode is supported, then StandardForm can convert the Mathics (compound) Expression into a SymPy expression after first removing the namespacing prefix _Mathics_User_System. If this expression can be converted to SymPy then simply callpprint on that.

This is simple and reliable.

Ok. So what am I missing?

[mmatera]

@rocky, the point is that in WL, formatted text is represented as Box expressions, which are agnostic about the front-end capabilities. If it is true that we can bypass the MakeBoxes mechanism, I think that the better and more compatible with WMA way to implement 2D text output would be to implement something that has as input a Box structure.

[rocky]

I never said anything about Boxes or bypassing MakeBoxes. Automatic Box is insertion should be a function of:

• the element: is this a leaf already a box some kind?
• the Function the element is in: is this in a list-like thing that boxes its elements?
• The top-level format that was specified: StandardForm? TeXForm?
• the capabilities declared: MathML supported?, unicode supported?

In the particular case of x^2, neither Symbol x nor Power nor Integer 2, nor Expression combining these should cause any kind of automatic boxing.

In TeXForm or LaTeXForm same thing is true for x^2 , even though TeX and LaTeX have their own internal notions of boxing. At the x^2 expression level just convert that to the TeX equivalent which is the same thing; x/2 is a better example because it becomes \frac{x}{2}. The main point is that TeX picks its own boxing and does a great job of it. And then we have the same thing for MathML.

Think of this like hard line breaks or carriage returns that a user might enter in text like I am doing in when writing this versus the soft line breaks that the system decides based on your browser and window dimensions.

The code in the past has made the mistake of trying to do too much in the core rather than delegate capabilities where they can be better dealt with. We saw this in code that rasterizes plots into SVG images inside core unconditionally. Both slowly, and poorly. When it is to be done, it should be done on demand and not in core but by some special sort of backend rendering module.

The place where Boxes and Formatting interact is when there have been explicit Boxes specified rather than implicit ones. Or in very special combinations where the front-end capabilities, Form, Function, and Element have all been deemed to allow it.

If a user inserts Boxes, such as you might find when a user wants to represent some complicated HTML-like table, or structure, then those boxes on formatting need to get converted to the equivalent representation in the underlying format choice. This could be conversion to TeX kinds of boxes, or MathML boxes, or CSS styling boxes. For an ASCII terminal this is doable by rearranging snippets of ASCII text, or by using terminal curses command but it is complicated. But the point is that x^2 however it is rendered as say ASCII text with its hard-coded \n and special unicode characters stays pretty much the same as the text that SymPy pprint returns.

[rocky]

Let me also say that this discussion I understood it to be about using SymPy's pprint to handle OutputForm

Then all of a sudden we are into talking about MakeBox.

the tricky part here would be to convert a box structure into something that pretty print can handle...

I would maintain the tricky part is to focus on conceptually simple solutions that are clean, general, easy to implement but also leaves room for scaling to more complex situations.

We aren't going to solve the general 2D situation on terminals or the general Boxing problems any time soon. Sheesh, we haven't been even able to isolate all of the leaf elements/objects properly and that's a prerequisite for being able to handle Boxing.