[WIP] PR: Add SymPy support

[oscargus]

I plan to add better SymPy support to Spyder. What I want to obtain is that SymPy expressions are "supported" and shown in the Variable explorer, ideally using SymPy pretty printing, e.g.

        2    
 3   3⋅x    x
x  - ──── + ─
      2     2

As far as I can tell the only way to do this is to add support for it in spyder-kernels as these expressions will not be returned among the supported variables otherwise. Is this correct? Or is there a way to tell spyder-kernels to include even more types? Is this a good idea?

Furthermore, I do not really see a way to edit these expressions at the moment, I am more interested in them showing up in the Variable explorer. Hence, should I add a new type, displayable(?) here in addition to editable and picklable or should I just ignore editing attempts in Spyder?

As part of this, I am also planning to update the string truncation mechanism. Right now, a 3 x 3 matrix with zeros will print on three rows, but only as the total string length is below 70 characters. I plan to change that to that each row is evaluated and truncated at 70 characters, and rows above a certain threshold (10?) will be truncated in a sensible way. Hence, the introduction of the truncated flag with the purpose to say that the result is already truncated (for nparrays and SymPy expressions at least, more can be added).

However, I have some problems testing spyder with my edited spyder-kernel. Any hints on how to obtain this? The problem seems to come from that pip(?) seems to think that I have Spyder 3.3.3 installed and therefore do not want to install a version > 1 using pip install -e .. (I do have 3.3.3 installed, but I want to run 4 from source...)

(I also have some Spyder PRs upcoming with e.g. the ability to show pretty printed expressions in the Object explorer, which may be another way to obtain it, but as the SymPy expressions does not show in the Variable explorer by default it seems like a better idea to get them there first...)

[oscargus]

This is what it looks like now:

What remains is to look into how/if SymPy supports pickling and to add tests.

[goanpeca]

Thaks for working on this @oscargus. However I think it is difficult to have this as it is taking too much space. We could add this maybe as a tooltip, but adding the whole pretty printed for sympy seems like too much. Thoughts @spyder-ide/core-developers ?

[bcolsen]

I think it's too large as well. I don't think that it should be larger an one line.

I like the tool tip though. We could even put the head(first ten lines) of a dataframe or a 2darray in a tool tip

[oscargus]

The table is not automatically resized though and the number of lines can be adjusted (right now 10). (And I selected some large examples to show the potential...) I think the main problem is what to show otherwise. Of course one can show the string representation (dynamically, if it is deemed that it is better). Either way, I can easily replace the two instances of pretty and put str instead. Better than not seeing the variables at all.

Note that NumPy currently prints over multiple lines. It is just that there is a total character limit, so one need to be a bit careful to select examples... So this works

but a 4 x 5 one-matrix gets truncated. (This PR fixes that as well by the way...)

With this, Spyder + SymPy would be quite useful for symbolic mathematics, as one can actually see partial results and edit them inline (it happened more or less magically with a few lines). (Also, one should trim the class names, but I'm not sure if that is done in spyder-kernels or Spyder, but I've notices that other special types do not use full paths...)

Tools tips are sorted out in Spyder, so that cannot be dealt with here. However, Spyder can of course generate the str representation locally and show the pretty printed as a tooltip if deemed to be too large, but if it isn't sent at all it will be hard to generate locally... If I understand the role of spyder-kernels and the architecture. Still, adding a tooltip_to_display function probably ends up here.

(You do not use computer algebra systems much? These are extremely small equations...)

[goanpeca]

(You do not use computer algebra systems much? These are extremely small equations...)

Precisely the point... these are small "toy" examples. Real examples would be even larger.

I am 👎 on this change.

[oscargus]

That is why it truncates at (currently) 80 x 10. Still, I'd claim that it is useful to see simpler expression as this. But probably it is better to make a guess which representation is the best. Truncating a few lines may not be a problem, but if 100 lines are truncated, it is probably better to show it in a different format.

[oscargus]

Btw, why is there a "Resize row to contents" if things should be single line?

[oscargus]

I'll just leave this figure illustrating the current strategy:

I'm not saying that my approach is without flaws, but considering the current status, it doesn't deviate fundamentally from what currently is allowed.

[jitseniesen]

Does this pretty print the whole SymPy expression and then truncate it? That could take a long time with large expressions, probably so long that in many cases it becomes annoying.

As for the size, I don't think it needs to be one line, but ten lines seems too much. Perhaps details need to be discussed in an issue instead of this PR. My initial feeling is that anything pretty is hard to do, so maybe stick with the repr (provided performance is not an issue).

[ccordoba12]

My initial feeling is that anything pretty is hard to do, so maybe stick with the repr (provided performance is not an issue).

I totally agree with this. What we could is that by doing double-click in a Sympy object, we show its text representation in our text editor. That way performance won't be an issue.

[oscargus]

Does this pretty print the whole SymPy expression and then truncate it? That could take a long time with large expressions, probably so long that in many cases it becomes annoying.

Correct. Clearly, pretty takes longer time than e.g. str or repr, but on the other hand gives a better idea of the expression, especially when truncating to 70/80 characters (the repr is printed using function names etc, so there could be a quite huge difference). One could possibly use something like count_ops() which returns the number of operations in the expression to do some filtering. Could still be useful for e.g. getting a str/repr-result quickly (probably no need to render for expressions with more than 100 operations anyway...). Will be a bit of a heuristic though.

As for the size, I don't think it needs to be one line, but ten lines seems too much. Perhaps details need to be discussed in an issue instead of this PR. My initial feeling is that anything pretty is hard to do, so maybe stick with the repr (provided performance is not an issue).

This is easily adaptable. However, I do not believe that multi-line is of much benefit when doing repr.

How should NumPy be treated? Currently, as shown, it plots up to 10 lines.

The benefit of a PR is that one can actually try it out.

[oscargus]

My initial feeling is that anything pretty is hard to do, so maybe stick with the repr (provided performance is not an issue).

I totally agree with this. What we could is that by doing double-click in a Sympy object, we show its text representation in our text editor. That way performance won't be an issue.

That happens right now, the SymPy objects are fully editable (except that there is some incorrect printing in SymPy for a few functions, but that is worked on). I also have a Spyder branch where one can display the pretty representation of the expression in the object explorer (with warnings that it may take a long time etc). Ideally, one can also add LaTeX rendering there (or as a right-click in the variable explorer), possibly with a windowed viewer for large expressions.

[ccordoba12]

One could possibly use something like count_ops() which returns the number of operations in the expression to do some filtering. Could still be useful for e.g. getting a str/repr-result quickly (probably no need to render for expressions with more than 100 operations anyway...). Will be a bit of a heuristic though.

We definitely need to do something like that before merging this (i.e. to optimize as much as possible the display of Sympy objects). The problem is that value_to_display is run on every execution and its result sent to Spyder. So if you have 100 big Sympy objects in your namespace, then even evaluating 1+1 would be perceived by the user as very, very slow.

except that there is some incorrect printing in SymPy for a few functions, but that is worked on

Where is that worked on? On Sympy or in Spyder?

I also have a Spyder branch where one can display the pretty representation of the expression in the object explorer (with warnings that it may take a long time etc)

That's very interesting! I'd really like to have that in Spyder!!

Ideally, one can also add LaTeX rendering there (or as a right-click in the variable explorer), possibly with a windowed viewer for large expressions.

I agree with this too. It'd be a terrific addition!

---

All in all I have to say I find this contribution very interesting because it could have a lot of value for heavy Sympy users.

Pinging @asmeurer (Sympy maintainer) to know his opinion about this.

[asmeurer]

It probably wouldn't be hard to write a "truncating" printer (or add support to the printers in SymPy), that stops printing after a certain number of characters.

Also I'm not sure if printing is currently cached in SymPy, but that would definitely help for this use-case.

[asmeurer]

You might want to set num_columns here to MAX_LINE_LENGTH.

[ccordoba12]

It probably wouldn't be hard to write a "truncating" printer (or add support to the printers in SymPy), that stops printing after a certain number of characters.
Also I'm not sure if printing is currently cached in SymPy, but that would definitely help for this use-case.

Thanks @asmeurer! Both suggestions are really good.

@oscargus, could you take care of implementing them on Sympy?