Merge pull request #799 from Mathics3/update-FUTURE

Start noting what's up and what goals have been met

FUTURE.rst

@@ -2,6 +2,120 @@
 
 .. contents::
 
+The following 2023 road map that appears the 6.0.0 hasn't gone through enough discussion. This provisional.
+Check the github repository for updates.
+
+
+2023 Roadmap
+============
+
+
+When the release settles, "Forms, Boxing, And "Formatting" is the next
+large refactor slated.  Having this will allow us to supporting Jupyter or other front
+ends. And it is something that is most visibly wrong in Mathics3 output.
+
+See ``PAST.rst`` for how the 2023 Roadmap compares to the 2022 Roadmap.
+
+Forms, Boxing and Formatting
+----------------------------
+
+This remains the biggest holdover item from 2022, and seems easily doable.
+It hinders interaction with Jupyter or other front ends.
+
+Right now "Form" (a high-level specification of how to format) and
+"format" (a low level specification of how output is encoded) are sometimes muddied.
+
+For example, TeXForm may be a "Form", but output encoded for AMS-LaTeX is done by a *formatter*.
+So AMS-LaTeX rendering and other kinds of rendering should be split into its own rendering for formatter module.
+Currently we have asymptote, and svg "format" modules.
+
+Back to high-level again, Boxing is something that can be written in Mathics3, and doing this at
+least initially ensures that we have design that fits more naturally
+into the Wolfram Language philosophy.
+
+
+Performance
+-----------
+
+While this is probably more of an overall concern, for now, big refactoring needed here, such as
+going over pattern matching, will get done after Forms, Boxing and Formatting .
+
+Forms, Boxing and Formatting will however contain one improvement that
+should speed up our performance: separating M-Expression evaluation from
+Box "evaluations).
+
+We expect there will be other little opportunities here and there as we have seen in the past.
+
+
+More Custom kinds of (compound) Expressions
++++++++++++++++++++++++++++++++++++++++++++
+
+We scratched the surface here with ListExpression. Associations and Python/Sympy/numpy literals can be customized with an aim towards reducing conversions from and to M-expressions.
+A number of compound expressions, especially those which involve literals are more efficiently represented in some other way. For example,
+representing a Mathics3 Association as a Python ordered dictionary, a Mathics3 List as a Python list or tuple, or as a numpy array.
+
+
+Further Code Reorganization in Core and Eval
+--------------------------------------------
+
+Core object like ``BaseElement`` and possibly ``Symbol``, (and
+probably others) are too "fat": they have too many custom methods that
+are not applicable for most of the subclasses support.  It is likely
+another pass will be made over this.
+
+We have started moving "eval" code out of the "eval" methods and into its own module.
+
+Mathics3 Module Enhancement
+---------------------------
+
+While we have put in quite a bit of effort to get these to be 6.0.0 compliant. There is still more work to do, and there are numerous bugs there.
+Refactoring code to generate Graphs in ``pymathics.graph`` might happen. Porting the ``pymathics.graph`` code to use NetworkX 3.0 would be nice;
+``pymathics.natlang`` could also use a look over in terms of the libraries we are using.
+
+Python upgrades
+---------------
+
+After Mathics3 Version 6.0.0, Python 3.6 will be dropped and possibly 3.7. Changes are needed to support 3.11 so we will be focusing on 3.8 to 3.11.
+
+We have gradually been using a more modern Python programming style
+and idioms: more type annotation, use of ``isort`` (order Python
+imports), ``black`` (code formatting), and ``flake8`` (Python lint
+checking).
+
+
+Deferred
+--------
+
+As mentioned before, pattern-matching revision is for later. `This
+discussion
+<https://github.com/Mathics3/mathics-core/discussions/800>`_ is a
+placeholder for this discussion.
+
+Overhauling the documentation to use something better supported and
+more mainstream like sphinx is deferred. This would really be nice to
+have, but it will require a bit of effort and detracts from all of the other work that is needed.
+
+We will probably try this out in a limited basis in one of the Mathics3 modules.
+
+Speaking of Mathics3 Modules, there are probably various scoping/context issues that Mathics3 modules make more apparent.
+This will is deferred for now.
+
+Way down the line, is converting to a more sequence-based interpreter which is needed for JIT'ing and better Compilation support.
+
+Likewise, speeding up startup time via saving and loading an image is something that is more of a long-term goal.
+
+Things in this section can change, depending on the help we can get.
+
+
+Miscellaneous
+-------------
+
+No doubt there will be numerous bug fixes, and builtin-function additions especially now that we have a better framework to support this kind of growth.
+Some of the smaller deferred issues refactorings may get addressed.
+
+As always, where and how fast things grow here depends on help available.
+
+
 2022 Roadmap
 =============
 
@@ -57,11 +171,11 @@ The current home-grown documentation should be replaced with Sphynx and autodoc.
 Compilation
 -----------
 
-Complation is a rather unsophisticated process by trying to speed up Python code using llvmlite. The gains here will always be small compared the kinds of gains a compiler can get. However in order to even be able to contemplate writing a compiler (let alone say a JIT compiler), the code base needs to be made to work more like a traditional interpreter. Some work will be needed just to be able or create a sequence of instructions to run.
+Compilation is a rather unsophisticated process by trying to speed up Python code using llvmlite. The gains here will always be small compared the kinds of gains a compiler can get. However in order to even be able to contemplate writing a compiler (let alone say a JIT compiler), the code base needs to be made to work more like a traditional interpreter. Some work will be needed just to be able or create a sequence of instructions to run.
 
-Right now the interpreter is strictly a tree interperter.
+Right now the interpreter is strictly a tree interpreter.
 
-Simpiler Things
+Simpler Things
 ---------------
 
 There have been a number of things that have been deferred:

PAST.rst

@@ -0,0 +1,55 @@
+While ``FUTURE.rst`` gives our current roadmap, this file ``PAST.rst``
+looks the other way for what we have accomplished when compared to what _was_ planned in ``FUTURE.rst``
+
+While this is also listed in ``CHANGES.rst``, here we extract that to
+make it easier to see the bigger picture without the details that are
+in ``CHANGES.rst``.
+
+Progress from 2022
+==================
+
+A fair bit of code refactoring has gone on so that we might be able to
+scale the code, get it to be more performant, and more in line with
+other interpreters.
+
+The buitin Functions have been organized into grouping akind to what is found in WMA.
+This is not just for documentation purposes, but it better modularizes the code and keep
+the modules smaller while suggesting where functions below as we scale.
+
+Image Routines have been gone over.
+
+A number of Built-in functions that were implemented were not accessible for various reasons.
+
+Mathics3 Modules are better integrated into the documentation.
+Existing Mathics3 modules pymathics.graph and pymathics.natlang have
+had a major overhaul, although more is needed. And will continue after th 6.0.0 release
+
+We have gradually been rolling in more Python type annotations and
+current Python practices such as using ``isort``, ``black`` and ``flake8``.
+
+
+Boxing and Formatting
+---------------------
+
+While some work on formatting is done has been made and the change in API reflects a little of this.
+However a lot more work needs to be done.
+
+Excecution Performance
+----------------------
+
+This has improved a slight bit, but not because it has been a focus, but
+rather because in going over the code organization, we are doing this
+less dumb, e.g. using Symbols more where symbols are intended. Or
+fixing bugs like resetting mpmath numeric precision on operations that
+need to chnage it temporarily.
+
+Simpler Things
+--------------
+
+A number of items here remain, but should not be thought as independent items, but instead part of
+"Forms, Boxing and Formatting".
+
+"Making StandardOutput of polynomials match WMA" is really are Forms, Boxing and Formatting issue;
+"Working on Jupyter integrations" is also very dependant this.
+
+So the next major refactor will be on Forms, Boxing and Formatting.