Trac #12103: Use MeatAxe as an optional back end for dense matrices o…

…ver `GF(p^n)`, p odd, n>1, `p^n<255`

Sage has (or will soon have) fairly good implementations of dense
matrices over `GF(2)`, over `GF(2^e)` (#9562) and over `GF(p)` (p prime,
#4260). However, it uses generic code for dense matrices over `GF(p^n)`,
p odd, n>1, `p^n<255`.

The original suggestion was to use a major modification of `MeatAxe
Release 2.2.4` instead of the basic implementation. The timings below
are with that old version (it is identical with 2.2.3 except for the GPL
licence, and 2.2.3 was before 1998).

I now suggest to try and do the same with the latest !MeatAxe release
2.4.24, which is from 2011. There also is an experimental 2.5.0 from
2003, but I think we shouldn't rely on that.

'''__Sources__'''

The upstream sources http://www.math.rwth-
aachen.de/~MTX/meataxe-2.4.24.tar.gz needed to be repackaged, in order
to unpack into a single folder. Use attachment:meataxe-2.4.24.tar.gz

'''__What is done__'''

There is no spkg-check. However, if SAGE_CHECK=yes or of one does `sage
-i -c meataxe`, then a test suite is executed as part of building the
package.

It is my experience that the tests pass most of the time. I can not
explain why sometimes they don't.

'''__What is missing__'''

Currently, the spkg installs libmtx.a and installs some binaries.
However, I also intend to add a Cython wrapper so that one can use
!MeatAxe matrices in Sage.

Here is the original ticket description:

This is awfully slow:
{{{
sage: MS = MatrixSpace(GF(5^3,'y'),2000)
sage: %time A = MS.random_element()
CPU times: user 6.36 s, sys: 0.02 s, total: 6.39 s
Wall time: 6.41 s
sage: type(A)
<type 'sage.matrix.matrix_generic_dense.Matrix_generic_dense'>
sage: B = MS.random_element()
sage: %time A*B    # using 6.3% of my computer's memory
CPU times: user 744.20 s, sys: 1.18 s, total: 745.38 s
Wall time: 747.69 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
sage: %time ~A     # using 10.4% of my computer's memory
CPU times: user 1096.74 s, sys: 1.30 s, total: 1098.05 s
Wall time: 1101.24 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
sage: %time A.echelon_form() # using 10.4% of my computer's memory
CPU times: user 378.62 s, sys: 0.33 s, total: 378.95 s
Wall time: 380.06 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
}}}

With the optional spkg and the patch, one gets a clear improvement.
{{{
sage: MS = MatrixSpace(GF(5^3,'y'),2000)
sage: %time A = MS.random_element()
CPU times: user 0.32 s, sys: 0.00 s, total: 0.32 s
Wall time: 0.33 s
sage: type(A)
<type 'sage.matrix.matrix_modpn_dense.Matrix_modpn_dense'>
sage: B = MS.random_element()
# The following uses Strassen-Winograd multiplication
sage: %time A*B      # using 3.5% of my computer's memory
CPU times: user 7.68 s, sys: 0.01 s, total: 7.69 s
Wall time: 7.72 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
# The following is school book multiplication;
# that's more or less the original meataxe speed:
sage: %time A._multiply_classical(B)   # using 3.6% of my computer's
memory
CPU times: user 11.68 s, sys: 0.02 s, total: 11.70 s
Wall time: 11.73 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
# Strassen is not implemented for inversion and echelon form.
sage: %time ~A    # using 3.8% of my computer's memory
CPU times: user 23.55 s, sys: 0.00 s, total: 23.55 s
Wall time: 23.62 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
sage: %time A.echelon_form() #using 3.9% of my computer's memory
CPU times: user 11.73 s, sys: 0.01 s, total: 11.74 s
Wall time: 11.78 s
2000 x 2000 dense matrix over Finite Field in y of size 5^3
}}}

URL: http://trac.sagemath.org/12103
Reported by: SimonKing
Ticket author(s): Simon King
Reviewer(s): Jeroen Demeyer, Travis Scrimshaw

build/pkgs/meataxe/SPKG.txt

@@ -0,0 +1,27 @@
+= MeatAxe =
+
+== Description ==
+
+The MeatAxe is a set of programs for working with matrix representations
+over finite fields. Permutation representations are supported to some
+extent, too.
+
+The MeatAxe is developed for the UNIX operating system. Supported platforms
+include Linux (x86), SunOS/Solaris (Sparc), HP/UX, DEC OSF/1 (Alpha), and
+Windows NT 4.0 (x86, Alpha, PPC).
+
+== License ==
+
+The C Meat-Axe is free software: you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free Software
+Foundation, either version 2 of the License, or (at your option) any later
+version.
+
+== Upstream Contact ==
+
+Michael Ringe ([email protected])
+
+== Special Update/Build Instructions ==
+
+The original upstream tarball was re-packaged, so that it unpacks into a single
+folder called meataxe-2.4.24

build/pkgs/meataxe/checksums.ini

@@ -0,0 +1,4 @@
+tarball=meataxe-VERSION.tar.gz
+sha1=0aa4313cc430c78e058068feba805428ef2324aa
+md5=e0f384e37a69671c73c2904e4e69dc01
+cksum=3083268116

build/pkgs/meataxe/dependencies

@@ -0,0 +1 @@
+# no dependencies

build/pkgs/meataxe/package-version.txt

@@ -0,0 +1 @@
+2.4.24.p1

build/pkgs/meataxe/patches/IO_fixes.patch

@@ -0,0 +1,78 @@
+Read and create library files in the directory given by MtxLibDir.
+
+The patch keeps a promise given by upstream.
+
+AUTHOR: Simon King 2015-09-18, [email protected]
+
+diff --git a/src/maketabF.c b/src/maketabF.c
+index fa03eda..d7af83e 100644
+--- a/src/maketabF.c
++++ b/src/maketabF.c
+@@ -319,7 +319,7 @@ static void writeheader()
+     int i, j;
+
+     sprintf(filename,"p%3.3ld.zzz",Q);
+-    fd = SysFopen(filename,FM_CREATE);
++    fd = SysFopen(filename,FM_CREATE|FM_LIB);
+     if (fd == NULL)
+     {
+ 	perror(filename);
+diff --git a/src/os.c b/src/os.c
+index a7f4271..b07b971 100644
+--- a/src/os.c
++++ b/src/os.c
+@@ -227,25 +227,31 @@ FILE *SysFopen(const char *name, int mode)
+ 	MTX_ERROR1("Invalid file mode %d",mode);
+ 	return NULL;
+     }
+-    f = fopen(name,fmodes[m]);
+-    if (f != NULL) 
+-	return f;
+
+     /* Search library directory
+        ------------------------ */
+     if ((mode & FM_LIB) != 0) 
+     {
+-	strcpy(buf,MtxLibDir);
+-	strcat(buf,"/");
+-	strcat(buf,name);
+-	f = fopen(buf,fmodes[m]);
++        if (*MtxLibDir != 0)
++            {
++            strcpy(buf,MtxLibDir);
++            strcat(buf,"/");
++            strcat(buf,name);
++            f = fopen(buf,fmodes[m]);
++            }
++        else
++            f = fopen(name,fmodes[m]);
+     }
+-
++    else
++    {
++        f = fopen(name,fmodes[m]);
++    }
++    if (f != NULL)
++	return f;
+     /* Error handling
+        -------------- */
+     if (f == NULL && (mode & FM_NOERROR) == 0)
+-	MTX_ERROR1("%s: %S",name);
+-
++    MTX_ERROR1("%s: %S",name);
+     return f;
+ }
+
+diff --git a/src/zcv.c b/src/zcv.c
+index a9ad7a3..763c9fb 100644
+--- a/src/zcv.c
++++ b/src/zcv.c
+@@ -584,7 +584,7 @@ static int Init(int argc, const char **argv)
+     inpname = App->ArgV[0];
+     if (strcmp(inpname,"-"))
+     {
+-	src = SysFopen(inpname,FM_READ|FM_TEXT|FM_LIB);
++	src = SysFopen(inpname,FM_READ|FM_TEXT);
+ 	if (src == NULL)
+ 	{
+ 	    MTX_ERROR1("Cannot open %s",inpname);