better printing of sparse matrices?

[stevengj]

Compare:

julia> A = Diagonal(rand(1000))
1000×1000 Diagonal{Float64,Array{Float64,1}}:
 0.0341907   ⋅         ⋅         ⋅        …   ⋅         ⋅         ⋅      
  ⋅         0.758919   ⋅         ⋅            ⋅         ⋅         ⋅      
  ⋅          ⋅        0.700876   ⋅            ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅        0.453502      ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅            ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅        …   ⋅         ⋅         ⋅      
 ⋮                                        ⋱                              
  ⋅          ⋅         ⋅         ⋅            ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅        …   ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅            ⋅         ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅           0.489115   ⋅         ⋅      
  ⋅          ⋅         ⋅         ⋅            ⋅        0.791967   ⋅      
  ⋅          ⋅         ⋅         ⋅            ⋅         ⋅        0.164329

julia> sparse(A)
1000×1000 SparseMatrixCSC{Float64,Int64} with 1000 stored entries:
  [1   ,    1]  =  0.0341907
  [2   ,    2]  =  0.758919
  [3   ,    3]  =  0.700876
  [4   ,    4]  =  0.453502
  [5   ,    5]  =  0.560549
  [6   ,    6]  =  0.832382
  ⋮
  [995 ,  995]  =  0.270777
  [996 ,  996]  =  0.0664044
  [997 ,  997]  =  0.689441
  [998 ,  998]  =  0.489115
  [999 ,  999]  =  0.791967
  [1000, 1000]  =  0.164329

Why can't a SparseMatrixCSC be printed in the same way as the structured sparse-matrix types? It would be a heck of a lot more readable.

PS. On an unrelated note, I also notice that the SparseMatrixCSC output isn't setting the :typeinfo property of the IOContext, so it screws up e.g. #30575. Update: This is fixed by #30589.

[rfourquet]

Not kidding, I had the exact same thought earlier today while reviewing some sparse code!

[abraunst]

Note that for a matrix with density p, the probability of having all k shown entries to be zero is (1-p)^k which for p small behaves as exp(-k p). Assuming k~200 (about the number of 0.0 that julia fits in my terminal), in an arguably realistic scenario for an NxN matrix with N=10^6, p=5/N this probability is ~ 0.999. I think that in a good fraction of real cases all shown entries would be zero. So I don't think that the current representation is so bad. ;-)

[fredrikekre]

I think that in a good fraction of real cases all shown entries would be zero.

True, but the diagonal is quite often filled though.

[andreasnoack]

I also don't like the current printing. Another possibility that we discussed a few years ago was to print a text spy plot instead of the numerical values. Sparse matrices are often quite large and it can often be more interesting to see the distribution of non-zeros than a few of the elements.

[stevengj]

@abraunst, in real sparse matrices it's pretty rare to have a random distribution like that. One option would be to print a portion starting at the first nonzero entry.

[ViralBShah]

I think people working on PDEs and such have the structured sparse matrices, but the folks working on graphs usually don't have that structure.

I do agree that what we do today should be replaced and I also like the idea of the text spy plot.

What if the portion around the first nonzero entry is also extremely sparse? (you could easily have just 1 nonzero in there).

[StefanKarpinski]

We could use Braille patterns to print a fairly compact text spy plot: https://en.wikipedia.org/wiki/Braille_Patterns

[stevengj]

Yes, similar to https://github.com/Evizero/UnicodePlots.jl

[ViralBShah]

Off-topic - I would love to ship UnicodePlots in stdlib. That would also give us these spy plots for little effort.

[JeffBezanson]

Using output like Diagonal when all (or most?) of the matrix fits on the screen, then switching to a text spy plot for larger matrices would be ideal.

[pablosanjose]

Another possibility for the Diagonal-like form is to print the first and last nonzeroes together with their column and row indices, but in matrix form. Something similar to

julia> sparse(A)
1000×1000 SparseMatrixCSC{Float64,Int64} with 1000 stored entries:
_______|11 ______ 14 _______ 102 ______ 932 ____
27     |⋅         0.758919   ⋅          ⋅        
96     |0.389739  ⋅          ⋅          ⋅        
203    |⋅         ⋅          0.700876   ⋅        
701    |⋅         ⋅          ⋅          0.453502

with perhaps some form of color highlighting (or bold, or italic) for the column/row indices

[abraunst]

Another possibility for the Diagonal-like form is to print the first and last nonzeroes together with their column and row indices, but in matrix form. Something similar to

Seems promising, although it is not exactly clear what "first" and "last" indices are. For example, what would be the output for sparse(1:n,n:-1:1,fill(1,n),n,n) (antidiagonal) or a matrix with a + shape?
EDIT: sorry, the antidiagonal shape is easy

[pablosanjose]

Seems promising, although it is not exactly clear what "first" and "last" indices are.

I think it wouldn't be difficult to define this in a sensible way. For example take the N non-zeroes closest to the four corners of the matrix, and create a list of their respective rows and columns, removing duplicates.

[stevengj]

As soon as you put the reader in charge of interpreting row and column indices, I don't see much advantage over what we are currently doing. The visual layout doesn't actually add anything if it bears no relationship to the arrangement of the matrix.

[StefanKarpinski]

It does have a few advantages:

1. It's more visually obvious which are the row and which are the column indices
2. It's more compact since it can potentially show many more non-zeros in the same space
3. It gives some visual sense of the shape of the data (same row, same column, etc.)

The remaining problem is choosing which rows and columns to select.

[ViralBShah]

Personally, to me, the spy plot is the thing I like seeing when I have a sparse matrix. I would love UnicodePlots in stdlib and just have this at my fingertrips (but I can always have that with a few lines of code and installing a package)

[mbauman]

The thing about using a spy plot for the default output is that it's backwards from all our other structured matrices — they use dots to represent locations with no stored value. This becomes particularly apparent if we switch between the two output types depending upon matrix size.

So I also like where @pablosanjose's idea is going.

[andreasnoack]

The thing about using a spy plot for the default output is that it's backwards from all our other structured matrices

1. I doubt that this will be confusing in practice
2. Sparse matrices are also very different from the structured matrices in that they don't have a fixed structure. The structure of a sparse matrix is probably the most important thing to know about it.

Using a pattern similar to the structured matrices would become something like

10974×10974 Symmetric{Float64,SparseMatrixCSC{Float64,Int64}}:
 1.0        ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …       ⋅              ⋅               ⋅
  ⋅        2.27861e7   -2.66358e-7  -32711.2          ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅       -2.66358e-7   3.94135e5        4.33531e-7   ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅   -32711.2          4.33531e-7       1.84354e7    ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅          1.0   ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅   1.0   ⋅   …       ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅   1.0          ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …       ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …       ⋅              ⋅               ⋅
 ⋮                                                        ⋮         ⋱
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅          1.80032e7  27001.2              ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …   5080.43           3.58957e7        ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅             -1.17899e5
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅          2.34176e6      1.03842e8        ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …      3.6964e6       4.14671e7        ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅         -6.21564e6      3.76573e7        ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅         -66123.9
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅               ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅   …  -1924.62       -5001.35             ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅          1.3521e8   -5050.15             ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅      -5050.15           1.3521e8         ⋅
  ⋅         ⋅            ⋅                ⋅           ⋅    ⋅    ⋅           ⋅              ⋅              1.05842e6

and a text spy plot (from UnicodePlots) would look like (except that the character width seems to be off on GitHub).

                                                   Sparsity Pattern
         ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
       1 │⣟⠿⣜⠓⠦⣄⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ > 0
         │⠙⢷⡈⠛⢶⣠⣄⠙⠳⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ < 0
         │⠀⠀⠩⣄⡈⢻⣿⣯⡳⣿⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠈⠛⠲⠻⢶⡻⡿⣜⡻⢧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⢬⡙⠷⣍⠳⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠳⢮⣻⢦⣿⢶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣯⡻⣯⣝⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠲⣏⣷⣾⣛⢦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢮⣹⠾⣌⡷⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⣮⡻⡷⣜⢷⢤⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢳⣜⠻⢧⣘⡳⠄⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢧⠙⠻⢦⡈⠻⢦⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢷⣄⠹⠿⣦⣀⠹⢶⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠷⣤⡈⠙⢶⣤⡈⠙⠖⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢲⣄⠉⠻⣦⣆⡨⠹⣶⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠲⣦⣒⡙⠷⢦⣐⡋⠷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⢯⣌⢛⡿⣿⣛⡷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⠻⣞⣿⡿⣞⡿⢦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠻⢮⣻⣷⣟⢷⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣿⣛⣷⣟⣲⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠿⣿⣿⣿⣟⢦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠿⣿⣷⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⡿⢿⣯⡿⢦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢾⣯⣿⣾⣯⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠺⣿⢿⣯⣿⣶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠷⣟⣿⣾⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣿⡿⣿⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠻⣿⣿⣷⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣿⣿⣶⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠿⣿⣿⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢿⣿⣷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⣿⣷⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⢻⣽⢾⣽⠦⢤⣀⠀⠀⠀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢻⣬⠷⢧⣈⣿⢦⡀⠀⠀│
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠳⣷⣯⣿⣿⣯⣀⠀│
   10974 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠙⢿⣿⣷│
         └────────────────────────────────────────────────────────────────────────────────────────────────────┘
         1                                                                                                10974
                                                      nz = 428650

[Evizero]

Found my way here through discourse. Let me know if I can help. I am also not opposed to simple copy&paste of specific code excerpts. Note though that I am not the original author of the spy function in UnicodePlots; that would be @dpo

[dpo]

I would be delighted to see spy (and all of the amazing UnicodePlots really) find its way into base!

[StefanKarpinski]

I'm afraid that what's going to happen here is that many people are insisting on wanting a spy plot for this, and as a result better sparse matrix printing is going to get coupled with bringing UnicodePlots or some subset of it into Base or stdlib. However, that won't and shouldn't happen. If a package was a sports car, then being in stdlib—or worse, Base—is like it having to drag a a ten ton trailer around behind it all the time. It brings all development to a near complete halt. If you want spy plots, install UnicodePlots and use the spy function. Let's keep this issue focused on improvements to sparse matrix printing that don't require bring the development of UnicodePlots to a standstill.

[Evizero]

(Part of) my point was that the code needed for producing a braille-character spy visualization exists and is decently straighforward. So if the wish is to go towards that route, then there is the option to factor it out of UnicodePlots and into something self contained that can live whereever (instead of reinventing the wheel)

[StefanKarpinski]

If we can get a simple printing of just the spy plot part, I would be fine with that. It's hard to tell from the outside how straightforward it is or isn't.

[pablosanjose]

To see if the Diagonal-type printing could work I've written a small package (https://github.com/pablosanjose/SparseShow) to test it out. It selects and prints a subset of non-zeroes close to the corners. It's just a proof of concept to see if that approach gains any traction, feel free to ignore this if you think we should go in a different direction.

The non-zero selection is currently not very efficient, in the sense that there is an intermediate conversion of the sparse matrix s to indices and values with findnz(s) (but it could be done in a better way without so much allocation, and a bit more code). The complete list of nonzeroes is then sorted according to a distance function dist (Metropolis distance from corners at the moment, can be changed to whatever).

This is the kind of rough output it generates (on a small-size repl)

julia> using SparseShow, SparseArrays

julia> sp = sprand(100, 100, .05)
100×100 SparseMatrixCSC{Float64,Int64} with 509 stored entries:
  [59 ,   1]  =  0.893654
  [3  ,   2]  =  0.604112
  [46 ,   2]  =  0.760531
  ⋮
  [41 , 100]  =  0.345492
  [82 , 100]  =  0.506576
  [85 , 100]  =  0.709841

julia> spshow(sp)
8×5 Array{Union{Spdef, Float64, Int64},2}:
             2         4         5          96       
   1             ⋅     0.375638  0.0951865       ⋅   
   3         0.604112  0.360448      ⋅           ⋅   
   6             ⋅         ⋅         ⋅       0.504533
  91         0.170411      ⋅         ⋅           ⋅   
  92         0.990792      ⋅         ⋅           ⋅   
  99             ⋅         ⋅     0.36217         ⋅   
 100             ⋅         ⋅     0.600704        ⋅   

(Ignore for the time being the summary Array{Union{Spdef, Float64, Int64},2})
Before putting any real work into this, do you think this kind of thing is interesting? There seems to be a 50/50 split in the post above

[ViralBShah]

Certainly an improvement. I think it is interesting and good to take it all the way.

[pablosanjose]

Ok. One difficulty I've hit is how to place the dotted ellipses at arbitrary rows/columns, other than the middle of the matrix. As far as I can tell, the Base machinery (Base.print_matrix) doesn't allow me to choose their position, correct? Also, it doesn't allow me to prepend the display of a matrix with an index column/row (what I did up there is an inelegant hack).

[StefanKarpinski]

I like it. One issue is that it makes it look like this is all of the data in the sparse matrix. If there are non-zeros between some rows or columns, I think that it should print ⋮ and ⋯ in between. The non-zero selection should be done such that you get at most a single row and/or column ellipsis.

[pablosanjose]

Yes, see comment above. The correct spot for the ellipsis is not the middle of the displayed matrix, however. It should be chosen as the split between indices i<N/2 and i>N/2 (where N is the size along the relevant axis). Do we need to extend Base.print_matrix to allow for this? It seems like quite a complicated function.

[stevengj]

That would be really nice, but is it at all possible to do that at the Braille-dot level? I would have thought you can only shade a full character, right?

True. A simpler option would just divide the matrix into MxN bins and put a braille dot for any bin that contains any nonzero entry. But I guess that's what @maxbennedich already proposed?

For sparse matrices, there is also the argument that only the nonzero pattern really matters, not the magnitude of the nonzeros, so a dot for nonzero (regardless of magnitude) might be the best option anyway.

[maxbennedich]

A simpler option would just divide the matrix into MxN bins and put a braille dot for any bin that contains any nonzero entry. But I guess that's what @maxbennedich already proposed?

It's what my first suggestion resulted in, and what I've seen other spy implementations do. However, all implementations I've seen, do this by iterating and plotting each individual non-zero entry, which I think is problematic. But now that you rephrase the problem this way... yes obviously that's a better way to solve it! It should be fairly straightforward and efficient to figure out if a given bin contains any nonzero (by analyzing colptr and rowval).

[pablosanjose]

For sparse matrices, there is also the argument that only the nonzero pattern really matters, not the magnitude of the nonzeros, so a dot for nonzero (regardless of magnitude) might be the best option anyway.

I agree. After all we want to use the same method to show sparse matrices of non-numeric types too. +1 to showing their density, however, as you said, not just whether there is any nonzero in a given bin. It's true that that requires O(nnz) complexity (must collect information from all nonzeros), but I don't agree that is a problem. If you ask to see a humongous sparse matrix, well, that's your responsibility, right? I don't think it will take more that a fraction of a second in any realistic case (though that is just a guess). Moreover, this kind of density sampling could be done with a single allocation of the (small) output matrix, and a sequential (cache friendly) scan of the sparse matrix.

I'm really interested in the ▀ (U+2580, 'Upper Half Block') approach using grayscales, as an analogue sort of the Braille spy plot. I think the result could be truly useful and beautiful, and probably quite simple to code. Actually, my only concern is that the U+2580 symbol does not fill the whole upper half block of a character in my terminal, but leaves a small empty bit above the block (font-specific I'd assume). It quite spoils the whole procedure (see e.g. the upper edge of the pixel matrix in this example). Is there a way to avoid that?

EDIT: cross post with @maxbennedich!

[StefanKarpinski]

When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero. Another way to represent it might be to split the matrix into M×N bins and count how many non-zeros there are in each and print dots only for bins have more than the median number of non-zeros per bin. That way what you're plotting is representative of the non-zero density.

[pablosanjose]

Another PoC using nonzero densities encoded in grayscale half-blocks:

using SparseArrays
sparsedensityshow(S::SparseMatrixCSC, gamma = 1.0) = sparsedensityshow(stdout, S, gamma)
function sparsedensityshow(io::IO, S::SparseMatrixCSC, gamma = 1.0)
    (m, n) = size(S)
    canvassize = (displaysize(io) .- (4, 0)) .* (2, 1)
    bin = maximum(((m, n) .+ canvassize .- (1, 1)) .÷ canvassize)
    colorsize = (round(Int, (m ÷ bin + 1)/2) * 2, n ÷ bin + 1)
    density = fill(0.0, colorsize)
    rows = rowvals(S)
    for col in 1:n
        for ptr in nzrange(S, col)
            (i, j) = (rows[ptr] - 1) ÷ bin,  (col - 1) ÷ bin
            bintotal = (i == colorsize[1] - 1 ? rem(colorsize[1] - 1, bin) : bin) * 
                       (j == colorsize[2] - 1 ? rem(colorsize[2] - 1, bin) : bin)
            density[i + 1, j + 1] += 1/bintotal
        end
    end
    for row in 1:2:colorsize[1]
        for col in 1:colorsize[2]
            printpixel(io, density[row, col], density[row + 1, col], gamma)
        end
        resetpixel(io); print(io, "\n")
    end
    return nothing
end

printpixel(io, fg, bg, gamma) = print(io, "\e[38;5;", 
    232 + round(Int, (23 * bg^gamma)), ";48;5;", 
    232 + round(Int, (23 * fg^gamma)), "m", "▄")
resetpixel(io) = print(io, "\e[39;49m")

This gives this result (EDIT: embedded a bitmap directly here).

It's not rendered correctly in Github (no fancy ANSI codes, I guess). The contrast of bins can be adjusted with parameter gamma. By using a lower-half block "▄" character instead of an upper-half block we can avoid the problem mentioned above. An sprand(10000,10000,0.01) is rendered in 0.1 seconds in my machine.

[maxbennedich]

When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero.

Indeed. Do you think it's common in practice that every bin will include non-zeros, or would most matrices tend to have a more defined structure?

Another way to represent it might be to split the matrix into M×N bins and count how many non-zeros there are in each and print dots only for bins have more than the median number of non-zeros per bin. That way what you're plotting is representative of the non-zero density.

Interesting idea. Are you not concerned about iterating over all non-zeros? For large matrices, it could mean that show becomes a bit sluggish.

I've created two more Braille based PoCs; one that prints a dot for any non-zero in the bin (as above, but much more efficient), and one that prints a dot if the count is above the median. Code available here, and some sample output here:

julia> show_any_nonzero(sprandn(500,1000,0.5); maxw = 24)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 250016

julia> show_above_median_nonzero(sprandn(500,1000,0.5); maxw = 24)
⠸⠶⠒⢀⢲⢤⠐⣓⢍⣰⢦⡖⢸⣦⠴⢠⢔⡹⠴⣜⠤⠐⣲⣽
⣉⡹⡚⠒⣘⢉⢠⣗⢨⣜⣓⣣⢨⣛⣱⠐⡑⢓⢄⡛⣿⢜⣃⢛
⠘⠜⢢⢨⣿⢵⠸⠟⣼⢸⢆⢿⢴⡞⡴⠽⡽⠘⣶⠰⡺⠘⣯⡗
⠒⣶⢐⠒⢃⠴⣰⣲⣷⢰⢚⣠⢠⣦⡾⢖⢄⡕⢠⣁⣌⠀⢾⢾
⢁⢑⣣⢨⣏⡻⢂⡓⡍⢨⣛⣙⣘⠦⢞⢘⢛⠛⣔⡘⡛⠛⡙⡂
⠱⢗⢝⠰⣒⢦⠜⣫⡛⢨⣁⣥⠩⢷⣻⠺⢖⣋⣰⡞⡏⠙⢿⡅
nz = 250433

For the following, the median was 0, so the results were identical for both implementations:

julia> show_any_nonzero(my_sample_matrix; maxw = 32)
⢿⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠹⣿⣿⣶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠘⢿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠻⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠈⢻⣿⣿⣿⣿⣧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠈⠿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢻⣿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣿⣿⣄⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣷⡀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣿⣿⣆⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢿⣷⡄⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣦
nz = 137821

julia> show_any_nonzero(sparse([(z=(x-93)/61+(y-65)/74im;(1:60).|>_->z=z^2-.8+.156im;abs(z)<2) for y=1:130,x=1:184]); maxw = 24)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣼⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⢠⣴⣤⡿⣿⣦⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⢠⣼⣿⣶⡀⣴⣿⣿⣿⣿⣿⣿⣷⠀⠀⣄⠀⣤⠀⠀⠀⠀
⣤⣿⣾⡟⣿⣿⠣⣿⣥⠉⢻⣿⣿⣿⠏⢠⣾⣿⣿⣿⣦⢀⣀⠀
⠈⠹⠟⢹⣿⣿⣦⣭⣿⠀⣾⣿⣿⣿⠀⣿⣶⡆⣽⣿⣷⣿⣿⣦
⠀⠀⠀⠈⣻⡟⢻⠛⠁⢸⣿⣿⣿⣿⣷⣤⣾⠗⣿⣿⣿⡟⠛⠁
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢙⣿⣿⣛⢿⣿⣿⠃⠀⠈⠛⠃⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⣿⡇⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
nz = 4729

Performance:

julia> A = sprand(10000, 10000, 0.5);

julia> @time show_any_nonzero(A; maxw = 100);
...
nz = 50004593  0.005193 seconds (41 allocations: 93.609 KiB)

julia> @time show_above_median_nonzero(A; maxw = 100);
...
nz = 50004593  0.348106 seconds (43 allocations: 406.188 KiB)

[stevengj]

When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero.

I think this is fine. If you have an sprand matrix where the nonzeros are spread out equally over the whole matrix with sufficient density, the correct result is for the spy plot to show nonzeros everywhere. (A density plot would show the same thing.) This, combined with the density of nonzero entries (which should be printed in the summary line), is a reasonable summary of the sparsity structure.

The suggestion for showing only counts above the median is interesting, and I'm not completely opposed, but there is also the possibility that it might be more confusing to explain.

[maxbennedich]

A potential problem with the graphical approaches: Poor Unicode support in Windows command prompt. Braille characters don't display with the default font (have to change to "MS Gothic"), and I wasn't able to get the half blocks to display with any font. It seems to work out of the box in Cygwin.

Would this rule out the graphical approaches?

Then again, built-in functions ∘ (function composition) and ∈ (in) among others also display as garbage in Windows command prompt.

[stevengj]

Yet another reason to finally address #7267.

[stev47]

The suggestion for showing only counts above the median is interesting, and I'm not completely opposed, but there is also the possibility that it might be more confusing to explain.

Why not combine the two: Use braille for signaling any nonzeros and color shading (per braille character) for signaling nonzero density (median normed)?

@maxbennedich for me it seems to be often way faster to just loop over all nonzeros and count into buckets instead of the searchsortedfirst thing, since the number of nonzero rows for one column is rather small and often even less than the plot height. As long as the matrix is not very row-dense, this might not be that bad.

[pablosanjose]

Why not combine the two: Use braille for signaling any nonzeros and color shading (per braille character) for signaling nonzero density (median normed)?

That would be nice. However, one problem with this is that the Braille shading should not be a grayscale color, but rather a transparency alpha channel, since you don't know what the background color of the terminal is (it is user-dependent). Very few terminals (any?) allow control of the character alpha. In the half-block approach you control both background and foreground colors, so this is not an issue, as long as your terminal supports at least 8-bit colors.

Scratch that. You can just override the background color of the Braille characters.

[maxbennedich]

I like the coloring approaches, but am concerned it might look strange depending on the color scheme used in people's terminals.

One way to solve the density plot without using colors is to use sampling. Consider a matrix with random density increasing linearly from 0 to 1 towards the right. The "any nonzero" approach renders it like this:

julia> show_any_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 4999242

The "above median" approach renders it like this:

julia> show_above_median_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⢻⣽⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠠⢐⠿⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢰⣹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠐⢘⣹⣺⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 5001619

A sample based approach (such as in this post above) looks better:

julia> show_sampled_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⠀⠀⠀⠀⢀⠀⠈⠀⡁⠀⠈⠈⡅⠤⠀⠠⢄⠁⠙⠄⠈⢰⡛⡀⢂⠰⠄⡤⢬⡀⠘⠈⣔⢌⡻⢇⣀⡥⠉⡗⢪⠵⡓⣯⣿⢠⢯⣣⣁⠁⣽⢏⣟⣝⡡⣧⣧⡯⠯⢿⣫⣿⢟⢾⣻⢿⣿⣿⣿⡿⣷⡿⣟⣿⣯⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠐⢀⠀⠄⠐⠠⠀⣄⠕⠂⢀⡄⠀⢀⠁⠁⠃⠘⠖⠉⡑⠾⡀⣊⠋⠹⢢⣽⡑⢏⢀⣰⣄⣇⣏⡮⡋⢻⣛⢢⣄⣝⣖⡷⠮⡿⣣⣕⣯⣡⡙⡎⠥⣾⣟⡷⡻⢽⣿⢁⣖⣷⣷⣷⣿⣷⣽⣿⣿⣿⣟⣿⣿⣻⣿⣿
⠀⢀⠠⠀⠀⠌⠠⡂⠀⠀⠀⡐⠀⠐⡀⢁⠐⢒⠈⢩⠥⣋⠀⠨⢓⣅⠡⠂⠈⢋⠳⣣⠦⡂⡁⣋⠁⣠⣼⣲⡉⠐⢹⣨⣽⡲⠥⣣⢿⣉⣾⣋⢿⠯⣗⢓⡽⢾⢌⣹⠯⢺⢛⣿⡻⣸⣷⣿⣙⡿⣫⣿⣏⣽⣿⣾⣿⣿⣿⣿
⠀⠀⠀⠄⠀⠀⠄⢂⢠⠠⠀⠀⠀⢂⠀⠈⠪⠠⠄⢨⠀⢀⣅⠢⢥⠕⣀⠠⡬⢆⠳⡀⠘⠯⠁⠰⠷⣪⠳⣄⢭⢦⠆⢳⡯⢅⣘⣿⡹⣵⡽⣷⠽⣘⢳⢣⡾⠿⡹⣿⡹⢯⣬⢭⢛⣟⣻⣽⣷⣿⣾⣿⣯⣿⣿⣿⣹⣿⣿⣿
nz = 5000152

However, this approach might also be quite confusing since it could miss non-zeros. I also don't know if this makes a difference for real-world sparse matrices.

@maxbennedich for me it seems to be often way faster to just loop over all nonzeros and count into buckets instead of the searchsortedfirst thing

I think you're right for matrices with a small/medium amount of non-zeros, but for larger matrices this is not the case. For a 10k x 10k matrix with 50 million random non-zeros, my half-optimized code loops over and buckets all non-zeros in ~0.35 seconds on my system, or 20 CPU cycles per non-zero. IMO, this is not acceptable for a show implementation. For example, if you're working in Atom and using the Workspace view, every single command you do in the REPL will call show on all variables, and thus incur a 0.35 second complete freeze, with one CPU pegged at 100 %. Compare to the searchsortedfirst implementation (show_any_nonzero) which does the same thing in ~0.005 seconds (see benchmark above).

I updated the gist I linked above with a show_any_nonzero_loop_nz method that loops over all non-zeros; feel free to take a look and comment in case you had a better implementation in mind.

[stev47]

For a 10k x 10k matrix with 50 million random non-zeros [...]

To me that is a rather dense matrix. For row-dense matrices searchsortedfirst along the columns is indeed faster. If one wants to optimize printing a large dense SparseMatrix, one could get information about row-density through diff(S.colptr) and use the appropriate method.

[maxbennedich]

Ah, I see what you mean. Good point.

[ViralBShah]

I wonder where we finally ended on this one. It would be nice to get something in. Or should these things go into UnicodePlots or elsewhere?

[maxbennedich]

If there's interest in a Braille-based plot, I'm happy to volunteer some time to do a proper implementation. Before proceeding, I have these two doubts:

1. Font may not support Braille characters, in particular Windows command prompt.
2. show might become slow for large, dense matrices (example above: 10k x 10k matrix with 50 million entries could render in 0.35 second on a consumer laptop). A fix would be to use a sample-based approach, i.e. don't look at every nonzero, but sample the matrix at a bunch of indexes to create the plot. Faster, but less accurate. Or a hybrid between the two approaches.

[stev47]

Using sixels for plotting might also be an idea, see also e.g. lsix. Terminal support might vary, but at least it's font-independent.

[StefanKarpinski]

I think I'd prefer to go with @pablosanjose's approach where we print the non-zeros. That is less of a departure from what we've done and continues to show actual values in the matrix. I would also be in favor of having built-in spy functionality for matrices in general, but I think that can be added separately.

[stevengj]

I still don't see @pablosanjose's approach as much of an improvement over the current output. spy-like output, on the other hand, would be a huge improvement.

[pablosanjose]

Whatever the final decision, I think it should not rely on processing all non-zeros of the sparse matrix. That would likely bog down the display of really huge matrices that often arise in real world use. show should be snappy. Hence, I think the spy-plot should just do partial density sampling. Same for the gray-scale version using half-blocks.

[StefanKarpinski]

This is a clear case of the perfect being the enemy of the good. Regardless of how much people may love spy plots for sparse matrices, we don't show any other kinds of data structures using that kind of visualization. No one is saying that we shouldn't have spy but people insisting on that being the default way to show sparse matrices has prevented any improvement from happening here.

[StefanKarpinski]

That sounded a little bit too salty. My point is that having two options here is preventing any progress and printing sparse matrices as spy plots by default is controversial and unusual enough for it not to actually happen, leaving us where we are with no improvement at all.

[pablosanjose]

Well, I do see @stevengj's argument: why is it better to show four matrix corners than just the beginning and end of first and last columns?

But I would point out that the alternative sparseshow style does have at least two advantages over the current show: it makes better use of screen real state (uses all the available REPL width, and can thus show "more" information), and can potentially also reuse column/row indices when more than one shown matrix element share them. Whether those advantages are actually worth the change is a matter of debate, I guess.

[ViralBShah]

Do we have a PR for the spy like printing?

[dkarrasch]

Yes we have #33821. Everybody is kindly invited to review.