julialinearalgebra / matrixdepot.jl Goto Github PK

View Code? Open in Web Editor NEW

73.0 73.0 22.0 4.21 MB

An Extensible Test Matrix Collection for Julia

Home Page: http://matrixdepotjl.readthedocs.org/

License: Other

Julia 84.24% HTML 15.76%

matrixdepot.jl's People

Contributors

Stargazers

Watchers

matrixdepot.jl's Issues

Wilkison test fails on Julia 0.3.5

In running the tests, I noticed that under Julia 0.3.5, I get the following error message:

$ julia runtests.jl
...
'parter' passed test...
'chow' passed test...
'randcorr' passed test...
'poisson' passed test...
'neumann' passed test...
'rosser' passed test...
'sampling' passed test...
ERROR: key not found: "wilkinson"
 in getindex at dict.jl:617
 in matrixdepot at /Users/lev/.julia/v0.3/MatrixDepot/src/MatrixDepot.jl:69
 in include at /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/sys.dylib
 in include_from_node1 at /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/sys.dylib
 in include at /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/sys.dylib
 in include_from_node1 at loading.jl:128
 in process_options at /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/sys.dylib
 in _start at /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/sys.dylib
while loading /Users/lev/Projects/github/MatrixDepot.jl/test/test_wilkinson.jl, in expression starting on line 3
while loading /Users/lev/Projects/github/MatrixDepot.jl/test/runtests.jl, in expression starting on line 36

Using one of the latest distributed nightly binaries (Julia-0.4.0-dev-02c0501ba9), all of the test pass fine.

Vandermonde performance

While performance might not be the primary goal of this package, I thought I'd mention that while trying to get a handle on Julia's performance, I benchmarked a number of Vandermonde implementations, including the one in MatrixDepot against numpy's np.vander. The results are here:

http://nbviewer.ipython.org/gist/synapticarbors/26910166ab775c04c47b

I'd be happy to submit a PR implementing vander3 if you're interested.

register v1.0.11

@JuliaRegistrator register
v1.0.11

Creating and modifying user groups

If the deprecated user extension for user defined functions and groups is not used (the corresponding directories/files are missing)
it is not possible to create/modyfy/remove user defined groups.
It is also impossible to add a user function to such a group.

MatrixDepot does not initialize under julia1.4

EDIT: reason for the following misbehavior was an obsolete cached data file db.data.
WORKAROUND: remove MatrixDepot/data/db.data file. It will be re-created when module is initialized next time.

Under julia1.4 the module is not usable.
Under 1.2 no problems, status for julia1.3 is unknown.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0-DEV.595 (2019-12-17)
 _/ |\__'_|_|_|\__'_|  |  krc/awmax/5fc8a05be4* (fork: 3 commits, 17 days)
|__/                   |

julia> using MatrixDepot
[ Info: Precompiling MatrixDepot [top-level]
include group.jl for user defined matrix generators
verify download of index files...
┌ Warning: exception during initialization: 'KeyError(MatrixDepot [b51810bb-c9f3-55da-ae3c-350fc1fbce05])'
└ @ MatrixDepot ~/.julia/dev/MatrixDepot/src/MatrixDepot.jl:135

julia> MatrixDepot.DATA_DIR
"/home/crusius/.julia/dev/MatrixDepot/data"

julia> MatrixDepot.mdinfo()
ERROR: MatrixDepot.DataError("no matrix according to baart found")

julia> MatrixDepot.init()
include group.jl for user defined matrix generators
verify download of index files...
ERROR: KeyError: key MatrixDepot [b51810bb-c9f3-55da-ae3c-350fc1fbce05] not found
Stacktrace:
 [1] getindex at ./dict.jl:477 [inlined]
 [2] root_module at ./loading.jl:960 [inlined]
 [3] deserialize_module(::Serialization.Serializer{IOStream}) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:912
 [4] handle_deserialize(::Serialization.Serializer{IOStream}, ::Int32) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:812
 [5] deserialize(::Serialization.Serializer{IOStream}) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:735
 [6] deserialize_datatype(::Serialization.Serializer{IOStream}, ::Bool) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:1210
 [7] handle_deserialize(::Serialization.Serializer{IOStream}, ::Int32) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:788
 [8] deserialize(::Serialization.Serializer{IOStream}) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:735
 [9] handle_deserialize(::Serialization.Serializer{IOStream}, ::Int32) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:795
 [10] deserialize(::Serialization.Serializer{IOStream}) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:735
 [11] handle_deserialize(::Serialization.Serializer{IOStream}, ::Int32) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:838
 [12] deserialize(::Serialization.Serializer{IOStream}) at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:735
 [13] deserialize at /home/julia/julia/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:722 [inlined]
 [14] (::MatrixDepot.var"#100#101"{MatrixDepot.MatrixDatabase})(::IOStream) at /home/crusius/.julia/dev/MatrixDepot/src/download.jl:101
 [15] open(::MatrixDepot.var"#100#101"{MatrixDepot.MatrixDatabase}, ::String, ::Vararg{String,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:298
 [16] open at ./io.jl:296 [inlined]
 [17] readdb at /home/crusius/.julia/dev/MatrixDepot/src/download.jl:100 [inlined]
 [18] downloadindices(::MatrixDepot.MatrixDatabase; ignoredb::Bool) at /home/crusius/.julia/dev/MatrixDepot/src/download.jl:125
 [19] init(; ignoredb::Bool) at /home/crusius/.julia/dev/MatrixDepot/src/MatrixDepot.jl:128
 [20] init() at /home/crusius/.julia/dev/MatrixDepot/src/MatrixDepot.jl:102
 [21] top-level scope at REPL[4]:1

julia>

Implement matrices from CONTEST toolbox

CONTEST (http://strathprints.strath.ac.uk/15066/) contains 9 graph models (symmetric or unsymmetric matrices with elements equal to zero or one). They have one or more parameters that affect features such as sparsity and characteristic pathlength and all can be of arbitrary dimension.

Version 1.0.12

@JuliaRegistrator register
v1.0.12

Generic infrastructure

I was about to ask about having graph datasets. They are mostly matrices but the edges might be ordered and have other metadata on them. What part of the infrastructure in this project is reusable and can be shared between RDatasets and MatrixDepot and a hypothetical GraphsDepot? You have a some capacity for making personal collections, labeling, indexing, querying, and IO. Also there are features for generators and static matrices.

Static compilation fails on MatrixDepot.jl

I’m trying to create a system image of Julia with MatrixDepot.jl embedded, I encountered following error while building Julia (using release-0.5 branch).

â  make -j 4
    PERL base/version_git.jl.phony
    JULIA usr/lib/julia/sys.o
coreio.jl
exports.jl
essentials.jl
base.jl
generator.jl
reflection.jl
options.jl
promotion.jl
tuple.jl
range.jl
expr.jl
error.jl
bool.jl
number.jl
int.jl
operators.jl
pointer.jl
refpointer.jl
WARNING: Method definition (::Type{#T<:Any})(Any) in module Inference at coreimg.jl:39 overwritten in module Base at sysimg.jl:53.
checked.jl
abstractarray.jl
subarray.jl
array.jl
hashing.jl
rounding.jl
float.jl
complex.jl
rational.jl
multinverses.jl
abstractarraymath.jl
arraymath.jl
float16.jl
simdloop.jl
reduce.jl
reshapedarray.jl
bitarray.jl
intset.jl
dict.jl
set.jl
iterator.jl
build_h.jl
version_git.jl
osutils.jl
c.jl
sysinfo.jl
io.jl
iostream.jl
iobuffer.jl
char.jl
intfuncs.jl
strings/strings.jl
strings/errors.jl
strings/string.jl
strings/types.jl
strings/basic.jl
strings/search.jl
strings/util.jl
strings/io.jl
strings/utf8proc.jl
parse.jl
shell.jl
regex.jl
pcre.jl
show.jl
base64.jl
nullable.jl
version.jl
libc.jl
libdl.jl
env.jl
libuv.jl
uv_constants.jl
event.jl
task.jl
lock.jl
threads.jl
weakkeydict.jl
stream.jl
socket.jl
filesystem.jl
process.jl
multimedia.jl
grisu/grisu.jl
methodshow.jl
floatfuncs.jl
math.jl
cartesian.jl
multidimensional.jl
permuteddimsarray.jl
reducedim.jl
ordering.jl
collections.jl
sort.jl
WARNING: Method definition searchsortedfirst(AbstractArray{T<:Any, 1}, Any) in module Sort at sort.jl:184 overwritten at sort.jl:187.
WARNING: Method definition searchsortedlast(AbstractArray{T<:Any, 1}, Any) in module Sort at sort.jl:184 overwritten at sort.jl:187.
WARNING: Method definition searchsorted(AbstractArray{T<:Any, 1}, Any) in module Sort at sort.jl:184 overwritten at sort.jl:187.
gmp.jl
mpfr.jl
combinatorics.jl
hashing2.jl
dSFMT.jl
random.jl
printf.jl
meta.jl
Enums.jl
serialize.jl
channels.jl
clusterserialize.jl
multi.jl
workerpool.jl
pmap.jl
managers.jl
asyncmap.jl
loading.jl
mmap.jl
sharedarray.jl
datafmt.jl
deepcopy.jl
interactiveutil.jl
replutil.jl
test.jl
i18n.jl
initdefs.jl
Terminals.jl
LineEdit.jl
REPLCompletions.jl
REPL.jl
client.jl
util.jl
linalg/linalg.jl
broadcast.jl
statistics.jl
irrationals.jl
dft.jl
dsp.jl
quadgk.jl
fastmath.jl
libgit2/libgit2.jl
pkg/pkg.jl
stacktraces.jl
profile.jl
dates/Dates.jl
sparse/sparse.jl
threadcall.jl
deprecated.jl
docs/helpdb.jl
docs/helpdb/Base.jl
docs/basedocs.jl
markdown/Markdown.jl
docs/Docs.jl
/home/julia/TopStarredImage/julia/base/precompile.jl
LoadError("sysimg.jl",381,LoadError("/home/julia/TopStarredImage/julia/base/userimg.jl",1,LoadError("/home/julia/TopStarredPackages/v0.5/MatrixDepot/src/MatrixDepot.jl",30,UndefVarError(:uv_evjulia> Pkg.status("OS = Ubuntu 14.04

entloop))))
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [/home/julia/TopStarredImage/julia/usr/lib/julia/sys.o] Error 1
make: *** [julia-sysimg-release] Error 2

OS = Ubuntu 14.04
MatrixDepot version = 0.5.6

circul takes two vectors but uses entries of only the first one

It uses the length of the second one. This is kind of weird, but I guess not technically a bug. Is this interface choice deliberate?

Certificates for https://sparse.tamu.edu/ expired?

Is anyone else getting curl errors like this?

An example from loading MatrixDepot.

┌ Warning: exception during initialization: 'ProcessFailedException(Base.Process[Process(`sh -c 'curl "https://sparse.tamu.edu/files/ss_index.mat" -Lso "/home/runner/.julia/scratchspaces/b51810bb-c9f3-55da-ae3c-350fc1fbce05/data/ss_index.mat"'`, ProcessExited(60))])'

This is due to expired certificates from the suitesparse website. A browser might experience similar issues.

A quick fix is to put -k right after curl in the download_command() function, but I'm not sure if that's safe because it essentially says to ignore the certificates. Any ideas on how we can fix this one?

"no metadata .mtx" error on second access

When I access a SuiteSparse matrix for the first time, everything is fine. When I quit and restart Julia, though, I get a fatal error. Example:

driscoll@~\> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3 (2018-12-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MatrixDepot
include group.jl for user defined matrix generators
verify download of index files...
used remote site is https://sparse.tamu.edu/?per_page=All
populating internal database...

julia> A = matrixdepot("Gset/G24");
downloading: https://sparse.tamu.edu/MM/Gset/G24.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60541  100 60541    0     0   119k      0 --:--:-- --:--:-- --:--:--  120k
sx G24/G24.mtx
ize(A
julia> size(A)
(2000, 2000)

julia> eltype(A)
Bool

julia>
driscoll@~\> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3 (2018-12-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MatrixDepot
include group.jl for user defined matrix generators
verify download of index files...
used remote site is https://sparse.tamu.edu/?per_page=All
populating internal database...

julia> A = matrixdepot("Gset/G24");
ERROR: MatrixDepot.DataError("Gset/G24 has no metadata G24.mtx")
Stacktrace:
 [1] daterr(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:6
 [2] _metareader(::MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:35
 [3] (::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}})() at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:12
 [4] get!(::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}}, ::Dict{Any,Any}, ::String) at ./dict.jl:453
 [5] metareader(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:11
 [6] getproperty(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::Symbol) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:62
 [7] (::getfield(MatrixDepot, Symbol("##86#87")))(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:511
 [8] mdopen(::getfield(MatrixDepot, Symbol("##86#87")), ::MatrixDepot.MatrixDatabase, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:457
 [9] matrixdepot at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:510 [inlined]
 [10] matrixdepot(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:508
 [11] top-level scope at none:0

julia> md = mdopen("Gset/G24");

julia> propertynames(md)
4-element Array{Symbol,1}:
 :m
 :n
 :nnz
 :dnz

julia> md.A
ERROR: MatrixDepot.DataError("Gset/G24 has no metadata G24.mtx")
Stacktrace:
 [1] daterr(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:6
 [2] _metareader(::MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:35
 [3] (::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}})() at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:12
 [4] get!(::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}}, ::Dict{Any,Any}, ::String) at ./dict.jl:453
 [5] metareader(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:11
 [6] getproperty(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::Symbol) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:62

julia>

[PkgEval] MatrixDepot may have a testing issue on Julia 0.4 (2015-06-30)

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.4

On 2015-06-27 the testing status was Tests pass.
On 2015-06-30 the testing status changed to Tests fail.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("MatrixDepot")' log
INFO: Cloning cache of MatrixDepot from git://github.com/weijianzhang/MatrixDepot.jl.git
INFO: Cloning cache of MatrixMarket from git://github.com/JuliaSparse/MatrixMarket.jl.git
INFO: Installing GZip v0.2.15
INFO: Installing MatrixDepot v0.2.6
INFO: Installing MatrixMarket v0.0.1
INFO: Package database updated

>>> 'Pkg.test("MatrixDepot")' log
Julia Version 0.4.0-dev+5700
Commit 147fa0b* (2015-06-29 20:31 UTC)
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
INFO: Testing MatrixDepot
'cauchy' passed test...
'circul' passed test...
'hadamard' passed test...
'hilb' passed test...
'dingdong' passed test...
'frank' passed test...
'invhilb' passed test...
'forsythe' passed test...
'magic' passed test...
ERROR: LoadError: LoadError: test error in expression: matrixdepot("grcar",n,3) == matrixdepot("grcar",n)
BoundsError
 in tril! at linalg/dense.jl:92
 in grcar at /home/vagrant/.julia/v0.4/MatrixDepot/src/higham.jl:201
 in matrixdepot at /home/vagrant/.julia/v0.4/MatrixDepot/src/common.jl:78
 in anonymous at test.jl:89
 in do_test at test.jl:49
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:133
 in include at ./boot.jl:254
 in include_from_node1 at loading.jl:133
 in process_options at ./client.jl:305
 in _start at ./client.jl:405
 in tril! at linalg/dense.jl:92
 in grcar at /home/vagrant/.julia/v0.4/MatrixDepot/src/higham.jl:201
 in matrixdepot at /home/vagrant/.julia/v0.4/MatrixDepot/src/common.jl:78
 in anonymous at test.jl:89
 in do_test at test.jl:49
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:133
 in include at ./boot.jl:254
 in include_from_node1 at loading.jl:133
 in process_options at ./client.jl:305
 in _start at ./client.jl:405
while loading /home/vagrant/.julia/v0.4/MatrixDepot/test/test_grcar.jl, in expression starting on line 2
while loading /home/vagrant/.julia/v0.4/MatrixDepot/test/runtests.jl, in expression starting on line 13
=============================[ ERROR: MatrixDepot ]=============================

failed process: Process(`/home/vagrant/julia/bin/julia --check-bounds=yes --code-coverage=none --color=no /home/vagrant/.julia/v0.4/MatrixDepot/test/runtests.jl`, ProcessExited(1)) [1]

================================================================================
INFO: No packages to install, update or remove
ERROR: MatrixDepot had test errors
 in error at ./error.jl:21
 in test at pkg/entry.jl:746
 in anonymous at pkg/dir.jl:31
 in cd at file.jl:22
 in cd at pkg/dir.jl:31
 in test at pkg.jl:71
 in process_options at ./client.jl:281
 in _start at ./client.jl:405

>>> End of log

Tag for updated compats

minor README suggestion

Thank you for the work.
Some users benefit with this extra hand-holding:

Basic Usage

To see all the matrices in the collection, type

julia> using MatrixDepot
julia> matrixdepot()

Use Scratch.jl for default data dir?

I almost always use a custom data directory for MatrixDepot (my supercomputer filesystem for scratch data) because I don't want to redownload the matrices when the package updates, and I need to ensure the data path is writable. However, there is now a more automatic solution for this kind of problem, Scratch.jl. I think that Scratch.jl may provide a more useful default data directory which persists across updates to the package.

Currently, the default is in the package itself set on line 140 of "data.jl"

const DATA_DIR = abspath(dirname(@__FILE__),"..", "data")

I think it would make a lot of sense to change that line to

const DATA_DIR = @get_scratch!("data")

why the nightly version tests fail

The nightly version now incorporates RowVectors taking vector transposes seriously. (search for RowVector).

Request: Hansen's test matrices for regularization methods

You might be interested in implementing some matrices in Per-Christian Hansen's paper (see Reference). In that paper, Hansen describes the generation of oscillatory matrices, which are matrices A with some powers A^p that are totally positive.

The paper also describes other test matrices available in Matlab's Regularization Toolbox.

These matrices would have been nice to have back when JuliaLang/julia#8859 was being discussed. It might still be useful for other purposes.

Reference: doi:10.1137/0916032

Random matrices for testing purposes

Hi weijian, I have this old file https://github.com/mschauer/SDE.jl/blob/v3/src/Randm.jl with which generates random symmetric, positive definite, stable, normal, orthogonal and unitary matrices for testing purposes. Maybe you are interested, or I can make a pull request, if you wish.

Problems with Windows

Just a quick note that the download command using sh and curl is causing lots of problems for people who use windows. Even installing MinGW and MSYS2 doesn't seem to fix the issue.

e.g. that will still leave sh unresolved on the path as that isn't a standard path addition.

I imagine there is a good reason to do it this way, but perhaps there could be alternatives?

(Sorry for the sloppier than typical bug report, this is on behalf of the students in my class that use windows and so I've been trying to help them and coming up empty handed...)

Symmetric matrices

It's probably not a good idea to declare symmetric matrices as Symmetric{...} for the time being because it seems sparsity is then ignored. See https://discourse.julialang.org/t/slow-matrix-vector-product-with-symmetric-matrices/3968 for some timings and an explanation.

python?

This might be a dumb question, but is there a Python interface? This possibility was mentioned in https://media.readthedocs.org/pdf/matrix-repository/latest/matrix-repository.pdf which seems to have been merged into MatrixDepot.jl.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Documentation at https://matrixdepotjl.readthedocs.io/en/latest/interface.html is old

Site https://matrixdepotjl.readthedocs.io/en/latest/interface.html shows very old content not the latest one.

Warn about non-structural zeros

I just realized that some coefficients are stored even though their value is zero. Is that intended behavior?

julia> using MatrixDepot, SparseArrays

julia> mat = matrixdepot("Bai/af23560");

julia> nnz(mat)
484256

julia> sum(!iszero, mat)
460598

julia> nnz(dropzeros(mat))
460598

Error occurs in "download.jl" on Windows cause of "tar" function

Hello, I got an error when trying to download data using:
mdopen("SNAP/web-Google")
Error Message: tar: Cannot connect to C: resolve failed
failed process: Process(`tar -vxf 'C:\path\path' -C 'C:\path\path' ....)

According to tar manual at chapter 6.1 Choosing and Naming Archive Files, "If the archive file name includes a colon (‘:’), then it is assumed to be a file on another machine" and also ". Filenames with a colon are common for Windows users (e.g., C: ) and if you need to use a file whose name includes a colon, then the remote tape drive behavior can be inhibited by using the ‘--force-local’ option".

The problem is solved by editing the download.jl line199 to run(tar --force-local -vxf $tarfile -C $dir). Maybe you should modify this line of code.

Thank you.

v1.0.7

@JuliaRegistrator register

Improve Code Coverage

Code coverage dropped from 99% to 66%. That is not acceptable.
New target: 100%

Obtaining all positive definite matrices from UFL?

Is there a way to ask MatrixDepot to fetch all symmetric positive definite matrices from the UFL collection? I searched and never found a way, so I obtained a list of the names of those 216 matrices using #29 and downloaded them with the matrixdepot command. But now

julia> mdlist(:posdef)
14-element Array{String,1}:
 ...

only ever gives a list of 14 matrices. Does it not know about the other ones? I must be misunderstanding what groups are for.

Thanks.

new release 1.0.10

@JuliaRegistrator register

new release 1.0.10

correct typos in documentation
adapt to ChannelBuffers v0.3 and newer

mmreadheader should not throw for empty files, but loadmatrix should

The attempt to download not-existing files can leave an empty *.mtx file as an indication of that.
This should not terminate the loading of meta data when the database is initialized.

Matrices without values load as -1

If the mtx file has a blank 3rd column, matrixdepot loads the matrix with values -1.
For example the Newman/polbooks matrix now edge weights (values column is empty)
This works on Newman/football which provides the values column as ones(n).
Is this a bug or the intended behavior? If intended, why?

julia> using MatrixDepot; a = matrixdepot("Newman/polbooks", :r)[1:10,1:10]
10x10 Array{Float64,2}:
  0.0  -1.0  -1.0  -1.0  -1.0  -1.0  -1.0   0.0   0.0   0.0
 -1.0   0.0   0.0  -1.0   0.0  -1.0  -1.0   0.0   0.0   0.0
 -1.0   0.0   0.0   0.0  -1.0  -1.0   0.0  -1.0   0.0   0.0
 -1.0  -1.0   0.0   0.0   0.0  -1.0   0.0   0.0  -1.0  -1.0
 -1.0   0.0  -1.0   0.0   0.0  -1.0  -1.0   0.0   0.0   0.0
 -1.0  -1.0  -1.0  -1.0  -1.0   0.0  -1.0  -1.0   0.0   0.0
 -1.0  -1.0   0.0   0.0  -1.0  -1.0   0.0  -1.0   0.0   0.0
  0.0   0.0  -1.0   0.0   0.0  -1.0  -1.0   0.0   0.0   0.0
  0.0   0.0   0.0  -1.0   0.0   0.0   0.0   0.0   0.0  -1.0
  0.0   0.0   0.0  -1.0   0.0   0.0   0.0   0.0  -1.0   0.0

Here are the files

> head -n 50 polbooks.mtx                                                    /h/j/./v/M/d/u/Newman (git)-[master] 
%%MatrixMarket matrix coordinate pattern symmetric
%-------------------------------------------------------------------------------
% UF Sparse Matrix Collection, Tim Davis
% http://www.cise.ufl.edu/research/sparse/matrices/Newman/polbooks
% name: Newman/polbooks
% [Books about US politics, compiled by V. Krebs]
% id: 2403
% date: 2001
% author: V. Krebs
% ed: M. Newman
% fields: name title A id date author kind notes aux ed
% aux: nodename nodevalue
% kind: undirected graph
%-------------------------------------------------------------------------------
% notes:
% Network collection from M. Newman                                         
% http://www-personal.umich.edu/~mejn/netdata/                              
%                                                                           
% Books about US politics                                                   
% Compiled by Valdis Krebs                                                  
%                                                                           
% Nodes represent books about US politics sold by the online bookseller     
% Amazon.com.  Edges represent frequent co-purchasing of books by the same  
% buyers, as indicated by the "customers who bought this book also bought   
% these other books" feature on Amazon.                                     
%                                                                           
% Nodes have been given values "l", "n", or "c" to indicate whether they are
% "liberal", "neutral", or "conservative".  These alignments were assigned  
% separately by Mark Newman based on a reading of the descriptions and      
% reviews of the books posted on Amazon.                                    
%                                                                           
% These data should be cited as V. Krebs, unpublished,                      
% http://www.orgnet.com/.                                                   
%-------------------------------------------------------------------------------
105 105 441
2 1
3 1
4 1
5 1
6 1
7 1
4 2
6 2
7 2
5 3
6 3
8 3
6 4
9 4
10 4

> head -n 50 football.mtx                                                    /h/j/./v/M/d/u/Newman (git)-[master] 
%%MatrixMarket matrix coordinate integer symmetric
%-------------------------------------------------------------------------------
% UF Sparse Matrix Collection, Tim Davis
% http://www.cise.ufl.edu/research/sparse/matrices/Newman/football
% name: Newman/football
% [American football games between Div IA colleges, Fall 2000]
% id: 2397
% date: 2002
% author: M. Girvan, M. Newman
% ed: M. Newman
% fields: name title A id date author kind notes aux ed
% aux: nodename nodevalue
% kind: undirected multigraph
%-------------------------------------------------------------------------------
% notes:
% Network collection from M. Newman                                        
% http://www-personal.umich.edu/~mejn/netdata/                             
%                                                                          
% The graph football contains the network of American football games       
% between Division IA colleges during regular season Fall 2000, as compiled
% by M. Girvan and M. Newman.  The nodes have values that indicate to which
% conferences they belong.  The values are as follows:                     
%                                                                          
%   0 = Atlantic Coast                                                     
%   1 = Big East                                                           
%   2 = Big Ten                                                            
%   3 = Big Twelve                                                         
%   4 = Conference USA                                                     
%   5 = Independents                                                       
%   6 = Mid-American                                                       
%   7 = Mountain West                                                      
%   8 = Pacific Ten                                                        
%   9 = Southeastern                                                       
%  10 = Sun Belt                                                           
%  11 = Western Athletic                                                   
%                                                                          
% If you make use of these data, please cite M. Girvan and M. E. J. Newman,
% Community structure in social and biological networks,                   
% Proc. Natl. Acad. Sci. USA 99, 7821-7826 (2002).                         
%-------------------------------------------------------------------------------
115 115 613
2 1 1
5 1 1
10 1 1
17 1 1
24 1 1
34 1 1
36 1 1
42 1 1
66 1 1

Two failing tests on Julia 1.7

On macOS Big Sur and Julia 1.7:

running remote.jl ...
[ Info: downloading head of https://sparse.tamu.edu/MM/HB/494_bus.tar.gz
downloading svd for */494_bus
[ Info: downloading: https://sparse.tamu.edu/svd/HB/494_bus_SVD.mat
[ Info: downloading: https://sparse.tamu.edu/MM/HB/494_bus.tar.gz
[ Info: downloading head of ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/psadmit/494_bus.mtx.gz
gzip: (stdin): unexpected end of file
MatrixDepot real remote matrix tests: Test Failed at /Users/Peter/Projects/MatrixDepot.jl/test/remote.jl:19
  Expression: loadinfo(pattern) == 1
   Evaluated: 0 == 1
Stacktrace:
 [1] macro expansion
   @ ~/Projects/julia/usr/share/julia/stdlib/v1.7/Test/src/Test.jl:445 [inlined]
 [2] top-level scope
   @ ~/Projects/MatrixDepot.jl/test/remote.jl:19
[ Info: downloading: ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/psadmit/494_bus.mtx.gz
┌ Warning: could not load Harwell-Boeing/psadmit/494_bus: ProcessFailedException(Base.Process[Process(`sh -c 'curl "ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/psadmit/494_bus.mtx.gz" -Lso "/var/folders/s4/k_gk1yr92qdby7zrf07blxpr0000gn/T/jl_p427Ta/data/mm/Harwell-Boeing/psadmit/494_bus.mtx.gz"'`, ProcessExited(28))])
└ @ MatrixDepot ~/Projects/MatrixDepot.jl/src/common.jl:439
MatrixDepot real remote matrix tests: Test Failed at /Users/Peter/Projects/MatrixDepot.jl/test/remote.jl:20
  Expression: load(pattern) == 1
   Evaluated: 0 == 1
Stacktrace:
 [1] macro expansion
   @ ~/Projects/julia/usr/share/julia/stdlib/v1.7/Test/src/Test.jl:445 [inlined]
 [2] top-level scope
   @ ~/Projects/MatrixDepot.jl/test/remote.jl:20
finished remote.jl
Test Summary:                        | Pass  Fail  Total
MatrixDepot real remote matrix tests |    3     2      5
ERROR: LoadError: Some tests did not pass: 3 passed, 2 failed, 0 errored, 0 broken.
in expression starting at /Users/Peter/Projects/MatrixDepot.jl/test/runtests.jl:53
ERROR: Package MatrixDepot errored during testing

This same error is also showing up in the CI on #79.

Classification and regression

Hi!

It would be very nice to have access also to "training" matricies for Machine Learning, where we have an additional vector containing data. For example the data available on http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ is widely used and there is no convenient way for loading them at this point. It would be very nice to have that kid of feature!

Cheers,
Dominik

Search UFL Collection

I didn't find a way to filter through the UFL collection with this package. I put together a makeshift solution that others might find useful or improve upon. I downloaded the data file used by the web interface at http://yifanhu.net/GALLERY/GRAPHS/search.html and converted it to something that resembles a CSV file. Then, JuliaDB lets users filter matrices that have certain properties.

In case it's useful, the files are here:

https://gist.github.com/99ec73923425a82d6f07beee3ab63cde

I hope something like that can be added to this repository in the future.

Download of MatrixMarket on ARM fails

Download fails on ARM with Armbian 20.11.3 Focal with Linux 5.9.14-rockchip64 and a fresh install of Julia v1.5.3 and MatrixDepot v1.0.3

dowloading index file http://math.nist.gov/MatrixMarket/matrices.html
┌ Warning: exception during initialization: 'ProcessFailedException(Base.Process[Process(`sh -c 'curl "http://math.nist.gov/MatrixMarket/matrices.html" -Lso "/home/mdpetter/.julia/packages/MatrixDepot/1equL/data/mm_matrices.html"'`, ProcessExited(60))])'
└ @ MatrixDepot ~/.julia/packages/MatrixDepot/1equL/src/MatrixDepot.jl:144

The command curl "http://math.nist.gov/MatrixMarket/matrices.html" -Lso "/home/mdpetter/.julia/packages/MatrixDepot/1equL/data/mm_matrices.html" works as expected when executed in sh or bash.

Remove duplicate sources of repositories

There are currently 3 methods to access the Suite Sparse collection.

the obsolete URL of the University of Florida
the current URL of TAMU, using as index data the HTML files
the current URL of TAMU, using as index data a dedicated file in MAT format

Only the last option should be used by the software; corresponding configuration and code should be removed.

release 1.0.9

@JuliaRegistrator register

Request: Rosser's matrix

Rosser's matrix is a classic challenge problem for eigenvalue algorithms. Would be interesting to provide more general test matrices from a similar construction, see Appendix A of [1] and, e.g., David Gross's blog post on the topic. The most general case would use Sylvester's method for constructing orthogonal matrices [2] to mix up the matrices of interest; however, it may be simpler to forego staying in the integer field and simply apply a random arbitrary rotation matrix Q = qrfact!(randn(n,n))[:Q].

References:

Rosser, Lanczos, Hestenes and Karush, J. Res. Natl. Bur. Stand. 47 (1951), 291-297. archive.org
Sylvester, Phil. Mag. 34 (1867), 461. hathitrust

Remove the deprecated way of keeping user defined local functions

The extension of the package be user defined functions, which are stored in source files in a special directory
shall be disabled.
There is now the option to for user extensions in a user-owned package, which uses MatrixDepot.

Obsolete example in docs

In the online docs, under Usage, the example for mdinfo(),
the code block shows matrixdepot() rather than mdinfo().
(And there is no method with signature matrixdepot())

MatrixDepot v0.8.0

release 1.0.8

@JuliaRegistrator register

Ability to fetch and save selected sets of matrixes to a directory

This is an enhancement request (if such a capability doesn't already exist that I haven't discovered yet), to be able to specify a set of matrixes (say, all of the sparse matrixes from UF), and then to fetch them all from the repository and save them as files in a directory, instead of creating the matrix.
I would then be able to (with a PR that I added to MatrixMarket JuliaSparse/MatrixMarket.jl#13) get the sparse matrix in coordinate format (I,J,V) to use in benchmarking different sparse matrix construction code (and operations).
See JuliaLang/julia#13400.
I have manually downloaded a few of the sparse matrixes from UF, but there are over 500 of them, so I'd like to be able to automate this task, and MatrixDepot looked like a very good starting point!
(If this capability is not already present, I'd be happy to work on making a PR to do this)

Pascal triangle generated only till upto small size

matrixdepot("pascal", 50) will generate an OverFlowError()

One possible fix is to change the convert function at the end of the binomial function in intfuncs.jl in Base. Instead of converting to an Int to should convert to Float

Use Documenter

Switch documentation from readthedocs.org to Documenter.jl.

Setup docs folder and make.jl
Remove redundancies form documentation
Make use of Documenter-features as far as appropriate

julialinearalgebra / matrixdepot.jl Goto Github PK

matrixdepot.jl's People

Contributors

Stargazers

Watchers

Forkers

matrixdepot.jl's Issues

On Julia 0.4

Basic Usage

Recommend Projects

Recommend Topics

Recommend Org