Add support for optional 1-bit normalization instead of Windsorization

add:
On Ubuntu 16.04 u need to add this at the end of my.cnf file:

[mysqld]
sql_mode="TRADITIONAL,NO_AUTO_CREATE_USER"

Tests don't cover all steps, and don't include testing of the results (values) !

Hi Esteban,

Yes, this line is the culprit:
https://github.com/ROBelgium/MSNoise/blob/master/msnoise/s05compute_mwcs.py#L109

Should be replaced with a dynamic extension coming from the database. 1.3.2 it will be

Thomas

Le 7/05/2015 04:14, Esteban Chaves a écrit :

Hi All;

Just wondering if we came up with a solution for this:

IOError: [Errno 2] No such file or directory: 'STACKS/01/REF/ZZ/YZ_HRIU_YZ_JUDI.MSEED'

1. Running: compute_mwcs
2. If one has SAC data and the output format is SAC not miniseed.
   Thomas, I did modified this part in the previous version, but for this one
   it seems like isn’t going through the if loop...? Any thoughts?

Cheers;

-Esteban

Database setup utility (and possibly other gui apps) fails with a
segmentation fault when running under Ubuntu 12.04. Tracking this down
showed that it stems from exceptions occurring internally within
traitsui. I.e. this is not a msnoise issue per se.

Replacing the system traisui (4.0) with upstream (4.5) resolves this
issue.

setup.py (line 17) should be updated so that the traitsui dependency is
versioned. I cannot determine if this behaviour is present with versions
4.1 to 4.4. So:

    traitsui>=4.5   will definitely work
    traitsui>4.0    might work

This issue presents a tiny change in the add_corr methods, which pass the stations in order to later be able to add the distance in the SAC headers:

 # Add stations s1, s2 for keep_all :
add_corr(s1, s2, db, station1.replace('.', '_'), station2.replace('.', '_'), 
               filterid, thisdate, thistime, min30 / fe, components, corr, fe)

 # Add stations s1, s2 for keep_days :
add_corr(s1, s2,  db, station1.replace('.', '_'), station2.replace('.', '_'), filterid,
               thisdate, thistime, min30 / fe, components, corr, fe, day=True, ncorr=ncorr)

in database tools, edit:

def add_corr(s1, s2, db, station1, station2, filterid, date, time, duration, components, CF, sampling_rate, day=False, ncorr=0):
    ...
     if sac:
          export_sac(s1, s2, db, path, pair, components, filterid, CF/ncorr, ncorr)

the export_sac too:

def export_sac(s1, s2, db, filename, pair, components, filterid, corr, ncorr=0, sac_format=None, maxlag=None, cc_sampling_rate=None):
    ....
    dist, azim, bazim = gps2DistAzimuth(s1.Y, s1.X, s2.Y, s2.X)
    ....
    tr.SetHvalue('DIST', dist)
    tr.SetHvalue('AZ', azim)
    tr.SetHvalue('BAZ', bazim)

in msnoise-admin -> station page, the decimals get rounded off to 2 decimal places.

ref flask-admin/flask-admin#1141 (comment)

scan_archive changes files in data_availability table to 'M' even if they haven't changed (if they fall within the time range identified by 'mtime'.) This is apparently related to the date precision when Python reads the metadata and compares to the starttime and endtime columns in MySQL.

The IO of MWCS and dt/t are done directly from the scripts, not via the API !

Might be easier to understand for new users ?

• Proof the documentation
• Test setup on a fresh install (Python + NumPy only)
• Run the suite on PdF Undervolc

The title says it all. Preprocessing and processing functions need to be generalized and moved to obspy. This shouldn't be too hard, but requires testing!

Bug in scan_archive : ls command is not available under Windows, replace "ls","-1" by "dir","/b"

write documentation in rst and link it to the current development. Once done, either build the statics pages for WP on msnoise.org, or host the doc on github pages.

Because of #21, the min&max time lags can't be defined inline anymore, so...

Step 1:
implement the "relative time lag window" vs interstation distance (cf OVPF install)

Step 2:
Modifiy the configuration/Configurator to enable/disable this feature.

examples:

if enabled:
config:velocity: 1 km/s
window:30s
sides:both

else:
minlag: 10 s
maxlag: 40s
sides: both

remaining parameters should be shared:
minCoh = 0.5
maxErr = 0.1
maxDt = 0.5

bug arises with bokeh>=0.10 at least, needs to be checked !

Improvement:

• What to do if already exists ?
• Provide a Deleter ?
• Etc

find a way to know which one is the right one or default to the new find.exe command?

this should never happen:

2015-04-02 15:22:48 [INFO] *** Starting: Compute CC ***
2015-04-02 15:22:48 [INFO] Will compute ZZ
2015-04-02 15:22:48 [INFO] New CC Job: 2006-01-25 (1 pairs with 2 stations)
2015-04-02 15:22:55 [INFO] Pre-Whitening Traces
2015-04-02 15:22:56 [INFO] Processing CC
2015-04-02 15:22:56 [INFO] Job Finished. It took 7.45 seconds
2015-04-02 15:22:56 [INFO] New CC Job: 2006-01-26 (1 pairs with 2 stations)
2015-04-02 15:23:03 [INFO] Pre-Whitening Traces
2015-04-02 15:23:03 [INFO] Processing CC
2015-04-02 15:23:03 [INFO] Job Finished. It took 7.37 seconds
2015-04-02 15:23:03 [INFO] New CC Job: 2006-01-27 (1 pairs with 2 stations)
2015-04-02 15:23:10 [INFO] Pre-Whitening Traces
2015-04-02 15:23:11 [INFO] Processing CC
2015-04-02 15:23:11 [INFO] Job Finished. It took 7.30 seconds
2015-04-02 15:23:11 [INFO] New CC Job: 2006-01-28 (1 pairs with 2 stations)
Traceback (most recent call last):
  File "/home/silvio/anaconda/bin/msnoise", line 9, in <module>
    load_entry_point('msnoise==1.3.1', 'console_scripts', 'msnoise')()
  File "/home/silvio/anaconda/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 393, in run
    cli(obj={})
  File "/home/silvio/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/home/silvio/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/home/silvio/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/silvio/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/silvio/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/silvio/anaconda/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 174, in compute_cc
    main()
  File "/home/silvio/anaconda/lib/python2.7/site-packages/msnoise/s03compute_cc.py", line 271, in main
    basetime, tramef_Z = preprocess(db, stations, comps, goal_day, params, tramef_Z)
  File "/home/silvio/anaconda/lib/python2.7/site-packages/msnoise/s03compute_cc.py", line 130, in preprocess
    stream[gap[0]] = stream[gap[0]].__add__(stream[gap[1]], method=0, fill_value="interpolate")
  File "/home/silvio/anaconda/lib/python2.7/site-packages/obspy-0.10.1-py2.7-linux-x86_64.egg/obspy/core/trace.py", line 681, in __add__
    raise TypeError("Sampling rate differs")
TypeError: Sampling rate differs

still some empty parts in the documentation, that should be fixed for next release !

Replace the current way of providing options to scripts by argparse. Example for stack.py:
python 04.stack.py --interval=5
will search for CC jobs being done in the last 5 days, instead of the last day (default).

Would it make sense NOT to export the stacks... but only use the 1day stack and MWCS step would "dynamically" compute the moving averages...

Currently MWCS step opens N stack files per day, (N mov_stack values). With the new approach, it would read N times a different number of files (1, 2, 30 ?, the mov_stack value)...

• the first time it would make sense to load the whole "matrix" and process from there
• the next times, just load the necessary date span

Or always read the whole matrix and compute the sliding window on the fly ?

OK, one last before MSNoise 1.3 goes out, add the support

--show : bool (default True)
--outfile: str (default None): if "?.png" is provided, ? is replaced by: "pair-components-filterid-mov_stack" automatically.

[41] ∰ msnoise stack -r -m -i 100
Lets STACK !
/Users/ech/github/obspy/obspy/init.py:150: ObsPyDeprecationWarning: Module 'obspy.sac' is deprecated and will stop working with the next ObsPy version. Please import module 'obspy.io.sac' instead.
ObsPyDeprecationWarning)
/Users/ech/github/obspy/obspy/core/util/deprecation_helpers.py:55: ObsPyDeprecationWarning: Function 'obspy.core.util.gps2DistAzimuth' is deprecated and will stop working with the next ObsPy version. Please use 'obspy.geodetics.gps2dist_azimuth' instead.
ObsPyDeprecationWarning)
2015-10-14 12:28:11 [DEBUG] Starting the ref stack
2015-10-14 12:28:11 [DEBUG] Processing YZ_INDI:YZ_LAFE-ZZ-1
2015-10-14 12:28:11 [DEBUG] Found 057 updated days
2015-10-14 12:28:11 [DEBUG] New Data for YZ_INDI:YZ_LAFE-ZZ-1
2015-10-14 12:28:11 [DEBUG] Processing YZ_INDI:YZ_SAJU-ZZ-1
2015-10-14 12:28:11 [DEBUG] Found 057 updated days
2015-10-14 12:28:11 [DEBUG] New Data for YZ_INDI:YZ_SAJU-ZZ-1
2015-10-14 12:28:11 [DEBUG] Processing YZ_LAFE:YZ_SAJU-ZZ-1
2015-10-14 12:28:11 [DEBUG] Found 057 updated days
2015-10-14 12:28:11 [DEBUG] New Data for YZ_LAFE:YZ_SAJU-ZZ-1
2015-10-14 12:28:12 [DEBUG] Starting the step stack
Traceback (most recent call last):
File "/Users/ech/anaconda/bin/msnoise", line 9, in
load_entry_point('msnoise==1.3', 'console_scripts', 'msnoise')()
File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/scripts/msnoise.py", line 392, in run
cli(obj={})
File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 610, in call
return self.main(_args, *_kwargs)
File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 590, in main
rv = self.invoke(ctx)
File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 936, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 782, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 416, in invoke
return callback(_args, **kwargs)
File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/scripts/msnoise.py", line 191, in stack
main('step', interval)
File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/s04stack.py", line 117, in main
mov_stacks = [int(mi) for mi in mov_stack.split(',')]
ValueError: invalid literal for int() with base 10: ''

Case:

1. One process starts the MWCS (DTT) jobs, gets STA1-STA2, marks the corresponding jobs as "In Progress" in the DB
2. Another process starts before all jobs are marked "In Progress", and gets the same STA1-STA2 job list.

Result:
MWCS is calculated two times (or more, if N process are used)...

First time trying to update with new data (had already done an initialization) w/ sqlite database,
msnoise scan_archive

scan_archive
2016-10-11 13:48:26 [INFO] *** Starting: Scan Archive ***
2016-10-11 13:48:26 [INFO] Will work on 1 threads

% ADDED THIS PRINT STATEMENT

b'data/NC/MCB/MCB.NC..HHZ.2016.281\ndata/NC/MCB/MCB.NC..HHZ.2016.282\ndata/NC/MCB/MCB.NC..HHZ.2016.283\ndata/NC/MCB/MCB.NC..HHZ.2016.284\n'

%

Traceback (most recent call last):
File "/usr/local/bin/msnoise", line 9, in
load_entry_point('msnoise==1.4.1', 'console_scripts', 'msnoise')()
File "/usr/local/lib/python3.5/dist-packages/msnoise-1.4.1-py3.5.egg/msnoise/scripts/msnoise.py", line 681, in run
cli(obj={})
File "/home/ashton/.local/lib/python3.5/site-packages/click/core.py", line 716, in call
return self.main(_args, *_kwargs)
File "/home/ashton/.local/lib/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/home/ashton/.local/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ashton/.local/lib/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/home/ashton/.local/lib/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(_args, *_kwargs)
File "/home/ashton/.local/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, *_kwargs)
File "/usr/local/lib/python3.5/dist-packages/msnoise-1.4.1-py3.5.egg/msnoise/scripts/msnoise.py", line 325, in scan_archive
main(init, threads=ctx.obj['MSNOISE_threads'])
File "/usr/local/lib/python3.5/dist-packages/msnoise-1.4.1-py3.5.egg/msnoise/s01scan_archive.py", line 182, in main
files = sorted(stdout.replace('\r', '').split('\n'))
TypeError: a bytes-like object is required, not 'str'

Looks like the file list is returned as bytes type object and needs to be recast as a string before being converted into a list. From what I remember this was a change in subprocess (?).

So in, s01scan_archive.py
Ln 181, change;

                if len(stdout) != 0:
                    files = sorted(stdout.replace('\r', '').split('\n'))

to;

                if len(stdout) != 0:
                    if type(stdout) is not str:
                        stdout = stdout.decode("utf-8")
                        files=[]
                        [files.append(file.split('/')[-1]) for file in stdout.split('\n')]
                        files=sorted(files) 
                    else:
                        files = sorted(stdout.replace('\r', '').split('\n'))

This was using the BUD directory structure, and since the find mtime returned the relative path into the BUD data directory (e.g. 'data/NETWORK/STA/file) it was necessary to parse just the file name out so that we didnt end up with a data/NETWORK/STA/data/NETWORK/STA/file, when the folder was later prepended.

The fact that the new job procedure inserts and doens't update existing fields yields an error for the following procedures, expecting unique day-pair-jobtype keys...

Adding threading support to the "stack" step would be very usefull upon initialisation, at least.

seems to fail

Hi

I'm now trying to create new station in the configuration page in web browser, but there's a 'builtins.TypeError' bug.

Bug says:
"TypeError: init() missing 8 required positional arguments: 'net', 'sta', 'X', 'Y', 'altitude', 'coordinates', 'instrument', and 'used' "

Is this about instantiating the class?
How can I fix the problem?

I'm a beginner in both python and msnoise, and this bug really worry me.
I'm useing Centos6.5 and python3.5(Anaconda)

Shuye

MSNoise should be importable like any other python package : installable using package managers, pip, easy_install, eggs or whatever.

This requires:

• Moving all codes in main() functions, calling main() from if name == 'main'.
• Propagating the db.ini path location to all codes so they find the right location
• really using the output_directory (currently only used for CROSS_CORRELATIONS when keep_all is True
• Adding information about that in the documentation
• Provide examples

This will allow MSNoise to be installed by sysadmins, and used by users, with no output placed in the folder of the package.

werkzeug should be 0.10.4 in order to avoid

an operation was attempted on something that is not a socket

when running msnoise admin

not sure all dependencies are properly identified (certainly with msnoise-admin) !

logging needs:

• logging level defined by verbosity level of top level msnoise command (-v -vv -vvv -vvvv)
• logging to console only, user can redirect to file if needed
• logging multiprocess (is that doable)

Three methods do about the same thing: save a CCF to the disk... To be investigated/cleaned !

Review the code to respect PEP8

• Admin interface via web interface (e.g. #25)
• Instrument response removal (e.g. #16)
• Support for Inventory/StationXML/DatalessSeed
• Support for Plugins
• Preprocessing in ObsPy (e.g. #29)
• Threading Stacks (e.g. #12)
• Data Availabiltiy Path as relative to the archive configuration value

instead of populating the station table with a complete scan of the archive, provide a modern parser.

In order to simplify the installation process, AND because the database usage is somehow limited, it doesn't really make sense to require a full MySQL server to run MSNoise. Switching to SQLAlchemy will allow to use sqlite, e.g.

Setting dtt_lag to dynamic and using default station locations (0,0,0) does not issue a flag or warning for the user.

[ ] - only load data for the needed dates +- max(mov_stacks) should be enough, instead of loading the whole dataset in a huge matrix, and only using a few lines from it.

https://github.com/ROBelgium/MSNoise/blob/master/msnoise/s03compute_cc.py#L657 this is not clear, it's actually depending on rms_threshold ...

If the obspy's resample is close to samplerate + not much slower, it should be prefered, as this would definitively ease the installation of MSNoise!

Because of #21, it's no longer possible to edit the data_structure.py. I need to find a way to let users define their own structure AND parser for s002 and s01 !

There's a problem when exporting/creating sac files during Cross-correlation:

msnoise compute_cc
/Users/ech/github/obspy/obspy/init.py:150: ObsPyDeprecationWarning: Module 'obspy.sac' is deprecated and will stop working with the next ObsPy version. Please import module 'obspy.io.sac' instead.
? ObsPyDeprecationWarning)
/Users/ech/github/obspy/obspy/core/util/deprecation_helpers.py:55: ObsPyDeprecationWarning: Function 'obspy.core.util.gps2DistAzimuth' is deprecated and will stop working with the next ObsPy version. Please use 'obspy.geodetics.gps2dist_azimuth' instead.
? ObsPyDeprecationWarning)
2015-10-13 19:48:12 [INFO] *** Starting: Compute CC ***
2015-10-13 19:48:12 [INFO] Will compute ZZ
2015-10-13 19:48:12 [INFO] New CC Job: 2012-01-01 (1 pairs with 2 stations)
2015-10-13 19:48:16 [INFO] Pre-Whitening Traces
2015-10-13 19:48:16 [INFO] Processing CC
Traceback (most recent call last):
? File "/Users/ech/anaconda/bin/msnoise", line 9, in
? ? load_entry_point('msnoise==1.3', 'console_scripts', 'msnoise')()
? File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/scripts/msnoise.py", line 392, in run
? ? cli(obj={})
? File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 610, in call
? ? return self.main(_args, *_kwargs)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 590, in main
? ? rv = self.invoke(ctx)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 936, in invoke
? ? return _process_result(sub_ctx.command.invoke(sub_ctx))
? File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 782, in invoke
? ? return ctx.invoke(self.callback, *_ctx.params)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/click-3.3-py2.7.egg/click/core.py", line 416, in invoke
? ? return callback(_args, **kwargs)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/scripts/msnoise.py", line 173, in compute_cc
? ? main()
? File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/s03compute_cc.py", line 363, in main
? ? thisdate, thistime, params.min30 / params.goal_sampling_rate, 'ZZ', daycorr, params.goal_sampling_rate, day=True, ncorr=ndaycorr)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/api.py", line 851, in add_corr
? ? ncorr)
? File "/Users/ech/anaconda/lib/python2.7/site-packages/msnoise-1.3-py2.7.egg/msnoise/api.py", line 891, in export_sac
? ? tr = SacIO(filename)
? File "/Users/ech/github/obspy/obspy/io/sac/sacio.py", line 298, in init
? ? self.read_sac_file(filen)
? File "/Users/ech/github/obspy/obspy/io/sac/sacio.py", line 608, in read_sac_file
? ? self.hf = from_buffer(fh.read(4 * 70), dtype=native_str('<f4'))
AttributeError: 'str' object has no attribute 'read'

• Remove intructions for mysql-python, no longer needed (thanks to pymysql)
• Correct typos identified at NTU workshop
• Add doc for web admin
• Add doc for plugins
• Complete the "TODO" parts in the doc (moslty in how-to's)
• Comments from Z. Spica (phpmyadmin install) - might be not needed with webmin, check
• Update readme !
• Document Phase Weighted Stack (configuration and effect)
• Add example of the dt/t plot
• Instrument Correction

add an header line to the MWCS output, like for the DTT output