nmayorov / ufcnn Goto Github PK
View Code? Open in Web Editor NEWImplementation of Undecimated Fully Convolutional Neural Network for time series modeling
Implementation of Undecimated Fully Convolutional Neural Network for time series modeling
In general it's advised to scale down the initial weights of neurons by the sqrt(k * n), where n is the number of inputs and k could could be 1 or 2 depending on exact approach, see http://cs231n.github.io/neural-networks-2/ and https://github.com/fchollet/keras/blob/master/keras/initializations.py.
https://github.com/nmayorov/ufcnn/blob/master/ufcnn/datasets.py#L19
I think it's better to rename n_stamps to n_samples or n_timesteps.
In ufcnn.py +line271
C_biases = init_conv_weights([n_outputs], random_seed)
Is this right ? Seems to be:
C_biases = init_conv_bias([n_outputs], random_seed)
don't whether will cause errors.
dilation = 1 for w, b in zip(H_weights, H_biases): x = tf.nn.relu(conv(x, w, b, filter_length, dilation)) H_outputs.append(x) dilation *= 2
Doesn't this code make the first G conv layer G4 rather than G3? Since H3 already has dilation of 2**(3-1), the last line would make G3 has dilation of 2**(4-1). Although this doesn't affect the performance very much, since it's just one more resolution level.
I found an issue on my implementation of your UFCNN that caused the gradients to be 0 for all but the output layer. I think this is related to Python handling the stddev as an integer math and rounding it to 0 most of the time:
def init_conv_weights(shape, seed):
n = np.prod(shape[:-1])
initial = tf.random_normal(shape, stddev=(2 / n)**0.5, seed=seed)
return tf.Variable(initial)
Changing the initial line from 2 to 2.0 as in this:
def init_conv_weights(shape, seed):
n = np.prod(shape[:-1])
initial = tf.random_normal(shape, stddev=(2.0 / n)**0.5, seed=seed)
return tf.Variable(initial)
There are probably a few ways to force the 2/n to be treated as floats, but this worked for me. The network now converged.
Thanks for the great work putting this together.
Hi @nmayorov,
are you still working on this? construct_ufcnn give me this error if n_levels is different from 1.
I use python 2.7 and tensorflow: 1.1.0.
Thanks!
construct_ufcnn(n_inputs=1, n_outputs=2, n_levels=1, n_filters=10, filter_length=5, random_seed=0)
(<tf.Tensor 'Placeholder_8:0' shape=(?, ?, 1) dtype=float32>, <tf.Tensor 'Squeeze_4:0' shape=(?, ?, 2) dtype=float32>, [<tf.Variable 'Variable_60:0' shape=(1, 5, 1, 10) dtype=float32_ref>, <tf.Variable 'Variable_62:0' shape=(1, 5, 10, 10) dtype=float32_ref>, <tf.Variable 'Variable_64:0' shape=(1, 5, 10, 2) dtype=float32_ref>], [<tf.Variable 'Variable_61:0' shape=(10,) dtype=float32_ref>, <tf.Variable 'Variable_63:0' shape=(10,) dtype=float32_ref>, <tf.Variable 'Variable_65:0' shape=(2,) dtype=float32_ref>])
construct_ufcnn(n_inputs=1, n_outputs=2, n_levels=2, n_filters=10, filter_length=5, random_seed=0)
Traceback (most recent call last):
File "", line 1, in
File "ufcnn/ufcnn.py", line 264, in construct_ufcnn
x = tf.concat(3, [x_prev, x])
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1029, in concat
dtype=dtypes.int32).get_shape(
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 639, in convert_to_tensor
as_ref=False)
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 704, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 113, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 102, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 370, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/Users/pastorea/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 302, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).name))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.