google / prettytensor Goto Github PK
View Code? Open in Web Editor NEWPretty Tensor: Fluent Networks in TensorFlow
Pretty Tensor: Fluent Networks in TensorFlow
I'm not sure how regularization works in PrettyTensor. I've done the following (simplified code):
x_pretty = pt.wrap(x)
with pt.defaults_scope(activation_fn=tf.nn.relu, l2loss=0.01):
y_pred = x_pretty. \
conv2d(kernel=8, depth=32, stride=4). \
flatten(). \
fully_connected(size=512). \
fully_connected(size=10, activation_fn=None)
loss_regression = y_pred.l2_regression(target=y_true)
regularizers = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss_regularizer = tf.reduce_sum(regularizers)
loss = loss_regression + 0.01 * loss_regularizer
optimizer = tf.train.RMSPropOptimizer(learning_rate=1e-3).minimize(loss)
But I don't think this is the correct way of doing it in PrettyTensor.
Could you modify this code to show how it should be done? And perhaps explain why?
Thanks.
Hi,
I just wanted to try the introductory example and got the following error:
Traceback (most recent call last):
File "testpt.py", line 15, in <module>
softmax_classifier(CLASSES, labels=label_tensor, name="sm"))
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 2019, in method
return func(input_layer, *args, **self.fill_kwargs(input_layer, kwargs))
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_loss_methods.py", line 401, in softmax_classifier
init=weight_init, bias_init=bias_init)
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1980, in method
result = func(non_seq_layer, *args, **kwargs)
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_methods.py", line 333, in __call__
dt=dtype)
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1694, in variable
collections=variable_collections)
File "/home/code/ml/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 334, in get_variable
collections=collections)
File "/home/code/ml/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 257, in get_variable
collections=collections, caching_device=caching_device)
File "/home/code/ml/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 118, in get_variable
name, "".join(traceback.format_list(tb))))
ValueError: Variable weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1694, in variable
collections=variable_collections)
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_methods.py", line 333, in __call__
dt=dtype)
File "/home/.local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1980, in method
result = func(non_seq_layer, *args, **kwargs)
The code that I try to run is as follows
import tensorflow as tf
import prettytensor as pt
import numpy as np
BATCH_SIZE = 100
DATA_SIZE = 28*28
CLASSES = 10
input_tensor = tf.placeholder(np.float32, shape=(BATCH_SIZE, DATA_SIZE))
label_tensor = tf.placeholder(np.float32, shape=(BATCH_SIZE, CLASSES))
pretty_input = pt.wrap(input_tensor)
softmax, loss = (pretty_input.
fully_connected(100, name="fc1").
softmax_classifier(CLASSES, labels=label_tensor, name="sm"))
It seems that variable tensors are not scoped properly. Am I doing something wrong?
Kind regards,
Tobias
I believe that there is an error when using phase
in the default_scope
coupled with batch_normalize=True
.
Basically it looks like this:
def encoder(self, inputs, latent_size, activ=tf.nn.elu, phase=pt.Phase.train):
with pt.defaults_scope(activation_fn=activ,
batch_normalize=True,
learned_moments_update_rate=0.0003,
variance_epsilon=0.001,
scale_after_normalization=True,
phase=phase):
params = (pt.wrap(inputs).
reshape([-1, self.input_shape[0], self.input_shape[1], 1]).
conv2d(5, 32, stride=2).
conv2d(5, 64, stride=2).
conv2d(5, 128, edges='VALID').
flatten().
fully_connected(self.latent_size * 2, activation_fn=None)).tensor
Full code here: https://github.com/jramapuram/CVAE/blob/master/cvae.py
If I remove phase=phase
within the scope assignment my model produces the following:
However, when setting the phase appropriately I get the following:
This is trained for the same number of iterations using the same model.
Hi,
I recently upgraded to tensorflow version 0.7.1 and immediately I got this error when importing prettytensor.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/prettytensor/__init__.py", line 22, in <module>
from prettytensor import funcs
File "/usr/local/lib/python2.7/dist-packages/prettytensor/funcs.py", line 19, in <module>
from prettytensor.pretty_tensor_image_methods import *
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_image_methods.py", line 17, in <module>
from prettytensor import pretty_tensor_class as prettytensor
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_class.py", line 1829, in <module>
tf.ops.register_tensor_conversion_function(
AttributeError: 'module' object has no attribute 'ops'
Please let me know any fix for it. Thanks in advance!
Hi,
I'm really happy with prettytensor's templates and unbounded variable, which makes it different from other wrappers built upon tensorflow, so i'm using it a lot in my project. There are several questions I have now that I could not figure out by reading through the source, so I'm trying to ask here.
Hi,
Thanks for a great library!
Is there a way I could send a single input to multiple towers without splitting/dividing the input and not have them join at the end?
Thanks!
This may be outside the scope of the intended use of this project, but is it possible to specify which GPU card to run on from prettytensor? I have a long running training (non-tensorflow) on GPU:0, but the default for PT seems to be to run (only) on GPU:0.
what is difference between pretty tensor and tflearn?
what are the pros and cons of pretty tensor?
It would be great if prettytensor also had the He initialization
http://arxiv.org/pdf/1502.01852v1.pdf
each time tried to use pretty tensor I faced this error
with pt.defaults_scope(activation_fn=tf.nn.relu):
y_pred, loss = x_pretty.\
conv2d(kernel=5, depth=16, name='layer_conv1').\
max_pool(kernel=2, stride=2).\
conv2d(kernel=5, depth=36, name='layer_conv2').\
max_pool(kernel=2, stride=2).\
flatten().\
fully_connected(size=128, name='layer_fc1').\
softmax_classifier(num_classes=num_classes, labels=y_true)
Error here:
`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-22-ab75212e54e0> in <module>()
1 with pt.defaults_scope(activation_fn=tf.nn.relu):
----> 2 y_pred, loss = x_pretty. conv2d(kernel=5, depth=16, name='layer_conv1'). max_pool(kernel=2, stride=2). conv2d(kernel=5, depth=36, name='layer_conv2'). max_pool(kernel=2, stride=2). flatten(). fully_connected(size=128, name='layer_fc1'). softmax_classifier(num_classes=num_classes, labels=y_true)
/home/mahmoodsalah/anaconda3/envs/CarND-TensorFlow-Lab/lib/python3.5/site-packages/prettytensor/pretty_tensor_class.py in method(*original_args, **kwargs)
1968 result = self.create_deferred(func, non_seq_layer, args, kwargs, name)
1969 else:
-> 1970 result = func(non_seq_layer, *args, **kwargs)
1971 return input_layer._method_complete(result)
1972 # Exit with because Template handles that.
/home/mahmoodsalah/anaconda3/envs/CarND-TensorFlow-Lab/lib/python3.5/site-packages/prettytensor/pretty_tensor_image_methods.py in __call__(self, input_layer, kernel, depth, activation_fn, stride, l2loss, weights, bias, edges, batch_normalize, phase, parameter_modifier, name)
209 params = parameter_modifier(
210 'weights',
--> 211 self.variable('weights', size, weights, dt=dtype),
212 phase)
213 y = tf.nn.conv2d(input_layer, params, stride, edges)
/home/mahmoodsalah/anaconda3/envs/CarND-TensorFlow-Lab/lib/python3.5/site-packages/prettytensor/pretty_tensor_class.py in variable(self, var_name, shape, init, dt, train)
1660 train = _defaults.get('trainable_variables', True)
1661 variable_collections = _defaults.get('variable_collections', ())
-> 1662 if tf.GraphKeys.GLOBAL_VARIABLES not in variable_collections:
1663 variable_collections = list(variable_collections) + [
1664 tf.GraphKeys.GLOBAL_VARIABLES]
AttributeError: type object 'GraphKeys' has no attribute 'GLOBAL_VARIABLES'`
Thanks for the making TF simple :)
I am just curious as to how I can pre-initialize the weights before performing any operations.
Thanks in advance!
Hi,
Can anyone tell me how to implement residual blocks or skip-connection with prettytensor? An idea or a code example will be helpful.
I'm trying to run the StackGAN code (https://github.com/hanzhanggit/StackGAN) using TF1.0 and prettytensor (0.7.2) and I receive the following error
"TypeError: Expected int32, got <prettytensor.pretty_tensor_class.Layer object at 0x7f396c1ea590> of type 'Layer' instead." when I run the (sh demo/birds_demo.sh).
A friend who ran the same in TF 0.12 had no problems. Is this because of TF1.0? Do you have any idea what's the problem or what can be done to overcome it?
Thank you
Hi,
I was wondering if there was support for using pre-trained weights in prettytensor. I read the documentation but couldn't find anything about it. Maybe I glossed over some details?
Thanks!
version = '0.7.1'
import prettytensor as pt
File "lib\prettytensor_init_.py", line 25, in
from prettytensor import funcs
File "lib\prettytensor\funcs.py", line 25, in
from prettytensor.pretty_tensor_image_methods import *
File "lib\prettytensor\pretty_tensor_image_methods.py", line 135, in
class conv2d(prettytensor.VarStoreMethod):
File "lib\prettytensor\pretty_tensor_image_methods.py", line 145, in conv2d
bias=tf.zeros_initializer(),
TypeError: zeros_initializer() missing 1 required positional argument: 'shape'
Installing PrettyTensor seems to have broken my GPU installation of TensorFlow. After installing PrettyTensor it would use the CPU version of TensorFlow.
I'm using a conda env named tf-gpu
for the GPU version and another env named tf
for the CPU version.
Here's the output of the pip install:
(tf-gpu) magnus@torpedo:~/development/TensorFlow-Tutorials$ pip install prettytensor
Collecting prettytensor
Downloading prettytensor-0.7.1-py3-none-any.whl (273kB)
100% |████████████████████████████████| 276kB 1.1MB/s
Collecting enum34>=1.0.0 (from prettytensor)
Downloading enum34-1.1.6-py3-none-any.whl
Requirement already satisfied: six>=1.10.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from prettytensor)
Collecting tensorflow>=0.12.0rc0 (from prettytensor)
Using cached tensorflow-0.12.0rc1-cp35-cp35m-manylinux1_x86_64.whl
Requirement already satisfied: numpy>=1.11.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow>=0.12.0rc0->prettytensor)
Requirement already satisfied: protobuf==3.1.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow>=0.12.0rc0->prettytensor)
Requirement already satisfied: wheel>=0.26 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow>=0.12.0rc0->prettytensor)
Requirement already satisfied: setuptools in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg (from protobuf==3.1.0->tensorflow>=0.12.0rc0->prettytensor)
Installing collected packages: enum34, tensorflow, prettytensor
Successfully installed enum34-1.1.6 prettytensor-0.7.1 tensorflow-0.12.0rc1
The culprit seems to be this:
Collecting tensorflow>=0.12.0rc0 (from prettytensor)
Using cached tensorflow-0.12.0rc1-cp35-cp35m-manylinux1_x86_64.whl
I tried pip uninstall tensorflow
and then pip install tensorflow_gpu-0.12.0rc0-cp35-cp35m-linux_x86_64.whl
but it doesn't work because it says it is already installed:
(tf-gpu) magnus@torpedo:~/Downloads$ pip install tensorflow_gpu-0.12.0rc0-cp35-cp35m-linux_x86_64.whl
Requirement already satisfied: tensorflow-gpu==0.12.0rc0 from file:///home/magnus/Downloads/tensorflow_gpu-0.12.0rc0-cp35-cp35m-linux_x86_64.whl in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages
Requirement already satisfied: six>=1.10.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow-gpu==0.12.0rc0)
Requirement already satisfied: protobuf==3.1.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow-gpu==0.12.0rc0)
Requirement already satisfied: wheel>=0.26 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow-gpu==0.12.0rc0)
Requirement already satisfied: numpy>=1.11.0 in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages (from tensorflow-gpu==0.12.0rc0)
Requirement already satisfied: setuptools in /home/magnus/anaconda3/envs/tf-gpu/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg (from protobuf==3.1.0->tensorflow-gpu==0.12.0rc0)
How do I fix this, please?
I have tensorflow 10.0 installed but installing prettytensor throws the following error:
Could not find any downloads that satisfy the requirement tensorflow>=0.9.0rc0
I hope it's alright that I ask this question here. I don't think it would get answered on StackOverflow and this forum is not very busy.
I define the following Convolutional Neural Network for CIFAR-10 using Pretty Tensor:
with pt.defaults_scope(activation_fn=tf.nn.relu):
y_pred, loss = x_pretty.\
conv2d(kernel=5, depth=64, name='layer_conv1').\
max_pool(kernel=2, stride=2).\
conv2d(kernel=5, depth=64, name='layer_conv2').\
max_pool(kernel=2, stride=2).\
flatten().\
fully_connected(size=256, name='layer_fc1').\
fully_connected(size=128, name='layer_fc2').\
softmax_classifier(class_count=10, labels=y_true)
I want to extract the images that are output from the convolutional layers and plot them, not in TensorBoard, but using my own plotting.
Someone on StackOverflow gave the following code for printing the names of all the nodes in the graph:
foo = [node.name for node in tf.get_default_graph().as_graph_def().node]
print(foo)
But the list is really long and it's not clear to me which node-name represents the output of e.g. layer_conv1.
Is there an easy way of getting the images that are output from these layers? How about the tensors that are output from the fully-connected layers?
Thanks!
It seems that in pretty_tensor_methods.py
a deprecated tensorflow function is tried to register
prettytensor.Register(tf.complex_abs)
Any input on this?
Here, optimizer is not in arguments of apply_optimizer
Is it possible to scale the initial random values for the weights in PrettyTensor?
I have a network of a few convolutional layers followed by a few fully-connected layers. They all use relu-activations except for the last layer which is just linear output. I would like the initial output values of the network to be random and close to zero. I think the best way would be to init the random weights in the output-layer to be close to zero.
I see that there is a weights
parameter to the fully_connected()
class but it is not clear to me how to use it.
Could you give an example in this code? Thanks.
with pt.defaults_scope(activation_fn=tf.nn.relu):
self.q_values = x_pretty.\
conv2d(kernel=8, depth=16, stride=4, name='layer_conv1').\
conv2d(kernel=4, depth=32, stride=2, name='layer_conv2').\
flatten().\
fully_connected(size=256, name='layer_fc1').\
fully_connected(size=num_actions, name='layer_fc2', activation_fn=None)
Could you please add lstm rnn example for Titanic dataset
I need to make Deconvolution Network Layer.
Now I'm working with PT, how can I add deconvolution layer.
Please make some conv2d_transpose() API :)
Hello Everyone,
I'm currently in the process of learning how to use TensorFlow (TF) and wanted to use PrettyTensor (PT) to make the graph construction much easier. The idea of PT is great! - but the documentation is short and confusing so it is difficult to learn.
Here's a few comments I have so far. I hope the comments may help you guys understand what new users find confusing so you can improve it. I might add to the list as I get further with PT.
__version__
variable.I haven't gotten around to sequence-construction, towers, etc. in PrettyTensor, so I can't really comment on that at this point. However, from having briefly looked at the documentation it seems just as confusing as the rest.
I think it would be well worth the effort to significantly improve the documentation. Perhaps have someone else in Google go over it so it's written with a fresh set of eyes. This may take someone a week or two to complete. But this is a tiny cost compared to the combined time wasted by thousands of new users who are trying to figure out how to do even simple things in TensorFlow and PrettyTensor. If 10.000 new users waste 1 hour each on average, then that's a combined 10.000 hours they could have spent on more productive things - and I think that's a very low estimate of the gain that could be had simply by improving the documentation. That's a pretty big contribution to the community, if you look at it that way.
Also, why isn't PrettyTensor an integrated part of TensorFlow? Why would anyone want to do the low-level coding required by TF and not use something like PT? In PT I can define my graph concisely in a few lines of code and not have to worry about allocating variables of the right shape, etc. The idea of PT is great, it just needs a more polished documentation.
I also looked briefly at Keras. Although its documentation seems to be more elaborate and polished than PrettyTensor, Keras has quite different syntax and semantics. In some cases PT seems to be simpler and better than Keras. So I'm hoping you guys will greatly improve the documentation for PrettyTensor so it's easier to learn for beginners.
I will take a look at TFLearn next. The documentation appears to be more polished than PrettyTensor.
I've seen a video about Microsoft's CNTK. It appears to be much simpler and more polished than TensorFlow + PrettyTensor. It'll be interesting to see when their Python version comes out in a few months.
Anyway, I hope some of these comments were useful.
This question on Stack Overflow observes that using Pretty Tensor with the latest version of TensorFlow will lead to uninitialized variable errors. As far as I can tell, the problem stems from using tf.GraphKeys.VARIABLES
as the default variable collection, which doesn't work with the (new) recommended way to initialize all variables (tf.global_variables_initializer()
.
Can you please update the implementation to use tf.GraphKeys.GLOBAL_VARIABLES
, to match the default TensorFlow implementation, or suggest a workaround? Thanks!
Once again I hope it's OK that I ask this question here instead of on StackOverflow.
I don't know if batch-normalization is really useful, there seems to be different opinions on the matter. But I'd like to try it. I can see that it's implemented in Pretty Tensor:
https://github.com/google/prettytensor/blob/master/docs/PrettyTensor.md#batch_normalize
But I can't figure out how to use it for the following Convolutional Neural Network:
with pt.defaults_scope(activation_fn=tf.nn.relu):
y_pred, loss = x_pretty.\
conv2d(kernel=5, depth=64, name='layer_conv1').\
max_pool(kernel=2, stride=2).\
conv2d(kernel=5, depth=64, name='layer_conv2').\
max_pool(kernel=2, stride=2).\
flatten().\
fully_connected(size=256, name='layer_fc1').\
fully_connected(size=128, name='layer_fc2').\
softmax_classifier(class_count=10, labels=y_true)
Any help would be appreciated.
In tensorflow, it is possible to visualize the filters using the image summary.
How can I do the same in prettytensor ?
Edit:
To rephrase the question, I wanted to access the 'weights' form the first convolution layer. So if the name of my first conv layer was conv1
as in
conv2d(9, 64, stride=1, init=tf.truncated_normal_initializer(stddev=0.001),edges='SAME',name='conv1')
To access the variable - conv1/weights
, I used
with tf.variable_scope('conv1') as scope:
tf.get_variable_scope().reuse_variables()
weights = tf.get_variable('weights')
But got an error:
ValueError: Variable conv1/weights does not exist, disallowed
What am I doing wrong here. Please suggest the correct way to extract the weights!
Thanks in advance!
Hi @eiderman !
In data_utils
file_name = maybe_download('http://cs.stanford.edu/people/karpathy/char-rnn/',
'shakespear.txt')
downloads a file of size 99993. But this is not compatible with the numpy.reshape
command if the chunk size is relatively prime to the file size. Downloading a different shakespeare file instead (shakespeare_input.txt
from the same directory) yields a file divisible by the chunk size, but the right fix is probably to pad or trim the array before reshaping.
Successfully downloaded shakespear.txt 99993 bytes.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-12-7111020859c7> in <module>()
----> 1 data_utils.shakespeare(2)
.../python2.7/site-packages/prettytensor/tutorial/data_utils.pyc in shakespeare(chunk_size)
130 arr = np.array([convert_to_int(c) for c in shakespeare_full])[
131 0:len(shakespeare_full) / chunk_size * chunk_size]
--> 132 return arr.reshape((len(arr) / chunk_size, chunk_size))
133
134
ValueError: total size of new array must be unchanged
I need more control over the summaries I write, but I can't seem to find an option that disables the summaries automatically added to e.g. relu layers.
Hi, can you please clarify how to use the saved model files if I want to load the model from a checkpoint?
Here is my code:
# mock input
mock_input = np.ones(input_shape)
# build model
accuracy, cost, inference_input, label_tensor, inferences, train_op = \
build_network(input_shape, learning_rate, specs, pt.Phase.test)
# set gpu and config options
gpu_options = \
tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=.9)
config = \
tf.ConfigProto(allow_soft_placement=True, gpu_options=gpu_options,
log_device_placement=True)
# load from checkpoint
model_ckpt = './models/first/-3036.data-00000-of-00001'
runner = pt.train.Runner(initial_checkpoint=model_ckpt)
with tf.Session(config=config), tf.device('/gpu:2'):
predictions = runner.run_model(
op_list=[inferences], num_steps=1,
feed_vars=(inference_input,), print_every=0,
feed_data=[(mock_input,)])
The inspection of the three saved files:
file.data-00000-of-00001
nor in file.meta
since both yield DataLoss error
file.index
must be the correct one but it gives NotFoundError: Tensor name "conv3d_1/bias/Adam_1" not found in checkpoint files ...
I thought that all variables are saved during training. what am I doing wrong?
Whenever I inserted batch normalization in prettytensor's conv2d function, it fails assertion.
My simple test code is
import tensorflow as tf
import prettytensor as pt
x = tf.placeholder(tf.float32, [None, 224, 224, 3])
net = pt.wrap(x)
net = net.conv2d(7, 64, batch_normalize=True)
Then the console log prints
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_class.py", line 1980, in method
result = func(non_seq_layer, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_image_methods.py", line 246, in __call__
y = input_layer.with_tensor(y).batch_normalize()
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_class.py", line 1980, in method
result = func(non_seq_layer, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_image_methods.py", line 59, in __call__
assert isinstance(learned_moments_update_rate, tf.compat.real_types)
AssertionError
Thank you
When passed in an initializer that's not callable, a tensor is created as opposed a variable:
prettytensor/prettytensor/pretty_tensor_class.py
Line 1699 in 5649411
any plans to implement it soon?
If i want to use conv2d kernel size 1x5 and 5x1 then how can i set the 'kernel' value?
Now i try it like below:
conv2d(kernel=[1, 5], depth=32, activation_fn = tf.nn.relu, edges='SAME', name='layer_conv1_a', batch_normalize = True).
conv2d(kernel=[5, 1], depth=32, activation_fn = tf.nn.relu, edges='SAME', name='layer_conv1_b'). \
is this right setting for 1x5 and 5x1 conv kernel?
Since tensorflow supports Py3, it would be great if prettytensor would as well.
Hi,
there appears to be a bug with the combination of template, batch_normalize=True, and phase=Unbound. I believe this is a valid use case for constructing graphs. I understand a similar issue had been raised before, but I'm quite positive about the issue.
Code to reproduce the bug:
import prettytensor as pt
import tensorflow as tf
with pt.defaults_scope(activation_fn=tf.nn.relu, phase=pt.UnboundVariable('phase'), batch_normalize=True):
out=pt.template('input').conv2d(4, 32, stride=2, name='conv1', bias=None)
test=out.construct(phase=pt.Phase.train, input=tf.placeholder(tf.float32, [20, 16, 16, 1]))
After stepping through the executions, I believe that the bug originates from the following interaction:
As the conv layer is being constructed, the line
y = pretty_tensor_normalization_methods.batch_normalize_with_arguments(
y, batch_normalize)
within the conv2d method is being called, as there is a default unbounded variable in the chainDict, the layer constructed by the batch_normalization method is a deferred layer, instead of a layer object. This causes any subsequent activation_fn to fail.
I have implemented a local fix to add the conv layer's phase information into batch_normalize argument. Essentially, if batch_normalize is True or a BatchNormalzationArgument without phase info, I would pass in the phase info from the conv layer's arguments.
Could anyone verify the issue and my proposed fix? I could create a pull-request once the issue is confirmed.
Thanks a lot
Hi,
This question is a post-op of the issue #6 .
How do we load the weights back into the model once they are saved. I had them saved this way.
vars = sess.run(tf.get_collection(tf.GraphKeys.VARIABLES))
pickle.dump(vars, open('vars.npy','wb'))
Thanks in advance!
lstm example
A few ops (embedding_lookup and the summaries in softmax) don't work when explicitly assigned to gpus. Evaluate which ops are not supported and create work-arounds or upstream fixes.
Reading the code, it looks like the evaluate_model
method here doesn't take an average over all of the minibatches that it evaluates. It calls run_model
, which iterates over minibatches and gets the loss functions by calling sess.run
here. This variable is then just reassigned on the next minibatch (although it might be printed or saved when _log_and_save
is called). Then, once we've iterated over all minibatches the most recent results
is returned. In evaluate_model
this is just indexed to output whatever loss function we wanted to evaluate over the test set.
There's no point in iterating over the whole test set if we're not taking an average when we run evaluate_model
.
Thanks for this great library! I've been intensively using it for a while and like it a lot.
I wonder, you are going to continue supporting it on Github?
Hi all,
First of all thanks for a great library.
Now, looking at the code for the rnn methods, the weights are always created anew with every call of {gru, lstm}_cell method, is that correct? If so, shouldn't the weights be created only once and reused at each timestep?
Running out of the box, the shakespeare tutorial crashes with a broken feed value. I am running the tensorflow (0.7.1) and pretty tensor (0.5.3) versions that are the latest right now.
mnist.py does not have the same issue, despite the apparent use of the same data_utils frameworks (permute_data, etc). Help?
$ python /usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/prettytensor/tutorial/shakespeare.py --epochs 2 --save_path /home/jeremy/notebooks/shakespeare/
Starting Shakespeare
W tensorflow/core/common_runtime/executor.cc:1102] 0x15629680 Compute status: Invalid argument: You must feed a value for placeholder tensor 'shakespeare_2/Placeholder' with dtype int32
[[Node: shakespeare_2/Placeholder = Placeholder[dtype=DT_INT32, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
[[Node: _send_shakespeare/cross_entropy/truediv_2_0 = _Send[T=DT_FLOAT, client_terminated=true, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=-2958238964605934168, tensor_name="shakespeare/cross_entropy/truediv_2:0", _device="/job:localhost/replica:0/task:0/cpu:0"](shakespeare/cross_entropy/truediv_2)]]
Traceback (most recent call last):
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/prettytensor/tutorial/shakespeare.py", line 249, in <module>
tf.app.run()
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/tensorflow/python/platform/default/_app.py", line 30, in run
sys.exit(main(sys.argv))
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/prettytensor/tutorial/shakespeare.py", line 226, in main
print_every=10)
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/prettytensor/local_trainer.py", line 216, in train_model
print_every=print_every)[2:]
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/prettytensor/local_trainer.py", line 166, in run_model
dict(zip(feed_vars, data)))
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 315, in run
return self._run(None, fetches, feed_dict)
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 511, in _run
feed_dict_string)
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 564, in _do_run
target_list)
File "/usr/share/anaconda/anaconda2/envs/tf2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 586, in _do_call
e.code)
Hi,
When running this tutorial on pretty tensor, I get FailedPreconditionError. I am using tensorflow_gpu-0.12.0rc0-cp27-none-linux_x86_64 and prettytensor-0.6.2 with python2.7.
When I searched on internet, I see that this error can occur if the session variables are not initialized.
In your tutorial they are initialized. Moreover I haven't received this error when I do not use prettytensor and define the network in a conventional way using default tensorflow library.
Below is the error message.
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:122 in _create_initializers.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2016-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:126 in _create_initializers.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2016-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:127 in _create_initializers.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:128 in _create_initializers.: initialize_local_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.local_variables_initializer` instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:130 in _create_initializers.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2016-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:122 in _create_initializers.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2016-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py:122 in _create_initializers.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2016-03-02.
Instructions for updating:
Please use tf.global_variables instead.
W tensorflow/core/framework/op_kernel.cc:975] Failed precondition: Attempting to use uninitialized value fully_connected/weights
[[Node: fully_connected/weights/read = Identity[T=DT_FLOAT, _class=["loc:@fully_connected/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](fully_connected/weights)]]
Exception -- stopping threads: Attempting to use uninitialized value fully_connected/weights
[[Node: fully_connected/weights/read = Identity[T=DT_FLOAT, _class=["loc:@fully_connected/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](fully_connected/weights)]]
Caused by op u'fully_connected/weights/read', defined at:
File "mnist_prettytensor.py", line 134, in <module>
tf.app.run()
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "mnist_prettytensor.py", line 84, in main
result = multilayer_fully_connected(image_placeholder, labels_placeholder)
File "mnist_prettytensor.py", line 54, in multilayer_fully_connected
return (images.flatten().fully_connected(100).fully_connected(100)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1981, in method
result = func(non_seq_layer, *args, **kwargs)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_methods.py", line 355, in __call__
dt=dtype)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1695, in variable
collections=variable_collections)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1024, in get_variable
custom_getter=custom_getter)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 850, in get_variable
custom_getter=custom_getter)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 346, in get_variable
validate_shape=validate_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 331, in _true_getter
caching_device=caching_device, validate_shape=validate_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 677, in _get_single_variable
expected_shape=shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 224, in __init__
expected_shape=expected_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 370, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1424, in identity
result = _op_def_lib.apply_op("Identity", input=input, name=name)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value fully_connected/weights
[[Node: fully_connected/weights/read = Identity[T=DT_FLOAT, _class=["loc:@fully_connected/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](fully_connected/weights)]]
Traceback (most recent call last):
File "mnist_prettytensor.py", line 134, in <module>
tf.app.run()
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "mnist_prettytensor.py", line 123, in main
print_every=100)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py", line 334, in train_model
print_every=print_every)[2:]
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/local_trainer.py", line 286, in run_model
results = sess.run(ops, dict(zip(feed_vars, data)))
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value fully_connected/weights
[[Node: fully_connected/weights/read = Identity[T=DT_FLOAT, _class=["loc:@fully_connected/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](fully_connected/weights)]]
Caused by op u'fully_connected/weights/read', defined at:
File "mnist_prettytensor.py", line 134, in <module>
tf.app.run()
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "mnist_prettytensor.py", line 84, in main
result = multilayer_fully_connected(image_placeholder, labels_placeholder)
File "mnist_prettytensor.py", line 54, in multilayer_fully_connected
return (images.flatten().fully_connected(100).fully_connected(100)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1981, in method
result = func(non_seq_layer, *args, **kwargs)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_methods.py", line 355, in __call__
dt=dtype)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1695, in variable
collections=variable_collections)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1024, in get_variable
custom_getter=custom_getter)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 850, in get_variable
custom_getter=custom_getter)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 346, in get_variable
validate_shape=validate_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 331, in _true_getter
caching_device=caching_device, validate_shape=validate_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 677, in _get_single_variable
expected_shape=shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 224, in __init__
expected_shape=expected_shape)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 370, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1424, in identity
result = _op_def_lib.apply_op("Identity", input=input, name=name)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/badami/Codes/deeplearning/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value fully_connected/weights
[[Node: fully_connected/weights/read = Identity[T=DT_FLOAT, _class=["loc:@fully_connected/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](fully_connected/weights)]]
I've used PrettyTensor for several of my tutorials on TensorFlow:
https://github.com/Hvass-Labs/TensorFlow-Tutorials
I don't think I've updated PT in several months because everything seemed to work fine for me. But recently I started getting reports that softmax_classifier()
did not work with the class_count
keyword anymore. It appears it has changed to num_classes
instead. I prefer the new keyword, but it was a rather significant API change which broke all my code, and it is not listed in the change-log:
https://github.com/google/prettytensor/blob/master/CHANGELIST.md
In the future, please list all important changes in the log.
I'm going through my tutorials now to update them, but I would prefer to wait until the deprecation warnings are removed, see #41.
How do I not create create a scope for new variables? I've looked into the code, but I simply cannot see how .variable()
creates a new scope for the variable.
This is undesired behavior for me, since I want to use the current scope that I assigned with tf.variable_scope()
to share some variables.
Given that prettytensor combined with TF and ipython makes very very easy to do large scale, iterative exploration, I wonder if it would be in scope to incorporate a default implementation of Reusable Holdout [1] into the runner class and provided examples -- specially for less-experienced people to get a head-start in doing analysis in a good way.
[1] http://googleresearch.blogspot.com/2015/08/the-reusable-holdout-preserving.html
I've trained a 3-layer fully connected layer as a test and it works perfectly. I'd like to see what the weights used in each layer are but I'm not sure how access them through prettytensor. In standard tensorflow the variables are explicit, but here they hidden through the "pretty" interface.
An example in the docs to shows how to access the weights would be very useful for others. We can use the example you have in the docs:
result = (pretty_tensor.wrap(input_data, m)
.flatten()
.fully_connected(200, activation_fn=tf.nn.relu)
.fully_connected(10, activation_fn=None)
.softmax(labels, name=softmax_name))
as an example to extract the weights.
Hi, I use tensorflow_gpu-0.12.0rc1-cp34-cp34m-linux_x86_64
and prettytensor-0.7.1
with python 3.4
.
when running the tensorboard I get following warning
WARNING:tensorflow:From /home/badami/Codes/deeplearning/tensorflow3/lib/python3.4/site-packages/prettytensor/bookkeeper.py:243 in add_scalar_summary.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
I also do not see any graph on the tensorboard. I am not sure if my code is incorrect or this warning is the reason I do not see the graph.
I'm running the baby_names tutorial, and it is failing with the following error (excerpt):
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/GPU:0'
[[Node: Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad = SparseApplyAdagrad[T=DT_FLOAT, Tindices=DT_INT32, use_locking=false](baby_names/embedding_lookup/params, baby_names/embedding_lookup/params/Adagrad, ExponentialDecay, gradients/concat, gradients/concat_1)]]
Caused by op u'Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad', defined at:
File "tutorial/baby_names.py", line 193, in
tf.app.run()
It was previously erroring due to the .csv not being found (so I copied into
/usr/local/lib/python2.7/dist-packages/prettytensor/tutorial/)
Any suggestions for how to fix this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.