pytorch / tutorials

PyTorch tutorials.

PyTorch Tutorials

All the tutorials are now presented as sphinx style documentation at:

https://pytorch.org/tutorials

Asking a question

If you have a question about a tutorial, post in https://dev-discuss.pytorch.org/ rather than creating an issue in this repo. Your question will be answered much faster on the dev-discuss forum.

Submitting an issue

You can submit the following types of issues:

• Feature request - request a new tutorial to be added. Please explain why this tutorial is needed and how it demonstrates PyTorch value.
• Bug report - report a failure or outdated information in an existing tutorial. When submitting a bug report, please run: python3 -m torch.utils.collect_env to get information about your environment and add the output to the bug report.

Contributing

We use sphinx-gallery's notebook styled examples to create the tutorials. Syntax is very simple. In essence, you write a slightly well formatted Python file and it shows up as an HTML page. In addition, a Jupyter notebook is autogenerated and available to run in Google Colab.

Here is how you can create a new tutorial (for a detailed description, see CONTRIBUTING.md):

1. Create a Python file. If you want it executed while inserted into documentation, save the file with the suffix tutorial so that the file name is your_tutorial.py.
2. Put it in one of the beginner_source, intermediate_source, advanced_source directory based on the level of difficulty. If it is a recipe, add it to recipes_source. For tutorials demonstrating unstable prototype features, add to the prototype_source.
3. For Tutorials (except if it is a prototype feature), include it in the toctree directive and create a customcarditem in index.rst.
4. For Tutorials (except if it is a prototype feature), create a thumbnail in the index.rst file using a command like .. customcarditem:: beginner/your_tutorial.html. For Recipes, create a thumbnail in the recipes_index.rst

If you are starting off with a Jupyter notebook, you can use this script to convert the notebook to Python file. After conversion and addition to the project, please make sure that section headings and other things are in logical order.

Building locally

The tutorial build is very large and requires a GPU. If your machine does not have a GPU device, you can preview your HTML build without actually downloading the data and running the tutorial code:

1. Install required dependencies by running: pip install -r requirements.txt.

If you want to use virtualenv, in the root of the repo, run: virtualenv venv, then source venv/bin/activate.

• If you have a GPU-powered laptop, you can build using make docs. This will download the data, execute the tutorials and build the documentation to docs/ directory. This might take about 60-120 min for systems with GPUs. If you do not have a GPU installed on your system, then see next step.
• You can skip the computationally intensive graph generation by running make html-noplot to build basic html documentation to _build/html. This way, you can quickly preview your tutorial.

If you get ModuleNotFoundError: No module named 'pytorch_sphinx_theme' make: *** [html-noplot] Error 2 from /tutorials/src/pytorch-sphinx-theme or /venv/src/pytorch-sphinx-theme (while using virtualenv), run python setup.py install.

Building a single tutorial

You can build a single tutorial by using the GALLERY_PATTERN environment variable. For example to run only neural_style_transfer_tutorial.py, run:

GALLERY_PATTERN="neural_style_transfer_tutorial.py" make html

or

GALLERY_PATTERN="neural_style_transfer_tutorial.py" sphinx-build . _build

The GALLERY_PATTERN variable respects regular expressions.

About contributing to PyTorch Documentation and Tutorials

• You can find information about contributing to PyTorch documentation in the PyTorch Repo README.md file.
• Additional information can be found in PyTorch CONTRIBUTING.md.

License

PyTorch Tutorials is BSD licensed, as found in the LICENSE file.

tutorials's Issues

Numerically unstable initialized values for uninitialized tensors?

I was trying to follow the tutorial when I noticed that if I just create an "uninitialized matrix", its values are not numerically stable. I guess since we will have to initialize the matrix later, it doesn't really matter, but I'm just wondering if this is intentional.

I'm running pyTorch with anaconda python 3.6, CUDA v8 on Linux.

from __future__ import print_function
import torch

x = torch.Tensor(5, 3)

x = torch.rand(5, 3)

x

-1.6775e+31  4.5895e-41  4.0929e-37
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 5x3]

y = torch.Tensor(5, 3); y

-1.6775e+31  4.5895e-41  4.2770e-37
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 5x3]

x * y

inf   0   0
  0   0   0
  0   0   0
  0   0   0
  0   0   0
[torch.FloatTensor of size 5x3]

Running .backward() without a CUDA-capable device

Should it be possible to run Variable.backwards() without having a CUDA-enabled GPU? I was trying to run the code from the autograd tutorial . I didn't find it mentioned that one must have a CUDA GPU to run the source/notebooks.

Training on GPU in deep learning notebook - inputs/labels need cuda()

In working through the deep learning notebook, it's not obvious at first how to get the learning working once you put the net on the GPU.

After some trial and error, this worked

    inputs, labels = Variable(inputs).cuda(), Variable(labels).cuda()

I could make a PR with this addition if desired

"Chat with other users on Slack" link directs to a 404 not found page

"Chat with other users on Slack" on the Deep Learning with PyTorch.ipynb directs to a 404 not found page on Safari

The ONNX tutorial does not appear in the topic list

It is not listed on http://pytorch.org/tutorials/.

Which means if we don't the address, it's hard to access the page (http://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html).

'MSELoss' object has no attribute 'next_functions'

I am reading tutorial of learning pytorch, but I get stuck in one error as follow.

https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/neural_networks_tutorial.py

Lines from 184 to 186 are my source code. And I have replace grad_fn to creator as grad_fn throws error out.

Paste my error info below.

Traceback (most recent call last):
  File "./network.py", line 56, in <module>
    print(loss.creator.next_functions[0][0])
AttributeError: 'MSELoss' object has no attribute 'next_functions'

AttributeError: 'TimeLimit' object has no attribute 'x_threshold'

I have a problem when I run the code in PyTorch DQN tutorial,the code and error are :

resize = T.Compose([T.ToPILImage(), T.Scale(40, interpolation=Image.CUBIC), T.ToTensor()])

# This is based on the code from gym.
screen_width = 600
def get_cart_location():
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0) # MIDDLE OF CART
    
    
def get_screen():
    screen = env.render(mode='rgb_array').transpose((2, 0, 1)) # transpose into torch order (CHW)
    # Strip off the top and bottom of the screen
    screen = screen[:, 160:320]
    view_width = 320
    cart_location = get_cart_location()
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width,None)
    else:
        slice_range = slice(cart_location - view_width // 2, cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescare, convert to torch tensor (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0)

env.reset()
plt.imshow(get_screen().squeeze(0).permute(1, 2, 0).numpy(), interpolation='none')
plt.show()

AttributeError                            Traceback (most recent call last)
<ipython-input-5-0076b471379d> in <module>()
     30 
     31 env.reset()
---> 32 plt.imshow(get_screen().squeeze(0).permute(1, 2, 0).numpy(), interpolation='none')
     33 plt.show()

<ipython-input-5-0076b471379d> in get_screen()
     14     screen = screen[:, 160:320]
     15     view_width = 320
---> 16     cart_location = get_cart_location()
     17     if cart_location < view_width // 2:
     18         slice_range = slice(view_width)

<ipython-input-5-0076b471379d> in get_cart_location()
      4 screen_width = 600
      5 def get_cart_location():
----> 6     world_width = env.x_threshold * 2
      7     scale = screen_width / world_width
      8     return int(env.state[0] * scale + screen_width / 2.0) # MIDDLE OF CART

AttributeError: 'TimeLimit' object has no attribute 'x_threshold'

I don't know how this happened,since I had installed the gym,and run the demo in gym successfully,any advice of you would be appreciated

Feature Request: tutorial on loading datasets

A tutorial outlining how to make use of the torch.utils.data.Dataset and torch.utils.data.DataLoader on your own data (not just the torchvision.datasets) would be good. The documentation page is quite obscure, and it is not entirely clear how these can be made use of on your own data.

Also outlining what would be good practices for when your data is:

• A numpy array
• A folder full of image files

And if pytorch has built in functions for creating queues of data, for when the data is too big to all fit in memory in one go (eg in the case of a folder full of image files).

Neural style transfer question

Hi, not sure if this is the right place to ask questions, but I'm working through the neural style transfer tutorial and am confused about something.

What is the purpose of the backward method in ContentLoss and StyleLoss?

If we remove the backward method, won't this work as well for the closure function in run_style_transfer?

    def closure():
            # correct the values of updated input image
            input_param.data.clamp_(0, 1)

            optimizer.zero_grad()
            model(input_param)
            style_score = 0
            content_score = 0

            for sl in style_losses:
                style_score += sl.loss
            for cl in content_losses:
                content_score += cl.loss

            run[0] += 1
            if run[0] % 50 == 0:
                print("run {}:".format(run))
                print('Style Loss : {:4f} Content Loss: {:4f}'.format(
                    style_score.data[0], content_score.data[0]))
                print()

            total_score = style_score+content_score
            total_score.backward()

            return total_score

On a related note, won't multiple backward calls in the original code accumulate the gradients for the image? Why is it okay to do this? Am I wrong in assuming that you should only call backward once? I'm new to Pytorch so I apologize if I'm missing anything fundamental. Thanks!

EDIT: Tagging the author @alexis-jacq if you don't mind :)

Reinforcement Q-learning issue on GPU

when running on GPU, torch.cuda.is_available() == True, I run into the following problem, in training loop:

---

TypeError Traceback (most recent call last)
in ()
50 for t in count():
51 # Select and perform an action
---> 52 action = select_action(state)
53 _, reward, done, _ = env.step(action[0,0])
54 reward = torch.Tensor([reward])

in select_action(state)
27 steps_done += 1
28 if sample > eps_threshold:
---> 29 return model(Variable(state, volatile=True)).data.max(1)[1].cpu()
30 else:
31 return torch.LongTensor([[random.randrange(2)]])

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
200
201 def call(self, *input, **kwargs):
--> 202 result = self.forward(*input, **kwargs)
203 for hook in self._forward_hooks.values():
204 hook_result = hook(self, input, result)

in forward(self, x)
11
12 def forward(self, x):
---> 13 x = F.relu(self.bn1(self.conv1(x)))
14 x = F.relu(self.bn2(self.conv2(x)))
15 x = F.relu(self.bn3(self.conv3(x)))

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
200
201 def call(self, *input, **kwargs):
--> 202 result = self.forward(*input, **kwargs)
203 for hook in self._forward_hooks.values():
204 hook_result = hook(self, input, result)

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/conv.pyc in forward(self, input)
235 def forward(self, input):
236 return F.conv2d(input, self.weight, self.bias, self.stride,
--> 237 self.padding, self.dilation, self.groups)
238
239

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/functional.pyc in conv2d(input, weight, bias, stride, padding, dilation, groups)
35 f = ConvNd(_pair(stride), _pair(padding), _pair(dilation), False,
36 _pair(0), groups)
---> 37 return f(input, weight, bias) if bias is not None else f(input, weight)
38
39

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/_functions/conv.pyc in forward(self, input, weight, bias)
33 if k == 3:
34 input, weight = _view4d(input, weight)
---> 35 output = self._update_output(input, weight, bias)
36 if k == 3:
37 output, = _view3d(output)

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/_functions/conv.pyc in _update_output(self, input, weight, bias)
95
96 self._bufs = [[] for g in range(self.groups)]
---> 97 return self._thnn('update_output', input, weight, bias)
98
99 def _grad_input(self, input, weight, grad_output):

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/_functions/conv.pyc in _thnn(self, fn_name, input, weight, *args)
154 impl = _thnn_convs[self.thnn_class_name(input)]
155 if self.groups == 1:
--> 156 return impl[fn_name](self, self._bufs[0], input, weight, *args)
157 else:
158 res = []

/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/_functions/conv.pyc in call_update_output(self, bufs, input, weight, bias)
232 args = parse_arguments(self, fn.arguments[5:], bufs, kernel_size)
233 getattr(backend, fn.name)(backend.library_state, input, output, weight,
--> 234 bias, *args)
235 return output
236 return call_update_output

TypeError: FloatSpatialConvolutionMM_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, torch.FloatTensor, torch.cuda.FloatTensor, torch.cuda.FloatTensor, torch.FloatTensor, torch.FloatTensor, long, long, int, int, int, int), but expected (int state, torch.FloatTensor input, torch.FloatTensor output, torch.FloatTensor weight, [torch.FloatTensor bias or None], torch.FloatTensor finput, torch.FloatTensor fgradInput, int kW, int kH, int dW, int dH, int padW, int padH)

"NameError: name 'base' is not defined" in DQN tutorial

Thanks for pretty nice tutorial, but I have a problem when running the DQN tutorial.
I ran it on an Ubuntu server then got an error message bellow at the ** Input extraction ** section.

NameError                                 Traceback (most recent call last)
<ipython-input-7-ca07103bb7e3> in <module>()
     30 
     31 
---> 32 plt.imshow(get_screen().squeeze(0).permute(1, 2, 0).numpy(), interpolation='none')
     33 # plt.show()

<ipython-input-7-ca07103bb7e3> in get_screen()
     10 
     11 def get_screen():
---> 12     screen = env.render(mode='rgb_array').transpose((2, 0, 1)) # transpose into torch order (CHW)
     13     # Strip off the top and bottom of the screen
     14     screen = screen[:, 160:320]

/home/USER/anaconda3/lib/python3.5/site-packages/gym/core.py in render(self, mode, close)
    172             raise error.UnsupportedMode('Unsupported rendering mode: {}. (Supported modes for {}: {})'.format(mode, self, modes))
    173 
--> 174         return self._render(mode=mode, close=close)
    175 
    176     def close(self):

/home/USER/anaconda3/lib/python3.5/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode, close)
    111 
    112         if self.viewer is None:
--> 113             from gym.envs.classic_control import rendering
    114             self.viewer = rendering.Viewer(screen_width, screen_height)
    115             l,r,t,b = -cartwidth/2, cartwidth/2, cartheight/2, -cartheight/2

/home/USER/anaconda3/lib/python3.5/site-packages/gym/envs/classic_control/rendering.py in <module>()
     21 
     22 try:
---> 23     from pyglet.gl import *
     24 except ImportError as e:
     25     reraise(prefix="Error occured while running `from pyglet.gl import *`",suffix="HINT: make sure you have OpenGL install. On Ubuntu, you can run 'apt-get install python-opengl'. If you're running on a server, you may need a virtual frame buffer; something like this should work: 'xvfb-run -s \"-screen 0 1400x900x24\" python <your_script.py>'")

/home/USER/anaconda3/lib/python3.5/site-packages/pyglet/gl/__init__.py in <module>()
    222     else:
    223         from .carbon import CarbonConfig as Config
--> 224 del base
    225 
    226 # XXX remove

NameError: name 'base' is not defined

xvfb-run may work with python command but how to manage or fix this problem with Jupyter? Thank you.

Build failed

Sphinx version: 2.5.6

Python version: 2.7.13 (CPython)

Docutils version: 0.13.1 release

Jinja2 version: 2.9.6

Last messages:

Loaded extensions:

Traceback (most recent call last):
File "/home/xwy/anaconda2/lib/python2.7/site-packages/Sphinx-1.5.6-py2.7.egg/sphinx/cmdline.py", line 295, in main
opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
File "/home/xwy/anaconda2/lib/python2.7/site-packages/Sphinx-1.5.6-py2.7.egg/sphinx/application.py", line 244, in init
self._init_builder(self.buildername)
File "/home/xwy/anaconda2/lib/python2.7/site-packages/Sphinx-1.5.6-py2.7.egg/sphinx/application.py", line 315, in _init_builder
self.emit('builder-inited')
File "/home/xwy/anaconda2/lib/python2.7/site-packages/Sphinx-1.5.6-py2.7.egg/sphinx/application.py", line 589, in emit
results.append(callback(self, *args))
File "/home/xwy/anaconda2/lib/python2.7/site-packages/sphinx_gallery/gen_gallery.py", line 237, in generate_gallery_rst
seen_backrefs)
File "/home/xwy/anaconda2/lib/python2.7/site-packages/sphinx_gallery/gen_rst.py", line 400, in generate_dir_rst
this_entry = _thumbnail_div(build_target_dir, fname, intro) + """
File "/home/xwy/anaconda2/lib/python2.7/site-packages/sphinx_gallery/backreferences.py", line 181, in _thumbnail_div
thumbnail=thumb, ref_name=ref_name)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 74: ordinal not in range(128)
~
~
~
~
~
~
7,19 All

Potential improvement to 60 minute blitz for pasteability?

Hello! I'm very much a newbie to this:

https://github.com/pytorch/tutorials/blob/master/Deep%20Learning%20with%20PyTorch.ipynb

I followed this guide with Anaconda 3.5 and got to this point: out = net(input)
I got a NotImplementedError from the original nn module that the class was supposed to override.

Turns out I skipped the error messages I got in interactive python where the indentation was wrong (so forward function wasn't implemented in my Net class).

If we removed the spaces between the functions or used comments we could avoid the issue:

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5) # 1 input image channel, 6 output channels, 5x5 square convolution kernel
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120) # an affine operation: y = Wx + b
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv2(x)), 2) # If the size is a square you can only specify a single number
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    def num_flat_features(self, x):
        size = x.size()[1:] # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
net

Unicode error in python2.7

Hi,
I got the following error while running char_rnn_classification_tutorial.py

print(unicodeToAscii('Ślusàrski'))
 File "src/char_rnn.py", line 17, in unicodeToAscii
   c for c in unicodedata.normalize('NFD', s)
TypeError: normalize() argument 2 must be unicode, not str

To circumvent the error I changed the code:

def unicodeToAscii(s):
    try:
        s = unicode(s, 'utf-8')
    except NameError:
        pass
    s = unicodedata.normalize('NFD', s)
    s = s.encode('ascii', 'ignore')
    s = s.decode('utf-8')
    return str(s)

ImportError: No module named pyglet in RL tutorial

Hi! Running the reinforcement_q_learning.ipynb tutorial however, may fail with the error:
File "/home/gym/gym/envs/classic_control/rendering.py", line 18, in <module> import pyglet gym.utils.reraise.ReraisedException: The original exception was: exceptions.ImportError: No module named pyglet

I fixed it, after installing the full version of GYM in Ubuntu by doing also:
pip install -e .[all]

Shouldn't the following command on Ubuntu 14.04 install all dependencies?

apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig

cuda issue

When i run this code

a = torch.Tensor(5, 3) # construct a 5x3 matrix, uninitialized
b = torch.Tensor(5, 3) # construct a 5x3 matrix, uninitialized
if torch.cuda.is_available():
aa= a.cuda()
bb = b.cuda()
aa + bb

I get the following error message
RuntimeError: cuda runtime error (8) : invalid device function at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.6_1484802121799/work/torch/lib/THC/generated/../generic/THCTensorMathPointwise.cu:246

Port new seq2seq tutorial

From https://github.com/spro/practical-pytorch/tree/master/seq2seq-translation

cc @pytorch/team-text-core @Nayef211

"Training on the GPU": "RuntimeError: Need input of dimension 4..."

Towards the end of "Deep Learning with PyTorch", there's the statement:

Exercise: Try increasing the width of your network (argument 1 and 2 of nn.Conv2d, see what kind of speedup you get.

Following these instructions, going back and making such a change, and then clicking through the rest of the notebook, it eventually fails when you get to "3. Train the network", producing the errors such as those listed below.

The tutorial is unclear. There are two nn.Conv2d layers given (self.conv1 and self.conv2), and the tutorial is unclear about which one should be changed. So far, my changing any of the numbers for the first two parameters of these layers seems to produce errors. For example, doubling the first two values of the second Conv2d line from (6,16,5) to (12,32,5) as in...

def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool  = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(12, 32, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

...produces these errors...

---

RuntimeError Traceback (most recent call last)
in ()
13
14 # forward + backward + optimize
---> 15 outputs = net(inputs)
16 loss = criterion(outputs, labels)
17 loss.backward()

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
208
209 def call(self, *input, **kwargs):
--> 210 result = self.forward(*input, **kwargs)
211 for hook in self._forward_hooks.values():
212 hook_result = hook(self, input, result)

in forward(self, x)
11 def forward(self, x):
12 x = self.pool(F.relu(self.conv1(x)))
---> 13 x = self.pool(F.relu(self.conv2(x)))
14 x = x.view(-1, 1655)
15 x = F.relu(self.fc1(x))

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
208
209 def call(self, *input, **kwargs):
--> 210 result = self.forward(*input, **kwargs)
211 for hook in self._forward_hooks.values():
212 hook_result = hook(self, input, result)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.pyc in forward(self, input)
235 def forward(self, input):
236 return F.conv2d(input, self.weight, self.bias, self.stride,
--> 237 self.padding, self.dilation, self.groups)
238
239

/usr/local/lib/python2.7/dist-packages/torch/nn/functional.pyc in conv2d(input, weight, bias, stride, padding, dilation, groups)
35 f = ConvNd(_pair(stride), _pair(padding), _pair(dilation), False,
36 _pair(0), groups)
---> 37 return f(input, weight, bias) if bias is not None else f(input, weight)
38
39

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in forward(self, input, weight, bias)
32 if k == 3:
33 input, weight = _view4d(input, weight)
---> 34 output = self._update_output(input, weight, bias)
35 if k == 3:
36 output, = _view3d(output)

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in _update_output(self, input, weight, bias)
89
90 self._bufs = [[] for g in range(self.groups)]
---> 91 return self._thnn('update_output', input, weight, bias)
92
93 def _grad_input(self, input, weight, grad_output):

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in _thnn(self, fn_name, input, weight, *args)
148 impl = _thnn_convs[self.thnn_class_name(input)]
149 if self.groups == 1:
--> 150 return impl[fn_name](self, self._bufs[0], input, weight, *args)
151 else:
152 res = []

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in call_update_output(self, bufs, input, weight, bias)
226 args = parse_arguments(self, fn.arguments[5:], bufs, kernel_size)
227 getattr(backend, fn.name)(backend.library_state, input, output, weight,
--> 228 bias, *args)
229 return output
230 return call_update_output

RuntimeError: Need input of dimension 4 and input.size[1] == 12 but got input to be of shape: [4 x 6 x 14 x 14] at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THNN/generic/SpatialConvolutionMM.c:47

Whereas doubling the first two parameters of the first nn.Conv2d line...

def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(6, 12, 5)
        self.pool  = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

produces a similar error...

---

RuntimeError Traceback (most recent call last)
in ()
13
14 # forward + backward + optimize
---> 15 outputs = net(inputs)
16 loss = criterion(outputs, labels)
17 loss.backward()

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
208
209 def call(self, *input, **kwargs):
--> 210 result = self.forward(*input, **kwargs)
211 for hook in self._forward_hooks.values():
212 hook_result = hook(self, input, result)

in forward(self, x)
10
11 def forward(self, x):
---> 12 x = self.pool(F.relu(self.conv1(x)))
13 x = self.pool(F.relu(self.conv2(x)))
14 x = x.view(-1, 1655)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
208
209 def call(self, *input, **kwargs):
--> 210 result = self.forward(*input, **kwargs)
211 for hook in self._forward_hooks.values():
212 hook_result = hook(self, input, result)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.pyc in forward(self, input)
235 def forward(self, input):
236 return F.conv2d(input, self.weight, self.bias, self.stride,
--> 237 self.padding, self.dilation, self.groups)
238
239

/usr/local/lib/python2.7/dist-packages/torch/nn/functional.pyc in conv2d(input, weight, bias, stride, padding, dilation, groups)
35 f = ConvNd(_pair(stride), _pair(padding), _pair(dilation), False,
36 _pair(0), groups)
---> 37 return f(input, weight, bias) if bias is not None else f(input, weight)
38
39

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in forward(self, input, weight, bias)
32 if k == 3:
33 input, weight = _view4d(input, weight)
---> 34 output = self._update_output(input, weight, bias)
35 if k == 3:
36 output, = _view3d(output)

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in _update_output(self, input, weight, bias)
89
90 self._bufs = [[] for g in range(self.groups)]
---> 91 return self._thnn('update_output', input, weight, bias)
92
93 def _grad_input(self, input, weight, grad_output):

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in _thnn(self, fn_name, input, weight, *args)
148 impl = _thnn_convs[self.thnn_class_name(input)]
149 if self.groups == 1:
--> 150 return impl[fn_name](self, self._bufs[0], input, weight, *args)
151 else:
152 res = []

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.pyc in call_update_output(self, bufs, input, weight, bias)
226 args = parse_arguments(self, fn.arguments[5:], bufs, kernel_size)
227 getattr(backend, fn.name)(backend.library_state, input, output, weight,
--> 228 bias, *args)
229 return output
230 return call_update_output

RuntimeError: Need input of dimension 4 and input.size[1] == 6 but got input to be of shape: [4 x 3 x 32 x 32] at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THNN/generic/SpatialConvolutionMM.c:47

Would love to know how to make this work. Submitting as an Issue for the tutorial because following instructions produces errors.
I'd suggest clarifying the instructions so that new adopters don't get stuck trying to resolve this.

why did the 60-minute tutorial disappear？

Minor error in Math

Minor correction in the math in 'Deep Learning with PyTorch.ipynb':
https://github.com/pytorch/tutorials/blob/master/Deep%20Learning%20with%20PyTorch.ipynb

You should have got a matrix of 4.5. Let's call the out Variable "$o$".
We have that $o = \frac{1}{4}\sum_i z_i$, $z_i = 3/4(x_i+2)^2$
(Missing 1/4)

Reinforcement demo issue

When I run the following line in the cart-pole demo using pytorch 0.1.6 - I get the following error:

plt.imshow(get_screen().squeeze(0).permute(1, 2, 0).numpy(), interpolation='none')

/home/brad/anaconda3/lib/python3.5/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode, close)
    135             self.viewer.add_geom(self.track)
    136 
--> 137         x = self.state
    138         cartx = x[0]*scale+screen_width/2.0 # MIDDLE OF CART
    139         self.carttrans.set_translation(cartx, carty)

AttributeError: 'CartPoleEnv' object has no attribute 'state'

however the rest of the demo seems to work ok

Also I had to insert 'import torch' in order to avoid a downstream error.

Lovely fun demo

A tutorial on writing custom Datasets + Samplers and using transforms

takeaway from thread: https://discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/2

Regards to Reinforcement DQN

Does USE_CUDA need to be defined?
Was an easy fix, to just define manually, but was wondering why the variable was left undefined in the demo.

Thank you very much for the walk through.

Confusing examples in "Introduction to PyTorch for former Torchies.ipynb"

Regarding to the effects of "retain_variables" argument, the notebook says:

# this would fail if we didn't specify 
# that we want to retain variables

however when I set retain_variables=False in y.backward(), it seems that nothing wrong happens. I can still get x.grad

THFloatTensor_add(tensor,tensor,float)

I tried the code in the c code tutorial. It seems the definition of THFloatTensor_add is different in the version of pytorch I used. The input2 should be a single float number in the version I installed using conda. I modified the my_lib.c and my_lib.h as following to pass the compile.

my_lib.h:
int my_lib_add_forward(THFloatTensor *input1, float input2, THFloatTensor *output);

my_lib.c:
int my_lib_add_forward(THFloatTensor *input1, float input2, THFloatTensor *output) { THFloatTensor_resizeAs(output, input1); THFloatTensor_add(output, input1, input2); return 1; }

Indentation error

The code displayed on the site http://pytorch.org/tutorials/beginner/pytorch_with_examples.html for custom functions has some indentation error. I checked the code given in the repo and there its correct (proper indentation). How should I correct this .
PS I want to begin contributing to open source so maybe the issue is a trivial one. Kindly help me in finding a way to correct it.

Transfer learning tutorial -- momentum handling

In the tutorial, a new SGD optimizer is generated each epoch, which means that the momentum from the last epoch is discarded. As far as I can tell, this is unusual behavior in deep network training. The same optimizer should be used, but with learning rate modified.

Web page for Tutorials

Hi,

I've been working on beautifying/integrating all the tutorials on pytorch into one. see pytorch/pytorch#778. These tutorials are based on sphinx-gallery and tutorials are executed during build time.

I've created a separate repo for the tutorials and used gh-pages to host them: http://chsasank.github.io/pytorch-tutorials. I also added my own transfer learning tutorial

After a discussion with @soumith, he suggested we should host these tutorials at tutorials.pytorch.org. He also requested a change:

• Categorize tutorials by level instead of source

If indeed this is to be front face of tutorials, we'll need to figure out

• how to modify the repo
• how to build the sources and host

For hosting, we shouldn't probably use github pages as it will mess up the git history with all the html files.

Since these tutorials are executed at build, we might need a decently powered build environment. Most tutorials take may be 5 min on my macbook air. Except seq2seq tutorial, which took 40 min on CPU/25 min on GPU. Note that a tutorial is re-excuted only if changes are made to the tutorial file.

Thanks,
Sasank.

Typo in neural_style loss function (style_score + style_score)

Based on my understanding of the optimization,

return style_score + style_score

should read:

return style_score + content_score

on line: https://github.com/pytorch/tutorials/blob/master/advanced_source/neural_style_tutorial.py#L584

Problem in advanced NLP tutorial

@ZhixiuYe pointed out a problem in the Viterbi Decoder in the original repository:
rguthrie3/DeepLearningForNLPInPytorch#8

Him and me proposed some possible fixes. What do you guys think about that?

Where is Variable?

In Reinforcement (Q-)Learning with PyTorch2, the section Training hyperparameters and utilities claim the cell providing Variable which is "a simple wrapper around torch.autograd". But I can't found it in the cell. Then I encounter NameError: name 'Variable' is not defined, anyway I import Variable from torch.autograd instead. So where is Variable? Or how can I implement it by scratch?

Regarding exercises in Character-Level RNN

I was wondering where I can find the dataset for the exercises given in Classifying Names with Character-Level RNN.
For example:
Any word -> language
First name -> gender
Character name -> writer
Page title -> blog or subreddit

To complete this task, do I have to create my own dataset or is there any repo where I can download those datasets?

Slowdown in DQN RL Tutorial

After about 5 episodes on latest master build of Pytorch, the time to execute each step t in the main loop slows way down. I tried a pip install of Pytorch as well to test if it was just my version and same thing. I am on OSX with no cuda. Is slowdown normal? I don't see anything in the optimization step that should really slow this down over time. Didn't know if this could be gym related as well.

If this isn't normal I will try to dig in some more and see what is causing this for me.

Thanks

invalid shortlink target

The link https://goo.gl/uGOksc referring to a document 'Train neural nets to play video games' redirects to a non-existing notebook https://github.com/pytorch/tutorials/blob/master/Reinforcement%20(Q-)Learning%20with%20PyTorch.ipynb

This link appears in these files:

• tutorials/beginner_source/former_torchies/parallelism_tutorial.py

  Line 120 in 01b89c9

  # .. _Train neural nets to play video games: https://goo.gl/uGOksc

• tutorials/beginner_source/blitz/cifar10_tutorial.py

  Line 312 in 01b89c9

  # .. _Train neural nets to play video games: https://goo.gl/uGOksc

  .

Probably what it should link to is https://github.com/pytorch/tutorials/blob/master/intermediate_source/reinforcement_q_learning.py .

(if this is correct and you want to eliminate the short url link like for the other links in these files, I can provide a pull request if wanted)

There is any fine tune tutorials?

Fine tune is very easy in Torch and Caffe, but I can't find how do fine tune in pytorch. Is there any fine tune examples or tutorials?

Broken Simple Tutorial Example

I have just installed PyTorch, and this example doesn't work: http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-variables-and-autograd

Running the example returns:

0 26220132.0
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-44-6cc21ebb6010> in <module>()
     36 
     37   # Manually zero the gradients before running the backward pass
---> 38   w1.grad.data.zero_()
     39   w2.grad.data.zero_()
     40 

AttributeError: 'NoneType' object has no attribute 'data'

as w1.grad.data doesn't seem to be defined.

Include explanation of view in the tutorial

I had the issue that I had no idea what x.view() in the code. I believe it's not explained properly. Tutorial links - http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html, http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html

There's a SO question that has an answer to this -
https://stackoverflow.com/questions/42479902/how-view-method-works-for-tensor-in-torch

Perhaps, the tutorial could link to it, or include the same information.

Errors with outdated with creator and previous_function in tutorial

There's an error with the release docker version of pytorch, and the tutorial. The problem is that the tutorial displayed here is not in sync. I assume what's pulled when building the docker image is the latest release version.

The fix is to the following replacements:
s/creator/grad_fn/g
s/previous_function/next_function/g

Traceback (most recent call last):
  File "neural_networks_tutorial.py", line 184, in <module>
    print(loss.creator)  # MSELoss
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/autograd/variable.py", line 64, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Variable' object has no attribute 'creator'

Initialize with t7 files?

If I trained a model with Torch and stored the weights using t7 format. Is it possible to use this as initialization in pytorch? Thank you.

update beginner tutorial to most recent pytorch version?

This beginner tutorial uses y.grad_fn where, from googling around it seems like it should now use y.creator. The image is updated, but the text/code isn't.

Regardless, the tutorial should probably say what version of PyTorch it's for and how to check, right?

I'm happy to make the modifications and do a pull request, but wasn't sure what kind of solution was desired.

How to save the model in Classifying name tutorial?

I am 100% successfully run the tutorial and I make some problem change, where I fixed the sequence to 10 and just 3 feature. It almost same with the tutorial. I have successfully save the model, but I have problem when loading it.

import torch.nn as nn
from torch.autograd import Variable
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__() 
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))`

I saved the model using this code. I put it in the end of training.
torch.save(rnn.state_dict(),'./halo.pkl')
The network is still same. Here is the code to load the model.

def restore_net(filename):
    n_hidden = 128
    n_letters = 3
    n_categories = 2
    rnn = RNN(n_letters, n_hidden, n_categories)
    rnn.load_state_dict(filename)
    return rnn

However I got this error.

Anyone can have a suggestion how should I save it?
-Thank you-

Add new LR schedulers in transfer learning tutorial

unused hidden variables of LSTM on deep_learning_nlp_tutorial.py examples

Hi,

I am just curious about line 1197.

self.hidden = self.init_hidden()

This value has not been used to update parameters of network in the program. But you are saying also, we need to clear out the hidden state of the LSTM, detaching it from its history on the last instance. Is it correct?

Best regards,

Image not found in Neural Transfer with PyTorch webpage

In webpage:

http://pytorch.org/tutorials/advanced/neural_style_tutorial.html

The links of this image got a 404 not found error:

http://pytorch.org/_static/img/neural-style/picasso.jpg
http://pytorch.org/_static/img/neural-style/dancing.jpg

Translation tutorial

I find the implementation of EncoderRNN to be very strange when n_layers > 1.
See forward method below.

It applies the same GRU cell several times, so it is like doubling the sequence:
word -> wwwooorrrddd, but not adding new layers.

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers=1):
        super(EncoderRNN, self).__init__()
        self.n_layers = n_layers
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        for i in range(self.n_layers):
            output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        <skipped>

Enhancement: cannot run downloaded tutorial scripts

Hi,
After downloading tutorial scripts, I cannot run the Python scripts such as autograd_tutorial.py. They have the following error:

SyntaxError: Non-ASCII character '\xe2' in file /home/tranlaman/Desktop/pytorch-workspace/tutorials/pytorch-60mins/autograd_tutorial.py on line 7, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details.

blog tutorial and slides

Couldn't find you on twitter so raising this here.

I wrote a beginner's first steps blog and a presentation for the pydata london monthly meetup:

• https://goo.gl/EmSfNk
• http://makeyourownneuralnetwork.blogspot.co.uk/2017/05/learning-mnist-with-gpu-acceleration.html

Perhaps these could be the basis for a very beginner-friendly gentle introduction to PyTorch and it's concepts?

Myself I couldn't find beginner-friendly guides with a logical progression, The existing tutorials are not really for complete (but intelligent or interested) beginners.

How do I help?

is the grad value wrong in beginner_source/blitz/autograd_tutorial.py line 92?

in line 92: z_i = 3(x_i+2)^2 and z_i\bigr\rvert_{x_i=1} = 27.

I think z_i\bigr\rvert_{x_i=1} = 6(x_i+2)\rvert_{x_i=1} = 6*(1+2) = 18, please correct me if I am wrong, otherwise i will submit a pull request.

'TimeLimit' object has no attribute 'x_threshold' error running reinforcement_q_learning.ipynb tutorial

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-2f6091ee96f6> in <module>()
     37 
     38 env.reset()
---> 39 plt.imshow(get_screen().squeeze(0).permute(
     40     1, 2, 0).numpy(), interpolation='none')
     41 plt.show()

<ipython-input-5-2f6091ee96f6> in get_screen()
     19     screen = screen[:, 160:320]
     20     view_width = 320
---> 21     cart_location = get_cart_location()
     22     if cart_location < view_width // 2:
     23         slice_range = slice(view_width)

<ipython-input-5-2f6091ee96f6> in get_cart_location()
      8 
      9 def get_cart_location():
---> 10     world_width = env.x_threshold * 2
     11     scale = screen_width / world_width
     12     return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART

AttributeError: 'TimeLimit' object has no attribute 'x_threshold'

attention weights calculation in seq2seq_translation_tutorial

When I was walking through the seq2seq tutorial, It seems to me that the attention weights calculation is different from the original method in Bahdanau et al. paper .

In the paper, the attention energies are calculated based on hidden states of the decoder and annotations h_i, which is a concatenated forward and backward states of the encoder(see s3.2 in the paper).

However, in the tutorial, it seems that attention energies are calculated based on inputs and hidden states of the decoder.

attn_weights = F.softmax(
            self.attn(torch.cat((embedded[0], hidden[0]), 1)))

I'm confused here. Did I miss something?