Question about RNN search spaces about deep_architect

[negrinho]

Hi Pratik,

If you want to consume multiple time steps instead of a single one, the best way is to do this directly, i.e., to consume the whole tensor (batch_size x input_dim x num_steps, or whatever ordering of the dimensions you are using) and output whole processed tensors your computation performs. I would probably take that route. In the future, I want to define modules that can have other search spaces inside, which would help in this use-case, but right now, there is a flat nesting hierarchy in the search space, i.e., eventually the search space bottoms out to basic modules and then you run that. I've thought about the nested case but haven't implemented yet.

The way to do this would be either to search over the cell, compile it to Pytorch code, and then use that, or search directly over something that consumes all time steps and does the computation you want. After the implementation of the recursive case, wrapping search spaces will be possible. I've thought about it, but haven't got around implementing it. Additionally, while it might not be super relevant here, the most recent branch is this one. Haven't updated the documentation yet though.

For the multi-layer example that you mention, I would do the same thing: stack the RNN that consume the whole input that contains all time steps and outputs the results of processing all time steps. Inside the module, you can enforce the locality of the computation by just looking at the relevant positions.

I'll check the code on Google Groups.

All the best,
Renato