What is sle_spatial? about lightweight-gan

[lucidrains]

There's a bug in my code and SLE spatial is always on 🙈🙈

[lucidrains]

SLE spatial is basically squeeze excitation, but done in the spatial dimension rather than the channel dimension

[woctezuma]

Oops.

[woctezuma]

I realize that this explains what SLE means, but not whether I should activate Spatial SLE.

lightweight-gan/lightweight_gan/lightweight_gan.py

Lines 273 to 295 in 9a48e87

	class SLE(nn.Module):
	def __init__(
	self,
	*,
	chan_in,
	chan_out
	):
	super().__init__()
	self.avg_pool = nn.AdaptiveAvgPool2d((4, 4))
	self.max_pool = nn.AdaptiveMaxPool2d((4, 4))
	chan_intermediate = chan_in // 2
	self.net = nn.Sequential(
	nn.Conv2d(chan_in * 2, chan_intermediate, 4),
	nn.LeakyReLU(0.1),
	nn.Conv2d(chan_intermediate, chan_out, 1),
	nn.Sigmoid()
	)
	def forward(self, x):
	pooled_avg = self.avg_pool(x)
	pooled_max = self.max_pool(x)
	return self.net(torch.cat((pooled_max, pooled_avg), dim = 1))

lightweight-gan/lightweight_gan/lightweight_gan.py

Lines 296 to 322 in 9a48e87

	class SpatialSLE(nn.Module):
	def __init__(self, upsample_times, num_groups = 2):
	super().__init__()
	self.num_groups = num_groups
	chan = num_groups * 2
	self.net = nn.Sequential(
	nn.Conv2d(chan, chan, 3, padding = 1),
	upsample(2 ** upsample_times),
	nn.Conv2d(chan, chan, 3, padding = 1),
	nn.LeakyReLU(0.1),
	nn.Conv2d(chan, 1, 3, padding = 1),
	nn.Sigmoid()
	)
	def forward(self, x):
	b, c, h, w = x.shape
	num_groups = self.num_groups
	mult = math.ceil(c / num_groups)
	padding = (mult - c % mult) // 2
	x_padded = F.pad(x, (0, 0, 0, 0, padding, padding))
	x = rearrange(x_padded, 'b (g c) h w -> b g c h w', g = num_groups)
	pooled_avg = x.mean(dim = 2)
	pooled_max, _ = x.max(dim = 2)
	pooled = torch.cat((pooled_avg, pooled_max), dim = 1)
	return self.net(pooled)

[bogdan-ivan]

I have seen in your thread "Optimal parameters for Google Colab" that you ended up dropping sle_spatial, have you noticed a difference in the results ?

[woctezuma]

I have seen in your thread "Optimal parameters for Google Colab" that you ended up dropping sle_spatial, have you noticed a difference in the results ?

No, I cannot talk about that. I dropped this parameter for now only because:

• I was not sure about its effect,
• I wanted to have a the simplest command-line possible, to make sure that the long run-times were not due to my choice of command-line options.

[lucidrains]

ok, sle-spatial should be fixed in the latest version

[Dok11]

There's a bug in my code and SLE spatial is always on 🙈🙈

@lucidrains why SLE spatial disabled by default?