Top 19 Nn Modulelist All Answers

You are looking for information, articles, knowledge about the topic nail salons open on sunday near me nn modulelist on Google, you do not find the information you need! Here are the best content compiled and compiled by the https://chewathai27.com/to team, along with other related topics such as: nn modulelist Nn Linear, Nn sequential, Nn ModuleList, Torch nn, ModuleDict, Torch nn Conv2d, Nn sequential append, Torch nn parameter

8.1.2 Deeper Neural Networks : nn.ModuleList() – Deep Networks | Coursera

• Article author: www.coursera.org
• Reviews from users: 22585 Ratings
• Top rated: 3.2
• Lowest rated: 1
• Summary of article content: Articles about 8.1.2 Deeper Neural Networks : nn.ModuleList() – Deep Networks | Coursera The course will teach you how to develop deep learning models using Pytorch. The course will start with Pytorch’s tensors and Automatic differentiation … …
• Most searched keywords: Whether you are looking for 8.1.2 Deeper Neural Networks : nn.ModuleList() – Deep Networks | Coursera The course will teach you how to develop deep learning models using Pytorch. The course will start with Pytorch’s tensors and Automatic differentiation …
• Table of Contents:

812 Deeper Neural Networks nnModuleList()

Explore our Catalog

Coursera Footer

Read More

Python Examples of torch.nn.ModuleList

• Article author: www.programcreek.com
• Reviews from users: 45858 Ratings
• Top rated: 4.3
• Lowest rated: 1
• Summary of article content: Articles about Python Examples of torch.nn.ModuleList def __init__(self, in_channels, out_channels, dilations=(1, 3, 6, 1)): super().__init__() assert dilations[-1] == 1 self.aspp = nn.ModuleList() for dilation … …
• Most searched keywords: Whether you are looking for Python Examples of torch.nn.ModuleList def __init__(self, in_channels, out_channels, dilations=(1, 3, 6, 1)): super().__init__() assert dilations[-1] == 1 self.aspp = nn.ModuleList() for dilation … This page shows Python examples of torch.nn.ModuleListPython Examples of torch.nn.ModuleList
• Table of Contents:

Read More

GitHub – FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict: Code for my medium article

• Article author: github.com
• Reviews from users: 36224 Ratings
• Top rated: 3.5
• Lowest rated: 1
• Summary of article content: Articles about GitHub – FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict: Code for my medium article We are going to start with an example and iteratively we will make it better. All these four es are contained into torch.nn. import torch.nn as nn … …
• Most searched keywords: Whether you are looking for GitHub – FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict: Code for my medium article We are going to start with an example and iteratively we will make it better. All these four es are contained into torch.nn. import torch.nn as nn … Code for my medium article. Contribute to FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict development by creating an account on GitHub.
• Table of Contents:

Latest commit

Git stats

Files

READMEmd

Module the main building block

Sequential stack and merge layers

Dynamic Sequential create multiple layers at once

ModuleList when we need to iterate

ModuleDict when we need to choose

Final implementation

Conclusion

About

Releases

Packages 0

Languages

Footer

Read More

ModuleList and Sequential in PyTorch: differences and usage scenarios

• Article author: programming.vip
• Reviews from users: 26229 Ratings
• Top rated: 3.9
• Lowest rated: 1
• Summary of article content: Articles about ModuleList and Sequential in PyTorch: differences and usage scenarios First, let’s talk about NN Modulelist this , you can put any NN Subes of module (such as nn.Conv2d, nn.Linear, etc.) are added to this … …
• Most searched keywords: Whether you are looking for ModuleList and Sequential in PyTorch: differences and usage scenarios First, let’s talk about NN Modulelist this , you can put any NN Subes of module (such as nn.Conv2d, nn.Linear, etc.) are added to this … PytorchPyTorch has some basic concepts that are important when building networks, such as NN Module, nn.ModuleList, nn.Sequential, these classes are called containers because we can add modules to them. These containers are easily confused. In this article, we mainly study NN Modulelist and NN SequentUTF-8…
• Table of Contents:

Read More

ModuleList — PyTorch master documentation

• Article author: 49.235.228.196
• Reviews from users: 25776 Ratings
• Top rated: 3.6
• Lowest rated: 1
• Summary of article content: Articles about ModuleList — PyTorch master documentation ModuleList. torch.nn. ModuleList (modules: Optional[Iterable[torch.nn.modules.module.Module]] = None)[source]. Holds submodules in a list. …
• Most searched keywords: Whether you are looking for ModuleList — PyTorch master documentation ModuleList. torch.nn. ModuleList (modules: Optional[Iterable[torch.nn.modules.module.Module]] = None)[source]. Holds submodules in a list.
• Table of Contents:

Docs

Tutorials

Resources

Read More

See more articles in the same category here: Chewathai27.com/to/blog.

FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict: Code for my medium article

Pytorch: how and when to use Module, Sequential, ModuleList and ModuleDict

Effective way to share, reuse and break down the complexity of your models

Updated at Pytorch 1.5

You can find the code here

Pytorch is an open source deep learning frameworks that provide a smart way to create ML models. Even if the documentation is well made, I still see that most people don’t write well and organized code in PyTorch.

Today, we are going to see how to use the three main building blocks of PyTorch: Module, Sequential and ModuleList . We are going to start with an example and iteratively we will make it better.

All these four classes are contained into torch.nn

import torch . nn as nn # nn.Module # nn.Sequential # nn.Module

Module: the main building block

The Module is the main building block, it defines the base class for all neural network and you MUST subclass it.

Let’s create a classic CNN classifier as example:

import torch . nn . functional as F class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , n_classes ): super (). __init__ () self . conv1 = nn . Conv2d ( in_c , 32 , kernel_size = 3 , stride = 1 , padding = 1 ) self . bn1 = nn . BatchNorm2d ( 32 ) self . conv2 = nn . Conv2d ( 32 , 64 , kernel_size = 3 , stride = 1 , padding = 1 ) self . bn2 = nn . BatchNorm2d ( 64 ) self . fc1 = nn . Linear ( 64 * 28 * 28 , 1024 ) self . fc2 = nn . Linear ( 1024 , n_classes ) def forward ( self , x ): x = self . conv1 ( x ) x = self . bn1 ( x ) x = F . relu ( x ) x = self . conv2 ( x ) x = self . bn2 ( x ) x = F . relu ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . fc1 ( x ) x = F . sigmoid ( x ) x = self . fc2 ( x ) return x

model = MyCNNClassifier ( 1 , 10 ) print ( model )

MyCNNClassifier( (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (fc1): Linear(in_features=50176, out_features=1024, bias=True) (fc2): Linear(in_features=1024, out_features=10, bias=True) )

This is a very simple classifier with an encoding part that uses two layers with 3×3 convs + batchnorm + relu and a decoding part with two linear layers. If you are not new to PyTorch you may have seen this type of coding before, but there are two problems.

If we want to add a layer we have to again write lots of code in the __init__ and in the forward function. Also, if we have some common block that we want to use in another model, e.g. the 3×3 conv + batchnorm + relu, we have to write it again.

Sequential: stack and merge layers

Sequential is a container of Modules that can be stacked together and run at the same time.

You can notice that we have to store into self everything. We can use Sequential to improve our code.

class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , n_classes ): super (). __init__ () self . conv_block1 = nn . Sequential ( nn . Conv2d ( in_c , 32 , kernel_size = 3 , stride = 1 , padding = 1 ), nn . BatchNorm2d ( 32 ), nn . ReLU () ) self . conv_block2 = nn . Sequential ( nn . Conv2d ( 32 , 64 , kernel_size = 3 , stride = 1 , padding = 1 ), nn . BatchNorm2d ( 64 ), nn . ReLU () ) self . decoder = nn . Sequential ( nn . Linear ( 64 * 28 * 28 , 1024 ), nn . Sigmoid (), nn . Linear ( 1024 , n_classes ) ) def forward ( self , x ): x = self . conv_block1 ( x ) x = self . conv_block2 ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , 10 ) print ( model )

MyCNNClassifier( (conv_block1): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv_block2): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (decoder): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() (2): Linear(in_features=1024, out_features=10, bias=True) ) )

Much Better uhu?

Did you notice that conv_block1 and conv_block2 looks almost the same? We could create a function that reteurns a nn.Sequential to even simplify the code!

def conv_block ( in_f , out_f , * args , ** kwargs ): return nn . Sequential ( nn . Conv2d ( in_f , out_f , * args , ** kwargs ), nn . BatchNorm2d ( out_f ), nn . ReLU () )

Then we can just call this function in our Module

class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , n_classes ): super (). __init__ () self . conv_block1 = conv_block ( in_c , 32 , kernel_size = 3 , padding = 1 ) self . conv_block2 = conv_block ( 32 , 64 , kernel_size = 3 , padding = 1 ) self . decoder = nn . Sequential ( nn . Linear ( 64 * 28 * 28 , 1024 ), nn . Sigmoid (), nn . Linear ( 1024 , n_classes ) ) def forward ( self , x ): x = self . conv_block1 ( x ) x = self . conv_block2 ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , 10 ) print ( model )

MyCNNClassifier( (conv_block1): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv_block2): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (decoder): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() (2): Linear(in_features=1024, out_features=10, bias=True) ) )

Even cleaner! Still conv_block1 and conv_block2 are almost the same! We can merge them using nn.Sequential

class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , n_classes ): super (). __init__ () self . encoder = nn . Sequential ( conv_block ( in_c , 32 , kernel_size = 3 , padding = 1 ), conv_block ( 32 , 64 , kernel_size = 3 , padding = 1 ) ) self . decoder = nn . Sequential ( nn . Linear ( 64 * 28 * 28 , 1024 ), nn . Sigmoid (), nn . Linear ( 1024 , n_classes ) ) def forward ( self , x ): x = self . encoder ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , 10 ) print ( model )

MyCNNClassifier( (encoder): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (decoder): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() (2): Linear(in_features=1024, out_features=10, bias=True) ) )

self.encoder now holds booth conv_block . We have decoupled logic for our model and make it easier to read and reuse. Our conv_block function can be imported and used in another model.

Dynamic Sequential: create multiple layers at once

What if we can to add a new layers in self.encoder , hardcoded them is not convinient:

self . encoder = nn . Sequential ( conv_block ( in_c , 32 , kernel_size = 3 , padding = 1 ), conv_block ( 32 , 64 , kernel_size = 3 , padding = 1 ), conv_block ( 64 , 128 , kernel_size = 3 , padding = 1 ), conv_block ( 128 , 256 , kernel_size = 3 , padding = 1 ), )

Would it be nice if we can define the sizes as an array and automatically create all the layers without writing each one of them? Fortunately we can create an array and pass it to Sequential

class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , n_classes ): super (). __init__ () self . enc_sizes = [ in_c , 32 , 64 ] conv_blocks = [ conv_block ( in_f , out_f , kernel_size = 3 , padding = 1 ) for in_f , out_f in zip ( self . enc_sizes , self . enc_sizes [ 1 :])] self . encoder = nn . Sequential ( * conv_blocks ) self . decoder = nn . Sequential ( nn . Linear ( 64 * 28 * 28 , 1024 ), nn . Sigmoid (), nn . Linear ( 1024 , n_classes ) ) def forward ( self , x ): x = self . encoder ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , 10 ) print ( model )

MyCNNClassifier( (encoder): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (decoder): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() (2): Linear(in_features=1024, out_features=10, bias=True) ) )

Let’s break it down. We created an array self.enc_sizes that holds the sizes of our encoder. Then we create an array conv_blocks by iterating the sizes. Since we have to give booth a in size and an outsize for each layer we zip ed the size’array with itself by shifting it by one.

Just to be clear, take a look at the following example:

sizes = [ 1 , 32 , 64 ] for in_f , out_f in zip ( sizes , sizes [ 1 :]): print ( in_f , out_f )

1 32 32 64

Then, since Sequential does not accept a list, we decompose it by using the * operator.

Tada! Now if we just want to add a size, we can easily add a new number to the list. It is a common practice to make the size a parameter.

class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , enc_sizes , n_classes ): super (). __init__ () self . enc_sizes = [ in_c , * enc_sizes ] conv_blocks = [ conv_block ( in_f , out_f , kernel_size = 3 , padding = 1 ) for in_f , out_f in zip ( self . enc_sizes , self . enc_sizes [ 1 :])] self . encoder = nn . Sequential ( * conv_blocks ) self . decoder = nn . Sequential ( nn . Linear ( 64 * 28 * 28 , 1024 ), nn . Sigmoid (), nn . Linear ( 1024 , n_classes ) ) def forward ( self , x ): x = self . encoder ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , [ 32 , 64 , 128 ], 10 ) print ( model )

MyCNNClassifier( (encoder): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (2): Sequential( (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (decoder): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() (2): Linear(in_features=1024, out_features=10, bias=True) ) )

We can do the same for the decoder part

def dec_block ( in_f , out_f ): return nn . Sequential ( nn . Linear ( in_f , out_f ), nn . Sigmoid () ) class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , enc_sizes , dec_sizes , n_classes ): super (). __init__ () self . enc_sizes = [ in_c , * enc_sizes ] self . dec_sizes = [ 64 * 28 * 28 , * dec_sizes ] conv_blocks = [ conv_block ( in_f , out_f , kernel_size = 3 , padding = 1 ) for in_f , out_f in zip ( self . enc_sizes , self . enc_sizes [ 1 :])] self . encoder = nn . Sequential ( * conv_blocks ) dec_blocks = [ dec_block ( in_f , out_f ) for in_f , out_f in zip ( self . dec_sizes , self . dec_sizes [ 1 :])] self . decoder = nn . Sequential ( * dec_blocks ) self . last = nn . Linear ( self . dec_sizes [ – 1 ], n_classes ) def forward ( self , x ): x = self . encoder ( x ) x = x . view ( x . size ( 0 ), – 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , [ 32 , 64 ], [ 1024 , 512 ], 10 ) print ( model )

MyCNNClassifier( (encoder): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (decoder): Sequential( (0): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() ) (1): Sequential( (0): Linear(in_features=1024, out_features=512, bias=True) (1): Sigmoid() ) ) (last): Linear(in_features=512, out_features=10, bias=True) )

We followed the same pattern, we create a new block for the decoding part, linear + sigmoid, and we pass an array with the sizes. We had to add a self.last since we do not want to activate the output

Now, we can even break down our model in two! Encoder + Decoder

class MyEncoder ( nn . Module ): def __init__ ( self , enc_sizes ): super (). __init__ () self . conv_blocks = nn . Sequential ( * [ conv_block ( in_f , out_f , kernel_size = 3 , padding = 1 ) for in_f , out_f in zip ( enc_sizes , enc_sizes [ 1 :])]) def forward ( self , x ): return self . conv_blocks ( x ) class MyDecoder ( nn . Module ): def __init__ ( self , dec_sizes , n_classes ): super (). __init__ () self . dec_blocks = nn . Sequential ( * [ dec_block ( in_f , out_f ) for in_f , out_f in zip ( dec_sizes , dec_sizes [ 1 :])]) self . last = nn . Linear ( dec_sizes [ – 1 ], n_classes ) def forward ( self , x ): return self . dec_blocks () class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , enc_sizes , dec_sizes , n_classes ): super (). __init__ () self . enc_sizes = [ in_c , * enc_sizes ] self . dec_sizes = [ self . enc_sizes [ – 1 ] * 28 * 28 , * dec_sizes ] self . encoder = MyEncoder ( self . enc_sizes ) self . decoder = MyDecoder ( self . dec_sizes , n_classes ) def forward ( self , x ): x = self . encoder ( x ) x = x . flatten ( 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , [ 32 , 64 ], [ 1024 , 512 ], 10 ) print ( model )

MyCNNClassifier( (encoder): MyEncoder( (conv_blocks): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (decoder): MyDecoder( (dec_blocks): Sequential( (0): Sequential( (0): Linear(in_features=50176, out_features=1024, bias=True) (1): Sigmoid() ) (1): Sequential( (0): Linear(in_features=1024, out_features=512, bias=True) (1): Sigmoid() ) ) (last): Linear(in_features=512, out_features=10, bias=True) ) )

Be aware that MyEncoder and MyDecoder could also be functions that returns a nn.Sequential . I prefer to use the first pattern for models and the second for building blocks.

By diving our module into submodules it is easier to share the code, debug it and test it.

ModuleList : when we need to iterate

ModuleList allows you to store Module as a list. It can be useful when you need to iterate through layer and store/use some information, like in U-net.

The main difference between Sequential is that ModuleList have not a forward method so the inner layers are not connected. Assuming we need each output of each layer in the decoder, we can store it by:

class MyModule ( nn . Module ): def __init__ ( self , sizes ): super (). __init__ () self . layers = nn . ModuleList ([ nn . Linear ( in_f , out_f ) for in_f , out_f in zip ( sizes , sizes [ 1 :])]) self . trace = [] def forward ( self , x ): for layer in self . layers : x = layer ( x ) self . trace . append ( x ) return x

model = MyModule ([ 1 , 16 , 32 ]) import torch model ( torch . rand (( 4 , 1 ))) [ print ( trace . shape ) for trace in model . trace ]

torch.Size([4, 16]) torch.Size([4, 32]) [None, None]

ModuleDict: when we need to choose

What if we want to switch to LearkyRelu in our conv_block ? We can use ModuleDict to create a dictionary of Module and dynamically switch Module when we want

def conv_block ( in_f , out_f , activation = ‘relu’ , * args , ** kwargs ): activations = nn . ModuleDict ([ [ ‘lrelu’ , nn . LeakyReLU ()], [ ‘relu’ , nn . ReLU ()] ]) return nn . Sequential ( nn . Conv2d ( in_f , out_f , * args , ** kwargs ), nn . BatchNorm2d ( out_f ), activations [ activation ] )

print ( conv_block ( 1 , 32 , ‘lrelu’ , kernel_size = 3 , padding = 1 )) print ( conv_block ( 1 , 32 , ‘relu’ , kernel_size = 3 , padding = 1 ))

Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.01) ) Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() )

Final implementation

Let’s wrap it up everything!

def conv_block ( in_f , out_f , activation = ‘relu’ , * args , ** kwargs ): activations = nn . ModuleDict ([ [ ‘lrelu’ , nn . LeakyReLU ()], [ ‘relu’ , nn . ReLU ()] ]) return nn . Sequential ( nn . Conv2d ( in_f , out_f , * args , ** kwargs ), nn . BatchNorm2d ( out_f ), activations [ activation ] ) def dec_block ( in_f , out_f ): return nn . Sequential ( nn . Linear ( in_f , out_f ), nn . Sigmoid () ) class MyEncoder ( nn . Module ): def __init__ ( self , enc_sizes , * args , ** kwargs ): super (). __init__ () self . conv_blocks = nn . Sequential ( * [ conv_block ( in_f , out_f , kernel_size = 3 , padding = 1 , * args , ** kwargs ) for in_f , out_f in zip ( enc_sizes , enc_sizes [ 1 :])]) def forward ( self , x ): return self . conv_blocks ( x ) class MyDecoder ( nn . Module ): def __init__ ( self , dec_sizes , n_classes ): super (). __init__ () self . dec_blocks = nn . Sequential ( * [ dec_block ( in_f , out_f ) for in_f , out_f in zip ( dec_sizes , dec_sizes [ 1 :])]) self . last = nn . Linear ( dec_sizes [ – 1 ], n_classes ) def forward ( self , x ): return self . dec_blocks () class MyCNNClassifier ( nn . Module ): def __init__ ( self , in_c , enc_sizes , dec_sizes , n_classes , activation = ‘relu’ ): super (). __init__ () self . enc_sizes = [ in_c , * enc_sizes ] self . dec_sizes = [ 32 * 28 * 28 , * dec_sizes ] self . encoder = MyEncoder ( self . enc_sizes , activation = activation ) self . decoder = MyDecoder ( dec_sizes , n_classes ) def forward ( self , x ): x = self . encoder ( x ) x = x . flatten ( 1 ) # flat x = self . decoder ( x ) return x

model = MyCNNClassifier ( 1 , [ 32 , 64 ], [ 1024 , 512 ], 10 , activation = ‘lrelu’ ) print ( model )

MyCNNClassifier( (encoder): MyEncoder( (conv_blocks): Sequential( (0): Sequential( (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.01) ) (1): Sequential( (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.01) ) ) ) (decoder): MyDecoder( (dec_blocks): Sequential( (0): Sequential( (0): Linear(in_features=1024, out_features=512, bias=True) (1): Sigmoid() ) ) (last): Linear(in_features=512, out_features=10, bias=True) ) )

Conclusion

So, in summary.

Use Module when you have a big block compose of multiple smaller blocks

when you have a big block compose of multiple smaller blocks Use Sequential when you want to create a small block from layers

when you want to create a small block from layers Use ModuleList when you need to iterate through some layers or building blocks and do something

when you need to iterate through some layers or building blocks and do something Use ModuleDict when you need to parametise some blocks of your model, for example an activation function

That’s all folks!

Thank you for reading

ModuleList and Sequential in PyTorch: differences and usage scenarios

PyTorch has some basic concepts that are important when building networks, such as NN Module, nn.ModuleList, nn.Sequential, these classes are called containers because we can add modules to them. These containers are easily confused. In this article, we mainly study NN Modulelist and NN Sequential, and judge when to use which one is more appropriate. The example in this article uses PyTorch version 1.0.

nn.ModuleList

First, let’s talk about NN Modulelist this class, you can put any NN Subclasses of module (such as nn.Conv2d, nn.Linear, etc.) are added to this list. The method is the same as Python’s own list, which is nothing more than extend, append and other operations. But different from the general list, add to NN The module in the modulelist will be registered in the whole network, and the parameters of the module will be automatically added to the whole network. The description looks boring. Let’s look at a few examples.

The first network, let’s take a look at using NN Modulelist to build a small network, including three full connection layers:

class net1(nn.Module): def __init__(self): super(net1, self).__init__() self.linears = nn.ModuleList([nn.Linear(10,10) for i in range(2)]) def forward(self, x): for m in self.linears: x = m(x) return x net = net1() print(net) # net1( # (modules): ModuleList( # (0): Linear(in_features=10, out_features=10, bias=True) # (1): Linear(in_features=10, out_features=10, bias=True) # ) # ) for param in net.parameters(): print(type(param.data), param.size()) # torch.Size([10, 10]) # torch.Size([10]) # torch.Size([10, 10]) # torch.Size([10])

We can see that this network consists of two full connection layers, and their weight (weithgs) and bias (bias) are within this network. Next, let’s look at the second network, which uses Python’s own list:

class net2(nn.Module): def __init__(self): super(net2, self).__init__() self.linears = [nn.Linear(10,10) for i in range(2)] def forward(self, x): for m in self.linears: x = m(x) return x net = net2() print(net) # net2() print(list(net.parameters())) # []

Obviously, the full connection layers and their parameters added using Python’s list are not automatically registered in our network. Of course, we can still use forward to calculate the output. However, if the network instantiated by net2 is used for training, because the parameters of these layers are not in the whole network, the network parameters will not be updated.

OK, see here, we generally understand NN What does modulelist do: it is a container that stores different modules and automatically adds the parameters of each module to the network. However, we need to note that NN Modulelist does not define a network. It just stores different modules together. There is no order between these modules, such as:

class net3(nn.Module): def __init__(self): super(net3, self).__init__() self.linears = nn.ModuleList([nn.Linear(10,20), nn.Linear(20,30), nn.Linear(5,10)]) def forward(self, x): x = self.linears[2](x) x = self.linears[0](x) x = self.linears[1](x) return x net = net3() print(net) # net3( # (linears): ModuleList( # (0): Linear(in_features=10, out_features=20, bias=True) # (1): Linear(in_features=20, out_features=30, bias=True) # (2): Linear(in_features=5, out_features=10, bias=True) # ) # ) input = torch.randn(32, 5) print(net(input).shape) # torch.Size([32, 30])

According to the results of net3, we can see that the order in the ModuleList does not determine anything. The execution order of the network is determined according to the forward function. If you insist that the order of ModuleList and forward is different, PyTorch means it doesn’t matter, but people who review your code may have a big opinion in the future.

Let’s consider another case. Since the ModuleList can be called according to the sequence number, can a module be called multiple times in the forward function? Of course, the answer is yes, but the modules called multiple times use the same set of parameters, that is, their parameters are exactly the same, no matter how you update them later. The example is as follows, although we use NN in forward Linear (10,10) twice, but they have only one set of parameters. What’s the use of doing this? I don’t think of it at present

class net4(nn.Module): def __init__(self): super(net4, self).__init__() self.linears = nn.ModuleList([nn.Linear(5, 10), nn.Linear(10, 10)]) def forward(self, x): x = self.linears[0](x) x = self.linears[1](x) x = self.linears[1](x) return x net = net4() print(net) # net4( # (linears): ModuleList( # (0): Linear(in_features=5, out_features=10, bias=True) # (1): Linear(in_features=10, out_features=10, bias=True) # ) # ) for name, param in net.named_parameters(): print(name, param.size()) # linears.0.weight torch.Size([10, 5]) # linears.0.bias torch.Size([10]) # linears.1.weight torch.Size([10, 10]) # linears.1.bias torch.Size([10])

nn.Sequential

Now let’s study NN Sequential, different from NN Modulelist, which has implemented the forward function, and the modules in it are arranged in order, so we must ensure that the output size of the previous module is consistent with the input size of the next module, as shown in the following example:

class net5(nn.Module): def __init__(self): super(net5, self).__init__() self.block = nn.Sequential(nn.Conv2d(1,20,5), nn.ReLU(), nn.Conv2d(20,64,5), nn.ReLU()) def forward(self, x): x = self.block(x) return x net = net5() print(net) # net5( # (block): Sequential( # (0): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) # (1): ReLU() # (2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) # (3): ReLU() # ) # )

Here are two initialization examples from the official website tutorial. In the second initialization, we use OrderedDict to specify the name of each module instead of the default naming method (by sequence number 0,1,2,3…).

# Example of using Sequential model1 = nn.Sequential( nn.Conv2d(1,20,5), nn.ReLU(), nn.Conv2d(20,64,5), nn.ReLU() ) print(model1) # Sequential( # (0): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) # (1): ReLU() # (2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) # (3): ReLU() # ) # Example of using Sequential with OrderedDict import collections model2 = nn.Sequential(collections.OrderedDict([ (‘conv1’, nn.Conv2d(1,20,5)), (‘relu1’, nn.ReLU()), (‘conv2’, nn.Conv2d(20,64,5)), (‘relu2’, nn.ReLU()) ])) print(model2) # Sequential( # (conv1): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) # (relu1): ReLU() # (conv2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) # (relu2): ReLU() # )

Students may have found that, eh, is there any difference between your model1 and net instantiated from class net5? No. The two networks are the same because NN Sequential is a NN A subclass of module, that is, NN Module has all methods. And directly use NN Sequential does not need to write the forward function, because it has been written internally for you.

At this time, some students should say, since NN Sequential is so good that I will use it directly in the future. If you’re sure NN The order in sequential is what you want, and you don’t need to add some other processing functions (such as the function in nn.functional, what’s the difference between NN and nn.functional?), Then you can directly use NN Sequential. The cost of doing so is to lose some flexibility. After all, you can’t customize the contents of the forward function yourself.

In general, NN Sequential is used to form convolution blocks, and then assemble different blocks into the whole network like building blocks, making the code more concise and structured.

nn.ModuleList and NN Sequential: which one should I use

We have briefly introduced these two classes. Now let’s discuss which one is more appropriate in two different scenarios.

Scenario 1: sometimes there are many similar or repeated layers in the network. We will generally consider creating them with a for loop, such as:

class net6(nn.Module): def __init__(self): super(net6, self).__init__() self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(3)]) def forward(self, x): for layer in self.linears: x = layer(x) return x net = net6() print(net) # net6( # (linears): ModuleList( # (0): Linear(in_features=10, out_features=10, bias=True) # (1): Linear(in_features=10, out_features=10, bias=True) # (2): Linear(in_features=10, out_features=10, bias=True) # ) # )

This is a general method, but if we don’t want to be so troublesome, we can also use Sequential, as shown in net7! Note the * operator, which can split a list into independent elements. So in scenario 1, I personally think it is more convenient and tidy to use net7

class net7(nn.Module): def __init__(self): super(net7, self).__init__() self.linear_list = [nn.Linear(10, 10) for i in range(3)] self.linears = nn.Sequential(*self.linears_list) def forward(self, x): self.x = self.linears(x) return x net = net7() print(net) # net7( # (linears): Sequential( # (0): Linear(in_features=10, out_features=10, bias=True) # (1): Linear(in_features=10, out_features=10, bias=True) # (2): Linear(in_features=10, out_features=10, bias=True) # ) # )

Let’s consider scenario 2. When we need the information of the previous layer, such as the shortcut structure in ResNets or the skip architecture used in FCN, the results of the current layer need to be integrated with the results of the previous layer. Generally, it is more convenient to use ModuleList. A very simple example is as follows:

class net8(nn.Module): def __init__(self): super(net8, self).__init__() self.linears = nn.ModuleList([nn.Linear(10, 20), nn.Linear(20, 30), nn.Linear(30, 50)]) self.trace = [] def forward(self, x): for layer in self.linears: x = layer(x) self.trace.append(x) return x net = net8() input = torch.randn(32, 10) output = net(input) for each in net.trace: print(each.shape) # torch.Size([32, 20]) # torch.Size([32, 30]) # torch.Size([32, 50])

We use a trace list to store the output results of each layer of the network, so that it can be easily called if the later layer needs to be used.

summary

In this article, we learned the two nn containers of ModuleList and Sequential through some examples. ModuleList is a list that stores various modules. These modules have no connection and do not realize the forward function. However, compared with ordinary Python list, ModuleList can automatically register the modules and parameters added to it on the network. The modules in Sequential need to be arranged in order. To ensure that the input and output sizes of adjacent layers match, the internal forward function has been realized, which can make the code cleaner. In different scenarios, if both are applicable, it depends on personal preferences. It is highly recommended that you look at the model implementation code under PyTorch’s official TorchVision, and you can learn a lot of network construction skills.

So you have finished reading the nn modulelist topic article, if you find this article useful, please share it. Thank you very much. See more: Nn Linear, Nn sequential, Nn ModuleList, Torch nn, ModuleDict, Torch nn Conv2d, Nn sequential append, Torch nn parameter