EncoderBlock Class Reference

Inheritance diagram for EncoderBlock:

Public Member Functions
	__init__ (self, input_dim, num_heads, dim_feedforward, dropout, which_linear)
	Constructor.
	forward (self, x)
	forward

Public Attributes
	which_linear = which_linear
	which linear
	self_attn
	Attention layer.
	linear_net
	Two-layer MLP.
	norm1 = nn.LayerNorm(input_dim)
	norm1
	norm2 = nn.LayerNorm(input_dim)
	norm2
	dropout = nn.Dropout(dropout)
	dropout

Detailed Description

EncoderBlock

Definition at line 933 of file ieagan.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
			input_dim,
			num_heads,
			dim_feedforward,
			dropout,
			which_linear )

Constructor.

Inputs:
    input_dim - Dimensionality of the input
    num_heads - Number of heads to use in the attention block
    dim_feedforward - Dimensionality of the hidden layer in the MLP
    dropout - Dropout probability to use in the dropout layers

Definition at line 937 of file ieagan.py.

    def __init__(self, input_dim, num_heads, dim_feedforward, dropout, which_linear):
        """
        Inputs:
            input_dim - Dimensionality of the input
            num_heads - Number of heads to use in the attention block
            dim_feedforward - Dimensionality of the hidden layer in the MLP
            dropout - Dropout probability to use in the dropout layers
        """
        super().__init__()
 
        
        self.which_linear = which_linear
        
        self.self_attn = MultiheadAttention(
            input_dim, input_dim, num_heads, which_linear
        )
 
        
        self.linear_net = nn.Sequential(
            self.which_linear(input_dim, dim_feedforward),
            nn.Dropout(dropout),
            nn.ReLU(inplace=True),
            self.which_linear(dim_feedforward, input_dim),
        )
 
        # Layers to apply in between the main layers
        
        self.norm1 = nn.LayerNorm(input_dim)
        
        self.norm2 = nn.LayerNorm(input_dim)
        
        self.dropout = nn.Dropout(dropout)
 

Member Function Documentation

forward()

forward	(		self,
			x )

forward

Definition at line 971 of file ieagan.py.

    def forward(self, x):
        # Attention part
        x_pre1 = self.norm1(x)
        attn_out = self.self_attn(x_pre1)
        x = x + self.dropout(attn_out)
        # x = self.norm1(x)
 
        # MLP part
        x_pre2 = self.norm2(x)
        linear_out = self.linear_net(x_pre2)
        x = x + self.dropout(linear_out)
        # x = self.norm2(x)
 
        return x
 
 

Member Data Documentation

dropout

dropout = nn.Dropout(dropout)

dropout

Definition at line 968 of file ieagan.py.

linear_net

linear_net

Initial value:

=  nn.Sequential(
            self.which_linear(input_dim, dim_feedforward),
            nn.Dropout(dropout),
            nn.ReLU(inplace=True),
            self.which_linear(dim_feedforward, input_dim),
        )

Two-layer MLP.

Definition at line 955 of file ieagan.py.

norm1

norm1 = nn.LayerNorm(input_dim)

norm1

Definition at line 964 of file ieagan.py.

norm2

norm2 = nn.LayerNorm(input_dim)

norm2

Definition at line 966 of file ieagan.py.

self_attn

self_attn

Initial value:

=  MultiheadAttention(
            input_dim, input_dim, num_heads, which_linear
        )

Attention layer.

Definition at line 950 of file ieagan.py.

which_linear

which_linear = which_linear

which linear

Definition at line 948 of file ieagan.py.

---

The documentation for this class was generated from the following file:

• pxd/scripts/pxd/background_generator/models/ieagan.py