layers

sorix.nn.layers

Linear

Linear(
    features, neurons, bias=True, init="he", device="cpu"
)

Bases: Module

Applies a linear transformation to the incoming data.

Attributes:

• W (Tensor) –

  Weights of the layer.

• b (Tensor) –

  Biases of the layer.

Examples:

layer = Linear(10, 5)
x = tensor(np.random.randn(8, 10))
y = layer(x)
print(y.shape)  # (8, 5)

Source code in sorix/nn/layers.py

def __init__(
    self, 
    features: int, 
    neurons: int,
    bias: bool = True, 
    init: str = 'he',
    device: str = 'cpu'
) -> None:
    super().__init__()
    if device == 'cuda' and not _cupy_available:
        raise Exception('Cupy is not available')

    self.device = device
    xp = self.xp

    if init not in ['he', 'xavier']:
        raise ValueError(f'Invalid initialization method: {init}. Valid methods are "he" and "xavier"')

    if init == 'he':
        self.std_dev = xp.sqrt(2.0 / features)  # He init for ReLU
    elif init == 'xavier':
        self.std_dev = xp.sqrt(2.0 / (features + neurons))  # Xavier init for tanh

    self.bias = bias
    self.W = tensor(xp.random.normal(0, self.std_dev, size=(features, neurons)), 
                    device=self.device, requires_grad=True, dtype=float32)
    self.b = tensor(xp.zeros((1, neurons)), 
                    device=self.device, requires_grad=True, dtype=float32) if self.bias else None

coef_ property

coef_

Returns weights as a flattened numpy array (Scikit-Learn parity).

intercept_ property

intercept_

Returns biases as a flattened numpy array or scalar (Scikit-Learn parity).

ReLU

ReLU()

Bases: Module

Rectified Linear Unit activation function.

Source code in sorix/nn/net.py

def __init__(self) -> None:
    super().__init__()
    self.device: str = 'cpu'
    self.training: bool = True

LeakyReLU

LeakyReLU(negative_slope=0.01)

Bases: Module

Leaky Rectified Linear Unit activation function.

Attributes:

• negative_slope (float) –

  Controls the angle of a negative slope. Default: 0.01

Source code in sorix/nn/layers.py

def __init__(self, negative_slope: float = 0.01) -> None:
    super().__init__()
    self.negative_slope = negative_slope

Sigmoid

Sigmoid()

Bases: Module

Numerically stable Sigmoid activation function.

Source code in sorix/nn/net.py

def __init__(self) -> None:
    super().__init__()
    self.device: str = 'cpu'
    self.training: bool = True

Tanh

Tanh()

Bases: Module

Hyperbolic tangent activation function.

Source code in sorix/nn/net.py

def __init__(self) -> None:
    super().__init__()
    self.device: str = 'cpu'
    self.training: bool = True

BatchNorm1d

BatchNorm1d(
    num_features, eps=1e-05, momentum=0.1, device="cpu"
)

Bases: Module

Applies Batch Normalization over a 2D input.

Source code in sorix/nn/layers.py

def __init__(
    self, 
    num_features: int, 
    eps: float = 1e-5, 
    momentum: float = 0.1, 
    device: str = 'cpu'
) -> None:
    super().__init__()
    self.device = device
    xp = self.xp

    self.gamma = tensor(xp.ones((1, num_features)), requires_grad=True, dtype=float32)
    self.beta = tensor(xp.zeros((1, num_features)), requires_grad=True, dtype=float32)

    # buffers (captured by state_dict)
    self.running_mean = tensor(xp.zeros((1, num_features)), requires_grad=False, dtype=float32)
    self.running_var = tensor(xp.ones((1, num_features)), requires_grad=False, dtype=float32)

    self.momentum = momentum
    self.eps = eps
    self.device = device

    if self.device != 'cpu':
        self.to(self.device)

Dropout

Dropout(p=0.5)

Bases: Module

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.

This implementation uses Inverted Dropout, meaning that the output is scaled by 1/(1-p) during training. This ensures that the expected value of the activations remains constant, allowing the layer to act as an identity function during inference.

Parameters:

• p (float, default: 0.5 ) –

  Probability of an element to be zeroed. Default: 0.5

Source code in sorix/nn/layers.py

def __init__(self, p: float = 0.5) -> None:
    super().__init__()
    self.p = p

Embedding

Embedding(num_embeddings, embedding_dim, device='cpu')

Bases: Module

A learnable lookup table that stores embeddings of a fixed dictionary and size.

This module is typically used to store word or categorical embeddings and retrieve them using integer indices. Maps each index to a dense vector of size embedding_dim.

Parameters:

• num_embeddings (int) –

  Size of the dictionary of embeddings (vocabulary size).

• embedding_dim (int) –

  The size of each embedding vector.

• device (str, default: 'cpu' ) –

  'cpu' or 'cuda'. Default: 'cpu'.

Attributes:

• W (Tensor) –

  The embedding weight matrix of shape (num_embeddings, embedding_dim).

Examples:

emb = nn.Embedding(num_embeddings=100, embedding_dim=16)
# Token IDs: batch of 2 sequences of length 5
indices = tensor(np.array([[1, 5, 3, 0, 7],
                           [2, 4, 1, 6, 8]]))  # shape (2, 5)
out = emb(indices)  # shape (2, 5, 16)

Source code in sorix/nn/layers.py

def __init__(
    self,
    num_embeddings: int,
    embedding_dim: int,
    device: str = "cpu",
) -> None:
    super().__init__()
    if num_embeddings < 1:
        raise ValueError(f"num_embeddings must be >= 1, got {num_embeddings}")
    if embedding_dim < 1:
        raise ValueError(f"embedding_dim must be >= 1, got {embedding_dim}")
    self.device = device
    self.num_embeddings = num_embeddings
    self.embedding_dim = embedding_dim
    xp = self.xp

    # Kaiming-uniform initialisation (same as PyTorch default for Embedding)
    bound = xp.sqrt(xp.array(1.0 / num_embeddings))
    w_data = xp.random.uniform(-float(bound), float(bound),
                               size=(num_embeddings, embedding_dim)).astype(np.float32)
    self.W = tensor(w_data, device=self.device, requires_grad=True, dtype=float32)