unit 2 ss

1. Extreme Learning Machines (ELMs) Neural Network

Extreme Learning Machines (ELMs) are a type of neural network mainly used for classification and regression tasks. What makes ELMs unique is that they are extremely fast to train compared to traditional neural networks. The reason for this is that ELMs randomly assign the input weights and biases of the hidden neurons and never update them again. Only the weights connecting the hidden layer to the output layer are trained.

In a typical neural network, training can take hours or days depending on the data size and complexity. But in ELMs, the training happens in seconds or minutes. This is possible because solving for the output weights becomes a mathematical problem that can be done using simple matrix operations.

Structure-wise, ELMs consist of three layers: an input layer, a hidden layer (with randomly initialized neurons), and an output layer. Activation functions such as sigmoid, sine, or radial basis functions are used in the hidden layer.

For example, if we want to build a system that classifies emails as spam or not spam, we can train an ELM on historical email data. Because it doesn't need to adjust weights repeatedly, the model can be trained in seconds, giving us a ready-to-use spam filter quickly.

However, the performance of ELM depends on how well the initial random weights perform. If not chosen properly, ELM might give poor accuracy, and increasing the number of neurons too much may lead to overfitting.

In summary, ELMs provide a balance between speed and accuracy. They are best suited for situations where we need fast training, like real-time systems, or when computational power is limited.

2. Advantages & Disadvantages of Spiking Neural Networks (SNNs) / Architecture and Features

Spiking Neural Networks (SNNs) are the third generation of neural networks. Unlike traditional networks that work on continuous values, SNNs communicate using discrete events called spikes. This is similar to how biological neurons work in the human brain, which send electrical spikes across synapses.

Architecture:
SNNs have neurons and synapses like other neural networks, but instead of sending real-number outputs, they send spikes. These spikes are generated only when the neuron’s internal state (called membrane potential) crosses a certain threshold. After firing a spike, the neuron resets and waits for the next trigger.

Key Features:

1. Temporal Data Processing: SNNs process inputs based on time. Spike timing plays a major role in how information is transmitted.

2. Energy Efficiency: Since neurons only spike when necessary, the system saves energy, which is especially useful in mobile or embedded systems.

3. Biologically Inspired: Their design closely resembles the human brain and nervous system, making them suitable for brain-machine interfaces.

Advantages:

• Very low power consumption, ideal for real-time systems and battery-powered devices.

• Can process time-based data better than standard networks.

• Have the potential to support intelligent systems like robots that respond to environmental changes dynamically.

Disadvantages:

• Difficult to train due to the absence of gradients in spikes, which makes backpropagation hard.

• Limited software and hardware support. Specialized neuromorphic chips are often required.

• Still under research and development, so not yet mainstream in commercial applications.

Example: In a smart wearable device, SNNs can be used to process EEG brain signals for detecting emotions in real time using minimal battery.

3. Boltzmann: Architecture, Cauchy Machine and Functioning

A Boltzmann Machine is a type of neural network that is used to discover patterns and learn internal representations of data. It is based on a stochastic or random process, meaning it does not give the same output every time. It is named after Ludwig Boltzmann, the scientist known for his work in statistical mechanics.

Architecture:
It consists of two types of nodes: visible units (input data) and hidden units (learned features). Unlike typical feedforward networks, Boltzmann Machines are fully connected but symmetrical—each node is connected to every other node, except itself, and the connections are bidirectional.

How it Works:

• The network tries to minimize an energy function, where low energy corresponds to more probable states.

• During training, the machine learns the weights of the connections such that the probability of training data increases.

• It uses a technique called "Gibbs Sampling" to gradually update the weights.

Cauchy Machine:
This is a variation of the Boltzmann Machine where instead of the traditional Gaussian distribution used for weight updates, a Cauchy distribution is used. The Cauchy function has heavier tails, which means it can explore wider parts of the solution space. This is helpful in avoiding local minima during training and improves learning especially when there are outliers or noise in the data.

Functioning Example:
Suppose we want to build a recommendation system. A Boltzmann Machine can learn from a user’s movie preferences and detect hidden patterns such as "if a user likes movies A and B, they’re likely to enjoy movie C." Once trained, it can recommend movies to similar users by sampling from the learned distribution.

In short, Boltzmann Machines are powerful tools for learning deep representations but are slow to train and require complex computation. They were an early foundation for later models like Deep Belief Networks.

4. How PNNs and SNNs Contribute Towards Classification Tasks

Both Probabilistic Neural Networks (PNNs) and Spiking Neural Networks (SNNs) are used in classification, but they work in different ways.

PNNs:

• Based on statistical principles, especially Bayesian classification.

• They estimate the probability that an input belongs to a certain class based on training data.

• They use radial basis functions and store all training samples. During classification, they compare new input data with all stored examples.

• They are fast at classifying once trained and are very accurate if the training set is clean and well-structured.

Example: If we have a dataset of different flower types based on petal length and width, PNNs will calculate the probability of each new flower data point belonging to a known class and pick the highest.

SNNs:

• Use spikes to process data over time.

• Useful in real-time and sensor-based systems where inputs are events over time (e.g., sound signals, gestures).

• They can learn patterns in the timing of input signals.

• Better suited for data where timing and order matter.

Example: In gesture recognition using wearable sensors, the order and timing of muscle movements are important. SNNs can learn and classify gestures based on those spike sequences.

Comparison:

• PNNs are better for static data where accuracy and speed are important.

• SNNs are better for dynamic and temporal data where time plays a major role.

5. Principle of Simulated Annealing and Role in Optimization

Simulated Annealing is a method used to solve optimization problems by mimicking the way metal cools down and solidifies to become more stable. This idea comes from metallurgy, where slow cooling reduces internal stress in the material.

How it works:

1. The algorithm starts with a random solution.

2. It tries slightly different solutions by making small changes.

3. If the new solution is better, it accepts it.

4. If it's worse, it still might accept it with some probability, especially in the beginning.

5. Over time, the system "cools down" by reducing the probability of accepting worse solutions.

This process helps the algorithm escape local optima and search for the global best solution. The “temperature” is a control factor that slowly decreases during the process.

Role in Optimization:

• Useful in large search spaces where there are many local optima.

• Works well when traditional methods like gradient descent fail.

• Often used in feature selection, neural network training, scheduling problems, etc.

Example: Suppose we want to schedule classes in a college without overlapping rooms and professors. There are many possible schedules, but only one or two optimal ones. Simulated Annealing can explore different schedules and slowly settle on the best one.

It is slow but reliable in finding good solutions to hard problems.

6. Role of Convolutional Neural Network in Image Recognition. How is it used in PNN classification tasks?

Convolutional Neural Networks (CNNs) are a special type of neural network designed specifically for processing image data. Unlike traditional networks, CNNs can understand spatial relationships, like detecting edges, corners, or textures in an image.

Role in Image Recognition:

• CNNs automatically extract features from images using filters.

• These filters slide over the image to detect patterns such as lines, shapes, or textures.

• The network builds up from detecting simple patterns to complex objects.

• CNNs are used in applications like facial recognition, object detection, and medical diagnosis.

Structure:

1. Convolution Layer: Extracts features using filters.

2. Activation Layer (ReLU): Adds non-linearity.

3. Pooling Layer: Reduces image size while retaining important features.

4. Fully Connected Layer: Makes the final decision or classification.

Example: In a self-driving car, a CNN can look at the road image and identify objects like pedestrians, stop signs, or other vehicles.

Use in PNN Classification:

• CNN is used to preprocess and extract deep features from images.

• The output of CNN (usually a vector) is passed to a PNN.

• PNN then performs classification based on the statistical probability of those features belonging to different classes.

• This combination is powerful because CNN reduces the raw image to a compact feature representation, and PNN gives fast, probabilistic classification.

Example: In a handwritten digit recognition system, CNN extracts features like curves and lines from digits, and PNN classifies which digit it is based on learned probabilities.

This combination improves both accuracy and performance in image-based classification tasks.