Stochastic Gradient Descent With Python
Stochastic Gradient Descent (SGD) is an optimization technique used to minimize the cost function in machine learning algorithms.
Furthermore, it’s an iterative method that updates the model’s parameters by computing the gradient of the cost function.
Moreover, it does so with respect to each weight for a single training example. Even more, this process repeats for all training examples in the dataset, sometimes even more than once.
Comparison of SGD with batch gradient descent and mini-batch gradient descent
There are three primary gradient descent techniques:
- Batch Gradient Descent: Here we use the entire dataset to compute the gradient and update the model’s weights. Therefore, it’s computationally expensive for large datasets and can be slow to converge.
- Stochastic Gradient Descent: Updates the model’s weights after each training example. Thus, it’s computationally efficient and can handle large datasets. However, it may have a noisy convergence, and the optimization path may not be as smooth as batch gradient descent.
- Mini-Batch Gradient Descent: A compromise between batch and stochastic gradient descent. To clarify, it updates the model’s weights after a small batch of training examples. Consequently, it provides a balance between computational efficiency and a smoother optimization path.
Importance of SGD in linear regression and machine learning applications
SGD is quite popular in machine learning due to its ability to scale to large datasets. Furthermore, it can perform online learning, which makes it suitable for real-time applications.
Moreover, in linear regression, SGD offers a computationally efficient alternative to traditional optimization techniques. For instance, such as the normal equation or batch gradient descent.
Additionally, we can easily adapt SGD to handle various regularization techniques, making it a versatile optimization algorithm.
Mathematical foundation of stochastic gradient descent
Cost function and gradient descent
In linear regression, the cost function is typically the mean squared error (MSE).
Furthermore, gradient descent aims to minimize this cost function by iteratively adjusting the model’s parameters.
Stochastic gradient descent algorithm
SGD updates the model’s weights by computing the gradient of the cost function. It does so with respect to each weight for each training example.
Furthermore, this process repeats for all training examples in the dataset, and multiple dataset passes (epochs) to achieve convergence.
Updating model weights using SGD
The algorithm updates models weights using the following formula:
weight = weight - learning_rate * gradient
where learning_rate is a positive scalar that controls the step size, and gradient is the partial derivative of the cost function with respect to the weight.
Convergence properties of SGD
Although SGD has a noisy convergence, it eventually converges to the global minimum for convex cost functions.
On the other hand, for non-convex cost functions, it converges to a local minimum.
Advantages and limitations of stochastic gradient descent
Faster convergence and reduced computational complexity
SGD converges faster than batch gradient descent because it updates the model’s weights after processing each training example.
Thus we get reduced computational complexity in comparison to batch gradient descent, which requires the entire dataset for each update.
Handling large-scale datasets and online learning
SGD is useful for large-scale datasets and online learning, as it can process and update the model with new data points as they become available, without the need to retrain the entire model.
Noisy convergence and sensitivity to hyperparameters
Due to its stochastic nature, SGD has a noisy convergence, which can sometimes lead to overshooting the optimal solution.
It is also sensitive to hyperparameters, such as the learning rate, which requires careful tuning.
Regularization techniques in SGD
We can easily adapt SGD to handle various regularization techniques, such as L1 (Lasso) and L2 (Ridge) regularization.
Furthermore, we can do so by incorporating the regularization terms into the cost function and gradient update rule.
Practical applications of stochastic gradient descent in linear regression
Real-time data processing
SGD’s ability to perform online learning makes it suitable for real-time data processing.
Further making the model adapt to new data points as they become available.
Large-scale machine learning problems
SGD is effective in handling large-scale machine learning problems with millions or billions of training examples. In such cases batch gradient descent becomes computationally infeasible.
Recommender systems and collaborative filtering
SGD is popular in recommender systems and collaborative filtering to optimize large-scale matrix factorization problems, providing personalized recommendations to users.
Deep learning and neural networks
SGD is also a popular optimization algorithm in deep learning. Particularly, it’s useful for training neural networks with large numbers of parameters and layers.
Tuning and selecting hyperparameters in SGD
Learning rate and its scheduling
The learning rate is a critical hyperparameter in SGD that controls the step size.
Moreover, a learning rate that’s too small may result in slow convergence. On the other hand, a learning rate that’s too large may cause the model to overshoot the optimal solution.
Scheduling techniques, such as learning rate annealing or adaptive learning rates, can help achieve a more efficient convergence.
Momentum and adaptive learning rate methods
Momentum and adaptive learning rate methods, such as Nesterov Accelerated Gradient (NAG), AdaGrad, RMS Prop, and Adam, are extensions of SGD.
Furthermore, they aim to improve its convergence properties and reduce sensitivity to hyperparameters.
Cross-validation and model selection
Cross-validation is a useful technique for selecting the best hyperparameters, such as learning rate and regularization strength, as well as for assessing the model’s performance.
Additionally, by splitting the data into training and validation sets, cross-validation enables a more robust evaluation of the model’s generalization ability.
Best practices for implementing SGD in various scenarios
When implementing SGD in different scenarios, it is essential to consider following factors:
- dataset size
- the complexity of the model
- the desired level of model interpretability
Furthermore, properly tuning hyperparameters, using momentum or adaptive learning rate methods, and employing regularization techniques can help optimize models performance.
Stochastic gradient descent example in Python
Following code snippet demonstrates how to apply stochastic gradient descent for a linear regression problem. Furthermore, we’re going to download a housing dataset using Kaggle API and preprocess it.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from kaggle.api.kaggle_api_extended import KaggleApi
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
#authenticate API connection with Kaggle
api = KaggleApi()
api.authenticate()
#download the housing dataset from https://www.kaggle.com/datasets/yasserh/housing-prices-dataset
api.dataset_download_file(
'yasserh/housing-prices-dataset',
file_name='housing.csv',
path='datasets'
)
#import dataset and remove rows that have missing values, if there are any
df = pd.read_csv('datasets/Housing.csv')
df.dropna()
print(df.head())
#split dataset to dependent and independent values for linear regression
independent_df = df.iloc[:,1:5]
bool_categories = ['mainroad', 'guestroom', 'basement', 'prefarea', 'hotwaterheating', 'airconditioning']
for cat in bool_categories:
independent_df[cat] = df[cat].astype('category').cat.codes
print(independent_df)
#turn off pandas warning - doesn't effect the result, just cleans the console output
pd.set_option('mode.chained_assignment', None)
dependent_df = df[['price']]
dependent_df['price_log'] = np.log(dependent_df['price'] + 1)
print(dependent_df)
X = independent_df
y = dependent_df['price']
#split dataset into training and testing partitions
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipeline = Pipeline([
('std_scalar', StandardScaler())
])
X_train = pipeline.fit_transform(X_train)
X_test = pipeline.transform(X_test)
#import the model and train it
model = SGDRegressor(
n_iter_no_change=250,
penalty=None,
eta0=0.0001,
max_iter=100000
)
model.fit(X_train, y_train)
#make predictions on the test data
y_pred = model.predict(X_test)
#evaluate the results using MSE and R2
#lower MSE indicates better performance
#higher R2 indicates better performance (0 - 1 range)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Mean Squared Error: {mse:.2f}')
print(f'R2 Score: {r2:.2f}')
Conclusion
Recap of the significance of stochastic gradient descent in linear regression
Stochastic gradient descent plays a crucial role in linear regression and machine learning, offering faster convergence, reduced computational complexity, and the ability to handle large-scale datasets and online learning tasks.
Future research and advancements in SGD methodology
As machine learning continues to evolve, there is significant potential for future research and advancements in SGD methodology, including the development of more efficient and robust optimization algorithms and adaptive learning rate methods.
Final thoughts on the role of stochastic gradient descent in modern machine learning applications
Stochastic gradient descent remains a vital optimization technique in modern machine learning applications.
Furthermore, with its versatility and adaptability making it suitable for various problem domains, from linear regression to deep learning and beyond.
So by understanding its underlying principles and best practices, practitioners can harness the power of SGD to build effective and efficient machine learning models.
