Web Scraping and Image Classification with TensorFlow

Web Scraping and Image Classification with TensorFlow
This project demonstrates my ability to combine web scraping techniques with machine learning to classify images. The goal was to build a pipeline that scrapes images of shoes from a website and uses them for training a machine learning model.
Key Features:
- Web Scraping: I utilized
BeautifulSoupandrequeststo scrape images from an e-commerce site, focusing on categories like formal shoes, sneakers, and boots. The images were automatically downloaded and saved into organized directories. - Image Preprocessing: The images were stored in separate folders for training and testing. This organization sets the foundation for an image classification task.
- Model Setup: While the project focuses on scraping, the next logical step (which can be explored further) is to use
TensorFlowandKerasto create a Convolutional Neural Network (CNN) model to classify these images into categories such as sneakers, formal shoes, and boots. - Scalability: The design of the scraping function allows it to easily scale for other shoe categories or even other types of products. Simply by adjusting the URLs and folder names, new categories can be added to the pipeline.
Technologies Used:
- Python: The core language for web scraping and model building.
- TensorFlow & Keras: For machine learning model development.
- BeautifulSoup & Requests: For scraping data from the web.
- PIL & Matplotlib: For handling images and visualizing results.
Why This Project?
This project not only helped me refine my web scraping skills but also gave me hands-on experience in integrating web data with machine learning models. It's an excellent example of how data from websites can be harvested and used in AI-driven applications, such as in e-commerce for product categorization.
💼 Business Impact
This project has the potential to significantly enhance business operations, especially for e-commerce platforms. By automating the categorization and classification of product images, businesses can save time and resources that would otherwise be spent on manual labor. Here are some key business impacts: Improved Product Categorization: Automated image classification ensures accurate categorization of products, which improves the shopping experience for customers. This makes it easier for businesses to organize their inventory and for customers to find what they are looking for quickly. Enhanced Customer Experience: By having accurate and consistent product categories, customers can easily navigate through various types of shoes (e.g., sneakers, formal shoes, boots), leading to improved satisfaction and higher conversion rates.
Code Snippet
import requests
from bs4 import BeautifulSoup
from PIL import Image
from IPython.display import display
import re
import os
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt
import tensorflow as tf
# Load Data
def download_image(train_or_test,folder_name):
all_url=[]
res=requests.get(f'https://shoeplace.ng/10-formal-shoes').content
be=BeautifulSoup(res,'lxml')
for i in range(len(be.find_all('img'))):
try:
all_url.append(be.find_all('img')[i]['data-full-size-image-url'])
except:
pass
for i in range(len(all_url)):
r = requests.get(all_url[i])
with open(f"shoes/{train_or_test}/{folder_name}/{i}.jpg",'wb') as f:
f.write(r.content)
return
def download_image(start_page,end_page,item_name,folder_name,train_or_test):
all_url=[]
for i in range(start_page,end_page):
res=requests.get(f'https://en.afew-store.com/collections/{item_name}?offset={i}').content
be=BeautifulSoup(res,'lxml')
for i in range(len(be.find_all('img'))):
try:
all_url.append(be.find_all('img')[i]['data-src'])
except:
pass
# webscraping
for i in range(len(all_url)):
r = requests.get('https:'+all_url[i])
with open(f"shoes/{train_or_test}/{folder_name}/{i}.jpg",'wb') as f:
f.write(r.content)
return
# TensorFlow
train=keras.utils.image_dataset_from_directory('/content/shoes/train',seed=seed,image_size=(180,180),validation_split=0.2,subset='training')
val=keras.utils.image_dataset_from_directory('/content/shoes/train',seed=seed,image_size=(180,180),validation_split=0.2,subset='validation')
model=keras.Sequential([
layers.Rescaling(1./255),
layers.Conv2D(60,5,activation='relu'),
layers.MaxPool2D(),
layers.Conv2D(80,6,activation='relu'),
layers.MaxPool2D(),
layers.Conv2D(100,8,activation='relu'),
layers.MaxPool2D(),
layers.Flatten(),
layers.Dense(6)
])



















