British Airways Customer Review Web Scraping, Text Classification, Sentiment Analysis, and Customer Satisfaction

Web Scraping, Text Classification, Sentiment Analysis, and Customer Satisfaction
This project combines web scraping, text classification, and sentiment analysis to evaluate customer satisfaction based on reviews or other text data. The project is built around extracting relevant data from airline websites and analyzing customer sentiments to classify whether the sentiment is positive, neutral, or negative.
Key Features:
- Web Scraping: Using
BeautifulSoupandRequests, I scraped customer reviews and other relevant data from various airline websites (such as British Airways, Air India, Emirates, and more) to gather a rich dataset for analysis. - Sentiment Analysis: With sentiment analysis techniques, the project identifies and classifies customer sentiments from the scraped text. This can be used to analyze feedback, detect issues, and measure customer satisfaction.
- Text Classification: The project also focuses on binary classification, where the system identifies positive and negative customer feedback based on review content.
- Model Development: I used
TensorFlowand other tools likePandas,NumPy, andRegexto preprocess the data, train models, and evaluate the performance of the sentiment classifier.
Technologies Used:
- Python: Core language for web scraping, data analysis, and machine learning tasks.
- BeautifulSoup & Requests: For scraping data from web pages.
- TensorFlow: For creating and training machine learning models to perform sentiment analysis and classification.
- Pandas & NumPy: For handling and manipulating datasets.
- Regex: For text pattern matching and cleaning the scraped data.
Business Impact:
This project offers a practical solution for businesses, especially in the airline industry, to better understand customer sentiment and improve their service offerings. By analyzing customer feedback, businesses can:
- Quickly identify areas for improvement.
- Measure the success of new services or features.
- Enhance customer satisfaction and loyalty by responding to customer concerns and recognizing positive feedback.
The ability to analyze customer sentiment on a large scale, across multiple platforms, can provide actionable insights that help businesses make data-driven decisions to enhance the customer experience.
💼 Business Impact
This project offers a practical solution for businesses, especially in the airline industry, to better understand customer sentiment and improve their service offerings. By analyzing customer feedback, businesses can: Quickly identify areas for improvement. Measure the success of new services or features. Enhance customer satisfaction and loyalty by responding to customer concerns and recognizing positive feedback. The ability to analyze customer sentiment on a large scale, across multiple platforms, can provide actionable insights that help businesses make data-driven decisions to enhance the customer experience.
Code Snippet
import requests
from bs4 import BeautifulSoup as bs
import re
from IPython.display import display
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import os
import numpy as np
import pandas as pd
import shutil
import matplotlib.pyplot as plt
import seaborn as sns
# Load Data/webscraping
def text_scapping(start_page,end_page,recommended_or_not_recommended,airline):
recommended=[]
not_recommended=[]
for i in range(start_page,end_page):
url=f'https://www.airlinequality.com/airline-reviews/{airline}/page/{i}/'
re=requests.get(url).content
result=bs(re,'lxml')
for i in range(len(result.find_all('div',class_='text_content'))):
import re
if result.find_all('td',class_=re.compile('^review-value rating-'))[i].string=='no':
not_recommended.append(re.split(r'[|]',result.find_all('div',class_='text_content')[i].text,maxsplit=2)[-1])
elif result.find_all('td',class_=re.compile('^review-value rating-'))[i].string=='yes':
recommended.append(re.split(r'[|]',result.find_all('div',class_='text_content')[i].text,maxsplit=2)[-1])
else:
pass
if 'recommended'==recommended_or_not_recommended:
recom_or_not_recom=recommended
else:
recom_or_not_recom=not_recommended
return recom_or_not_recom
def download_text(train_or_test,recommended_or_not_recommended,good_or_bad):
z=[0]
for d in (os.listdir(f'/content/drive/MyDrive/Colab_Notebooks/airline/{train_or_test}/{good_or_bad}')):
z.append(int(d.split('.')[0]))
if 'not_recommended'==recommended_or_not_recommended:
for k in range(len(not_recommended)):
with open(f'/content/drive/MyDrive/Colab_Notebooks/airline/{train_or_test}/{good_or_bad}/{k+np.max(z)}.txt', 'w') as f:
f.write(not_recommended[k])
if 'recommended'==recommended_or_not_recommended:
for k in range(len(recommended)):
with open(f'/content/drive/MyDrive/Colab_Notebooks/airline/{train_or_test}/{good_or_bad}/{k+np.max(z)}.txt', 'w') as f:
f.write(recommended[k])
return
# TensorFlow
def plot_text(train_or_test,good_or_bad,text_number):
c=[]
with open(f'/content/drive/MyDrive/Colab_Notebooks/airline/{train_or_test}/{good_or_bad}/{text_number}.txt') as r:
c.append(r.read())
text_len=[]
for k in c:
text=k.split()
for d in text:
text_len.append(len(d))
letter=[]
letter_count=[]
for c,d in sorted(zip(text_len,text),reverse=True):
letter_count.append(c)
letter.append(d)
plt.figure(figsize=(20,10))
sns.barplot(x=letter,y=letter_count)
plt.xticks(ticks=range(len(letter_count)),labels=letter,rotation=90)
plt.title('distribution of n-grams')
plt.xlabel('n-grams')
plt.ylabel('n-grams length')
unique_text,unique_count=np.unique(text,return_counts=True)
unique_text_=[]
unique_text_count=[]
for k,i in sorted(zip(unique_count,unique_text)):
unique_text_.append(i)
unique_text_count.append(k)
plt.figure(figsize=(20,10))
sns.barplot(x=unique_text_,y=unique_text_count)
plt.xticks(ticks=range(len(unique_text_count)),labels=unique_text_,rotation=90)
plt.title('Frequency distribution of n-grams')
plt.xlabel('n-grams')
plt.ylabel('Frequencies')
return plt.show()
#Model Training
model.compile(optimizer='adam',metrics=tf.metrics.BinaryAccuracy(threshold=0.0),loss=tf.keras.losses.BinaryCrossentropy(from_logits=True))



















