In this project, I implement 3 different machine learning models (Logistic Regression, Random Forest, and Multinomial Naive Bayes) using sklearn to classify data from an Amazon Product Review dataset. I used specific features (review time, review text, review summary, number of upvotes) from the dataset of Amazon product reviews to predict the overall score (1 to 5 stars) of each review. This involved extracting numeric features from text features (i.e. product text and summary text). I used 4 different score cutoffs (1,2,3, or 4 stars) to first label each review as ‘good’ or ‘bad’, which then allowed me to perform binary classification by giving the model 80% of the data (features + labels) for ‘training’, and then then testing its performance on the remaining 20% of the data (i.e. its ability to predict the labels using the given features). NOTE: a small subset (10%) of the training data was used for cross-validation and hyperparameter tuning. In addition to binary classification, I also performed multi-class classification, which involved a similar procedure, except the 5 possible review scores were used as classes directly (instead of using score cutoffs to label reviews as ‘good’ or ‘bad’). Finally, I performed k-means clustering, a type of unsupervised learning algorithm used to group together similar data points. The true clusters were taken to be the product categories (e.g. automotive), and the silhouette score and rand index were used to evaluate the model’s performance.
This repository contains the following files:
-
CS74 Final Project_Charles_Ihara.ipynb - this jupyter notebook contains my code, output, and written explanations of methods and results.
-
Training.csv - the training data
-
Test.csv - the testing data (without the overall review score). NOTE: the kaggle submissions were used to find out how my model performed on this test data.
-
Kaggle Submissions - this sub-directory contains csv files with the review scores predicted by my model, as well as the Google Colab notebooks used to produce these files.
-
Winter 2025 COSC74_274 Final Project.docx - this word document contains the full assignment instructions.
The project instructions are written below. Additional grading information can be found in the Winter 2025 COSC74_274 Final Project word document.
The goal of the course project is to implement machine learning models and concepts covered in this course for a real-world dataset. The project will utilize the Amazon product review dataset and focus on binary classification, multi-class classification, and clustering approaches to analyze and categorize product reviews. All code must be implemented in Python and all models must use the Scikit Learn toolkit - https://scikit- learn.org/stable/index.html. You are not allowed to use other toolkits, such as NLTK or transformer network architectures, for your project results.
Here are examples of some useful Scikit modules:
Text Feature Extraction:
- https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
- https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html#sklearn.feature_extraction.text.HashingVectorizer
- https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer
Machine Learning Models:
- https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
- https://scikit-learn.org/stable/modules/svm.html
- https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
Projects will be individual: each student will work on their own project. Students can discuss with each other for clarification, but they should make sure not to share codes. Any collaboration or sharing of ideas should be acknowledged in the project reports with sufficient details.
Link to dataset: https://tinyurl.com/22yau9r8
You will be given two files: Training.csv and Test.csv.
Training.csv: This file is a CSV file consisting of review-related information with the following fields:
- overall: This is the product’s rating on a scale of (1-5)
- verified: A boolean variable denoting if the review has been verified by Amazon
- reviewTime: time of review
- reviewerID: The unique ID of the Amazon reviewer (some have left multiple reviews)
- asin: Product ID. One product will have many different reviews
- reviewerName: Encoding of the Amazon reviewer’s username
- reviewText: The Amazon review
- unixReviewTime: unix time of review
- vote: How many people voted this review as being helpful
- image: If there is an image, link to the image
- style: If there is style information (e.g., size of shirt, color of phone), it is embedded in a dictionary here. Only available for some samples
- Category: The Amazon product category of the product.
Test.csv: This file contains all the same features as Training.csv, but the overall variable is withheld. You will submit your predictions of the overall for each product using this file and we will compare them with the true labels.
In this task, you have to develop binary classifiers to classify product reviews as good or bad. The cutoff of ‘goodness’ will be an input, i.e., you have to develop classifiers with the following cutoffs of product rating: 1,2,3,4. Note: The cutoff is not an input to the model, but to the experiment. For example, when cutoff=3, all samples with a rating <= 3 will have label 0, and all samples with a rating > 3 have label 1. You are expected to report the performance of at least three different classifiers for each of the four cutoffs. You need to perform cross-validation for hyperparameter tuning. Your report should describe why certain model parameters help or hurt the model performance. For each classifier, you should report in your report the confusion matrix, ROC, AUC, macro F1 score, and accuracy for the best combination of hyperparameters using 5-fold cross-validation. We will share a baseline macro F1 score for classification and at least one of your classification models must achieve at least the baseline score for full credit.
Turn the above classifier into a multiclass classifier where the target classes are 1,2,3,4,5. In other words, you want to classify the product rating on a five-class scale. For each classifier, you should report the confusion matrix, ROC, AUC, macro F1 score, and accuracy for the best combination of hyperparameters using 5-fold cross-validation. You are expected to report the performance of at least three different classifiers. We will share a baseline macro F1 score for classification, and at least one of your classification models must achieve at least the baseline score for full credit.
In this task, you will cluster the product reviews in the test dataset. You will need to create word features from the data and use that for k-means clustering. Clustering will be done by product types, i.e., in this case, the labels will be product categories. You will use the Silhouette score and Rand index to analyze the quality of clustering. We will share a baseline silhouette score for clustering, and your model must achieve at least the baseline score for full credit.
For classification tasks (4 binary classification tasks and 1 multiclass classification task), you should work on at least three different classifiers. For each classifier, below is the expected procedure.
-
You should decide which hyperparameters you want to tune.
-
For each combination, you should compute a 5-fold cross-validation score. (You should report the mean of the five-fold validation scores.) For example, let’s say there are 100 different combinations of hyperparameters for a classifier. You will then have 100 cross-validation scores.
-
Now, you can choose the best combination of hyperparameters based on the cross-validation scores. Note that you have to report all the validation scores and explain how you come up with the best combination of hyperparameters.
-
You should report the best model (with the chosen hyperparameters) in multiple metrics (e.g. confusion matrix, ROC, AUC, macro F1 score, and accuracy). You should report them on one of the validation sets (i.e. 20% of your data). For the multiclass classification task, you should show 6 curves in one plot: 5 curves from each category and the average curve. You will repeat the above procedures for at least two other classifiers. Now, you have to run your best models on the Kaggle competitions (i.e. test set) to achieve higher performance than our baseline scores. You should report the best test scores you achieved from Kaggle on your report.
To receive instantaneous feedback on the performance of your models on the test data as you are building them, we have provided five Kaggle competitions. There is one Kaggle competition for each of the five classification tasks (four binary classification tasks with cutoffs 1, 2, 3, 4 and one multiclass classification task). Links to join these competitions are given below. These competitions will allow you to obtain the macro F1 score of your model and compare it to the macro F1 score of the baseline for that classification task. Your model results in these competitions are not going to be directly correlated to your final score. Instead, your model results in these competitions will allow you to compare your model with the baseline macro F1 score you must beat. You are allowed to submit five times a day to each of the five competitions.
Kaggle Competition Links:
-
Binary Classification Cutoff 1: https://www.kaggle.com/competitions/dartmouth-cosc-74-274-w-2025-session-2-part-1
-
Binary Classification Cutoff 2: https://www.kaggle.com/competitions/dartmouth-cosc-74-274-w-2025-session-2-part-2
-
Binary Classification Cutoff 3: https://www.kaggle.com/competitions/dartmouth-cosc-74-274-w-2025-session-2-part-3
-
Binary Classification Cutoff 4: https://www.kaggle.com/competitions/dartmouth-cosc-74-274-w-2025-session-2-part-4
-
Multiclass Classification: https://www.kaggle.com/competitions/dartmouth-cosc-74-course-project-winter-25-p-5