michellesklee / wedfuly_recommender

Recommend wedding vendors based on vendor features and image clustering (CNN autoencoder + k-means)

---

Table of Contents

Background
Data Collection
Florist Image Processing
Convolutional Autoencoder
k-Means Clustering
Evaluating the Model
Building a Recommender
Recommender Application
Conclusion
Next Steps
Tech Stack

Background

On Wedfuly, clients work online with wedding planners who help with the planning process, including choosing wedding vendors such as florists, photographers, and bakeries.

The goal of this project was to build a recommender that facilitates the process of wedding planners suggesting vendors to their clients. For the first part of this project, I focused on florists with the goal of training a model to meaningfully cluster floral arrangements (bridal bouquets specifically). As a second step, I built a similar recommender for photographers using pre-labeled images of Wedfuly photographers.

Data Collection

Images of bridal bouquets collected online (train set: 2897, test set: 637) Images from Wedfuly wedding photographers (~1200)

Florist Image Processing

1. Cropped bouquets from images
2. Centered bouquets and cropped images to square
3. Resized images to 100x100
4. Converted to RGB
5. Augmentation: LR, UD

Convolutional Autoencoder

loss: 0.0140 - val_loss: 0.0142

---

Reconstructed Images

Building Attention Model

Determining where the model is paying the most attention

Isolate encoding layers:

get_encoded = K.function([cnn_model.layers[0].input], [cnn_model.layers[5].output])

Pool filters from the encoded layers to see where model is paying most attention. Taking max() of filters is standard method.

Next pool up to the last convolutional layer in the model. The images on the right show the max of the filters. Bright spots are where the model is activated the most -- seemingly bright things and contrasts.

k-Means Clustering

Clustering with k-Means - 7 was optimal k

Images from each cluster:

Cluster 0: "Moody and Dark"

Cluster 1: "Earthy Modern"

Cluster 2: "Simply Minimal"

Cluster 3: "Moody and Wild"

Cluster 4: "Traditional"

Cluster 5: "Light and Modern"

Cluster 6: "Colorful and Bold"

Evaluating the Model - Florists

To evaluate the model, I googled "bridal bouquet" and chose 40 of the top images to mimic the process of clients adding images to their Pinterest board.

Cluster 0: "Moody and Dark"

If you like this...

You might like these vendors...

---

Cluster 1: "Earthy Modern"

If you like this...

You might like these vendors...

---

Cluster 2: "Simply Minimal"

If you like this...

You might like these vendors...

---

Cluster 3: "Moody and Wild"

If you like this...

You might like these vendors...

---

Cluster 4: "Traditional"

If you like this...

You might like these vendors...

---

Cluster 5: "Light and Modern"

If you like this...

You might like these vendors...

---

Cluster 6: "Colorful and Bold"

If you like this...

You might like these vendors...

---

Building a Recommender

k-Means cluster labels became a feature of an overall model that also included the following features:

Continuous Variables:

1. Total price
2. Location of wedding - Boulder, Denver, Foothills, Summit County, Mountain Town Not Listed, Other

Categorical Variables (dummy coded):

3. Method of delivery - Delivery (full service), Delivery (no service)/Drop-off, or Pick-up
4. Services provided - Ceremony Decor, Reception Decor, and/or Handhelds
5. Size of wedding

Distance Metric

As clients enter specifics of their wedding as well as select images that match their style, cosine similarity will be calculated to find the vendor that has provided wedding services that are most similar to the clients needs.

Recommender Application

A recommender application built on Flask and Docker and hosted on Amazon Web Services allows clients and wedding planners to enter specifics of their wedding and select images to recommend vendors.

Conclusion

Even with limited training images, a k-Means algorithm on top of a CNN autoencoder has potential to cluster images for recommendation purposes. Using k-Means cluster labels in addition to features related to the wedding provides a more robust recommender system that takes into account the specifics of the wedding as well as clients' style.

Next Steps

1. As Wedfuly grows and more data is collected, the recommender will be further fine tuned and validated
2. While manually cropping bouquets was sufficient for this initial model, in the future, training a neural network to detect bouquets may be worth exploring
3. Using transfer learning such as XCeption may also be considered in the future to detect features in images

Tech Stack