A Multimodal Approach for Measuring Item Similarity

Measuring similarity between items using computer vision and natural language processing

Multimodal Item Similarity

Measuring similarity using computer vision and natural language processing

Measuring how similar two items are is more complex than it seems. People consider visual appearance, functionality, context, and subjective preferences—all simultaneously. We develop AI systems that combine computer vision and natural language processing to understand similarity the way humans do.

The Challenge

Traditional methods use simple categories or require extensive manual effort. They miss nuanced similarities that matter to users. Our solution: teach AI to analyze both visual and textual features, learning patterns that match human intuition.

Our Approach

Visual Analysis

Computer vision extracts features like architectural styles, landscapes, activities, and atmosphere from images.

Text Processing

NLP analyzes descriptions to understand cultural characteristics, offerings, climate, and context.

Multimodal Integration

Combining both creates similarity judgments that match human intuition.

Applications

Tourism & Travel

Destination recommendations that match travel preferences. Finding alternatives when favorites are unavailable.

E-commerce & Real Estate

Product and property recommendations based on visual and textual similarity.

General Purpose

Applicable to any domain requiring multi-faceted similarity measurement.

Why It Works

No single similarity measure captures human judgment. Our hybrid approach combines multiple dimensions, adapts to different domains, and learns from feedback. Validated against expert judgments and deployed at scale.

Our AI system analyzes visual and textual features of destinations to understand similarity patterns that match human intuition.

Related Publications

2024

  1. Warm Recommendation: Enhancing Cold Start Recommendations Using Multimodal Product Representations
    2024
    Anat Goldstein, Amit Alony, and Chen Hajaj
    International Conference on Information Systems (ICIS)
  2. Measuring Flight-Destination Similarity: A Multidimensional Approach
    2024
    Anat Goldstein, and Chen Hajaj
    Expert Systems with Applications