Improvement and integration of a deep learning-based semantic model into a hybrid lexical-semantic search engine

Abstract

This thesis presents a series of improvements to the semantic search system deployed on the mango.com e-commerce platform, with the goal of enhancing retrieval accuracy, robustness, and relevance across multiple languages and query types. The work focuses on addressing key limitations of the existing semantic component within a hybrid lexical¿semantic search architecture. The main contributions include the training of a multilingual BERT uncased model to improve robustness to capitalization and diacritics, as well as fine-tuning on fashion-specific data to enhance domain understanding. In addition, limitations in handling attribute-only queries are addressed through the use of intelligent image cropping and a weighted fusion strategy that combines image embeddings with short textual metadata. Furthermore, the CLIP image encoder is fine-tuned to generate semantically richer and more discriminative visual representations, leading to higher similarity scores and improved ranking stability. Experimental results, evaluated on a manually annotated multilingual dataset, demonstrate consistent improvements across locales, increased semantic similarity scores, and more stable ranking performance, without introducing catastrophic forgetting or degrading standard query retrieval. The integration of these enhancements into the full production search pipeline provides a robust foundation for future improvements, including tighter alignment between multilingual text embeddings and the fine-tuned image encoder.