Decision Models as Product Rerankers for E-Business Search and Recommendation: An Independent Evaluation on the Amazon Shopping Queries Dataset

Source abstract

Which products a shopper sees first, in search results or recommendations, shapes what they buy. Retailers increasingly use language models to judge product relevance. Decision models (Jev, Clef and Perplexity Decider) return a probability for each permitted judgement instead of generated text, which makes them natural rerankers for search and recommendation. We present, to our knowledge, the first independent, pre-specified evaluation of decision models on the Amazon Shopping Queries (ESCI) benchmark, with 1,000 US-English, 200 Spanish and 200 Japanese queries. Baselines were a dedicated reranker (Qwen3-Reranker-4B), general-purpose language models, a cross-encoder fine-tuned on 50,000 in-domain pairs, and BM25. Every decision model improved nDCG@10 over BM25, by 0.084 to 0.114 in English and by 0.100 to 0.149 in Spanish and Japanese (for Japanese, against a BM25 with character-bigram segmentation). Perplexity Decider ranked best (0.822) and significantly outperformed the dedicated reranker (0.787) by 0.032 to 0.035, whether run locally or through its hosted API. Asking for a four-way relevance label, an ordinal score or a yes/no exact-match judgement mattered little; the four-way label was best or tied in every comparison. All systems confused Exact with Substitute products; recall of Complement products, which underpin accessory recommendations, ranged from 15% to 68%. Clef’s labels were unaffected by the order of answer options, which it sorts internally, whereas a general-purpose language model changed up to 26% of its labels. The hosted Perplexity API reranked 1,000 queries for US$0.10–0.14 and, with the yes/no design, lay on the cost–quality frontier. We discuss implications for retailers, marketplaces, recommender systems and shopping agents.