Evaluating Whether LLM Recommendations Respond to the Decision Value of Missing Preferences
Kun-Yu Lee and Edward C. Malthouse
ManuscriptUnder review
Kun-Yu Lee
Master's Student in Machine Learning & Data Science, Northwestern University
I work with Edward C. Malthouse (Northwestern University) and Jing Yang (Boston University) on evaluating LLM-based recommender systems.
My research asks how LLM-based recommenders decide under uncertainty: when a missing preference is worth a clarifying question, which options they retrieve and rank, and whether this behavior can be reproducibly audited.
I am applying to PhD programs for Fall 2027.
Education
M.S. in Machine Learning and Data Science, Northwestern University
Sep 2025 – Expected Dec 2026GPA 3.94 / 4.00
Honors: Third Place, Northwestern MLDS AI Hackathon (2025)
B.S. in Computer Science and B.A. in Data Science, University of Nebraska–Lincoln
Aug 2021 – Dec 2024Minor in Mathematics · GPA 3.91 / 4.00
Honors: B.S. with High Distinction; B.A. with Distinction; Dean's List (2021–2024)
News
- Sep 2026Preprint on evaluating brand retrieval and ranking in LLM recommendations is now on arXiv. [arXiv]
Selected Publications
Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations
Edward C. Malthouse, Kun-Yu Lee, Jing Yang, Sanchary Pal, and Xueyan Feng
arXiv preprint arXiv:2609.16304, 2026Preprint
A shorter version is under review.
Proposes a framework for evaluating open-ended LLM brand recommendations that defines the competitive set independently of model outputs and estimates recommendation prevalence and prominence through repeated sampling (BRP@k, MRR@k), applied to six LLMs across five product categories.
PaperarXiv@misc{malthouse2026evaluating, title = {Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations}, author = {Malthouse, Edward and Lee, Kun-Yu and Yang, Jing and Pal, Sanchary and Feng, Xueyan}, year = {2026}, eprint = {2609.16304}, archivePrefix = {arXiv}, primaryClass = {cs.IR}, url = {https://arxiv.org/abs/2609.16304} }
Research
LLM Recommendations and the Decision Value of Missing Preferences
Northwestern University · Under review
When a consumer leaves a preference unstated, do LLM recommenders raise it because the recommendation depends on it, or only because the consumer mentioned it?
Auditing Brand Recommendations in LLMs
Northwestern University · Jan 2026 – Present · arXiv preprint, 2026; related manuscripts under review
When an LLM recommends brands without an explicit candidate set, which brands does it retrieve, how prominently are they ranked, and what does its implicit “brand image” look like?
Benchmarking and Advancing Agentic Orchestration
Northwestern University × Research Net.AI (industry capstone) · In progress
How can the performance of deployed multi-agent systems for automated software engineering be measured systematically, and where do their natural language understanding (NLU) pipelines break down?
Earlier Research
Viral Genomics Web Tool
May 2024 – May 2025Undergraduate Research Assistant, University of Nebraska–Lincoln · with Qiuming Yao
- Built a scalable backend with reproducible, schema-validated pipelines to collect, clean, and integrate heterogeneous, large-scale genomic metadata, including deduplication, missing-value handling, and field normalization.
- Implemented automated validation and indexing/caching for real-time responsiveness to user-uploaded datasets.
- Developed an interactive web tool integrating IGV (Integrative Genomics Viewer) and geographic mapping for querying mutation patterns, regional impacts, and temporal trends across virus lineages.
EV Adoption Study
Nov 2023 – May 2024Undergraduate Research Assistant, University of Nebraska–Lincoln · with Jason Fraser Hawkins
- Constructed analysis-ready features linking consumer adoption behavior to infrastructure, geography, and time, harmonized from multiple heterogeneous data sources.
- Evaluated regression and tree-based models with an emphasis on interpretable, decision-relevant patterns.
- Communicated findings through visual analysis to guide iterative model refinement.
Contact
I’m always happy to discuss research, collaboration, or PhD opportunities.
Northwestern University · Evanston, IL