the Streamlit app: a photo of a smiling face uploaded for emotion detection
why
an emotion label on its own doesn't say why someone feels that way. i wanted to connect a detected emotion to the feedback behind it.
how it works
a ResNet-50 fine-tuned on FER-2013 labels the face, the emotion becomes a query, FAISS pulls the 4 closest of 500 reviews, Flan-T5 summarizes them and RoBERTa scores each review's sentiment, all in a Streamlit app.
what i learned
the overall score hid the weak classes: 62% accuracy, but disgust, the rarest class, was right only 2 times out of 55, and none of the reviews ended up labelled disgust. per-class results told me more than the headline number.
results
62.2% accuracy on the 3,589-image FER-2013 test set (happiness 89%), with a live Streamlit demo.
disgust and fear are still the weak classes, and the README says so.
emotion-rag / core/models.py
docs = db.similarity_search(query, k=4)
context = "\n\n".join([doc.page_content for doc in docs])
prompt = PROMPT_TEMPLATE.format(context=context, question=query)
answer = llm.invoke(prompt)
