Cooking Made Easier Through Speech Recognition

Launched in 2023, TransforMA is a joint project in which the University of Mannheim and the University of Applied Sciences Mannheim work together in interdisciplinary teams. The project aims to position itself as a center of competence on issues related to transformation and to help shape transformations in Mannheim and the region through the dialogic transfer of knowledge. The project addresses five fields of action dedicated to issues such as responsibility and sustainability, public-private collaboration and technology.

One of the projects is Foodle, a virtual cooking assistant that analyzes YouTube cooking videos, automatically generating a list of ingredients and cooking instructions. In addition, Foodle can answer questions about the recipe – all via chatbot and speech recognition. The assistant was developed by the students Thilo Dieing, Aakriti Istwal, Priyanka Roy, Sharan Shyamsundar, Hanh Thi-Bich Nguyen and Mariam Arustashvil from the Mannheim Master of Data Science program, supervised by Prof. Dr. Heiner Stuckenschmidt and Darshit Pandya at the Chair of Artificial Intelligence. FORUM interviewed master’s student Thilo Dieing about his work on Foodle.

FORUM: You have worked on Foodle since March 2023, now the language assistant is ready for use. What was your role in the project?

Thilo Dieing: Everybody on the team works a little bit on everything. My two main responsibilities were designing the front end, meaning the app that users see, where speech recognition and chatbot are combined, and evaluating the speech recognition models.

FORUM: What was your highlight when working on Foodle?

Dieing: The very first time we had a web version of Foodle that we could present to our project leads. That was a highlight. And of course, the end of the project when we had a functioning model. That felt as if the “baby” we had worked on for one year had finally been born.

FORUM: Are there any plans to make Foodle available for public use at some point?

Dieing: Not now, because Foodle is very slow if you want to use it on a regular computer. One option might be to show Foodle to the public as a demonstrator on one or two days. What we do have in mind, however, is to take the dataset with the transcribed YouTube videos that we created manually and publish it together with the Chair of Artificial Intelligence. That would make the project valuable for the larger research community.

FORUM: What is your recommendation to students who want to get involved in similar projects?

Dieing: Just go ahead and give it a shot! Getting involved in such projects always means jumping in at the deep end, but I can say on behalf of the entire team that we learned so much in the process and would never have thought we would be able to eventually come up with such a product.

Interview: Rheia Martiny / August 2024