Discussion and Further Work
One of the key performance metrics for our work is the accuracy of food recognition. Over nearly two months, we have tested the fridge on various categories of foods, including fruits, vegetables, meat, noodles, pizza, burgers, etc. Rare did it give wrong label. We observe that the AWS Rekognition gives wrong answers mostly when we put in a piece of meat. It is obvious and not surprising since many kinds of raw meat really have similar appearance.
The user is able to see all relevant data visualized on App. And every time the user refresh the page, he gets all most updated data(collection of current food items, food nutrition information, expiration dates, temperature & humidity curve, relevant recipe recommendations) with low latency. What we are possibly going to improve in the future is add a trigger from Lambda to App: once there is any update for food data, Lambda triggers the App to automatically update its data visualization.
Our implementation for estimating the expiration date is to be improved, we will either develop a python crawler approach or deal with it by means of database.
Moreover, we plan to add more powerful functionality to the current version of our smart fridge: enable “foods to buy” based on what the user has bought recently. This is a tough problem because it has something to do with “healthy diet plan”.
For now, the users are limited to change one food every time they open the fridge door, it is a little bit inconvenient. Thus, we could add physical buttons inside the fridge in the future, so that users are able to use buttons to control the image capturing of multiple foods, and they can also re-capture the image if they did not place the food in the correct place in time.
Furthermore, the web app can be transformed into a hybrid app, and then can enhance the user’s experience. For instance, we can create a new page which is triggered by the click or touch on the food item in the home page, which includes more detailed information of the food and allows users to modify the informations on their demand.
The user is able to see all relevant data visualized on App. And every time the user refresh the page, he gets all most updated data(collection of current food items, food nutrition information, expiration dates, temperature & humidity curve, relevant recipe recommendations) with low latency. What we are possibly going to improve in the future is add a trigger from Lambda to App: once there is any update for food data, Lambda triggers the App to automatically update its data visualization.
Our implementation for estimating the expiration date is to be improved, we will either develop a python crawler approach or deal with it by means of database.
Moreover, we plan to add more powerful functionality to the current version of our smart fridge: enable “foods to buy” based on what the user has bought recently. This is a tough problem because it has something to do with “healthy diet plan”.
For now, the users are limited to change one food every time they open the fridge door, it is a little bit inconvenient. Thus, we could add physical buttons inside the fridge in the future, so that users are able to use buttons to control the image capturing of multiple foods, and they can also re-capture the image if they did not place the food in the correct place in time.
Furthermore, the web app can be transformed into a hybrid app, and then can enhance the user’s experience. For instance, we can create a new page which is triggered by the click or touch on the food item in the home page, which includes more detailed information of the food and allows users to modify the informations on their demand.
Conclusion
We have built up an intelligent fridge which has a high recognition accuracy for majority of food classes. The data is updated in real time with low latency. The operations for using the fridge is easy and friendly for users. The backend design is robust and efficient and has high scalability.
We have spent a lot of time and effort dealing with hardware issues. We now have rich experience in coping with physical data fluctuation and unstability. Also we get much more familiar with cloud computing and have obtained experience in AWS architecture design and realization. What’s more, we introduced and learned some state-of-art front-end development techniques while designing the App.
We will do some further improvement to our smart fridge, including estimating a more accurate expiration date, warning/alarming user when any food reaches expiration date, enabling users to edit or modify the food information data if they happen to think of it as incorrect or inaccurate.
We have spent a lot of time and effort dealing with hardware issues. We now have rich experience in coping with physical data fluctuation and unstability. Also we get much more familiar with cloud computing and have obtained experience in AWS architecture design and realization. What’s more, we introduced and learned some state-of-art front-end development techniques while designing the App.
We will do some further improvement to our smart fridge, including estimating a more accurate expiration date, warning/alarming user when any food reaches expiration date, enabling users to edit or modify the food information data if they happen to think of it as incorrect or inaccurate.
References
[1] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
[2] Working with Images - Amazon Rekognition. (2019). Retrieved from https://docs.aws.amazon.com/rekognition/latest/dg/images.html
[2] Working with Images - Amazon Rekognition. (2019). Retrieved from https://docs.aws.amazon.com/rekognition/latest/dg/images.html