Bake a cake with TensorFlow — ML part

Written by George Soloupis ML and Android GDE.

This blog post demonstrates the usage of an electronic nose (e-Nose) during the cake baking process. The custom device collected all the necessary data for training the final machine learning model. Establishing a connection between the e-Nose and an android device was crucial, allowing real-time machine learning inference on the phone and facilitating a comprehensive inspection of the cake baking procedure.

Essential ingredients for cake baking:

1. Butter
2. Sugar
3. Eggs
4. Flour
5. Milk
6. Vanilla essence
7. A custom-made electronic device equipped with air sensors and a spectrophotometer.

e-Nose with modules.

The electronic device mentioned above was developed by Akis Vlissidis, building upon the work we started last year to prototype a solution for food quality detection. For this iteration, we opted to replace the Raspberry Pi 4 with an ESP32 module, providing increased freedom due to its wireless capabilities. The ESP32 allows for a Bluetooth connection with a mobile device, enabling us to utilize more modules in a compact form factor while also utilizing the mobile device’s screen. This upgrade offers enhanced flexibility and convenience in our cake baking process.

At the beginning of every supervised machine learning process, data collection is a crucial step. The experiment was divided into 10 different steps, each representing a distinct class of our classification task. These steps were as follows:
1. Butter only
2. Butter with half sugar
3. Butter with whole sugar
4. Butter, sugar and 2 eggs
5. Butter, sugar and 4 eggs
6. Butter, sugar, eggs and half flour
7. Butter, sugar, eggs and whole flour
8. Butter, sugar, eggs, flour and half milk
9. Butter, sugar, eggs, flour and whole milk
10. Butter, sugar, eggs, flour, milk and vanilla essence

and we started the data collection:

e-Nose during data collection.

Data collection proved to be time-consuming and challenging due to air sensor restrictions. The device had to be confined within a controlled room, free from air drafts, and maintained at a stable temperature and humidity to ensure precise operation and accurate data collection. For more details about a typical air sensor and its parameters, you can refer to the blog post linked here. On the other hand, the Sparkfun spectrophotometer, with its digital outputs, appeared to be resilient against environmental factors.

A small screenshot of the data is like below:

Data sheet with sensors and values.

A diagram of the MQ2 sensor during the data collection is as follows:

MQ2 diagram.

Because the air sensors emitted high heat, data collection for each specimen had to be conducted three times, with each session lasting approximately 1.5 hours. The diagram clearly indicates that the data collection lacked repeatability, necessitating the utilization of machine learning algorithms for further processing. A total of 45,000 records were collected for all the aforementioned classes. These records were shuffled and then split into training and testing sets. To prepare the data for a classification task, normalization was performed using mean and standard deviation. The preprocessed data was then ready to be fed into a Keras machine learning model for further analysis and classification.

TensorFlow/Keras implementation

At the beginning we used a simple ML model with Dense layers:

Simple ml model with dense layers.

However, the data did not meet the required standards, resulting in an accuracy of only 40%. Consequently, we made the decision to transition to a more specialized machine learning model from the Keras examples, which we believed would better suit our needs. You can read more about this decision and our approach in the blog post linked here.

ML model with Conv1D layers.

The outcome was truly astonishing! The specialized machine learning model yielded an accuracy of over 90%, far surpassing our previous results.

Diagram of the losses.

Moreover, we successfully converted this model into a .tflite file, a critical step for enabling machine learning inference on the Android mobile phone. This achievement allowed us to deploy the model on the device and conduct real-time predictions with ease.

You can find all data files and a detailed python notebook at this github repository.

TensorFlow Decision Forests

Another remarkable library we explored ( library for classification, regression, and other tasks) was TensorFlow Decision Forests. Its user-friendly nature provided us with an excellent opportunity to validate the high accuracy achieved with the Keras model. Additionally, we leveraged the library’s visualizations and logs to gain deeper insights into our data, further enhancing our understanding of the underlying patterns and relationships within the dataset.

1. Variables’ importance

Visualization of variables’ importance.

2. Confusion matrix table

Confusion matrix.

3. Model plotting

Model plotting.

In contrast to the multiple Python notebook cells required for the Keras implementation, TensorFlow Decision Forests streamlined the entire process effortlessly and automatically. Tasks that used to demand significant manual effort are now accomplished with minimal intervention, significantly reducing the development time, human errors and enhancing overall productivity. Follow along the notebook at this github repository.

The latest implementation not only confirmed the high accuracy achieved with the Keras model but also provided a more efficient and user-friendly process. However, it’s worth noting that, at present, TensorFlow Decision Forests models cannot be directly converted to .tflite files for deployment on Android phones.

Therefore, for our Android mobile application, we continued and used the cake_model.tflite file generated during the previous implementation with the Keras model. This file allowed us to seamlessly integrate the machine learning model into the Android environment, enabling real-time predictions on the device. Explanation at part 2.

Mobile’s screen during real time inference.

With the mobile device successfully set up and connected to the e-Nose, we proceeded with the final experiment to assess the cake baking procedure. A family member participated in the experiment, making a cake, and providing us with random samples to evaluate the various cake baking steps. This verification process allowed us to assess the accuracy and effectiveness of the e-Nose in monitoring and analyzing the cake baking process in real-time.

Below is a table summarizing the real baking steps and the corresponding assessments made by the e-Nose after several minutes:

Real time results.

The results of the experiment demonstrated that the e-Nose performed exceptionally well in recognizing the various steps involved in cake baking. However, it encountered some challenges in distinguishing between certain subtle differences, such as discerning the quantity of eggs used, particularly between 2 and 4 eggs, in the specimen.

Overall, the e-Nose’s ability to accurately assess most of the cake baking stages was impressive. Nevertheless, further refinements might be necessary to improve its sensitivity and precision for specific aspects like distinguishing small variations in ingredient quantities.

Areas that require improvement include:

1. Developing a method to make the e-Nose independent of room temperature and humidity.
2. Enhancing real-time response for faster results during monitoring.
3. Improving the device’s speed in returning to default settings, expediting the process of evaluating new specimens.
4. Implementing a system for using smaller sample sizes, measured in grams, to optimize data collection and reduce waste.
5. Determining whether calibration is necessary between different samples to ensure accurate and consistent measurements.

Conclusion

In this blog post, we explored the application of an electronic nose (e-Nose) during a cake baking process. The custom device collected essential data for training a machine learning model. Establishing a connection between the e-Nose and an android device allowed real-time machine learning inference on the phone, enabling a comprehensive inspection of the cake baking procedure. The final experiment, baking a cake while the e-Nose assessed the steps, demonstrated the device’s capability in recognizing the baking stages. Small challenges in distinguishing subtle differences, such as the quantity of eggs used, were faced during the final assessment. Overall, the e-Nose proved to be a valuable tool in monitoring the cake baking process, offering insightful feedback for further device improvements.