Introduction
A recent study published on Diabettech.com highlights the inconsistencies of AI-powered carb counting. The author submitted 13 food photographs to four leading AI models, each with over 500 queries, and found significant variations in carbohydrate estimates. This raises concerns about the reliability of AI-powered carb counting for diabetes management.
The Study
The study used real food photographs, taken with a phone, and a prompt adapted from the iAPS open-source automated insulin delivery system. The four AI models used were OpenAI GPT-5.4, Anthropic Claude Sonnet 4.6, Google Gemini 2.5 Pro, and Google Gemini 3.1 Pro Preview. Each model was queried over 500 times, with the same prompt and photo, to assess the consistency of their carbohydrate estimates.
Results
The results showed that each model returned different carbohydrate estimates for the same photo across repeated queries. The degree of disagreement varied significantly between models, with Claude Sonnet 4.6 showing the least variation (2.4% median CV) and Gemini 2.5 Pro showing the most (11.0% median CV).
Implications
The inconsistencies in AI-powered carb counting pose significant risks for diabetes management. A difference of 42.9 units of insulin, as seen in the paella photo example, can be life-threatening. The study highlights the need for caution when relying on AI-powered carb counting and the importance of understanding the limitations of these models.
Discussion
The study's findings have sparked discussion on Hacker News, with some users criticizing the methodology and others acknowledging the importance of highlighting the limitations of AI-powered carb counting. The study serves as a reminder that AI models are not perfect and should be used with caution, especially in critical applications like diabetes management.
Conclusion
The study's results emphasize the need for further research into the inconsistencies of AI-powered carb counting and the development of more accurate and reliable models. In the meantime, users of AI-powered carb counting apps should be aware of the potential risks and take necessary precautions to ensure their safety.
Next Steps
If you're using an AI-powered carb counting app, it's essential to understand the limitations of these models and take steps to verify the accuracy of the estimates. Consider consulting with a healthcare professional or using alternative methods for carb counting. For custom AI and automation solutions, reach out to AImatic at hello@aimatic.dev.