ChatGPT passes the nutrition test, but experts remain irreplaceable

In a recent study published in the journal Nutrients, researchers evaluated the potential of chat generative pretrained transformer (ChatGPT) to provide nutritional guidance.

Non-communicable diseases (NCDs) are the foremost cause of mortality, accounting for 74% of deaths globally. The 2019 global burden of diseases study estimated there were 43.8 million cases of type 2 diabetes (T2D), 1.2 billion cases of non-alcoholic fatty liver disease (NAFLD), and 18.5 million cases of hypertension. Obesity prevalence has almost tripled between 1975 and 2016.

Various studies have consistently underscored the impact of lifestyle and dietary factors on NCD onset and progression. Of late, internet searches for information on health-related queries have been increasing. ChatGPT is a widely used chatbot that generates responses to textual queries. It can comprehend the context and provide coherent responses.

ChatGPT has emerged as an accessible and efficient resource for medical advice seekers. Chatbots can deliver real-time, interactive, personalized patient education and support, helping improve patient outcomes. Nevertheless, data on the utility of ChatGPT to improve nutrition among NCD patients have been limited.

The study and findings

In the present study, researchers compared the nutritional advice provided by ChatGPT with recommendations from international guidelines in the context of NCDs. Analyses were performed using the default ChatGPT model (version 3.5). The team included medical conditions requiring specific nutritional treatments, such as arterial hypertension, T2D, dyslipidemia, obesity, NAFLD, sarcopenia, and chronic kidney disease (CKD).

A set of prompts for these conditions, formulated by doctors and dieticians, was used to obtain dietary advice from the chatbot. Separate chat sessions were conducted for each prompt conversation. ChatGPT’s responses were compared with recommendations from international clinical guidelines. Two dieticians independently assessed and categorized ChatGPT’s responses. Responses were deemed “appropriate” if they aligned with the guidelines and “inappropriate” if contradictory.

Additionally, responses were classified as “unsupported” if they were not confirmed in the guidelines, “not fully matched” if they did not wholly fulfill guidelines, and “general advice” if they were non-specific and promoted a healthy diet overall. Besides, the team also assessed whether ChatGPT could substitute consultation with a dietician in managing complex cases and was presented with a scenario involving (a patient with) multiple coexisting conditions (CKD, obesity, and T2D).

Findings

Overall, the advice provided by ChatGPT was accurate. Appropriateness rates ranged between 55.5% for sarcopenia and 73.3% for NAFLD. One response each for NAFLD and obesity contradicted the guidelines. Regarding obesity, the chatbot suggested regular meals and snacks to stabilize blood sugar levels, whereas the guidelines emphasize avoiding snacks between meals.

Regarding NAFLD, it reported benefits from supplements like omega-3 fatty acids, antioxidants, and vitamin E under medical supervision. However, existing guidelines do not endorse antioxidants and omega-3 fatty acids for NAFLD treatment. One T2D-related recommendation was unsupported by the guidelines.

Specifically, ChatGPT suggested dividing meals into smaller, well-balanced portions as an alternative to three large meals. While this was not directly contradictory to the guidelines, existing T2D guidelines do not address this. Further, numerous responses were categorized as not fully matched with the guidelines.

For example, ChatGPT emphasized incorporating foods with low glycemic index and monitoring portion sizes for hypertriglyceridemia, while guidelines recommend addressing excess body weight and reducing carbohydrates. In addition, ChatGPT offered generic health advice for several conditions. Examples include staying hydrated, avoiding processed foods, and incorporating lean proteins. The chatbot also repeatedly stressed overall well-being and appetite management.

The general advice comprised foods to be included in the diet, which guidelines often do not report. Regarding the scenario of a patient with obesity, CKD, and T2D, a few suggestions by ChatGPT were inappropriate or conflicting. For instance, it emphasized prioritizing lean proteins for muscle health and subsequently suggested limiting overall protein intake. Its responses were generally generic, repeatedly emphasizing consultation with a dietician.

Conclusions

The findings highlight several points of agreement and divergences in ChatGPT’s responses to dietary guidelines. Responses were clear and comprised practical examples of foods to be included or excluded from the diet. Some recommendations by ChatGPT were partially complete. The chatbot failed to provide appropriate guidance in the case of multiple coexisting conditions. While ChatGPT was fairly accurate regarding dietary advice for NCDs, limitations were evident for more complex scenarios. Thus, while ChatGPT can have potential utility, it cannot replace the advice of experts.