Robotic Risk—but Is the Ultimate Answer Still a Human One?

By: Lauren Scribner

It now goes without saying that Artificial Intelligence (AI) can be a convenient solution for accomplishing certain tedious tasks, such as planning an upcoming trip, figuring out what to make for dinner with ingredients already on hand, or adding some professional flare to an email. But when it comes to some of the more high-stakes areas of life, such as obtaining financial, medical, or legal advice, the convenience of instantaneous feedback available at the push of the button may not be the be-all and end-all.

OpenAI just implemented a new personal finance feature on ChatGPT where users can link their accounts to the application and “get more personalized advice.”[1] The company has advertised a demonstrated need for this feature, stating that 200 million monthly users “already turn to ChatGPT for guidance on managing their money.”[2]

Or take medical advice. An April 2025 survey revealed that “nearly 8 in 10 adults say they’re likely to go online to answer a specific question about health symptoms or a health condition” and “[n]early two-thirds of this group say they have seen AI-generated responses” in the form of an “AI Overview” on Google or a “Copilot Answer” on Microsoft Bing.[3] Approximately 75% of respondents reported the AI-generated response at least sometimes, often, or always gives them the answer they need.[4] Nearly two-thirds of the respondents found “AI-generated results to be somewhat or very reliable.”[5]

And don’t forget legal advice, where there is a reported “wave of pro se litigants using AI tools to be their own advocate in court.”[6] Especially in civil cases where there is no option for a public defender or court-appointed attorney, AI is reported to “fill long-standing gaps in legal-aid services, which leave many would-be plaintiffs on the sidelines if they don’t have the means or ​wherewithal to hire an attorney.”[7]

But relying on AI for advice that was once strictly relegated to a financial, medical, or legal professional comes with major risks. At worst, the technology can be outright wrong,[8] as we have seen with the proliferation of AI-authored legal filings “with hallucinated case law.”[9] But even in situations where AI provides “accurate” advice, it may not be the best advice for an individual when taking a bird’s-eye view. One of the common threads between financial, medical, and legal advice is the lasting impact that decisions in these realms tend to have on people’s lives.

For one thing, a key and consistent feature of AI applications is their reliance on whatever a user inputs into their system. And as my seventh-grade math teacher used to say whenever we picked up our fancy graphing calculators, “junk in will give you junk out.” Much like a calculator,[10] AI’s helpfulness is directly proportional to the accuracy and sophistication of the information inputted by the user. When it comes to legal assistance, for example, if a pro se litigant is unfamiliar with the applicable law and procedural posture of a case, that litigant may still be left with junk no matter how much “help” AI offers.

Relatedly, pro se litigants may not fully grasp the legal implications of AI-generated outputs. A recent example is a life insurance company is seeking to hold Open AI responsible for a pro se litigant who violated her settlement agreement with the insurer.[11] In its filings, even OpenAI touts the necessity of legal counsel by highlighting that users must “expressly agree that they will not rely on [the application’s output] ‘as a substitute for professional advice’” and that “[d]espite  agreeing to those terms, [the pro se litigant] ‘used ChatGPT as a tool to help her . . . navigate the legal system without the benefit of counsel.’”[12]

So where does that leave us? When it comes to high-stakes decisions, a combination of both AI and human experts is likely necessary and ideal. AI tools are an easy and convenient way to gain initial insight into a situation that can ultimately make the interaction with a human expert more efficient and productive. Use AI to:

• process a concern with a chatbot to organize your thoughts in preparation for a conversation with the expert;
• generate a list of questions to ask the expert;
• learn the language of the industry so you can be a more informed participant in the decision-making process;
• get a first look at the potential pros and cons for pursuing a certain course of action; or
• research what courses of action have worked well for others in your situation.

Then, discuss that information with a human expert who can advise on the different courses of action and their potential long-range impacts.

In short, high-stakes scenarios like financial help, medical concerns, and legal needs all generally require a multifaceted and nuanced approach that considers a comprehensive view of the client’s “big picture.” There is a reason that expert advice in these areas is often expensive and time consuming—it is complicated. Investigating different solutions and determining how those courses of action could unfold and impact a client’s life requires dedicated attention unlikely to be achieved solely by using an application on a cellphone.

[1] Bonifacic, Igor, ChatGPT will offer personalized financial advice (if you connect your bank account), Engadget (May 15, 2026), https://www.engadget.com/2173768/chatgpt-will-offer-personalized-financial-advice-if-you-connect-your-bank-account/ (last accessed: May 25, 2026).

[2] Id.

[3] Many in U.S. Consider AI-Generated Health Information Useful and Reliable, University of Pennsylvania Annenberg Public Policy Center (July 14, 2025), https://www.annenbergpublicpolicycenter.org/many-in-u-s-consider-ai-generated-health-information-useful-and-reliable/ (last accessed: May 25, 2026).

[4] Id.

[5] Id.

[6] Randazzo, Sara, No lawyer? No money? More Americans are suing with AI help?, Reuters (May 15, 2026), https://www.reuters.com/legal/government/no-lawyer-no-money-more-americans-are-suing-with-ai-help-2026-05-15/ (last accessed: May 25, 2026).

[7] Id.

[8] Take, for example, the chatbot on Character.AI claiming to be a “licensed medical professional” that went as far as indicating “it had gone to medical school at Imperial College London and was licensed to practice medicine in the U.K. and Pennsylvania” and “even provided a fake Pennsylvania medical license number.” Bond, Shannon, Pennsylvania sues Character.AI over claims chatbot posed as doctor, NPR (May 5, 2026), https://www.npr.org/2026/05/05/nx-s1-5812861/characterai-chatbot-medical-advice-pennsylvania-lawsuit (last accessed: May 25, 2026).

[9] Id. As of May 25, 2026, at least 1,494 cases have been identified “where generative AI produced hallucinated content.” See Charlotin, Damien, AI Hallucination Cases (May 25, 2026), https://www.damiencharlotin.com/hallucinations/?q=&sort_by=-date&period_idx=0&page=2 (last accessed: May 25, 2026). Of concern is that “[t]he pattern emerging across these cases is that people keep trusting AI’s answers even when they know the systems can be wrong.” Melendez, Steven, AI keeps inventing fake cases. Lawyers keep citing them, Scientific American (May 22, 2026), https://www.scientificamerican.com/article/why-lawyers-keep-citing-fake-cases-invented-by-ai/ (last accessed: May 25, 2026).

[10] Current AI models are undoubtedly more sophisticated than a calculator, but the analogy still holds.

[11] Waiss, Alexis, OpenAI Dismissal Motion Says ChatGPT Is Mere Tool, Not Attorney, Bloomberg Law (May 18, 2026), https://news.bloomberglaw.com/litigation/open-ai-dismissal-motion-says-chatgpt-is-mere-tool-not-attorney (last accessed: May 25, 2026).

[12] Id.