Artificial intelligence may be transforming the way firms operate, but according to new research, it’s far from ready to replace human professionals—especially in high-stakes environments like financial advisory, investment analysis, and wealth management.
A study conducted by Carnegie Mellon University suggests that the current generation of AI agents—designed to act autonomously in workplace settings—still struggles to complete even the most basic tasks when evaluated in realistic, scenario-based environments.
In a simulation designed to mimic the dynamics of a small technology company, researchers deployed AI agents to complete routine assignments involving decision-making, file analysis, task delegation, and intra-office communication.
The setting included mock websites, a Slack-style chat interface, HR and CTO support bots, and a host of digital tools reflecting those used in modern workplaces. Tasks ranged from assigning developers to projects based on budget constraints and team availability, to navigating multimedia content and conducting performance evaluations.
Despite the carefully curated environment and clearly outlined objectives, the results were underwhelming. One illustrative example involved an AI agent failing to dismiss a basic pop-up window obstructing access to essential files. Instead of resolving the issue, the agent contacted the human resources bot and then abandoned the task entirely after receiving generic instructions to wait for IT—IT support that, in this case, never arrived. The work was left incomplete.
None of the participants were real. Instead, these were fully autonomous digital agents built using models developed by leading AI firms including OpenAI, Google, Anthropic, and Meta. Each was evaluated for its ability to manage tasks typically performed by employees in areas such as software engineering, corporate administration, and financial operations. In theory, these agents should have been capable of performing complex workflows independently, combining web browsing, file navigation, and internal communications without human oversight.
That’s the ambition behind agent-based AI: moving beyond the simple, one-off interactions of chatbots and into more integrated, decision-capable systems that can operate with autonomy in unpredictable environments.
These agents are designed to act on a user's behalf, taking real-time input and turning it into actionable output—ideally with minimal human prompting. It’s a model that excites business leaders. A recent Deloitte survey of more than 2,500 C-suite executives revealed that over a quarter are actively exploring how to integrate AI agents into their organizations at scale.
Executives across sectors, including tech and financial services, have publicly predicted that these tools will soon reshape back-office operations. Nvidia’s CEO, Jensen Huang, famously suggested that IT departments will evolve into “HR departments for AI agents.”
OpenAI’s Sam Altman went even further, stating that 2024 would be the year AI agents formally "join the workforce." However, this recent study offers a stark counterpoint to that enthusiasm. Despite the high expectations, today’s most advanced AI agents are falling short.
The data is sobering. Anthropic’s Claude 3.5 Sonnet, the top-performing model, completed fewer than 25% of the assigned tasks. Other leading platforms, including Google’s Gemini 2.0 Flash and the large language model that powers ChatGPT, failed to complete more than 10% of the tasks.
In no domain—whether finance, HR, or technical problem-solving—did any model demonstrate mastery or even consistent competence. Graham Neubig, a computer science professor at Carnegie Mellon and a co-author of the study, confirmed that there wasn’t a single category in which AI agents managed to complete a majority of tasks.
For wealth advisors and RIAs, these findings are significant. While AI undoubtedly offers powerful tools for portfolio analysis, client segmentation, and document generation, the notion of replacing professionals—or even fully automating back-office workflows—remains premature.
The current generation of AI agents is still heavily constrained by logic gaps, lack of contextual understanding, and an inability to adapt to unstructured problems. In the context of wealth management, where compliance, client trust, and nuanced interpretation of financial data are paramount, these limitations carry serious implications.
Importantly, the Carnegie Mellon research represents one of the first large-scale efforts to evaluate AI agents on their ability to function in practical, scenario-driven environments. Unlike prior studies, which often relied on subjective forecasts from industry insiders or broad extrapolations from static capabilities, this benchmark focused on measurable outcomes tied directly to task completion. The goal was to create a utility-based assessment that mirrors the complexity of real-world jobs, moving beyond speculative automation risk assessments.
Two years ago, OpenAI released a much-discussed paper forecasting that roles such as financial analysts, administrators, and research professionals were among the most vulnerable to AI disruption. However, those predictions were based on surveys and assumption-driven models rather than real-time testing. The Carnegie Mellon team sought to rectify that oversight by grounding their evaluation in live task execution. Their results reinforce the importance of separating theoretical automation potential from real-world capability.
That distinction matters deeply for advisory practices navigating questions around digital transformation. Many firms are actively experimenting with AI integration—from natural language processing to predictive analytics and client behavior modeling.
These tools can enhance productivity, but they are far from turnkey replacements for human expertise. For example, while AI may help prepopulate financial planning documents or surface risk signals in client portfolios, interpreting those outputs still requires professional judgment and regulatory awareness.
Moreover, the study underscores a central tension in the current AI discourse: while models can mimic human conversation and produce plausible-sounding outputs, their underlying reasoning and decision-making remain brittle. In finance, where errors carry fiduciary consequences and trust is non-negotiable, this fragility presents a real barrier to full automation.
That’s not to say AI agents don’t hold promise. Even in this study, partial successes point to where the technology might eventually make a meaningful impact. In structured environments with well-defined rules and outcomes—such as spreadsheet analysis, basic reporting, or document summarization—AI agents showed flashes of utility. These are precisely the types of tasks that could be selectively automated to free up time for deeper client engagement, holistic planning, and strategic advisory work.
Advisors should view the evolution of AI agents not as a threat, but as an opportunity to refocus their value proposition. As technology assumes greater responsibility for rote tasks, the differentiator for RIAs will be empathy, behavioral coaching, and bespoke financial strategy—skills that remain well outside the capabilities of today’s AI. This reinforces the importance of upskilling teams to work alongside AI rather than competing with it.
The broader message for the advisory industry is clear: while AI agents may one day serve as meaningful collaborators in the workplace, they are not ready to take over core responsibilities. For now, their most productive use lies in augmenting, not replacing, human intelligence. As tools mature and benchmarks evolve, advisors who remain adaptive and discerning will be best positioned to harness AI’s potential while preserving the human connection that defines exceptional financial service.
