Unlocking value from data: How AI agents are conquering 2024

If 2023 is the year of AI-powered chat and search bots, The year 2024 was all about AI agents. What started with Devin earlier this year has evolved into a full-fledged phenomenon, offering organizations and individuals a way to transform the way they work at different levels, from programming and development to personal tasks like planning and booking tickets for a vacation.

Among these wide-ranging applications, we have also seen the emergence of… Data agents This year – AI-powered agents handle different types of tasks across the data infrastructure stack. Some did basic data integration work while others handled downstream tasks, such as analysis and management in the pipeline, making things simpler and easier for enterprise users.

The benefits were improved efficiency and cost savings, leading many to ask: How will things change for data teams in the coming years?

Artificial general intelligence agents took over the data tasks

While agent capabilities have been around for some time, allowing organizations to automate some basic tasks, the emergence of… Generative artificial intelligence He completely took things to the next level.

With natural language processing capabilities and the use of AI generation tools, agents can go beyond simple reasoning and answer the actual planning of multi-step actions, interacting autonomously with digital systems to complete actions while collaborating with other agents and people at the same time. They also learn to improve their performance over time.

Cognition Artificial Intelligence Devin It was the first major offering to the dealership, enabling large-scale engineering operations. Subsequently, larger players began providing more targeted institutional and personal agents powered by their models.

In a conversation with VentureBeat earlier this year, Google Cloud’s Gerrit Kazmaier said he heard from customers that their data practitioners were constantly facing challenges including automating the manual work of data teams, reducing the cycle time of data and analysis pipelines and simplifying data management. Essentially, teams were not short of ideas about how to create value from their data, but they lacked the time to implement those ideas.

To solve this problem, Kazmaier explained, Google revamped BigQuery, its underlying data infrastructure offering, with Gemini AI. The resulting agent capabilities not only provide organizations with the ability to discover, cleanse, and prepare data for downstream applications—breaking down data silos and ensuring quality and consistency—but they also support pipeline management and analysis, freeing teams to focus on higher-value tasks.

Many organizations today use BigQuery’s Gemini agent capabilities, including a fintech company Golowhich exploited Gemini’s ability to understand complex data structures to automate its query generation process. Japanese IT company Henry It also uses BigQuery’s Gemini SQL generation capabilities to help its data teams deliver insights more quickly.

But discovering, preparing and helping with analysis was just the beginning. As core models have evolved, even fine-grained data operations — once pioneered by startups specializing in their own fields — have been targeted by deeper agent-based automation.

For example, AirByte and Fasten It made headlines in the data integration category. The first launched a helper that created data connectors from the API documentation link in seconds. At the same time, the latter enhanced its broader application development offering with agents that created enterprise-level APIs — whether to read or write information on any topic — using only natural language description.

Based in San Francisco Alternative artificial intelligenceFor its part, it targeted various data operations, including authentication, testing, and transformations, with the new DataMates technology, which used proxy AI to pull context from the entire data set. Many other startups, incl Redbird and RapidCanvasalso worked in the same direction, claiming to offer AI agents that can handle up to 90% of the data tasks required in the AI ​​and analytics pipelines.

Agents running RAG and more

In addition to large-scale data operations, proxy capabilities have also been explored in areas such as retrieval augmented generation (RAG) and back-end workflow automation. For example, the team behind the vector database Deaths I recently discussed an idea RAG Agenta process that allows AI agents to access a wide range of tools—such as web search, a calculator, or a software API (such as Slack/Gmail/CRM)—to retrieve and validate data from multiple sources to improve the accuracy of answers.

Moreover, at the end of the year, Snowflake intelligence emerged, giving organizations the option to set up data agents that can leverage not only BI data stored in a Snowflake instance, but also structured and unstructured data across siled third-party tools – such as sales transactions in a database, documents in knowledge bases like SharePoint, and information in tools Productivity like Slack, Salesforce, and Google Workspace.

With this additional context, agents display relevant insights in response to natural language questions and take specific actions around the insights generated. For example, a user can ask their data agent to enter the insights generated into an editable form and upload the file to their Google Drive. They may also be required to write to Snowflake tables and make modifications to data as needed.

More to come

While we may not have covered every data agent app seen or announced this year, one thing is very clear: the technology is here to stay. As AGI models continue to evolve, adoption of AI agents will move full steam ahead, as most organizations, regardless of their sector or size, choose to delegate repetitive tasks to specialized agents. This will translate directly into efficiency.

As evidence of this, in a recent survey of 1,100 technology executives Capgemini82% of respondents said they intend to integrate AI-driven agents across their groups within the next three years – up from 10% currently. More importantly, as many as 70 to 75% of respondents said they trust an AI agent to analyze and compile data on their behalf, as well as handle tasks like building and improving code on a recurring basis.

This agent-driven shift also means big changes in how data teams work. Currently, agents’ results are not at production level, which means a human must take over at some point to adjust the work to suit their needs. However, with some additional developments over the coming years, this gap will likely disappear, giving teams AI agents that will be faster, more accurate, and less prone to mistakes that humans typically make.

So, bottom line, the roles of data scientists and analysts we see today will likely change, with users likely moving into the realm of AI monitoring (where they can monitor the actions of the AI) or higher value tasks needed by the system. Can struggle to perform.

Daily insights into business use cases with VB Daily

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from organizational transformations to hands-on deployments, so you can share insights to maximize ROI.

Read our privacy policy

Thanks for subscribing. Check more VB newsletters here.

An error occurred.