Gemini 2.0 Flash heralds a new era of real-time multimedia AI

Google’s version Gemini 2.0 flash this weekwhich offers users a way to directly interact with the videos surrounding them, has set the stage for what could be a pivotal shift in how businesses and consumers interact with technology.

This release — along with announcements from OpenAI, Microsoft and others — is part of a transformative leap forward happening in technology called “multimodal AI.” This technology allows you to take video clips – or audio or photos – that arrive on your computer or phone, and ask questions about it.

It also signals an intensifying competitive race between Google and its main rivals – OpenAI and Microsoft – to dominate AI capabilities. But more importantly, it seems to define the next era of interactive and efficient computing.

This moment in AI feels to me like an “iPhone moment,” and by that I am referring to the period 2007-2008 when Apple released the iPhone which, with its Internet connectivity and great user interface, transformed everyday life by giving people the experience of a powerful computer in their pocket. .

While OpenAI’s ChatGPT may have ushered in AI’s latest moment with its powerful human-like chatbot in November 2022, Google’s release here at the end of 2024 feels like a major continuation of that moment — at a time when many observers were getting nervous. About the potential slowdown in AI technology improvements.

Gemini 2.0 Flash: The catalyst for the multimedia AI revolution

Google’s Gemini 2.0 Flash provides groundbreaking functionality, allowing real-time interaction with videos captured via a smartphone. Unlike previously organized demonstrations (such as Google’s Astra project in May), this technology is now available to ordinary users through Google Artificial Intelligence Studio.

I encourage you to try it for yourself. I used it to view and interact with my surroundings — which for me this morning were my kitchen and dining room. You can immediately see how this offers breakthroughs in education and other use cases. You can see why creator Jerrod Liu reaction On X yesterday I was amazed when he used Gemini 2.0 Realtime AI to edit a video in Adobe Premiere Pro. “This is absolutely crazy,” he said, after Google guided him within seconds on how to add a basic blur effect even though he was a novice user.

Sam Witteveen, a prominent AI developer and co-founder of Red Dragon AI, was given early access to test Gemini 2.0 Flash, and stressed that Gemini Flash’s speed — it’s twice as fast as Google’s flagship to date, Gemini 1.5 Pro — is “insanely so.” The “cheap price” makes it not just a showcase for developers to test new products, but a practical tool for organizations managing AI budgets. (To be clear, Google hasn’t actually announced Gemini 2.0 Flash pricing yet. It’s a free preview. But Witteveen is basing his assumptions on the precedent set by Google’s Gemini 1.5 series.)

For developers, the direct API of these live multimedia features offers great potential, as they enable seamless integration into applications. This API is also available for use; A Demo app available. Here it is Google blog post for developers.

Programmer Simon Willison The API is called Next Level Streaming: “These things are straight out of science fiction: being able to have a voice conversation with an LLM student about things he can ‘see’ through a camera is one of those ‘we live in the future’ moments.” He pointed to the way the API requires code execution mode to be enabled, which allows models to write Python code, run it, and consider the result as part of its response — all part of the agent’s future.

Clearly, this technology is a harbinger of new application ecosystems and user expectations. Imagine being able to analyze live video during a presentation, suggest edits, or troubleshoot problems in real time.

Yes, this technology is great for consumers, but it’s important for enterprise users and leaders to embrace it as well. New features represent the foundation for a whole new way of working and interacting with technology – signaling the coming productivity gains and creative workflows.

The competitive landscape: a race to define the future

The release of Google Gemini 2.0 Flash on Wednesday comes amid a flurry of releases from Google and by its major rivals, which are rushing to ship their latest technology by the end of the year. All promise to offer consumer-ready multimedia capabilities—live video interaction, image generation, and sound synthesis—but some are not fully developed or even fully available.

One reason for the rush is that some of these companies are offering their employees bonuses for delivering key products before the end of the year. Another is bragging rights when they get new features first. They can get significant user traction by being first, as OpenAI showed in 2022, when its ChatGPT became the fastest-growing consumer product in history. Although Google had similar technology, it was not ready for a public release, and it remained the same. Observers have since criticized Google harshly for being too slow.

Here’s what other companies have announced in the past few days, all helping to deliver this new era of multi-modal AI.

1. Advanced sound mode with vision in OpenAI: It was launched yesterday but is still in progress It offers features like real-time video analysis and screen sharing. Although early access issues were promising, they limited their immediate impact. For example, I haven’t been able to access it yet even though I’m a Plus subscriber.
2. Vision of Microsoft’s Copilot:last week, Microsoft launched similar technology in preview -Only for a select group of its professional users. Its in-browser design suggests enterprise applications but lacks the polish and accessibility of Gemini 2.0. Microsoft also released a The Phi-4 model is fast and powerful To boot.
3. Anthropic Claude 3.5 Haiku: Anthropic, so far in a hot race to lead the large language model (LLM) with OpenAI, hasn’t offered anything as cutting-edge on the multimodal side. I just did it Haiku version 3.5 is efficient and fast. But its focus on lower cost and smaller model sizes belies the boundary-pushing features in Google’s latest release, and those of OpenAI’s voice-to-vision mode.

Overcoming challenges and seizing opportunities

Although these technologies are revolutionary, challenges remain:

• Accessibility and scalability: OpenAI and Microsoft have faced bottlenecks in the rollout, and Google must ensure it avoids similar risks. Google has indicated that its live streaming feature (Project Astra) has a contextual memory limit of Up to 10 minutes of memory during a sessionalthough it is likely to increase over time.
• Privacy and security: AI systems that analyze video or personal data in real time need strong safeguards to maintain trust. Google’s Gemini 2.0 Flash model has native image generation, access to third-party APIs, and the ability to tap into a Google search and execute code. This is all powerful, but it can make it dangerously easy for someone to accidentally release private information while manipulating these things.
• Ecosystem integration: As Microsoft leverages its enterprise suite and Google embeds itself in Chrome, the question remains: which platform provides the most seamless experience for enterprises?

However, the potential benefits of the technology trump all these hurdles, and there is no doubt that developers and project companies will rush to embrace it over the next year.

Conclusion: A new dawn, currently led by Google

As developer Sam Witveen and I Discuss on our podcast Recorded Wednesday night after Google’s announcement, Gemini 2.0 Flash is a truly impressive release, marking the moment when multi-modal AI became a reality. Google’s advances have set a new standard, although this feature may be very fleeting. OpenAI and Microsoft are hot on their tail. We are still very early in this revolution, just as in 2008, when despite the release of the iPhone, it was not clear how Google, Nokia and RIM would respond. History has shown that Nokia and RIM didn’t do that, and they died. Google has responded really well, and given the iPhone a run.

Likewise, Microsoft and OpenAI are clearly competing strongly in this race with Google. Meanwhile, Apple has decided to partner on the technology, and this week announced further integration with ChatGPT — but it’s certainly not trying to win outright in this new era of multimedia displays.

On our podcast, Sam and I also cover the special strategic advantage Google has around browser territory. For example, Project Mariner, a Chrome extension, lets you do real-world web browsing tasks with more functionality than competing technologies offered by Anthropic (called Use of computer) and Microsoft OmniParser (still under research). (It’s true that Anthropic gives you greater access to your computer’s local resources.) All of this gives Google a head start in the race to push agent AI technologies forward in 2005 as well, even if Microsoft seems like… Moving forward on the actual implementation side of delivering proxy solutions to enterprises. AI agents perform complex tasks autonomously, with minimal human intervention – for example, they will soon perform advanced research tasks and scan databases before conducting e-commerce, trading stocks or even purchasing real estate.

Google’s focus on making these Gemini 2.0 capabilities available to both developers and consumers is smart, because it ensures it approaches the industry with a comprehensive plan. Until now, Google has suffered from a reputation for not being as aggressively developer-focused as Microsoft.

The question for decision makers is not whether or not to adopt these tools, but rather how quickly they can be integrated into the workflow. It will be great to see where next year takes us. Be sure to listen to our tips for enterprise users in the video below:

Daily insights into business use cases with VB Daily

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from organizational transformations to hands-on deployments, so you can share insights to maximize ROI.

Read our privacy policy

Thanks for subscribing. Check more VB Newsletters are here.

An error occurred.