The Future of AI Infrastructure: A Conversation with SambaNova Systems CEO Rodrigo Liang
To effectively compete, companies must take a hard look at what they can do to support an AI infrastructure. On this episode of the At the Edge podcast, SambaNova Systems cofounder and CEO Rodrigo Liang joins host and McKinsey Senior Partner Lareina Yee to discuss agentic AI, the S-curve of AI value, and why businesses must adopt a hybrid AI model.
The following transcript has been edited for clarity and length.
Rethinking AI infrastructure
Lareina Yee: SambaNova is an ambitious, completely exciting AI company addressing an enormous market. Can you tell us what you originally saw in the marketplace that inspired you to start SambaNova?
Rodrigo Liang: I have two amazing cofounders, Stanford Professors Kunle Olukotun and Christopher Ré, and the three of us got together and really started thinking about this worldwide transformation we’re going through. If you think about this AI-first, AI-centric world we’re building, it’s ultimately driving a scale of transformation we’ve only seen a few times over the last two or three decades.
So the genesis of SambaNova came from this brainstorming process to see if the computing infrastructure we’re running on was really the most efficient. And the conclusion, based on Stanford research, is that there are significantly better ways to enable AI. That’s when we decided to embark on this journey seven and a half years ago.
Lareina Yee: Seven years ago, there were some of us, myself included, who were super interested in data centers. Today, it’s become a hot topic, and everyone’s talking about infrastructure.
McKinsey calculates a roughly $5 trillion investment needed over the next five years to build all the data center infrastructure, including buildings, software, cooling systems, and energy plants, to power AI’s voracious appetite. How do you think about the dynamics of the cost, the innovation, and the moment we’re in?
Rodrigo Liang: There are three things I think are incredibly important for us to think about as we’re building out the scale.
In the last three years, we’ve already seen an incredible build-out of GPUs [graphics processing units], AI infrastructure, and teraflops [floating-point operations per second]. Most of this build-out has been for pretraining large models and is really dominated by the largest players in the world. But as you move forward, you’re seeing a world that wants to do inference, do test-time computing, and all these different things requiring the models we’ve trained.
But as we scale up, we’re now seeing other constraints start to appear, like a lack of sufficient power for these data centers. So people are talking about nuclear power plants and other sources of energy. But then you have to figure out how to get the cooling done as well.
And as you think about energy, you’ll also need to figure out how to update your entire grid to power those gigawatt data centers. And eventually, you’ve got to get all of that back-connected to where the users are, which is mainly in these large metropolitan areas—which is not where you’re going to put your gigawatt data center.
So, there are a lot of infrastructure challenges we have to figure out, and at SambaNova, we’re very focused on making it all easier. We’re dedicated to figuring out how to deliver the capacity you need at a fraction of the cost and a fraction of the power.
We all need to contribute to the solution because the answer can’t be, “Just build more power plants and build more data centers.” It’s too hard. You will need those, but the core tech also needs to be significantly more efficient.
Bookending the technology stack
Lareina Yee: Tell us about some of that magic sauce around efficiency that SambaNova is working on. If I’m an average layperson thinking about this, how do I understand the important role you play in this ecosystem?
Rodrigo Liang: Think of SambaNova as bookends on the technology stack. On the one hand, we build chips, and on the other, we create API services that allow the best open-source models to be accessed without having to actually invest in all of this custom-model work.
With SambaNova, you can go to cloud.sambanova.ai and use all the best open-source models, with all the benefits and full accuracy, with world-record speeds at a very efficient cost. Because as soon as you actually deploy AI, the cost of infrastructure acquisition, power, networking, and all the things that are required starts adding up.
And if you’re going to go from this world of training to what I think is going to be a tenfold increase in investment for inferencing, you have to be more efficient. You have to make the cost come down. Otherwise, it won’t scale.
Planning for a hybrid model
Lareina Yee: Let’s just fast-forward and assume businesses will figure out how to scale AI. So, if I’m a business leader, how do I plan?
Rodrigo Liang: The companies that win will use AI to provide better services in the market, engaging with customers faster and better and making customization easier. They will also change their operations so AI can give them a significantly better time to market and a significantly better customer experience.
So, your AI solution is going to be a hybrid model. Just like you have cloud and on-premises, you’re going to have large language models [LLMs] and custom LLMs. You’re also going to have text, vision, language, and voice models.
When you run a company, you have your own custom methods to accomplish your various operational needs. But to fully embrace hybrid, the data will anchor where your AI models run, whether it’s in cloud A, cloud B, or on-premises.
That’s how we think about how you should deploy infrastructure. Let the data reign and drive the solution you need because you’re going to be hybrid anyway.
The beauty of small language models
Lareina Yee: I’ve had conversations with large businesses that are enthusiastic about huge LLMs but say the secret sauce is in the user experience [UX] with small models that only tap into their internal data. They may not always need the internet at their fingertips, but there are other times when they actually do.
Another thing on people’s minds is agentic AI, and there are some really interesting ways to make that happen for businesses. Rodrigo Liang, an expert in the field, shared some valuable insights on how businesses can leverage agentic AI to drive value and transform workflows.
One key aspect highlighted by Liang is the ability to use large language models (LLMs) on-premises, giving businesses the flexibility to work with models trained on their own private data. This ensures that businesses are not restricted to small models and can access their own large-class LLMs securely and privately. This capability is particularly valuable as it allows businesses to explore a wide range of queries without limitations.
While smaller models may be accurate for specific tasks, they can be more brittle and prone to breaking when faced with prompts outside their training. This is where agentic AI shines, as it allows for the creation of specialized agents that can handle specific tasks efficiently. These agents, when connected in a workflow, can provide real-time responses to users, enhancing the overall user experience.
Maintaining workflow security
Security is another crucial factor when implementing agentic AI in workflows. Liang emphasized the importance of ensuring that access to sensitive information is controlled within the workflow. Businesses need to establish mechanisms to restrict access to certain agents based on user permissions, especially when dealing with regulated or confidential data.
The S-curve of AI value
Liang discussed the concept of the S-curve of AI value, where businesses start with simple, low-risk tasks before scaling up to more complex and valuable applications. He highlighted the potential for AI to drive significant cost savings and efficiency gains in areas such as compliance reporting, where automation can streamline processes and improve accuracy.
By focusing on transformational rather than basic automation, businesses can unlock greater value from their AI investments. Liang emphasized the need for enterprises to embrace change and integrate AI technology into their workflows effectively to realize the full potential of AI.
Basic versus transformational change
Liang also touched on the distinction between basic task automation and transformational workflow change. While many businesses may start with basic automation, they may struggle to see the expected business value. By shifting towards transformational change, businesses can drive innovation and create significant value through AI-driven workflows.
Overall, the insights shared by Rodrigo Liang underscore the potential of agentic AI to revolutionize business workflows and drive value across industries. By leveraging large language models, ensuring workflow security, and focusing on transformational change, businesses can harness the power of AI to enhance efficiency, improve decision-making, and stay ahead in a rapidly evolving digital landscape.
In today’s rapidly evolving business landscape, the integration of artificial intelligence (AI) has become a game-changer for companies looking to streamline processes, increase efficiency, and drive innovation. With the ability to scan and analyze vast amounts of data at an unprecedented speed, AI is revolutionizing industries across the board.
One of the key benefits of AI is its ability to handle complex tasks that would be time-consuming and labor-intensive for humans. For example, AI can scan 100,000 SKUs of all your products and learn every single product spec with precision. This level of analysis far surpasses what any human engineer could achieve, making AI an invaluable tool for product development and optimization.
During a recent discussion between Lareina Yee and Rodrigo Liang, the potential of AI in various industries was highlighted. From drug discovery to energy exploration, AI is being utilized to revolutionize processes that were previously slow and inefficient. For example, SambaNova’s work with the US government during COVID to create a surrogate AI scientist has significantly accelerated the drug discovery process, allowing for faster experimentation and results.
Furthermore, AI inferencing, the process of applying trained models to new data, is becoming increasingly important in the AI ecosystem. As open-source models continue to improve, companies can leverage existing models and customize them for their specific needs, saving time and resources on training new models from scratch.
Looking ahead, Liang predicts a future where every individual will have a suite of personalized AI agents to assist with various tasks and workflows. These agents, akin to templates in software design, will streamline processes and enhance productivity in both professional and personal settings.
When it comes to robotics, Liang envisions a future where specialized robots will be deployed in manufacturing and business settings, revolutionizing production processes and increasing efficiency. While humanoid robots may still be a bit further down the road, the integration of robots into everyday tasks is already underway.
For businesses looking to capitalize on the mainstreaming of AI, AI agents, and robotics, Liang advises taking an inventory of their operations and embracing a hybrid approach. By combining cloud-based solutions with on-premises capabilities, companies can leverage the best of both worlds to optimize their processes and drive innovation.
As AI continues to reshape industries and redefine business operations, companies that embrace these technologies and adapt to the changing landscape will be well-positioned for success in the future. By leveraging the power of AI, businesses can unlock new opportunities, drive efficiency, and stay ahead of the competition in an increasingly digital world.
The Strategic Imperative of Deploying Institutional Learning for Competitive Advantage
As organizations navigate the rapidly evolving landscape of technology and innovation, one thing is clear: the ability to leverage emerging technologies such as artificial intelligence (AI) will be a key differentiator in the market. However, simply having access to the latest tech is not enough. Success lies in the effective deployment of these technologies through institutional learning and change management.
Your ability to use the tech is going to differentiate you in the market. And every single one of your competitors in the market is also trying to do the same.
So you have to start deploying some things in certain locations that allow you to create that institutional learning. Behind the tech, the bigger thing is change management, which has to happen for the production. The faster you can get through that curve, the more effectively you’ll be able to take advantage of the technology.
AI Access for Everyone
Lareina Yee: We’ve talked a lot about AI, but I’d love to just talk a little bit about you. You’re from Brazil and have had an incredible career. Tell us a little bit about your wish list if you were bringing AI to Brazil.
Rodrigo Liang: AI’s going to be pervasive and should not be available only to those who can afford it. I think everybody should have access to this technology, regardless of where on the planet they live. Also, in every market we enter, SambaNova is very invested in linguistics, because most countries don’t want to be English-first; they want everything translated into their own language.
So when SambaNova enters a market, we come prepared for the native language, or we work with locals to help us operate in their language. Because whether it’s for customer support, interpreting documents, or translating audio and video, it needs to be native.
Lareina Yee: Final question. Tell us the origins of the SambaNova name.
Rodrigo Liang: My cofounder, Kunle, is of Nigerian background, and we had a company that was named Afara, which in his language meant “bridge.” So this time around, we were talking about something Brazilian because of my background.
And if you really want a word that immediately makes you think of Brazil, it’s down to either samba or rio. One thing led to another, and SambaNova came together. It’s a new dance. Since SambaNova technologies are about data flow, it’s all about allowing these models to operate on their own without having to parse them, cut them, or do all the legacy things we do to workflows.
So the name stuck because it’s at the essence of what we do. Let the technology flow out there and see how it goes.
The Path to Competitive Edge
In today’s hyper-competitive market, organizations must not only embrace emerging technologies like AI but also ensure that they have the capability to effectively deploy and utilize these tools to gain a competitive edge. Your ability to use the tech is going to differentiate you in the market. And every single one of your competitors in the market is also trying to do the same. The one that gets there first by using the tech most effectively will gain a competitive edge.
It is crucial for organizations to focus on institutional learning and change management to facilitate the successful integration of technology into their operations. This involves creating a culture of continuous learning, upskilling employees, and ensuring that all stakeholders are aligned towards the common goal of leveraging technology for business success.
Recommendations for Success
Based on industry insights and best practices, here are some actionable recommendations for organizations looking to deploy institutional learning for competitive advantage:
- Invest in comprehensive training programs to upskill employees on the latest technologies and tools.
- Establish clear communication channels to ensure alignment across all levels of the organization.
- Encourage a culture of experimentation and innovation to foster continuous learning and adaptation.
- Collaborate with external partners and experts to stay abreast of industry trends and best practices.
- Monitor and measure the impact of technology deployment on key performance indicators to drive continuous improvement.
Market Trends and Organizational Impact
The rapid advancement of technology, particularly in the field of AI, is reshaping industries and markets across the globe. Organizations that are able to harness the power of AI and other emerging technologies stand to gain a significant competitive advantage in terms of efficiency, productivity, and innovation.
By deploying institutional learning and change management strategies, organizations can not only adapt to the changing landscape of technology but also thrive in a highly competitive market. The ability to effectively utilize technology will be a key driver of success in the digital age.
FAQ
Q: What is the role of institutional learning in technology deployment?
A: Institutional learning plays a critical role in the successful deployment of technology within organizations. By fostering a culture of continuous learning and upskilling, organizations can ensure that their employees are equipped to effectively utilize emerging technologies for competitive advantage.
Q: How can organizations ensure effective change management during technology deployment?
A: Effective change management involves clear communication, stakeholder engagement, and a structured approach to implementing new technologies. By involving all relevant parties in the process and addressing potential challenges proactively, organizations can navigate the complexities of technology deployment more successfully.
Conclusion
In conclusion, the strategic imperative of deploying institutional learning for competitive advantage cannot be overstated. As organizations strive to differentiate themselves in a crowded market, the ability to effectively leverage technology through institutional learning and change management will be a key factor in their success.
By investing in training, communication, and a culture of innovation, organizations can position themselves at the forefront of technological advancement and gain a competitive edge in the digital age. The time to start deploying these strategies is now, as the market waits for no one.
