How to prevent your software update from being the next CrowdStrike

by with No Comment Tech

Times Square billboards displaying Windows blue screen of death after CrowdStrike outage on July 19, 2024.

Image Credits: Selcuk Acar/Anadolu / Getty Images

CrowdStrike released a relatively minor patch on Friday, and somehow it wreaked havoc on large swaths of the IT world running Microsoft Windows, bringing down airports, healthcare facilities and 911 call centers. While we know a faulty update caused the problem, we don’t know how it got released in the first place. A company like CrowdStrike very likely has a sophisticated DevOps pipeline with release policies in place, but even with that, the buggy code somehow slipped through.

In this case it was perhaps the mother of all buggy code. The company has suffered a steep hit to its reputation, and the stock price plunged from $345.10 on Thursday evening to $263.10 by Monday afternoon. It has since recovered slightly.

In a statement on Friday, the company acknowledged the consequences of the faulty update: “All of CrowdStrike understands the gravity and impact of the situation. We quickly identified the issue and deployed a fix, allowing us to focus diligently on restoring customer systems as our highest priority.”

Further, it explained the root cause of the outage, although not how it happened. That’s a post mortem process that will likely go on inside the company for some time as it looks to prevent such a thing from happening again.

Dan Rogers, CEO at LaunchDarkly, a firm that uses a concept called feature flags to deploy software in a highly controlled way, couldn’t speak directly to the CrowdStrike deployment problem, but he could speak to software deployment issues more broadly.

“Software bugs happen, but most of the software experience issues that someone would experience are actually not because of infrastructure issues,” he told TechCrunch. “They’re because someone rolled out a piece of software that doesn’t work, and those in general are very controllable.” With feature flags, you can control the speed of deployment of new features, and turn a feature off, if things go wrong to prevent the problem from spreading widely.

It is important to note however, that in this case, the problem was at the operating system kernel level, and once that has run amok, it’s harder to fix than say a web application. Still, a slower deployment could have alerted the company to the problem a lot sooner.

What happened at CrowdStrike could potentially happen to any software company, even one with good software release practices in place, said Jyoti Bansal, founder and CEO at Harness, a maker of DevOps pipeline developer tools. While he also couldn’t say precisely what happened at CrowdStrike, he talked generally about how buggy code can slip through the cracks.

Typically, there is a process in place where code gets tested thoroughly before it gets deployed, but sometimes an engineering team, especially in a large engineering group, may cut corners. “It’s possible for something like this to happen when you skip the DevOps testing pipeline, which is pretty common with minor updates,” Bansal told TechCrunch.

He says this often happens at larger organizations where there isn’t a single approach to software releases. “Let’s say you have 5,000 engineers, which probably will be divided into 100 teams of 50 or so different developers. These teams adopt different practices,” he said. And without standardization, it’s easier for bad code to slip through the cracks.

How to prevent bugs from slipping through

Both CEOs acknowledge that bugs get through sometimes, but there are ways to minimize the risk, including perhaps the most obvious one: practicing standard software release hygiene. That involves testing before deploying and then deploying in a controlled way.

Rogers points to his company’s software and notes that progressive rollouts are the place to start. Instead of delivering the change to every user all at once, you instead release it to a small subset and see what happens before expanding the rollout. Along the same lines, if you have controlled rollouts and something goes wrong, you can roll back. “This idea of feature management or feature control lets you roll back features that aren’t working and get people back to the prior version if things are not working.”

Bansal, whose company just bought feature flag startup Split.io in May, also recommends what he calls “canary deployments,” which are small controlled test deployments. They are called this because they hark back to canaries being sent into coal mines to test for carbon monoxide leakage. Once you prove the test roll out looks good, then you can move to the progressive roll out that Rogers alluded to.

As Bansal says, it can look fine in testing, but a lab test doesn’t always catch everything, and that’s why you have to combine good DevOps testing with controlled deployment to catch things that lab tests miss.

Rogers suggests when doing an analysis of your software testing regimen, you look at three key areas — platform, people and processes — and they all work together in his view. “It’s not sufficient to just have a great software platform. It’s not sufficient to have highly enabled developers. It’s also not sufficient to just have predefined workflows and governance. All three of those have to come together,” he said.

One way to prevent individual engineers or teams from circumventing the pipeline is to require the same approach for everyone, but in a way that doesn’t slow the teams down. “If you build a pipeline that slows down developers, they will at some point find ways to get their job done outside of it because they will think that the process is going to add another two weeks or a month before we can ship the code that we wrote,” Bansal said.

Rogers agrees that it’s important not to put rigid systems in place in response to one bad incident. “What you don’t want to have happen now is that you’re so worried about making software changes that you have a very long and protracted testing cycle and you end up stifling software innovation,” he said.

Bansal says a thoughtful automated approach can actually be helpful, especially with larger engineering groups. But there is always going to be some tension between security and governance and the need for release velocity, and it’s hard to find the right balance.

We might not know what happened at CrowdStrike for some time, but we do know that certain approaches help minimize the risks around software deployment. Bad code is going to slip through from time to time, but if you follow best practices, it probably won’t be as catastrophic as what happened last week.

How to prevent your software update from being the next CrowdStrike

by with No Comment Tech

Times Square billboards displaying Windows blue screen of death after CrowdStrike outage on July 19, 2024.

Image Credits: Selcuk Acar/Anadolu / Getty Images

CrowdStrike released a relatively minor patch on Friday, and somehow it wreaked havoc on large swaths of the IT world running Microsoft Windows, bringing down airports, healthcare facilities and 911 call centers. While we know a faulty update caused the problem, we don’t know how it got released in the first place. A company like CrowdStrike very likely has a sophisticated DevOps pipeline with release policies in place, but even with that, the buggy code somehow slipped through.

In this case it was perhaps the mother of all buggy code. The company has suffered a steep hit to its reputation, and the stock price plunged from $345.10 on Thursday evening to $263.10 by Monday afternoon. It has since recovered slightly.

In a statement on Friday, the company acknowledged the consequences of the faulty update: “All of CrowdStrike understands the gravity and impact of the situation. We quickly identified the issue and deployed a fix, allowing us to focus diligently on restoring customer systems as our highest priority.”

Further, it explained the root cause of the outage, although not how it happened. That’s a post mortem process that will likely go on inside the company for some time as it looks to prevent such a thing from happening again.

Dan Rogers, CEO at LaunchDarkly, a firm that uses a concept called feature flags to deploy software in a highly controlled way, couldn’t speak directly to the CrowdStrike deployment problem, but he could speak to software deployment issues more broadly.

“Software bugs happen, but most of the software experience issues that someone would experience are actually not because of infrastructure issues,” he told TechCrunch. “They’re because someone rolled out a piece of software that doesn’t work, and those in general are very controllable.” With feature flags, you can control the speed of deployment of new features, and turn a feature off, if things go wrong to prevent the problem from spreading widely.

It is important to note however, that in this case, the problem was at the operating system kernel level, and once that has run amok, it’s harder to fix than say a web application. Still, a slower deployment could have alerted the company to the problem a lot sooner.

What happened at CrowdStrike could potentially happen to any software company, even one with good software release practices in place, said Jyoti Bansal, founder and CEO at Harness Labs, a maker of DevOps pipeline developer tools. While he also couldn’t say precisely what happened at CrowdStrike, he talked generally about how buggy code can slip through the cracks.

Typically, there is a process in place where code gets tested thoroughly before it gets deployed, but sometimes an engineering team, especially in a large engineering group, may cut corners. “It’s possible for something like this to happen when you skip the DevOps testing pipeline, which is pretty common with minor updates,” Bansal told TechCrunch.

He says this often happens at larger organizations where there isn’t a single approach to software releases. “Let’s say you have 5,000 engineers, which probably will be divided into 100 teams of 50 or so different developers. These teams adopt different practices,” he said. And without standardization, it’s easier for bad code to slip through the cracks.

How to prevent bugs from slipping through

Both CEOs acknowledge that bugs get through sometimes, but there are ways to minimize the risk, including perhaps the most obvious one: practicing standard software release hygiene. That involves testing before deploying and then deploying in a controlled way.

Rogers points to his company’s software and notes that progressive rollouts are the place to start. Instead of delivering the change to every user all at once, you instead release it to a small subset and see what happens before expanding the rollout. Along the same lines, if you have controlled rollouts and something goes wrong, you can roll back. “This idea of feature management or feature control lets you roll back features that aren’t working and get people back to the prior version if things are not working.”

Bansal, whose company just bought feature flag startup Split.io in May, also recommends what he calls “canary deployments,” which are small controlled test deployments. They are called this because they hark back to canaries being sent into coal mines to test for carbon monoxide leakage. Once you prove the test roll out looks good, then you can move to the progressive roll out that Rogers alluded to.

As Bansal says, it can look fine in testing, but a lab test doesn’t always catch everything, and that’s why you have to combine good DevOps testing with controlled deployment to catch things that lab tests miss.

Rogers suggests when doing an analysis of your software testing regimen, you look at three key areas — platform, people and processes — and they all work together in his view. “It’s not sufficient to just have a great software platform. It’s not sufficient to have highly enabled developers. It’s also not sufficient to just have predefined workflows and governance. All three of those have to come together,” he said.

One way to prevent individual engineers or teams from circumventing the pipeline is to require the same approach for everyone, but in a way that doesn’t slow the teams down. “If you build a pipeline that slows down developers, they will at some point find ways to get their job done outside of it because they will think that the process is going to add another two weeks or a month before we can ship the code that we wrote,” Bansal said.

Rogers agrees that it’s important not to put rigid systems in place in response to one bad incident. “What you don’t want to have happen now is that you’re so worried about making software changes that you have a very long and protracted testing cycle and you end up stifling software innovation,” he said.

Bansal says a thoughtful automated approach can actually be helpful, especially with larger engineering groups. But there is always going to be some tension between security and governance and the need for release velocity, and it’s hard to find the right balance.

We might not know what happened at CrowdStrike for some time, but we do know that certain approaches help minimize the risks around software deployment. Bad code is going to slip through from time to time, but if you follow best practices, it probably won’t be as catastrophic as what happened last week.

US bans sale of Kaspersky software citing security risk from Russia 

by with No Comment Tech

Eugene Kaspersky, CEO of Kaspersky Lab, gives a keynote speech during the Mobile World Congress (MWC) fair in Barcelona on June 28, 2021.

Image Credits: PAU BARRENA/AFP / Getty Images

The U.S. government announced on Thursday that it is banning the sale of Kaspersky antivirus in the country, and is asking Americans who use the software to switch to a different provider.

The Commerce Department’s Bureau of Industry and Security said it imposed the “first of its kind” ban, arguing that Kaspersky threatens U.S. national security and users’ privacy because the company is based in Russia. 

“Russia has shown it has the capacity, and even more than that, the intent to exploit Russian companies like Kaspersky to collect and weaponize the personal information of Americans. And that’s why we are compelled to take the action that we’re taking today,” U.S. Commerce Secretary Gina Raimondo said in a call with reporters.

News of the ban was first reported by Reuters ahead of the announcement. A Kaspersky spokesperson did not immediately respond to TechCrunch’s request for comment. 

Kaspersky will be banned from selling its software to American consumers and businesses starting on July 20, but the company will be able to provide software and security updates to existing customers until September 29. After that, Kaspersky will no longer be permitted to push software updates to U.S. customers, according to Raimondo. 

“That means your software and services will degrade. That’s why I strongly recommend that you immediately find an alternative to Kaspersky,” Raimondo said. 

Raimondo said that U.S. consumers who already use Kaspersky’s antivirus are not violating the law.

“U.S. individuals and businesses that continue to use or have existing Kaspersky products and services are not in violation of the law, you have done nothing wrong and you are not subject to any criminal or civil penalties,” said Raimondo. “However, I would encourage you in the strongest possible terms, to immediately stop using that software and switch to an alternative in order to protect yourself and your data and your family.”

To inform consumers, Raimondo said the Department of Homeland Security and the Justice Department will work to notify U.S. consumers, and the U.S. government will set up a website, “so people who are impacted can find the information they need to understand why we’re doing what we’re doing, and help them take next steps.” 

A senior U.S. Commerce Department official said during the press call that federal cybersecurity agency CISA will do outreach to critical infrastructure organizations that use Kaspersky software in their operations to help them find alternatives. The official also said that they don’t plan on naming any specific action by Kaspersky that led to today’s decision. (The Commerce Department asked reporters not to name the official.)

The ban announced Thursday is the latest escalation in a long series of U.S. government actions against the Moscow-headquartered Kaspersky. 

In September 2017, the Trump administration banned U.S. federal agencies from using Kaspersky software over fears that the company could be compelled to help Russian intelligence agencies. Earlier in the year, it was reported that Russian government hackers had stolen U.S. classified documents stored on an intelligence contractor’s home computer because it was running Kaspersky’s antivirus, marking the first known incident of espionage resulting from use of the company’s software.

The decision to ban Kaspersky has been in the works since last year, according to a report by The Wall Street Journal in April 2023. 

According to Kaspersky, the company has more than 400 million individual customers, and over 240,000 corporate customers worldwide. The senior official declined to say how many U.S. customers Kaspersky has, but said there is a significant number, including critical infrastructure organizations, and state and local government entities. 

A red car illustration with a loading bar on the windshield.

WTF is a 'software-defined vehicle?'

by with No Comment Tech

A red car illustration with a loading bar on the windshield.

Image Credits: Lyudinka / Getty Images

Like any good buzz phrase, “software-defined vehicle” is as straightforward as it is obtuse. If you’re trying to figure out what automakers and suppliers mean by this term — which was all over CES 2024 — you’re in the right place.

A software-defined vehicle is a vehicle defined by software — okay, sorry, I can do better than that. 

The term “software-defined vehicle” (often irritatingly condensed to “SDV”) isn’t new; it gained ground in the mid-to-late 2010s to describe the evolution of cars from mechanical devices to largely software-run devices. Often, automakers and industry watchers compare this transformation to that of the smartphone. You might recall automakers talking about turning your car into a “smartphone on wheels” as far back as 2014 or so.

Why? Ages ago, phones came in many shapes and sizes, with a range of features. Now, they’re all pocketable rectangles differentiated by the underlying operating system and regularly updated software. People who talk about software-defined vehicles argue a similar transformation is happening with cars. That’s why consultants like to say that cars are becoming “smartphones on wheels.” Certainly, this hasn’t happened just yet.

Industry analysts, Tier 1 supplier execs and top leadership at automakers have all told TechCrunch the same thing: The industry is working on it, but most automakers, particularly those in North America and Europe, just aren’t there yet. 

The metaphor is apt, but don’t take it too seriously. For one, the rise of foldables, and the enduring love for mobile keyboards and standalone cameras tell us that software isn’t everything. But more to the point, smartphones aren’t known for longevity; a few years go by and they seem to lag after the latest big update. Automakers, on the other hand, are aiming to create software-defined vehicles that improve over time even as critical hardware ages. Bless their hearts; they have so much to prove.

Software-defined vehicles defined

Let’s start with what it doesn’t mean. It’s not about adding in a touchscreen or a voice assistant.

At its core, a software-defined vehicle is any car, truck or SUV with “capabilities that can be upgraded over time, fundamentally by software, instead of having to change physical parts,” said Gartner automotive analyst Mike Ramsey in a call with TechCrunch.

Another expert offered a similar definition: Software-defined means “you can actually introduce new functions into a vehicle that haven’t been there before,” said Moritz Neukirchner, a senior director at automotive software company Elektrobit. Neukirchner, whose formal title includes the buzz phrase, conceded that software-defined vehicle is an “awfully overused buzzword these days.” He argued that software-defined vehicles aren’t just about updates, “but actually introducing new value into a car.”

Getting to that level of capability, however is where automakers and suppliers have struggled, largely because it requires a complete change in the underlying electronic architecture of the vehicle. 

“Big part of SDV is not necessarily consumer facing,” said Dan Cauchy, who oversees automotive software at the nonprofit Linux Foundation. Instead, Cauchy said a vehicle is software-defined when the underlying tech enables automakers to build, deploy and manage software themselves across multiple car models, whether they’re older, newer, entry- or luxury-level. 

The starting point

According to Stefan Buerkle, who leads cross domain computing at Bosch, a software-defined vehicle has two key elements — the software running in the car and the electronic architecture behind it. 

Today, a well-equipped car has around 100 different electronic control units (ECUs), each with a specific purpose — think: power steering, activating airbags, locking doors and so on. This way of doing things has worked because “you developed it once, you defined the interfaces — who communicates with whom, and in which standard,” said Buerkle. When the electronic systems in the car are working and stable, “you bring it in the field and you sell it for the next five years,” Buerkle explained. “And in those five years, you do the next car.”

That’s business as usual, but the architecture in a software-defined vehicle should be way different, per Buerkle. “It is almost impossible to manage 90 to 100 different ECUs and ensure that when you update the one it still works with the other one,” he said. Instead, a software-defined vehicle needs a “streamlined architecture,” Buerkle said, to bring together all the functions “into one to two maximum ECUs.”

According to Buerkle, that’s how you “turn the car into something as updatable as your smartphone.”

What can software-defined vehicles do?

Already, touchscreens in vehicles are absorbing physical buttons, knobs, switches and dials, but the software-defined vehicle trend is supposed to be more than that. When automakers and analysts use the term, they tend to talk about big wireless software updates that introduce new functionality. The software-defined vehicle also comes up when automakers allude to all the money they hope to make by charging drivers subscription fees (see: BMW’s heated seats fiasco, Ford’s hands-off driving subscription and Rivian’s theoretical augmented reality upcharges).

Ramsey believes firms like GM and Stellantis are looking to Apple’s wildly profitable services business for inspiration. Apple users cough up subscription fees in exchange for everything from music streaming and game downloads to file storage and technical support. If smartphones are any indication, we may see automakers sell, or otherwise enable, a wide range of car-focused subscriptions in the future. Unlike Apple, however, “I do not expect any of those companies will achieve really high levels of revenue and profit from this,” Ramsey cautioned, “But they’ll probably achieve something that will be reasonably profitable,” he said.

This more or less jibes with what Hyundai software boss Chang Song said during the company’s press conference at CES 2024. The executive argued that “cars have evolved beyond their traditional role as a means of transportation. They’re not just for getting around anymore,” he said.

Song added, “We’re talking about cars that will work with your phone’s services for catching a ride or sharing a car, and managing big fleets for businesses.” According to song, “the heart of these solutions is software.”

Can I buy a software-defined vehicle yet?

Companies typically invoke the software-defined vehicle term in a forward-looking and aspirational way. Yet, there are a few automakers that produce software-defined vehicles today. However, that number varies depending on your definition.

Tesla vehicles are software-defined, according to Buerkle, Ramsey and a number of other automotive experts TechCrunch spoke to on the ground at CES 2024. Tesla has the ability to not only add new features via over-the-air software updates, like video games or pothole detection, it also can issue updates that can change how the vehicle performs and even its battery management.

Yet, you could try to argue otherwise. Last we heard, Tesla doesn’t push software to all of its cars at once, nor has it launched an in-car app store — though it appears to be laying the groundwork for one. This doesn’t sound quite like a smartphone on wheels, does it? 

By his own definition, Cauchy estimated that true software-defined vehicles are “maybe a couple of years” away. Neukirchner responded similarly, “I think we’re getting there, but no one is there completely yet.” 

Neukirchner elaborated, “When I look at the core technology, you need to speed up development practices. You need to speed up testing. You need to speed up integration. You need to be able to adapt your business models to this new world.” The Elektrobit director added, however, that he sees “progress throughout the entire supply chain.”

So, maybe it’s just not fully baked. Still, the software-defined trend is clearly underway.

Software-defined profits

Suppliers certainly want in on the software-defined hype. At CES, MotorTrend announced an “awards gala” for software-defined vehicle “innovators,” while LG pitched its “future mobility platform” and in-car displays in an SDV-laden press release. As I wrote this blog, Intel said it bought a chip maker to shore up its automotive business, and BlackBerry announced an audio and acoustics platform for software-defined vehicles. 

Neukirchner highlighted some business use-cases around software-defined vehicles, including fleet management and optimization tools. He also pointed to the role startups can play as automotive suppliers, by selling stuff like cloud-based testing tools, as well as the current limitations of automotive app stores.

“The problem we see these days is we are currently counting 15 different automotive OSes (operating systems) — so in the end,” Neukirchner warned, “you’re going to see quite a bit of fragmentation of that market, and this is something that may hinder success” for third-party software developers.

Is this all bullshit?

Yes and no. The auto industry loves buzzwords and acronyms; godless terms like “infotainment” and “PHEVs” abound. Luckily, the phrase “software-defined vehicles” seems too clunky to catch on beyond press releases and analyst reports. Still, it’s a very real trend behind the scenes.

Automakers are throwing billions of dollars into the software-defined vehicle effort in hopes of getting some return on that investment. Once unique to Tesla, built-in tablets now look pretty ordinary on new car and truck dashboards, and major automakers are gradually embracing over-the-air updates (albeit many have limited capability), driver-assist software and specialized app stores. As CES reminds us each year, basically everything is getting “smart” anyways — phones, TVs, watches, refrigerators and so on. A similar phenomenon is happening with cars, too; it’s just happening slower, in part because cars have way more moving parts than, say, a smart thermostat.

According to Buerkle, drivers’ expectations are changing, but this trend isn’t simply about giving consumers what they want.

As high prices push folks to keep vehicles longer, car companies want to make up for selling fewer cars by charging software subscription fees. GM, for example, wants its services revenue to top $25 billion per year by 2030, while Stellantis set its bar just slightly lower — $22.5 billion by the end of the decade. This isn’t quite Apple territory — the company’s services business brings in about that much in a single quarter. But still, these automakers aim to make a ton of revenue from software and related services.

The real test is whether customers ultimately derive meaningful value from software-defined vehicles, as well as if that value warrants the fees that come along with it.

Read more about CES 2024 on TechCrunch

A red car illustration with a loading bar on the windshield.

WTF is a 'software-defined vehicle?'

by with No Comment Tech

A red car illustration with a loading bar on the windshield.

Image Credits: Lyudinka / Getty Images

Like any good buzz phrase, “software-defined vehicle” is as straightforward as it is obtuse. If you’re trying to figure out what automakers and suppliers mean by this term — which was all over CES 2024 — you’re in the right place.

A software-defined vehicle is a vehicle defined by software — okay, sorry, I can do better than that. 

The term “software-defined vehicle” (often irritatingly condensed to “SDV”) isn’t new; it gained ground in the mid-to-late 2010s to describe the evolution of cars from mechanical devices to largely software-run devices. Often, automakers and industry watchers compare this transformation to that of the smartphone. You might recall automakers talking about turning your car into a “smartphone on wheels” as far back as 2014 or so.

Why? Ages ago, phones came in many shapes and sizes, with a range of features. Now, they’re all pocketable rectangles differentiated by the underlying operating system and regularly updated software. People who talk about software-defined vehicles argue a similar transformation is happening with cars. That’s why consultants like to say that cars are becoming “smartphones on wheels.” Certainly, this hasn’t happened just yet.

Industry analysts, Tier 1 supplier execs and top leadership at automakers have all told TechCrunch the same thing: The industry is working on it, but most automakers, particularly those in North America and Europe, just aren’t there yet. 

The metaphor is apt, but don’t take it too seriously. For one, the rise of foldables, and the enduring love for mobile keyboards and standalone cameras tell us that software isn’t everything. But more to the point, smartphones aren’t known for longevity; a few years go by and they seem to lag after the latest big update. Automakers, on the other hand, are aiming to create software-defined vehicles that improve over time even as critical hardware ages. Bless their hearts; they have so much to prove.

Software-defined vehicles defined

Let’s start with what it doesn’t mean. It’s not about adding in a touchscreen or a voice assistant.

At its core, a software-defined vehicle is any car, truck or SUV with “capabilities that can be upgraded over time, fundamentally by software, instead of having to change physical parts,” said Gartner automotive analyst Mike Ramsey in a call with TechCrunch.

Another expert offered a similar definition: Software-defined means “you can actually introduce new functions into a vehicle that haven’t been there before,” said Moritz Neukirchner, a senior director at automotive software company Elektrobit. Neukirchner, whose formal title includes the buzz phrase, conceded that software-defined vehicle is an “awfully overused buzzword these days.” He argued that software-defined vehicles aren’t just about updates, “but actually introducing new value into a car.”

Getting to that level of capability, however is where automakers and suppliers have struggled, largely because it requires a complete change in the underlying electronic architecture of the vehicle. 

“Big part of SDV is not necessarily consumer facing,” said Dan Cauchy, who oversees automotive software at the nonprofit Linux Foundation. Instead, Cauchy said a vehicle is software-defined when the underlying tech enables automakers to build, deploy and manage software themselves across multiple car models, whether they’re older, newer, entry- or luxury-level. 

The starting point

According to Stefan Buerkle, who leads cross domain computing at Bosch, a software-defined vehicle has two key elements — the software running in the car and the electronic architecture behind it. 

Today, a well-equipped car has around 100 different electronic control units (ECUs), each with a specific purpose — think: power steering, activating airbags, locking doors and so on. This way of doing things has worked because “you developed it once, you defined the interfaces — who communicates with whom, and in which standard,” said Buerkle. When the electronic systems in the car are working and stable, “you bring it in the field and you sell it for the next five years,” Buerkle explained. “And in those five years, you do the next car.”

That’s business as usual, but the architecture in a software-defined vehicle should be way different, per Buerkle. “It is almost impossible to manage 90 to 100 different ECUs and ensure that when you update the one it still works with the other one,” he said. Instead, a software-defined vehicle needs a “streamlined architecture,” Buerkle said, to bring together all the functions “into one to two maximum ECUs.”

According to Buerkle, that’s how you “turn the car into something as updatable as your smartphone.”

What can software-defined vehicles do?

Already, touchscreens in vehicles are absorbing physical buttons, knobs, switches and dials, but the software-defined vehicle trend is supposed to be more than that. When automakers and analysts use the term, they tend to talk about big wireless software updates that introduce new functionality. The software-defined vehicle also comes up when automakers allude to all the money they hope to make by charging drivers subscription fees (see: BMW’s heated seats fiasco, Ford’s hands-off driving subscription and Rivian’s theoretical augmented reality upcharges).

Ramsey believes firms like GM and Stellantis are looking to Apple’s wildly profitable services business for inspiration. Apple users cough up subscription fees in exchange for everything from music streaming and game downloads to file storage and technical support. If smartphones are any indication, we may see automakers sell, or otherwise enable, a wide range of car-focused subscriptions in the future. Unlike Apple, however, “I do not expect any of those companies will achieve really high levels of revenue and profit from this,” Ramsey cautioned, “But they’ll probably achieve something that will be reasonably profitable,” he said.

This more or less jibes with what Hyundai software boss Chang Song said during the company’s press conference at CES 2024. The executive argued that “cars have evolved beyond their traditional role as a means of transportation. They’re not just for getting around anymore,” he said.

Song added, “We’re talking about cars that will work with your phone’s services for catching a ride or sharing a car, and managing big fleets for businesses.” According to song, “the heart of these solutions is software.”

Can I buy a software-defined vehicle yet?

Companies typically invoke the software-defined vehicle term in a forward-looking and aspirational way. Yet, there are a few automakers that produce software-defined vehicles today. However, that number varies depending on your definition.

Tesla vehicles are software-defined, according to Buerkle, Ramsey and a number of other automotive experts TechCrunch spoke to on the ground at CES 2024. Tesla has the ability to not only add new features via over-the-air software updates, like video games or pothole detection, it also can issue updates that can change how the vehicle performs and even its battery management.

Yet, you could try to argue otherwise. Last we heard, Tesla doesn’t push software to all of its cars at once, nor has it launched an in-car app store — though it appears to be laying the groundwork for one. This doesn’t sound quite like a smartphone on wheels, does it? 

By his own definition, Cauchy estimated that true software-defined vehicles are “maybe a couple of years” away. Neukirchner responded similarly, “I think we’re getting there, but no one is there completely yet.” 

Neukirchner elaborated, “When I look at the core technology, you need to speed up development practices. You need to speed up testing. You need to speed up integration. You need to be able to adapt your business models to this new world.” The Elektrobit director added, however, that he sees “progress throughout the entire supply chain.”

So, maybe it’s just not fully baked. Still, the software-defined trend is clearly underway.

Software-defined profits

Suppliers certainly want in on the software-defined hype. At CES, MotorTrend announced an “awards gala” for software-defined vehicle “innovators,” while LG pitched its “future mobility platform” and in-car displays in an SDV-laden press release. As I wrote this blog, Intel said it bought a chip maker to shore up its automotive business, and BlackBerry announced an audio and acoustics platform for software-defined vehicles. 

Neukirchner highlighted some business use-cases around software-defined vehicles, including fleet management and optimization tools. He also pointed to the role startups can play as automotive suppliers, by selling stuff like cloud-based testing tools, as well as the current limitations of automotive app stores.

“The problem we see these days is we are currently counting 15 different automotive OSes (operating systems) — so in the end,” Neukirchner warned, “you’re going to see quite a bit of fragmentation of that market, and this is something that may hinder success” for third-party software developers.

Is this all bullshit?

Yes and no. The auto industry loves buzzwords and acronyms; godless terms like “infotainment” and “PHEVs” abound. Luckily, the phrase “software-defined vehicles” seems too clunky to catch on beyond press releases and analyst reports. Still, it’s a very real trend behind the scenes.

Automakers are throwing billions of dollars into the software-defined vehicle effort in hopes of getting some return on that investment. Once unique to Tesla, built-in tablets now look pretty ordinary on new car and truck dashboards, and major automakers are gradually embracing over-the-air updates (albeit many have limited capability), driver-assist software and specialized app stores. As CES reminds us each year, basically everything is getting “smart” anyways — phones, TVs, watches, refrigerators and so on. A similar phenomenon is happening with cars, too; it’s just happening slower, in part because cars have way more moving parts than, say, a smart thermostat.

According to Buerkle, drivers’ expectations are changing, but this trend isn’t simply about giving consumers what they want.

As high prices push folks to keep vehicles longer, car companies want to make up for selling fewer cars by charging software subscription fees. GM, for example, wants its services revenue to top $25 billion per year by 2030, while Stellantis set its bar just slightly lower — $22.5 billion by the end of the decade. This isn’t quite Apple territory — the company’s services business brings in about that much in a single quarter. But still, these automakers aim to make a ton of revenue from software and related services.

The real test is whether customers ultimately derive meaningful value from software-defined vehicles, as well as if that value warrants the fees that come along with it.

Read more about CES 2024 on TechCrunch

vast array of overhead electric wires

Germany's Instagrid, which uses software to supercharge portable batteries, raises $95M

by with No Comment Tech

vast array of overhead electric wires

Image Credits: Gerard Hermand/Moment/Getty Images

Energy supply is one of the bigger issues impacting how technology will evolve over time — a challenge that might be feel closer to home when you consider the batteries of objects like mobile phones or electric vehicles but is definitely not constrained just to consumer tech. A startup called Instagrid is using software to scale that mountain when it comes to enterprise-grade portable chargers. It’s been getting a lot of attention, selling 30,000 units of its flagship “One” product to date and growing at 100% annually, and today it’s announcing a Series C of $95 million to power up its growth strategy. The funding is being made at a valuation of between $400 million and $500 million, post-money.

For the moment, lithium ion remains the main game in town when it comes to rechargeable batteries, and while materials are bound on change and improve, the lithium ion battery, even as it also improves, has some basic limitations. “In lithium ion over the last 10 years, the costs have decreased by 90% and energy density has increased by 300%, and power density has even increased by 400%. But power conversion has not kept up,” said Andreas Sedlmayr, the co-founder and co-CEO, in an interview. “So for us it was really looking into why is that broken and how can we fix that? In the end, what we did is we took a hardware problem and converted it into a software.”

The company, he said, has written “roughly 500,000 lines of software code” to improve getting energy out of its battery, which today weighs 20kg, is just 42x21x42cm in size, and can be fully recharged in a few hours plugged into a normal electrical socket.

The funding is being led by Teachers’ Venture Growth (TVG) — the later-stage investing arm of the Ontario Teachers’ Pension Plan. Morgan Stanley Investment Management’s (MSIM) 1GT climate private equity strategy, plus previous investors Energy Impact Partners, SET Ventures, blueworld.group, Hightech Gründerfonds, and Pierre-Pascal Urbon (who chairs Instagrid’s advisory board), also participated.

Instagrid, based out of Stuttgart in Germany’s industrial heartland, had only raised around $55 million previously since being founded in 2018.

Sedlmayr co-founded and co-leads the company with Sebastian Berning. Both have PhDs in material science and were working on battery technology at Bosch, seeing the challenges from the other end: the batteries that power different devices, used by people on the go, which need recharging regularly.

“But Sebastian and I are entrepreneurs at heart, so we decided that this is the time that we step out and build something from scratch to help with that,” he said.

Instagrid’s focus on enterprise scenarios — industrial customers, utility providers, medical and emergency services, and media companies are among its customers — grew directly out of that. Sidestepping direct competition with its former employer, among others, it’s not looking (currently) at how to manage batteries in devices, but at generators that are used to charge that electrical machinery when mains electricity is unavailable. Sedlmayr, in fact, talked to me for this story from Las Vegas, where he was attending a conference for the concrete industry: That’s a mark of the company’s target user, but also of the region where the company is now focusing its business development.

One of the problems that Instagrid is going out to fix is the environmental one: existing, legacy generators are first of all expensive, loud, and bad for the environment, and organizations are looking for alternatives to power their work.

There are likely many tens of thousands of them already in use in the world — Instagrid’s main investor here puts the figure of combustible engines in the world, a more general range of products, at 50 million — and these are in need of replacing.

Just one study, covering the humanitarian aid sector, found that there are likely 11,000 fuel-powered generators in use across different aid organizations.

“Our initial conservative estimates are that humanitarian agencies spend more than $100 million on fuel per year, emitting almost 200,000 tonnes of CO2,” the report notes. (It also happens to suggest a different alternative to what Instagrid has built: solarized systems, which represents another alternative that exists today and Instagrid could also potentially adopt.)

The other problem that Instagrid is looking to solve is controlability: as “dumb” machines that you simply fill up and run, these generators can be hard to manage and control. The software element of Instagrid’s system essentially allows an organization to manage its batteries like a remote fleet: you can understand where power needs to be used, and where it can be powered down, how much power is left, and more.

That is the key aspect of what attracted TVG, essentially a tech investor, to the company.

“We were drawn to Instagrid’s novel use of software to extract higher performance from battery cells and became particularly excited after discussing the company’s technology with other parts of OTPP’s portfolio,” European MD Avid Larizadeh-Duggan told TechCrunch. “Through our real estate and infrastructure investments we were able to appreciate that thousands of generators are being deployed in the field and that these come with major drawbacks in terms of cost, health and safety and work environment on top of the associated carbon emissions. But until now there hadn’t been a viable portable battery solution for professional use. Existing portable battery packs target consumer users and lack the peak power performance for professional application. Instagrid solves this through proprietary power electronics and software which extract unrivaled peak power and allow Instagrid products to power anything with a plug.”

The focus on software and cloud-based access feels in some regards like the future of how batteries will develop and improve. (Research at a university in the U.K. for example has found a way to help better understand the health of batteries in portable devices, another indirect way of improving their efficiency.)

That also potentially opens the door to malicious hacking, but Sedlmayr said that it focuses on robust security around its system and that to date there have been no attempts at breaches. That threat could increase with its growth, however.

Larizadeh-Duggan said that areas of opportunity for the company include more accessories to enhance monitoring in the cloud, including more emissions and energy usage tracking and analytics; improving the startup’s energy management algorithms, to focus on “three-phase power and uninterruptable power supply as required in specialist use cases”; and more hardware.

This is an area which today focuses around lithium ion, although Sedlmayr describes the company as “material agnostic” and willing to work with other kinds of battery designs over time as they emerge.

Metronome founders

Metronome's usage-based billing software finds hit in AI as the startup raises $43M in fresh capital

by with No Comment Tech

Metronome founders

Image Credits: Metronome

Metronome, a startup that helps software companies offer usage-based billing, has raised $43 million in a Series B funding round led by NEA.

Existing backers Andreessen Horowitz and General Catalyst also participated in the financing, which brings its total amount raised to over $78 million since its 2019 inception. 

Founded by Dropbox alums Kevin Liu and Scott Woody, San Francisco-based Metronome says it saw a 6x increase in ARR last year as more companies transitioned from subscription to usage-based models, or a combination of both. Its customers include startups such as OpenAI and Anthropic and enterprise companies like Databricks and Nvidia. Initially, Metronome worked with startups but last year expanded to the enterprise.

“We were fortunate to see that growth during what was otherwise a tough year for SaaS,” said Liu. “Companies have been cutting spend on ‘nice-to-have’ software, but we’re seen as a core driver of revenue opportunities for our customers. The rise of AI has also been a big factor (many AI companies are adopting usage-based models), as has the desire from companies to move away from pure subscription and seat-based models to more hybrid and usage-based approaches.”

Unsurprisingly, Metronome itself has a usage-based model.

The startup declined to reveal its valuation, saying only that “it was a very healthy multiple above” its Series A valuation. 

“We still had nearly all of our Series A in the bank and were heavily oversubscribed,” said Woody. 

The draw for AI companies

Metronome claims to “dramatically reduce” the engineering investment required by companies for billing integration and maintenance.

“We help teams launch products quickly, offer any pricing and streamline quote-to-cash workflows, all without engineering effort,” said Liu. It does that with a data platform that it says offers integrations “out-of-the-box, so engineering teams can just point their data stream directly at Metronome and skip having to own and maintain a lot of their own infrastructure.”

For enterprises in particular, Metronome claimed that transitioning to cloud and/or usage-based revenue would typically require overhauling their financial stack. Its product, Liu said, helps facilitate that transition “while plugging into their existing tooling, minimizing disruption and drastically speeding up the process.”

AI companies in particular seem to be drawn to Metronome’s offering, the company claims.

“The entire AI stack has usage-based COGS, from APIs down to the GPU infrastructure layer, which means that AI businesses often turn to usage-based pricing to keep their margins consistent,” said Woody. “We’ve had a huge amount of inbound interest from companies looking to monetize new AI products.”

Growing headcount

To help meet that demand, over the last year, Metronome doubled its headcount to 66 full-time employees, growing its staff by more than 40% in the last quarter alone. It claims to “still have a lot of hiring to do this year,” particularly across its R&D and customer-facing teams.

The company also plans to use its new funding to advance on its product roadmap.

“This capital also gives us a tremendous amount of dry powder and runway, which is important in an uncertain environment like this,” said Liu. “We’re building critical infrastructure, so customers need to know that we’ll be around for the long haul.”

As part of the funding round, NEA partner Hilarie Koplow-McAdams has joined Metronome’s board of directors. 

“Billing is often under resourced internally and seen as a bottleneck for product launches and pricing changes. In reality, it’s a make-or-break revenue driver for any business,” she said in a written statement. “Metronome makes it possible for companies to operationalize new business models quickly. Every customer we spoke to shared how Metronome turned billing from a ‘hair-on-fire’ problem to a system that just works.” 

Metronome raises $30M to help software companies shift to usage-based pricing models

Want more fintech news in your inbox? Sign up for TechCrunch Fintech here.

AI investor survey

AI is going to save software companies' dreams of growth

by with No Comment Tech

AI investor survey

Image Credits: Bryce Durbin / TechCrunch

It appears emerging price points for AI-powered software products will boost the total addressable market (TAM) for technology products and help reaccelerate growth at tech companies big and small.

In late 2023, Battery Ventures noted that the pullback of revenue growth at software startups had reached its nadir, and growth levels were starting to stabilize in the fourth quarter. Around the same time, Scale Venture Partners reported that after several years of deceleration, early-stage software companies were expected to renew momentum in 2024. Taken together, it seemed that tech companies were all but out of the woods.

Today, we’re seeing early indications that those optimistic takes were in tune with how 2024 would at least start to unfold. Companies are reporting their Q4 2023 results, and Big Tech companies have posted better-than-expected revenue and profit so far. Microsoft did well, Meta blew the doors off, and Amazon had a great quarter as well. We’re still waiting on a host of smaller SaaS companies to report, but it does appear that 2023 ended on a better note than earlier in the year.

Building a viable pricing model for generative AI features could be challenging

There’s good reason to expect more of the same in 2024. It appears that the market is willing to accept that software imbued with new AI capabilities will cost more. So, yes, software companies of all sizes will have something new to upsell existing customers and potentially land new accounts, and it means that the TAM of software companies is widening.

A business can grow faster for longer in a larger market than it can in a smaller market. AI is therefore serving as a near-term growth boost for tech companies while raising the ceiling for how big they can become over time.

The price of AI

Charging for AI software products is not merely a way to squeeze customers; it’s also a great way to defend margins. AI is not cheap, and recently we’ve seen people once again debating whether AI startups have worse gross margins than traditional SaaS businesses.

If you are going to run lots of LLM cycles, you have to charge for it. Otherwise, you are going to take large bales of cash and set them on fire. So it should not be entirely surprising that we’re seeing AI services coming at a stiff price point. Microsoft’s Copilot Pro will run you $20 per month, as will Google’s new level of consumer service that includes its latest AI model. GitHub Copilot costs $10 per month, minimum. Box is charging for AI as well, to put an enterprise spin on the point. The list goes on.

The cost of enterprise-level access to AI services can be partially graded on the success of Microsoft 365’s AI tools. They seem to be doing well. And OpenAI’s revenue growth is a useful proxy for demand for its services, given how popular ChatGPT is. Both seem to indicate that we’re seeing real demand for AI-powered software, which, in turn, should convert to additive growth for tech companies.

Looping back to where we started: This dynamic is why I expect software growth to be solid this year.

There’s another way to consider the impact of AI-related software services on tech companies’ top lines. Observe the following chart from Altimeter investor Jamin Ball:

Image Credits: Clouded Judgement.

This chart parses how much growth AI is driving for Microsoft’s cloud platform. If we presume that all hyperscale cloud infrastructure providers are seeing similar boosts to compute demand, we can infer that lots of tech companies are ramping up their usage of AI models inside their products. As no company wants to see their gross margins erode as they grow, the above implied spend that Ball charted must be converted into higher prices. That means growth.

It seems that the market isn’t hesitating to pay for AI-related software services. And given that demand for AI-related tooling is high, the combination of the two factors should not only boost tech growth today, but also the total tech market itself over time. How about that for a good foot to start the year on?

Lucid Air Touring

Feds probing Lucid's software fix for defroster recall

by with No Comment Tech

Lucid Air Touring

Image Credits: Lucid Motors

The National Highway Traffic Safety Administration (NHTSA) has opened an investigation into a Lucid Motors windshield defroster recall from January, saying it’s “concerned” the company’s over-the-air update solution doesn’t go far enough to fix the problem. Lucid, meanwhile, told TechCrunch that the agency “understates the remedy being offered.”

The probe comes as some automakers are increasingly relying on software updates to fix problems that would normally require a visit to a dealership or service center. While the investigation is not as high-stakes as, say, the one the agency has opened into Tesla’s problems with its Autopilot driver-assistance system, it shows NHTSA is paying close attention to what automakers think over-the-air updates should — and shouldn’t — be used for. 

The recall involves 2,042 Lucid Air sedans built with a high-voltage coolant heater from supplier Webasto AG that, when it fails, removes the ability to defrost the windshield. Lucid’s solution for the recall was to ship an over-the-air update that could identify which cars had the part and reactively warn the driver if it fails.

NHTSA’s Office of Defects Investigation now says it is “concerned that by only providing a notification to drivers that their windshield defroster is not working, Lucid puts the driver in a situation where they could experience the failure during the same drive cycle as they receive the notification.” This could hypothetically cause drivers to find out the feature is unavailable at the moment they need to defrost their windshield — meaning they would also not be able to request a free replacement part until after a failure.

The agency notes that Jaguar Land Rover took a more proactive approach with its electric SUV, the I-Pace, which briefly used this same heater from Webasto. Jaguar announced in 2023 that it would alert all 668 affected owners and replace the part for free.

Lucid is framing its approach as innovative, though. The company said the over-the-air update introduced a “diagnostic software feature” that actively monitors the heater for faults, and it “intends to demonstrate that the analysis underlying this approach is sound and that customer safety is assured.” The company also said it is alerting all affected owners electronically and by mail, so they won’t find out after a failure, as NHTSA suggests.

Waymo jaguar ipace autonomous vehicle

Waymo recalls and updates robotaxi software after two cars crashed into the same towed truck

by with No Comment Tech

Waymo jaguar ipace autonomous vehicle

Image Credits: Waymo

Waymo is voluntarily recalling the software that powers its robotaxi fleet after two vehicles crashed into the same towed pickup truck in Phoenix, Arizona, in December. It’s the company’s first recall.

Waymo chief safety officer Mauricio Peña described the crashes as “minor” in a blog post, and said neither vehicle was carrying passengers at the time. There were no injuries. He also said Waymo’s ride-hailing service — which is live in Phoenix, San Francisco, Los Angeles, and Austin — “is not and has not been interrupted by this update.” The company declined to share video of the crashes with TechCrunch.

Waymo said it developed, tested, and validated a fix to the software that it started deploying to its fleet on December 20. All of its robotaxis received that software update by January 12.

“This voluntary recall reflects how seriously we take our responsibility to safely deploy our technology and to transparently communicate with the public,” Peña wrote.

The recall comes at a time when self-driving cars are facing intense scrutiny following a series of high-profile crashes and controversies — including one last week where a Waymo robotaxi struck a cyclist. This past weekend, a Waymo autonomous vehicle was vandalized and burned by a crowd of people in San Francisco. Rival operator Cruise, meanwhile, has suspended its operations as it deals with the fallout of a crash from last October. It hired its first chief safety officer this week.

The crashes that prompted the recall both happened on December 11. Peña wrote that one of Waymo’s vehicles came upon a backward-facing pickup truck being “improperly towed.” The truck was “persistently angled across a center turn lane and a traffic lane.” Peña said the robotaxi “incorrectly predicted the future motion of the towed vehicle” because of this mismatch between the orientation of the tow truck and the pickup, and made contact. The company told TechCrunch this caused minor damage to the front left bumper.

The tow truck did not stop, though, according to Peña, and just a few minutes later another Waymo robotaxi made contact with the same pickup truck being towed. The company told TechCrunch this caused minor damage to the front left bumper and a sensor. (The tow truck stopped after the second crash.)

Waymo said it contacted the Phoenix Police Department and the Arizona Department of Public Safety on the day of the crashes, and told the National Highway Traffic Safety Administration about them on December 15. The company and NHTSA staff have had “four subsequent conversations” about the crashes, before Waymo decided to voluntarily recall the software that was in use at the time.

The recall could increase public pressure on Waymo, as cities, citizens and government agencies are scrutinizing the promise of autonomous vehicle fleets. Robotaxis are already facing pushback in Los Angeles, including from the Teamsters union. The California Department of Motor Vehicles is already investigating Waymo’s aforementioned crash with a cyclist. And the Department of Justice and the U.S. Securities and Exchange Commission, among others, are probing Cruise’s handling of its October crash.