Elastic CEO reflects on Amazon spat, license switch, and the principles of open source

Let the OSS Enterprise newsletter guide your open source journey! Sign up here.

While the value of open source software (OSS) is clear for most to see, community-driven software often finds itself at the center of many heated debates, spanning everything from security deficiencies to license changes.

Way back at the start of 2021, one of those big “hot potato” OSS talking points reared its head when Elastic revealed that it was transitioning its database search engine Elasticsearch, alongside the Kibana visualization dashboard, from an open source Apache 2.0 license to a duo of proprietary “source available” licenses. The move was a long time coming, and it followed a host of other formerly “open source” companies that made similar switches to protect their business interests — MongoDB in 2018, and CockroachDB a year later, to name a couple.

With the dust now (mostly) settled in the wake of Elastic’s relicensing brouhaha, VentureBeat caught up with cofounder and CEO Shay Banon to get his thoughts on everything that went on: why they made the license change; what impact — if any — it has had on business, and what being a “free and open” company (vs. “open source”) really means.

License to search

Companies typically use Elasticsearch for any application that relies on the access and retrieval of data or documents — it’s a search engine, one that’s used by some of the world’s biggest companies from Netflix to Slack. As the project’s core developer, Elastic — which is dual-headquartered in the Netherlands and U.S. — sells premium features and fully managed services on top of Elasticsearch.

The “problem” with a pure, fully permissive open source license is that anyone — such as large cloud vendors — can take that software and more or less do what they like with it. This includes selling premium or hosted services on top of the open source project, cutting the core creator and project maintainer (e.g. Elastic) out altogether. This does make sense on many levels, as it helps the cloud vendor create a stickier platform and enables its customers to get all their computing services from a single provider. But for the core project maintainer, it means it’s putting the lion’s share of the spadework into the project, including security and feature upgrades, without getting any of the rewards.

But when a third-party builds a commercial service on top of an open source project, it can also create a lot of confusion, with end-users often not clear on which version of a product they are actually using. And that gets to the crux of what Elastic’s move was all about — it was about avoiding confusion between Elastic’s own commercial Elasticsearch offering and Amazon’s.

Amazon launched the Amazon Elasticsearch Service way back in 2015. At the time, Amazon’s chief technology officer announced — in a tweet that was later deleted — that it was in partnership with Elastic. This is something that Banon has previously taken great umbrage at, noting that there was no such partnership in place. In a blog post back in January, Banon wrote:

Imagine our surprise when the Amazon CTO tweeted that the service was released in collaboration with us — it was not. And over the years, we have heard repeatedly that this confusion persists.

What’s in a name?

The road to Elastic’s big license change in January was a long one. In early 2019, AWS announced it was launching a new “open distro” for Elasticsearch, with participation from other notable companies including Netflix, which was pitched as a “value-added” distribution (i.e. not a fork) that was 100% open source and supported by AWS — it also came with the promise that it would continue to send any code contributions and security patches back upstream to the original Elasticsearch project.

But why launch this distro when Elasticsearch was already open source? For that, we have to go further back.

In 2018, Elastic announced that it was making the code from a proprietary product called X-Pack openly available for anyone to inspect and contribute to. This is generally known as “source available,” rather than “open source,” but it served to “muddy the waters” between open source and proprietary code, according to Amazon VP Adrian Cockcroft, who wrote in a 2019 blog post:

Since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on.

For example, neither release notes nor documentation make it clear what is open source and what is proprietary. Enterprise developers may inadvertently apply a fix or enhancement to the proprietary source code. This is hard to track and govern, could lead to breach of license, and could lead to immediate termination of rights.

And it was the culmination of these shenanigans that, ultimately, led Elastic to change the license for Elasticsearch and Kibana in January — and it didn’t have to wait long for a response. Just one week later, Amazon revealed that it would begin work on an open source Elasticsearch fork, which would ship under a completely new name, OpenSearch, which eventually went to market in July.

“Bringing clarity was a big part of why we made the [license] change — it was just painful,” Banon told VentureBeat. “The problem was that lots of customers don’t necessarily focus on the details and distinctions between Elastic’s Elasticsearch offering and other third-party ‘as-a-service’ offerings.”

So while Elastic might push an update or patch out for its Elasticsearch, that wouldn’t necessarily find its way into the third-party offering immediately. By forcing a clearer distinction between the two versions of Elasticsearch, Elastic was safeguarding its brand and reputation.

“Amazon used the Apache 2.0 (open source) license and provided a service, they are totally entitled to do it — it’s perfectly fine and legal,” Banon added. “There were a few things that we did have a problem with that extended beyond the usage of the software. Calling the service ‘Amazon Elasticsearch Service’ — you go to any trademark lawyer, they’ll tell you that’s trademark infringement, and that created confusion in the market. Especially as Amazon and AWS grew more, [the confusion] just became massive. And that’s problematic.”

But if it was just a case of trademark infringement, couldn’t Elastic just tell Amazon not to use the Elasticsearch name in its own service? That was, in fact, a recourse the company was actively pursuing — Elastic filed a trademark infringement lawsuit against Amazon back in 2019. But the problem was, lawsuits take a long time to resolve, consuming significant resources in the process; changing the license was a way to speed things up, according to Banon, and get Amazon to stop using the Elasticsearch brand in its own product offering.

“The legal process was dragging its feet — to be honest, I was really frustrated with the progress,” Banon said. “The wheels of justice will take their turn, and they’ll happen, but at the end of the day, we had users that were confused — users and customers sometimes that didn’t know which one [Amazon’s Elasticsearch or Elastic’s Elasticsearch] is which, which features are being used where. They’d go to the Amazon Elasticsearch Service and think that it was something that we back.”

Believe it or not, this is a fairly common problem in the open source sphere — PrestoDB fork PrestoSQL was forced to change its name to Trino last year after Facebook asserted trademark ownership over the “Presto” name. And just last month, livestreaming software provider Streamlabs OBS had to drop “OBS” from its name after it was called out by the open source OBS project on which it is built. Ultimately, it was all about avoiding brand confusion, with the OBS project’s Twitter account revealing that some of its support volunteers had encountered “angry users” seeking refunds when it was actually the commercial Streamlabs product they had paid for.

We’re often faced with confused users and even companies who do not understand the difference between the two apps.

Support volunteers are sometimes met with angry users demanding refunds. We’ve had interactions with several companies who did not realize our apps were separate.

— OBS (@OBSProject) November 17, 2021

The principles of open source

There were few surprises when Amazon announced its plans for the Elasticsearch fork — “we totally expected it to happen,” Banon said — but Elastic had already bolstered its commercial offering to protect it against any future open source kerfuffles. This mostly involved investing in its proprietary IP, such as its security and application performance management (APM) capabilities, which it had already released under a “free and open” license, rather than an OSI-approved Apache license.

Put simply, customers weren’t necessarily using Elastic because of its open source license, which is why its revenues have continued to grow in the 12 months since it made the switch.

“I think that people were engaging with Elastic because of the quality of the products, and the quality of the community that we built around Elastic, not around one permissive license, to be completely honest,” Banon said.

It is worth noting, however, that some companies other than AWS did decide to ditch Elasticsearch, including CrateDB developer Crate.io, which revealed in February that it was transitioning from Elasticsearch to a “fully open source fork of Elasticsearch.”

Whenever any company switches its open source license, it nearly always riles at least some in the community. But Banon said that despite some of the naysaying, he’s noticed no real impact from a business or community perspective.

“I think the vast majority of the open source community were perfectly fine with our change,” Banon said. “[And] from the metrics that we track, like number of downloads, meetups, community engagement, things along those lines, everything remains the same — and it’s actually going up.”

This gets to the heart of what Banon said is the most important principle of open source — it’s not about the license.

“As a company, we never treated open source as a business model — open source is not a business model,” he said. “The first principle of open source is around engaging on GitHub, for example — you use open source to engage with the community, you use open source as a way to create communities, you use open source to collaborate with people.”

To the casual observer, it might appear that Elastic is against any third-party offering Elasticsearch “as-a-service,” but that isn’t the case. Other companies, including Google and Alibaba, already offer Elasticsearch-as-a-service in direct partnership with Elastic.

“If you take the software and provide it as a service, then I think it’s healthy for both of us to have skin in the game,” Banon said. “That means that when we fix a vulnerability, which has huge implications if you’re providing it ‘as-a-service’, we’ll reach out to you and work together with a vendor to do that. That’s so easy to do, because SaaS is the tide that lifts all boats.”

So does Banon care at all that its core product is no longer “open source” in the purest sense of the word?

“I don’t think it matters — ‘free and open’, I’m fine with that,” he said. “These things can be so distracting, and then you end up losing the things that really matter. Are we still engaging with our community the same way? Are we still engaging with them on GitHub? If these things are still true, then I’m perfectly fine.”