The challenges are changing for today’s data-driven organizations, but it’s still essential to drive digital transformation and get to market faster than the competition. Even in a post-COVID world, the key questions about how to do it remain the same. Is it better to develop your own data tools with Open Source technology, or purchase a commercial product?

Open Source solutions are a great way to build, deploy, and support data flows and other processes, but to make the most of them, you need a smart approach. They are highly complex and require a lot of love to make them work correctly. The technology delivers the very latest and greatest features. But, let’s be realistic, we all know that its documentation is not always the best, and sometimes its tooling needs improvement. Delivering on the potential of Open Source requires not only highly skilled people to operate these systems, but also people who understand your business. Before you choose an Open Source or a commercial solution, consider some of the strengths of both approaches.

Is time on your side?

Going Open Source versus non-Open Source is a business decision that your IT, engineering, and business leaders will need to make.  The big question that needs to be raised is: Is the upside of adopting open-source, and having a competitive advantage, above the risks and the slower time-to-market (and therefore opportunity costs) of going with commercial software? From my conversations with organizations, particularly since COVID, enterprises increasingly want turnkey cloud services that address specific use cases and can be delivered in days and weeks with low risk. They need to accelerate their digital transformation projects above everything else. 

In some cases, this need for agility may outweigh the upside and competitive advantage of adopting an Open Source technology. If the technology is not very mature, the opportunity costs can be too great in this climate to justify working with it. Time is of the essence, and businesses are under pressure to accelerate initiatives like digitizing their supply chain, improving forecasting, and automating manufacturing. At the same time, they must update their cybersecurity and compliance to keep pace with a changing threat landscape and new regulations. The stakes for these types of strategic initiatives are extremely high, and can even mean the difference between a business’ success or failure.

Are you ready to commit?

One of the main challenges of Open Source is that it requires a commitment to maintain it. I’ve talked with a police force in Europe that works primarily with Open Source technology. If their technology cannot provide the capabilities they need, they often wind up building workarounds, patches, and compromises to address issues in the code. They don’t want to fork the code and run it in house, because then they have to maintain it themselves. So, they end up building around it, which takes time and consumes valuable resources that could otherwise be focusing on improving police operations and maximizing public safety. Organizations are often excited about Open Source, but all too often they end up either building around it or abandoning it. 

One of the advantages of choosing a commercial solution is that you’ve got a contract. You can force your vendor to maintain the relationship, put pressure on them to build new features, and in some cases, you can acquire the source code if you need to work with it. Although some may be concerned that commercial solutions are expensive, vendors also play an important role as trusted business partners. The value of having a close partner to call when you’re in desperate need of help is tremendous—especially when compared to being left to resolve an issue on your own. 

Is your solution secure and compliant?

The threat landscape is a moving target, so you’ll always want to keep security at the top of mind. Although Open Source gives you access to bleeding-edge tech, it doesn’t always accommodate security and compliance requirements, especially if they are specific to a particular industry. According to recent research from Veracode, 70% of applications have Open Source security flaws. Most flawed libraries end up in code indirectly.

Commercial offerings can certainly lack adequate security as well, but if it’s designed for a specific use case, you will have better luck finding a solution that has the features you need to align with your compliance and governance needs. Most technology vendors put strict incident response procedures and SLAs in place to minimize vulnerabilities in their software. These vendors understand that a zero-day attack on commercial software can not only impact customers, but their own reputations, so they take security threats very seriously.

Do you have the resources you need—and can you keep them?

It’s one thing to acquire the skill and talent you need to support your data tools and technology. But you also need to be sure you have a strategy in place to retain the best people by providing a superior experience. In a recent Deloitte survey, research showed that enterprises with a top-quartile employee experience achieve twice the innovation, double customer satisfaction, and achieve 25% higher profits than organizations with bottom-quartile employee experience.

Engineers love to work with the latest and greatest technology, so Open Source tools can be a powerful way to keep them engaged and satisfied. At the same time, you’ll want to provide them a defined career path. Retaining the best professionals isn’t easy if your engineers are constantly polishing their resumes and chasing new technology. Focus on building a culture of engagement and retention at your organization, so you’ve got people in place who not only understand the technology you’re using but your business priorities. 

Getting the best of commercial and Open Source with DataOps

Ultimately, the best approach to today’s industry challenges will be a mixture of Open Source, complemented with commercial software. Thankfully, the data technology landscape now includes technologies such as Apache Kafka and Kubernetes that have gained a wide level of adoption. These technologies have proven their value after successfully overcoming early challenges with Big Data technologies. 

However, although these technologies are mature and widely adopted, they can only be operated by an elite set of engineers or operators, pulled from a small talent pool. In our post-COVID environment, digital transformation initiatives that are too dependent on highly technical teams can lead to organizations failing on their strategic commitments. 

Commercial tooling can be used to now expose these technologies beyond IT and engineering teams and foster DataOps practices. DataOps brings down the barriers to working with data, enabling a new set of data consumers to operate these data technologies and their data themselves, without deep technical knowledge—giving them far more in control over their critical digital transformation projects.  DataOps adopts some of the practices from DevOps, but is designed to adapt business users. It helps organizations, bringing non-engineering teams close to their data, allowing them to self-serve whilst at the same time ensuring the highest standards of governance and compliance.  

We’ve found that organizations that adopt DataOps practices have been able to significantly accelerate and improve the delivery of their strategic projects. By enhancing everything from multi-channel marketing campaigns, digitizing supply chains, automating manufacturing, and delivering new fraud-detection services, DataOps enables organizations to unleash the full potential of the technologies they choose, to drive transformation and stay competitive.