While the turbulence from the Covid-19 pandemic continues to shape the global business ecosystem, one important trend remains unchanged: our steady march toward a data-first world. Today, companies need to put easy access to reliable and consistent data in the hands of every person — and all in real time. As all leading companies continue transforming themselves into data-driven enterprises, here are five things they should keep in mind going into 2022.
1. Data connectors’ limitations will rear their ugly heads.
Data connectors are the gateways to data and provide meaningful business insights, but they can’t keep up with the pace of new systems. Take the challenges facing delivery companies, for example. Every time they add a new merchant partner to their platforms, they need to integrate data from various systems that those merchants use (for their inventory, operations management, product feeds, etc.). In order to do this, they need a variety of data connectors. However, hand-coding connectors one by one is inefficient, seriously delays making new products available to customers and raises concerns about code reusability and code maintainabilty.
This is why I believe automated data connector generation will eliminate our reliance on overworked data engineering teams to hand-code data connections and pipelines.
2. The modern data stack will become modern data stacks.
The modern data stack is emerging as the go-to cloud data architecture for analytics, machine learning and operations. However, as a business grows in complexity and data maturity, one data stack will no longer be enough. Consider a few scenarios:
• A sales team uses Salesforce as the single source of truth but consumes data and gets insights for quarterly planning in a Tableau dashboard that’s sitting on top of Redshift. This is a traditional analytics stack on a data warehouse.
• Contrast this with a modern retail marketing operations team that looks at all of the store-level events that originate from all stores in a particular geography and converts those events into actions (e.g., marketing offers with a real-time application). Here, we have a stack for streaming data connected to ad platforms.
• Meanwhile, a financial risk team is running an AI-based model to detect fraud in online purchases in real time. This is a real-time data stack in a transaction flow.
• All along, a supply chain team is tracking real-time inventory and shipment information through IoT systems.
Each of these use cases requires its own “modern data stack.” Hundreds of use cases mean hundreds of modern data stacks in the company.
3. The demand for no-code/low-code data solutions will grow.
No-code/low-code platforms, which make it possible for business users and non-data engineers to craft data flows for their needs, have been making headlines recently — and with good reason. I believe data control will continue to move toward data users. Let’s agree that there will never be enough engineers to hand-code all of our data needs, let alone maintain and update the blitzscaling data applications. In fact, an overwhelming majority of data engineers are getting burned out solving the same problems over and over again for an ever-growing number of data users.
For example, we worked with mobile advertising company Kargo to help it directly set up data flows with no coding required, reducing its data delivery time by 85% and boosting its ad campaigns’ ROI significantly.
4. APIs, streaming and bulk data will converge.
When you send a package via FedEx, you don’t worry about whether the package will go by air, train or truck. The same is happening with data. Application creators will soon support multiple data delivery systems so that customers can leverage whatever is fastest or easiest for their data stack. The superior method of delivering data depends on the use case. For example, data on real-time inventory should be streaming, not delivered in bulk, to ensure the most accurate count.
5. Multi-cloud innovation will bring the best of the cloud to large enterprises.
For large enterprises, a multi-cloud strategy helps reduce the risk of vendor lock-in and offers the best-of-breed solution across clouds. An enterprise may prefer AWS for their data lake, Google Cloud for their machine learning, Azure for familiar end-user tools and governance with Active Directory. However, running multi-cloud is both complex from an engineering perspective and expensive from an inter-cloud data egress cost perspective.
As enterprises begin to expand to multi-cloud tenants, interoperability between those clouds and other data systems will become exponentially more important and a high priority as sacrificing usability and performance for multi-cloud solutions will not be an option.
How can organizations better prepare for these trends? Here are a few guidelines:
• Think strategic — configure, don’t code. Instead of hand-coding core platforms, take advantage of the rapid innovation occurring in the enterprise technology domain. By focusing your engineering resources on differentiating strategic projects rather than challenges that have already been solved by platform providers, you can set up your enterprise with a sustained competitive advantage.
• Design for flexibility, agility and scale. Architectures are hard to change, so design for greater flexibility and agility. Implement a faster adoption process for new technology, but be prepared to pivot. Plan for at least 10 times more growth in data volume and variety than your business growth rates. As data powers more operational use cases, the need for real-time data will become urgent sooner than you expect.
• Consider incorporating no-code solutions. The future is moving toward more data and more users of data. Engineering talent will only get more scarce. Let go of the need to control, and decentralize your data by empowering the end users with no-code solutions.
Courtesy- https://www.forbes.com/sites/forbestechcouncil/2022/01/31/five-enterprise-data-predictions-for-2022/?sh=1725e0287f36