Breaking down data silos to ease data management

Given the exponential data growth today, organizations continue to invest heavily in data management tools. However, they still face many issues in organizing and managing their data. There are many reasons for this, but the primary cause of the struggle is simply because of the sheer volume of data they are dealing with today.

When it comes to organizing and managing data, organizations would first need to understand their data. This includes being able to differentiate which data really matters to them and being able to sort it out. Then, there is also the need to look at how they can store the data once they are done using it.

All of these processes may sound complicated. However, with the right tools and technology, most of these processes can be automated, enabling organizations to have a more seamless approach to data management.

To understand more about this, Tech Wire Asia speaks to Suvig Sharma, Area Vice President for Enterprise Sales for ASEAN at Confluent.

Is data deluge the biggest problem organizations are facing today?

With the emergence of new applications and solutions, data is generated from seemingly endless sources. It is understandable that organizations feel overwhelmed by this data overload.

The problem business leaders face today is not about having too much data; it is about unleashing the full potential that the data collected has to offer. Successful organizations are experts at mastering their data. They all continuously collect and analyze massive amounts of data to understand their customers, provide personalized customer experiences, improve operational efficiencies and make better business decisions.

To wade through the deluge, data management is critical for organizations to ensure that information is accurate, reliable and as timely as possible for response, analysis, reporting and decision-making.

Our data streaming platform allows organizations to connect data pipelines, and securely process and store large amounts of information in real-time. This allows teams to aggregate data quickly and have a centralized overview of digital transactions or events across the organization as they occur.  By doing so, teams can efficiently and precisely locate relevant data insights for their needs, effectively making data work smarter not harder, and maximizing the utility of information on hand.

What is data streaming and how can it help organizations deal with the challenges they face today?

Real-time data streaming processes, stores, analyses and acts on a constant flow of data that is generated from multiple sources. Compared to traditional batch processing, where data is scheduled for processing and analysis at fixed volumes or time periods, real-time data streaming reduces the lag time to provide more consistent and timely results.

By setting data in motion, organizations are able to bolster resilience and stay competitive. Having data-driven insights allows companies to closely track ever-changing shifts in all types of metrics, ranging from consumer preferences and inventory management to sales transactions and the use of online services. The modernization of legacy systems to automate data flows will leave organizations with more time and resources to focus on business innovation. With an accurate overview of customer, business and macroeconomic activity as they happen, organizations are able to make data-driven decisions when it comes to pricing strategy or streamlining digital experiences.

Another challenge we’ve seen more organizations face is also mitigating fraud and cybersecurity threats. With data streaming, large volumes of data can be rapidly sorted and analyzed in real-time to facilitate quicker and more accurate detection of anomalies. With such detailed information on hand at all times, organizations are able to make more proactive and on-the-fly decisions that help reduce risks and potential harm.

With organizations seemingly taking a multi-cloud or hybrid cloud approach to their data and workloads today, should they be concerned about complexities in data streaming?

One of the biggest challenges to hybrid and multi-cloud strategies is the ability to liberate their data from their silos and make it easily interoperable and enable self-service access to well-formatted data. Oftentimes, complex data architectures slow organizations down. Data movements using periodic batch data, APIs and custom engineering work result in brittle point-to-point interconnections that make data slow and not secure. The adoption of new cloud services can exacerbate these problems because the more point-to-point data integrations are created, new security challenges and siloed processes are introduced.

Data streaming addresses many of the issues we see with traditional data pipelines that have this point-to-point, brittle connections. It completely decouples the architecture, standardizes schemas for downstream compatibility and enables in-flight processing, combining any stream from anywhere for an enriched view of the data collected. If something happens in one part of the value stream, it can immediately trigger something to happen in another part with intelligence. Just like our central nervous system.

Confluent has built a differentiated, cloud-native platform that sets the industry standard for data in motion. With Kafka at its core, we deliver a fully-managed service that incorporates the infinite capacity, on-demand scalability, and global nature of public clouds into every aspect of its design. The result is a cloud-native solution with a complete enterprise-grade data streaming platform, that’s a fully managed software and service available everywhere. Through our connector ecosystem, we make it simple for our customers to tap into all of their data sources and destinations.

How can decentralization and data streaming break down data silos and the constraints caused by centralized, point-to-point batch architectures?

Understanding the data flow for batch-oriented systems can be complex due to scheduling, checkpoints and data grouping. The reasoning behind the data for these systems is also complex, which often results in discrepancies between what has been derived and what is happening now.

However, with decentralization, data can be processed as it arrives. Essentially, organizations can harness the power of data in motion, where data across business lines are distributed and can operate in their own domain. The continuous processing of data enables the capture of single data events that generate more accurate insights. Placing the decentralization of data ownership and management at the helm builds a self-serve platform that allows data to be utilized independently while ensuring greater interoperability across teams.

As application development takes on a decentralized approach, the capturing and storing of data through data streaming gives rise to a central nervous system that can be easily accessed across different teams, all while ensuring that applications and processes are closely interconnected. Real-time stream processing eventually generates greater data-driven synergy that will resonate throughout whole organizations.

Lastly, can you share with us examples from various APAC companies that have successfully employed data streaming to address these challenges

From enhancing customer experience to ensuring data governance and security, we’ve seen organizations and government agencies across industries optimize their operations and bolster business capabilities with data streaming.

In Singapore, Confluent Cloud is onboarded in Infocomm Media Development Authority’s (IMDA) Tech Acceleration Lab, which allows agencies to test, develop and deploy solutions within a controlled test sandbox on the Government Commercial Cloud (GCC). A Singapore government agency was able to modernize its existing data store with a data streaming platform, allowing real-time interoperability between modern and legacy systems as they gradually migrated to the cloud.

We have also seen financial services players develop scalable and real-time data pipelines that not only reduce downtime by processing transactions in real time but also pinpoint fraud detection and mitigate threats.

Another great example is the work we’ve done with Bank Rakyat Indonesia (BRI), the largest bank in Indonesia. They were in the midst of a digital transformation to improve their market position and improve financial inclusion across the country. With Confluent, they were able to leverage data streaming for real-time credit scoring, fraud detection and merchant assessment services. This resulted in real-time fraud detection, loan disbursements being cut from two weeks to two minutes and loan defaults being predicted proactively.

---

---

---

---