9TH BIT Consulting, specialists in implementing SOA solutions, middleware integration and products, has announced the local availability of StreamSets’ Data Connector Edge (SDC Edge) continuous ingest software, that effectively addresses the sinister challenge of data drift.
9TH BIT Consulting is the African partner for StreamSets, a global leader in data performance management.
Data drift, the rise of sprawling data architectures, and the need for reliable, real-time analytics have all intensified the pressure on data movement, and traditional integration solutions are struggling. Added to these challenges, hand-coded pipelines take too long to cultivate and collapse often, leaving data-driven applications at risk.
This release by StreamSets addresses these issues by simplifying the process of building and running modern data flows, helping companies speed up their time to insights by automatically managing the comprehensiveness, precision and reliability of their data as it flows from collection to consumption.
SDC Edge enables users to build data drift resistant pipelines in a matter of hours (not weeks or months) thereby reducing the costs associated with long lead developments. The system connects to reputable big data systems and can be deployed flexibly on-edge, on-cluster or in the cloud.
Features include the ability to:-
- Build adaptable pipelines with minimal coding and maximum flexibility
Users have access to a simplified graphic user interface through which they can build pipelines without the need to hand code, deploy built in transformations and advanced scripting and triggered execution of external code, and rapidly troubleshoot using data injection, snapshot and replay functions. - Operate continuously in the face of constant change
SDC Edge offers zero downtime for upgrades and migration, as well as direct integration with big data governance tools, including Cloudera Navigator and Apache Atlas, and it enables users to manage complex dataflow topologies with StreamSets Dataflow Performance Manager (DPM). - Execute pipelines wherever they are needed
SDC Edge enables flexible deployment in clusters, across multiple clouds, or on edge nodes and devices, allows users to deploy and scale efficiently against Kubernetes, and offers 100% in-memory operation for high throughput and low latency. - Monitor pipeline performance and data quality
SDC Edge pipelines automatically detect and adapt to data drift, the system offers customisable metrics on dataflow operations and data fidelity, and users are privy to early warning of anomalies via data introspection, samples, threshold rules and alerts.
Barry de Waal, chief executive of strategy and sales at 9TH BIT, said: “Today, moving data reliably and with quality is blocked by the reality of data drift. SDC Edge delivers performance management tools for data flows that feed the next generation of big data applications. This includes the superior management of data in motion, ensuring data arrives in good time, thereby accelerating analysis and simplifying decision making.”
Arvind Prabhakar, co-founder and CTO, StreamSets, said: “The massive volume of data created by the explosion of digital devices presents an invaluable opportunity for analytics and insight.
“However, harnessing this data for important efforts such as IoT and cybersecurity has been a challenge due to the lack of end-to-end data ingestion frameworks. We built SDC Edge to bring disciplined, well-managed data movement to huge populations of IoT sensors and personal devices so that the promised benefits of these critical initiatives are realised.”
Click below to share this article