Unstructured data is growing at an estimated 40-60% annually, according to Gartner and other market research sources. Yet the compute and network resources processing this data only boast an average 50% utilization, sitting idle while petabyte-scale AI data ingestion, data tiering, and migration jobs crawl to completion. IT organizations are struggling to not only maintain control for cost optimization but to make large-scale data mobilization faster and simpler for departmental needs.
The new Komprise Elastic Shares patent (US #12,566,637) solves this problem directly with a dynamic resource allocation innovation.
Komprise COO and Cofounder Krishna Subramanian discusses the latest Komprise patent and its implications for IT infrastructure teams.
What is the new Komprise patent and what problem does it solve?
Krishna Subramanian: Komprise Elastic Shares is a significant patent that addresses the core issues of preparing unstructured data for AI and accelerating unstructured data ingestion. Komprise Elastic Shares is a dynamic partitioning technology that continuously redistributes unstructured data processing tasks across a grid of machines in a streaming fashion. This ensures near-linear speed-up at scale without requiring prior knowledge of dataset size, structure, or processing time.
With CPU and GPU utilization on average 50%, it’s hard for enterprise IT teams to economize when requirements are growing quickly for AI and security needs. Meanwhile, storage costs are rising due to the ongoing memory shortage.
Idle machines during migration, tiering and data workflow jobs waste expensive compute and networking resources.
Key Metrics: Unstructured Data Processing Performance
- 50% average CPU/GPU utilization without Elastic Shares
- 20%+ annual unstructured data growth rate
- 25x Komprise Hypertransfer WAN speed improvement
- 2x: AI workflow ingest rate from Komprise Intelligent AI Ingest
- 70%+ data storage cost reduction with Komprise intelligent data tiering
How does Komprise Elastic Shares technology work?
KS: Unstructured data is difficult to partition because traditional techniques are unable to handle the uneven and unknown distribution of large-scale unstructured data trees. Komprise Elastic Shares technology optimizes resources during processing by continuously distributing unstructured data sets in a streaming fashion across a grid of machines. As soon as one machine finishes, new work can be assigned to it, which keeps all machines busy until the processing has finished.
Is Komprise Elastic Shares different from load balancing?
KS: Yes. Traditional static partitioning of unstructured data workloads falls short because the distribution of data sizes is not known upfront, and that distribution is often wildly uneven. Machines can go idle before the job has completed. Komprise Elastic Shares technology overcomes three limitations of load balancing approaches:
- Dynamic partitioning ensures expensive resources get assigned new tasks as soon as the resources become available;
- Komprise can process datasets without prior knowledge of their size, structure, and diverse processing times, which is essential for data streaming to AI;
- Komprise automatically rebalances resource allocation to address unstructured data hierarchies of unknown branch densities.
How will customers benefit?
KS: Enterprise IT organizations can reduce waste, lower costs and accelerate data mobilization tasks intrinsic to AI data workflows and ingestion, metadata enrichment, data migrations, data tiering and sensitive data management. This is critically important for large organizations that are often managing multiple or dozens of petabytes of data, and spending millions of dollars annually on network and compute.
Why is Komprise Elastic Shares patent important right now?
KS: The Elastic Shares patent comes amid heightened focus on unstructured data management as enterprises realize this data is an untapped asset. They need automation to discover, classify, and ingest the right data for AI ROI. Meanwhile, enterprises need to tier and right-place data and stretch existing capacity to curb costs as flash storage prices escalate during a memory shortage. Now, unstructured data management from Komprise can also help IT get more out of their networking and compute resources.
What are other significant technical innovations from Komprise?
KS: Transparent Move Technology (TMT)™ is a Komprise patented technology that is integral to the Komprise Intelligent Data Management platform. It eliminates user and app disruption from cold data tiering to archival storage, due to its dynamic handling of standard symlinks, called Komprise Dynamic Links.
TMT delivers file-object duality so users can access their data from its original location as a file, or at the destination as an object. TMT cuts 70%+ costs and extends storage capacity while also avoiding the dreaded rehydration penalty. Also important are:
- Komprise Intelligent AI Ingest: Ingestion engine delivers 2X-faster data flows to AI while lowering storage, transfer and compute costs.
- Komprise Hypertransfer: This data migration innovation brings 25X faster migrations for SMB data.
- Komprise Elastic Data Migration: Komprise delivers 27X faster migrations for NFS data.
How does Komprise Elastic Shares fit into the Komprise roadmap and overall technology direction?
KS: Unstructured data is crucial for AI and its footprint dominates resource consumption. To address both value and cost issues, techniques to optimize processing of unstructured data are crucial. Komprise is focused on unstructured data and continues to innovate and make significant strides to improve ROI, deliver value, unlock insights and cut costs of unstructured data.
Elastic Shares builds on other Komprise innovations such as Hypertransfer, which moves data 25 times faster over WANs, and Intelligent AI Ingest, which doubles the AI workflow ingest rates by optimizing use of precious AI resources in an enterprise. Stretching flash and GPU capacity to double the workload they can address by fully utilizing available resources is a significant milestone that Elastic Shares provides.
Read the press release:
