Unstructured data management is a relatively new category in the world of enterprise data storage and IT infrastructure. Komprise, founded in 2014, is a pioneer in this space. The company’s three founders launched Komprise Intelligent Data Management to address what they foresaw as an uncontrollable tsunami of data hitting the enterprise from digitization, mobile apps, IoT and more.

Nobody could have predicted the Pandemic of 2020, which accelerated digital processes, and thereby data growth virtually overnight. In late 2022, came another surprise: ChatGPT. AI is the impetus behind unleashing the value of unstructured data. AI, along with other forces like the growth of rich media, log data and unmanaged cloud sprawl, is also behind the rapid growth of unstructured data. Today, most enterprises are storing over 5PB of data.

In recent years, enterprise storage vendors have been talking a lot about data management. Traditionally, the conversations were more about hardware performance, reliability and access. Today, IT execs are betting on hybrid cloud storage as the most cost-effective, flexible way to manage different workloads and varying needs for access, performance and security.

And the only way to effectively optimize a hybrid IT environment is by instilling a data-centric focus and architecture. This entails the ability to see, understand, search, filter, and move unstructured data across cloud, on-premises and edge storage silos.

Komprise: Built for the AI age of unstructured data

Komprise is a storage-agnostic unstructured data management SaaS. It gives organizations complete visibility and control over unstructured data across any NAS, cloud, or SaaS platform. Komprise provides global data classification, analytics, and lifecycle management across file and object storage silos. With Komprise you can see where your data lives, how it’s growing and its true cost and value, independent of any infrastructure. Unlike storage-based data management, Komprise allows you to move data freely without facing lock-in or rehydration penalties when switching vendors.

Komprise is different from storage vendors and data management solutions in the following ways:

1. See & Save Across Silos; No Lock-In

Komprise scans across unstructured data storage silos (NAS, cloud, object) to extract system metadata and enriched metadata into a Global Metadatabase. This solves the traditional chaos of unstructured data: billions of files, disparate storage silos, and very little context about what the data truly contains or how it’s used. Komprise gives IT teams visibility, meaning, and structure across all their data silos without moving the data.

Unlike storage vendor technologies that use proprietary methods to move unstructured data, Komprise gives full transparency with zero lock-in. This is possible with Komprise Transparent Move Technology (TMT), a patented innovation that allows for file-object duality when tiering data.

• Komprise TMT transparently moves cold data from on-premises storage to cloud or secondary storage for an average 70% savings while ensuring that users access their data exactly as before.
• TMT uses dynamic symbolic links (not brittle stubs or agents) to make relocated files appear and behave the same as original files.
• Because it stays out of the “hot data path,” Komprise has minimal impact on primary storage performance. It supports native access to data as objects for cloud analytics, while reducing egress costs.
• Cold data tiering to immutable object storage also presents a highly cost-effective anti-ransomware strategy for file data. Read more here.
• Unlike storage tiering solutions which require you to buy more storage if switching vendors, Komprise allows you to move without rehydrating tiered data.

2. Cut Data Noise to Improve AI Results

Since AI relies upon unstructured data it’s critical to get the right data into pipelines. This is a challenge for unstructured data because this data has been piling up for decades across different places without anyone curating the data.

Komprise Intelligent AI Ingestion delivers a search, curate, classify and move workflow that protects sensitive data, filters 80% of the unwanted data noise, and delivers the data most suited to individual AI projects. In doing all of these things, Komprise reduces waste and cost in AI processing and minimizes security risks.

• The Komprise Global Metadatabase is a comprehensive file index with a Deep Analytics search interface. It delivers a surgical approach with rich filters, unlike traditional ETL and data ingestion approaches that provide connectors to blindly copy data from a source.
• Komprise delivers 2x data transfer speed, benchmarked against a data transfer tool from a major cloud provider.
• Komprise provides built-in standard and custom sensitive data classification so you can reduce the risk of PII and custom sensitive data leakage and avoid compliance violations.
• Komprise automatically maintains an audit trail of each ingestion workflow for data governance and auditing.

3. Prevent Sensitive Data Leaks

A top IT concern for AI is corporate data risk. It is simply too easy for employees to upload files with proprietary and customer data into AI tools. To prevent sensitive data leaks, you need easy ways to find and confine this data. Many storage and IT managers don’t have these tools at their disposal. Komprise Smart Data Workflow Manager has two built-in scanners as part of the Intelligent Data Management platform:

• PII Detection: Select which PII data types to scan for such as national IDs, credit card numbers and email addresses. Komprise supports multiple classifications to identify diverse PII types within any given file.
• Regex and Keyword Search: Find any text patterns across file and object data store silos via both keyword and regular expressions (regex) search. Identify specific data formats such as employee IDs, machine or instrument IDs, product or project codes, or personal health information (PHI) data such as patient record IDs from specific healthcare IT systems. Learn more here.

4. Manage & Move Petabyte-Scale Workloads

Komprise has a proven record of expertly handling complex, petabyte scale enterprise data migrations for organizations across all industries. Komprise Elastic Data Migration is known for delivering exceptionally fast NFS and SMB data transfers using a highly parallel, multi-threaded architecture.

It maximizes resources by breaking migrations into small tasks that run across Komprise Observers. Protocol-level optimizations reduce unnecessary NFS/SMB round trips, cutting overhead. Komprise Hypertransfer creates virtual WAN channels to overcome latency and SMB chattiness, dramatically increasing throughput.

Tests on small-file workloads show Komprise achieving up to 27x faster cloud migrations than other tools.

• Komprise is also different because it analyzes data before migrating anything. That way, organizations can choose to tier cold data that has not been accessed for one year or longer to enterprise archival storage and only migrate the warm and hot data to the new high-performance storage. This is not only cost-effective but ensures that data is always living in the right place at the right time. You can automate policies to continually tier data off as it ages to colder storage.
• Komprise ACE is a popular assessment program that happens pre-migration to identify potential bottlenecks from network and security configurations or other technologies in the customer’s environment. Komprise helps customers fix these issues prior to migration to prevent delays, errors or failures.

Choosing the Right Foundation for the Unstructured Data Era

Independent unstructured data management is the foundation for enterprise cost control, cyber-resilience and AI success. Komprise was designed to deliver a data-centric, storage-agnostic approach that gives enterprises complete visibility and intelligence on their unstructured data, wherever it lives and however it changes over time. By separating what Gartner calls data storage management services from storage infrastructure, organizations can reduce costs, shrink risk and prepare high-value data for AI and analytics without sacrificing flexibility or future choice.