Glossary of Terms

Analytics-driven Data Management

Analytics-driven data management is a core principle of the standard-based platform of Komprise Intelligent Data Management that’s based on data insight and automation to strategically and efficiently manage and move unstructured data at massive scale. With Komprise, you can know first, move smart, and take control of massive unstructured data growth while cutting 70% of your enterprise data storage costs, including backup and cloud costs.

Know First: Get insight into your data before you invest. See across your data storage silos, vendors, and clouds to make informed storage and backup decisions.

• Analyze any NAS, S3
• Plan and project storage cost savings
• Search, tag, build virtual data lakes with a global file index

Move Smart: Ensure the right data is in the right place at the right time. Establish analytics-driven policies to manage data based on its need, usage, and value.

• Transparently data tier and data archive cold data without users noticing any difference
• Migrate NFS file data 27 times faster and SMB file data workloads 25 times faster
• Replicate file data to the cloud for 50% less

Take Control: Get back to the business at hand while reducing your storage, backup, and cloud costs and get the fastest, easiest path to the cloud for your file and object data.

• Ensure you have data mobility and avoid storage-vendor lock-in
• Open, standards-based platform
• Native cloud access

Read the Komprise Architecture Overview white paper.

Want To Learn More?

BlueXP

NetApp BlueXP is a management console designed to unify a disparate set of hybrid cloud NetApp products. Many NetApp customers are taking advantage of BlueXP to help them monitor and manage NetApp data storage environments. The BlueXP interface provides a set of data management capabilities, each that are licensed separately:

• Tiering
• Copy/Sync
• Data Classification/PII
• OnTap to S3 Backup

BlueXP is ideally suited for organizations who have multiple NetApp tools and want to simplify the management with a unified console, sometimes called a NetApp control plane. As enterprises increasingly become multi-storage, multi-cloud and hybrid it’s important to also consider broader unstructured data management requirements and know the benefits of a storage-agnostic solution.

Read the 5 requirements of unified data control plane for unstructured data management.

NetApp BlueXP Tiering

Data tiering is one component of BlueXP. According to NetApp, BlueXP is able to:

• Tier to cloud object
• Set flexible policies
• Reduce data storage costs

First of all, it’s important to understand the differences and benefits of file-level tiering vs block-level tiering. Read the white paper. Secondly, it’s important to understand the tiering requirements. As Komprise co-founder and CEO Kumar Goswami wrote in this post, storage-based tiering has some benefits, especially for tiering snapshots, certain log files and other data from Flash storage – data that is proprietary and deleted in short order. But it’s important to understand that block-level tiering, rather than tiering entire files has many potential ramifications, including:

• Limited policies result in more data access from the cloud.
• Defragmentation of blocks leads to higher cloud costs.
• Sequential reads lead to higher cloud costs and lower performance.
• Data tiered to the cloud cannot be accessed from the cloud without licensing a storage file system.
• Tiering blocks impacts performance of the storage array.
• Data access results in re–hydration, thereby reducing potential cost savings.
• Block tiering does not reduce backup costs.
• Block tiering locks you into your storage vendor.
• Proprietary lock-in and cloud file storage licensing costs.

NetApp BlueXP Tiering Feature Comparison with Komprise Intelligent Data Management

Komprise is a storage-agnostic control plane across all your hybrid data estate that optimizes data storage costs and puts enterprise IT organizations in control of their data at all times with no lock-in. Here is a comparison of specific NetApp BlueXP functionality versus Komprise. Be sure to ask:

• Can you tier data that is more than 183 days old?
• Can you tier directly to Amazon S3 IA or Azure Blob Cool?
• Do you require a cooling period on rehydration?
• Do you have flexible data management policies at the share, directory and file levels?
• Can you access tiered files without additional licensing?
• Can you migrate data without rehydration?
• Do you tier files or blocks?

Learn more about Komprise Data Management for NetApp.

Webinar: NetApp + Komprise – Right Data, Right Place, Right Time

Watch a demo of Komprise Storage Insights.

Want To Learn More?

Cloud Data Management

What is Cloud Data Management?

Cloud data management is a way to manage data across cloud platforms, either with or instead of on-premises storage. A popular form of data storage management, the goal is to curb rising cloud data storage costs, but it can be quite a complicated pursuit, which is why most businesses employ an external company offering cloud data management services with the primary goal being cloud cost optimization.

Cloud data management is emerging as an alternative to data management using traditional on-premises software. The benefit of employing a top cloud data management company means that instead of buying on-premises data storage resources and managing them, resources are bought on-demand in the cloud. This cloud data management services model for cloud data storage allows organizations to receive dedicated data management resources on an as-needed basis. Cloud data management also involves finding the right data from on-premises storage and moving this data through data archiving, data tiering, data replication and data protection, or data migration to the cloud.

Advantages of Cloud Data Management

How to manage cloud storage? According to two 2023 surveys (here and here), 94% of respondents say they’re wasting money in the cloud, 69% say that data storage accounts for over one quarter of their company’s cloud costs and 94% said that cloud storage costs are rising. Optimal unstructured data management in the cloud provides four key capabilities that help with managing cloud storage and reduce your cloud data storage costs:

1. Gain Accurate Visibility Across Cloud Accounts into Actual Usage
2. Forecast Savings and Plan Data Management Strategies for Cloud Cost Optimization
3. Cloud Tiering and Archiving Based on Actual Data Usage to Avoid Surprises
   • For example, using last-accessed time vs. last modified provides a more predictable decision on the objects that will be accessed in the future, which avoids costly archiving errors.
4. Radically Simplify Cloud Migrations
   • Easily pick your source and destination
   • Run dozens or hundreds of migrations in parallel
   • Reduce the babysitting

The many benefits of cloud data management services include speeding up technology deployment and reducing system maintenance costs; it can also provide increased flexibility to help meet changing business requirements.

Challenges Faced with Enterprise Cloud Data Management

But, like other cloud computing technologies, enterprise cloud data management services can introduce challenges – for example, data security concerns related to sending sensitive business data outside the corporate firewall for storage. Another challenge is the disruption to existing users and applications who may be using file-based applications on premise since the cloud is predominantly object based.

Cloud data management service solutions should provide you with options to eliminate this disruption by transparently moving and managing data across common formats such as file and object.

Komprise Intelligent Data Management

Features of a Cloud Data Management Services Platform

Some common features and capabilities cloud data management solutions should deliver:

• Data Analytics: Can you get a view of all your cloud data, how it’s being used, and how much it’s costing you? Can you get visibility into on-premises data that you wish to migrate to the cloud? Can you understand where your costs are so you know what to do about them?
• Planning and Forecasting: Can you set policies for how data should get moved either from one cloud storage class to another or from an on-premises storage to the cloud. Can you project your savings? Does this account for hidden fees like retrieval and egress costs?
• Policy based data archiving, data replication, and data management: How much babysitting do you have to do to move and manage data? Do you have to tell the system every time something needs to be moved or does it have policy based intelligent automation?
• Fast Reliable Cloud Data Migration: Does the system support migrating on-premises data to the cloud? Does it handle going over a Wide Area Network? Does it handle your permissions and access controls and preserve security of data both while it’s moving the data and in the cloud?
• Intelligent Cloud Archiving, Intelligent Tiering and Data Lifecycle Management: Does the solution enable you to manage ongoing data lifecycle in the cloud? Does it support the different cloud storage classes (eg High-performance options like File and Cloud NAS and cost-efficient options like Amazon S3 and Glacier)?

In practice, the design and architecture of a cloud varies among cloud providers. Service Level Agreements (SLA) represent the contract which captures the agreed upon guarantees between a service provider and its customers.

It is important to consider that cloud administrators are responsible for factoring:

• Multiple billable dimensions and costs: storage, access, retrievals, API, transitions, initial transfer, and minimal storage-time costs
• Unexpected costs of moving data across different storage classes. Unless access is continually monitored and data is moved back up when it gets hot, you’ll face expensive retrieval fees.

This complexity is the reason why only a mere 20% of organizations are leveraging the cost-saving options available to them in the cloud.

How do Cloud Data Management Services Tools work?

As more enterprise data runs on public cloud infrastructure, many different types of tools and approaches to cloud data management have emerged. The initial focus has been on migrating and managing structured data in the cloud. Cloud data integration, ETL (extraction, transformation and loading), and iPaaS (integration platform as a service) tools are designed to move and manage enterprise applications and databases in the cloud. These tools typically move and manage bulk or batch data or real time data.

Cloud-based analytics and cloud data warehousing have emerged for analyzing and managing hybrid and multi-cloud structured and semi-structured data, such as Snowflake and Databricks.

In the world of unstructured data storage and backup technologies, cloud data management has been driven by the need for cost visibility, cost reduction, cloud cost optimization and optimizing cloud data. As file-level tiering has emerged as a critical component of an intelligent data management strategy and more file data is migrating to the cloud, cloud data management is evolving from cost management to automation and orchestration, governance and compliance, performance monitoring, and security. Even so, spend management continues to be a top priority for any enterprise IT organizing migrating application and data workloads to the cloud.

What are the challenges faced with Cloud Data Management security?

Most of the cloud data management security concerns are related to general cloud computing security questions organizations face. It’s important to evaluate the strengths and security certifications of your cloud data management vendor as part of your overall cloud strategy

Is adoption of Cloud Data Management services growing?

As enterprise IT organizations are increasingly running hybrid, multi-cloud, and edge computing infrastructure, cloud data management services have emerged as a critical requirement. Look for solutions that are open, cross-platform, and ensure you always have native access to your data. Visibility across silos has become a critical need in the enterprise, but it’s equally important to ensure data does not get locked into a proprietary solution that will disrupt users, applications, and customers. The need for cloud native data access and data mobility should not be underestimated. In addition to visibility and access, cloud data management services must enable organizations to take the right action in order to move data to the right place and the right time. The right cloud data management solution will reduce storage, backup and cloud costs as well as ensure a maximum return on the potential value from all enterprise data.

How is Enterprise Cloud Data Management Different from Consumer Systems?

While consumers need to manage cloud storage, it is usually a matter of capacity across personal storage and devices. Enterprise cloud data management involves IT organizations working closely with departments to build strategies and plans that will ensure unstructured data growth is managed and data is accessible and available to the right people at the right time.

Enterprise IT organizations are increasingly adopting cloud data management solutions to understand how cloud (typically multi-cloud) data is growing and manage its lifecycle efficiently across all of their cloud file and object storage options.

Analyzing and Managing Cloud Storage with Komprise

• Get accurate analytics across clouds with a single view across all your users’ cloud accounts and buckets and save on storage costs with an analytics-driven approach.
• Forecast cloud cost optimization by setting different data lifecycle policies based on your own cloud costs.
• Establish policy-based multi-cloud lifecycle management by continuously moving objects by policy across storage classes transparently (e.g., Amazon Standard, Standard-IA, Glacier, Glacier Deep Archive).
• Accelerate cloud data migrations with fast, efficient data migrations across clouds (e.g., AWS, Azure, Google and Wasabi) and even on-premises (ECS, IBM COS, Pure FlashBlade).
• Deliver powerful cloud-to-cloud data replication by running, monitoring, and managing hundreds of migrations faster than ever at a fraction of the cost with Elastic Data Migration.
• Keep your users happy with no retrieval fee surprises and no disruption to users and applications from making poor data movement decisions based on when the data was created.

A cloud data management platform like Komprise, named a Gartner Peer Insights Awards leader, that is analytics-driven, can help you save 50% or more on your cloud storage costs.

Learn more about your options for migrating file workloads to the cloud: The Easy, Fast, No Lock-In Path to the Cloud.

Want To Learn More?

Cloud Migration

Cloud migration refers to the movement of data, processes, and applications from on-premises data storage or legacy infrastructure to cloud-based infrastructure for storage, application processing, data archiving and ongoing data lifecycle management. Komprise offers an analytics-driven cloud migration software solution – Elastic Data Migration – that integrate with most leading cloud service providers, such as AWS, Microsoft Azure, Google Cloud, Wasabi, IBM Cloud and more.

Benefits of Cloud Migration

Migrating to the cloud can offer many advantages – lower operational costs, greater elasticity, and flexibility. Migrating data to the cloud in a native format also ensures you can leverage the computational capabilities of the cloud and not just use it as a cheap storage tier. When migrating to the cloud, you need to consider both the application as well as its data. While application footprints are generally small and relatively easier to migrate, cloud file data migrations need careful planning and execution as data footprints can be large. Cloud migration of file data workloads with Komprise allows you to:

• Plan a data migration strategy using analytics before migration. A pre-migration analysis helps you identify which files need to be migrated, plan how to organize the data to maximize the efficiency of the migration process. It’s important to know how data is used and to determine how large and how old files are throughout the storage system. Since data footprints often reach billions of files, planning a migration is critical.
• Improve scalability with Elastic Data Migration. Data migrations can be time consuming as they involve moving hundreds of terabytes to  petabytes of data.  Since storage that data is migrating from is usually still in use during the migration, the data migration solution needs to move data as fast as possible without slowing down user access to the source storage.  This requires a scalable architecture that can leverage the inherent parallelism of the data sets to migrate multiple data streams in parallel without overburdening any single source storage. Komprise uses a patented elastic data migration architecture that maximizes parallelism while throttling back as needed to preserve source data storage performance.
• Shrink cloud migration time. When compared to generic tools used across heterogeneous cloud and physical storage, Komprise cloud data migration is nearly 30x faster. Performance is maximized at every level with the auto parallelize feature, minimizing network usage and making migration over WAN more efficient.

• Reduce ongoing cloud data storage costs with smart migration, intelligent tiering and data lifecycle management in the cloud. Migrating to the cloud can reduce the amount spent on IT needs, storage maintenance, and hardware upgrades as these are typically handled by the cloud provider. Most clouds provide multiple storage classes at different price points – Komprise intelligently moves data to the right storage class in the cloud based on your policy and performs ongoing data lifecycle management in the cloud to reduce storage cost.  For example, for AWS, unlike cloud intelligent tiering classes, Komprise tiers across both S3 and Glacier storage classes so you get the best cost savings.
• Simplify storage management. With a Komprise cloud migration, you can use a single solution across your multivendor storage and multicloud architectures. All you have to do is connect via open standards – pick the SMB, NFS, and S3 sources along with the appropriate destinations and Komprise handles the rest. You also get a dashboard to monitor and manage all of your migrations from one place. No more sunk costs of point migration tools because Komprise provides ongoing data lifecycle management beyond the data migration.
• Greater resource availability. Moving your data to the cloud allows it to be accessed from wherever users may be, making your it easier for international businesses to store and access their data from around the world. Komprise delivers native data access so you can directly access objects and files in the cloud without getting locked in to your NAS vendor—or even to Komprise.

Cloud Migration Process

The cloud data migration process can differ widely based on a company’s storage needs, business model, environment of current storage, and goals for the new cloud-based system. Below are the main steps involved in migrating to the cloud.

Step 1 – Analyze Current Storage Environment and Create Migration Strategy

A smooth migration to the cloud requires proper planning to ensure that all bases are covered before the migration begins. It’s important to understand why the move is beneficial and how to get the most out of the new cloud-based features before the process continues.

Step 2 – Choose Your Cloud Deployment Environment

After taking a thorough look at the current resource requirements across your storage system, you can choose who will be your cloud storage provider(s). At this stage, it’s decided which type of hardware the system will use, whether it’s used in a single or multi-cloud solution, and if the cloud solution will be public or private.

Step 3 – Migrate Data and Applications to the Cloud

Application workload migration to the cloud can be done through generic tools.  However, since data migration involves moving petabytes of data and billions of files, you need a data management software solution that can migrate data efficiently in a number of ways including through a public internet connection, a private internet connection, (LAN or a WAN), etc.

Step 4 – Validate Data After Migration

Once the migration is complete, the data within the cloud can be validated and production access to the storage system can be swapped from on-premises to the cloud.  Data validation often requires MD5 checksum on every file to ensure the integrity of the data is intact after migration.

Komprise Cloud Data Migration

With Elastic Data Migration from Komprise, you can affordably run and manage hundreds of migrations across many different platforms simultaneously. Gain access to a full suite of high-speed cloud migration tools from a single dashboard that takes on the heavy lifting of migrations, and moves your data nearly 30x faster than traditional available services—all without any access disruption to users or apps.

Our team of cloud migration professionals with over two decades of experience developing efficient IT solutions have helped businesses around the world provide faster and smoother data migrations with total confidence and none of the headaches. Contact us to learn more about our cloud data migration solution or sign up for a free trial to see the benefits beyond data migration with our analytics-driven Intelligent Data Management solution.

Learn more about your options for migrating file workloads to the cloud: The Easy, Fast, No Lock-In Path to the Cloud.

Want To Learn More?

Cloud NAS

What is Cloud NAS?

Cloud NAS is a relatively new term – it refers to a cloud-based storage solution to store and manage files. Cloud NAS or cloud file storage is gaining prominence and several vendors have now released cloud NAS offerings.

What is NAS?

Network Attached Storage (NAS) refers to data storage that can be accessed from different devices over a network. NAS environments have gained prominence for file-based workloads because they provide a hierarchical structure of directories and folders that makes it easier to organize and find files. Many enterprise applications today are file-based, and use files stored in a NAS as their data repositories.

Access Protocols

Cloud NAS storage is accessed via the Server Message Block (SMB) and Network File System (NFS) protocols. On-premises NAS environments are also accessed via SMB and NFS.

Why is Cloud NAS gaining in importance?

While the cloud was initially used by DevOps teams for new cloud-native applications that were largely object-based, the cloud is now seen as a major destination for core enterprise applications. These enterprise workloads are largely file-based, and so moving them to the cloud without rewriting the application means file-based workloads need to be able to run in the cloud.

To address this need, both cloud vendors and third-party storage providers are now creating cloud-based NAS offerings. Here are some examples of cloud NAS offerings:

• Amazon Elastic File System (EFS) – A file system by AWS that runs the NFS protocol.
• Amazon FSx – Windows File Server running on AWS, for SMB based workloads.
• Amazon FSx for NetApp ONTAP – Fully managed file services built on NetApp’s popular ONTAP, which makes it easy and cost-effective to launch, run, and scale feature-rich, high-performance file systems in the cloud.
• NetApp Cloud Volumes ONTAP (CVO) – NetApp ONTAP based file system on AWS, Azure and Google
• Qumulo Cloud NAS – Qumulo NFS and SMB running in the cloud
• Azure Files – Fully managed serverless file shares.
• Azure NetApp Files – High-performance NFS based file storage running NetApp in the Azure cloud.

Cloud NAS Tiers

Cloud NAS storage is often designed for high-performance file workloads and its high performance Flash tier can be very expensive.

Many Cloud NAS offerings such as AWS EFS and NetApp CloudVolumes ONTAP do offer some less expensive file tiers – but putting data in these lower tiers requires some data management solution. As an example, the standard tier of AWS EFS is 10 times more expensive than the standard tier of AWS S3. Furthermore, when you use a Cloud NAS, you may also have to replicate and backup the data, which can often make it three times more expensive. As this data becomes inactive and cold data, it is very important to manage data lifecycle on Cloud NAS to ensure you are only paying for what you use and not for dormant cold data on expensive tiers.

Intelligent Data Archiving and Intelligent Data Tiering for Cloud NAS

An analytics-driven unstructured data management solution can help you get the right data onto your cloud NAS and keep your cloud NAS costs low by managing the data lifecycle with intelligent archiving and intelligent tiering.

As an example, Komprise Intelligent Data Management for multi-cloud does the following:

• Analyzes your on-premises NAS data so you can pick the data sets you want to migrate to the cloud
• Migrates on-premises NAS data to your cloud NAS with speed, reliability and efficiency
• Analyzes data on your cloud NAS to show you how data is getting cold and inactive
• Enables policy-based automation so you can decide when data should be archived and tiered from expensive Cloud NAS tiers to lower cost file or object classes
• Monitors ongoing costs to ensure you avoid expensive retrieval fees when cold data becomes hot again
• Eliminates expensive backup and DR costs of cold data on cloud NAS

Cloud NAS Migration

There are man potential advantages to migrated your NAS device to the cloud. But the right approach to cloud data migration is essential. Some of the common cloud NAS migration challenges are outlined in this post: Eliminating the Roadblocks of Cloud Data Migrations for File and NAS Data. Avoid unstructured data migration challenges and pitfalls with an analytics-first approach to cloud data migration and unstructured data management. With Komprise Elastic Data Migration you will:

• Know before you migrate – analytics drive the most cost-effective plans
• Preserve data integrity – maintain metadata, run MD5 checksums
• Save time and costs – multi-level parallelism provides elastic scaling
• Be worry-free – built for petabyte-scale that ensures reliability
• Migrate NFS 27X faster and Migrate SMB data 25X faster – forget slow, free tools that need babysitting

Get the fast, no lock-in path to the cloud with a unified platform for unstructured data migration.

———-

Want To Learn More?

Cold Data Storage

What is cold data?

Cold data refers to data that is infrequently accessed, as compared to hot data that is frequently accessed. As unstructured data grows at unprecedented rates, organizations are realizing the advantages of utilizing cold data storage devices instead of high-performance primary storage as they are much more economical, simple to set up & use, and are less prone to suffering from drive failure.

For many organizations, the real difficulty with cold data is figuring out when data should be considered hot and kept on primary storage or it can be labeled as cold and moved off to a secondary storage device. For this reason, it’s important to understand the difference between data types to develop a solution for managing cold data that is most cost effective for your organization.

Types of Data That Cold Storage is Typically Used For

Examples of data types for which cold storage may be suitable include information a business is required to keep for regulatory compliance, video, photographs, and data that is saved for backup, archival, big-data analytics or disaster recovery purposes. As this data ages and is less frequently accessed, it can generally be moved to cold storage. A policy-based data management approach allows organizations to optimize storage resources and reduce data storage costs by moving inactive data to more economical cold data storage.

Advantages of Developing a Cold Data Storage Solution

1. Prevent primary storage solutions from becoming overburdened with unused data
2. Reduce overall resource costs of data storage
3. Simplify data storage solution and optimize the management of its data
4. Efficiently meet governance and compliance requirements
5. Make use of more affordable & reliable mechanical storage drives for lesser used data

Reduce Strain on Primary Storage by Moving Cold Data to Secondary Storage

Affordable Costs of Cold Storage

When comparing costs for enterprise-level storage drives, the mechanical drives used in many cold data storage systems are just over 20% of the price that high-end solid-state drives (SSD) can cost on average. For SSD’s at the top tier of performance, storage still costs close to 10 centers per gigabyte whereas NAS-level mechanical drives cost only around 2 centers per gigabyte on average.

Simplify Your Data Storage Solution

A well-optimized cold data storage system can make your local storage infrastructure much less cluttered & easier to maintain. As the storage tools which help us automatically determine which data is hot and cold continue to improve, managing the movement of data between solutions or tiers is becoming easier every year. Some cold data storage solutions are even starting to automate the entirety of the unstructured data management process based on rules that the business establishes.

Meet Regulatory or Compliance Requirements

Many organizations in the healthcare industry are required to hold onto their data for extended periods of time, if not forever. With the possibility of facing litigation somewhere down the line based on having this data intact, corporations are opting to use a cold data storage solution which can effectively store critically important, unused data under conditions in which it cannot be tampered with or altered.

Increase Data Durability with Cold Data Storage

Reliability is one of the most important factors when choosing a data storage solution to house data for extended periods of time or indefinitely. Mechanical drives can be somewhat slower than SSD’s in providing file access, but they are still quick to be able to pull files and offer much more budget room for creating additional backup or parity within your storage system.

When considering storage hardware for cold data solutions, consider low cost, high-capacity options with a high degree of data durability so your data can remain intact for as long as it needs to be stored for.

Learn more about the your options when it comes to migrating file workloads to the cloud.

How Pfizer Saved Millions with a Cold Data Management Strategy

Pfizer needed to change the way it was managing petabytes of unstructured data to cut data storage costs and reinvest in areas with patients at the center. Read the blog.

Want To Learn More?

Data Classification

Data classification is the process of organizing data into tiers of information for data organizational purposes.

Data classification is essential to make data easy to find and retrieve so that your organization can optimize risk management, compliance, and legal requirements. Written guidelines are essential in order to define the categories and criteria to classify your organization’s data. It is also important to define the roles and responsibilities of employees in the data organization structure.

When data classification procedures are established, security standards should also be established to address data lifecycle requirements. Classification should be simple so employees can easily comply with the standard.

Examples of types of data classifications:

• 1st Classification: Data that is free to share with the public
• 2nd Classification: Internal data not intended for the public
• 3rd Classification: Sensitive internal data that would negatively impact the organization if disclosed
• 4th Classification: Highly sensitive data that could put an organization at risk

Data classification is a complex process, but automated systems can help streamline this process. The enterprise must create the criteria for classification, outline the roles and responsibilities of employees to maintain the protocols, and implement proper security standards. Properly executed, data classification will provide a framework for the data storage, transmission and retrieval of data.

Automation simplifies data classification by enabling you to dynamically set different filters and classification criteria when viewing data across your storage. For instance, if you wanted to classify all data belonging to users who are no longer at the company as “zombie data,” the Komprise Intelligent Data Management solution will aggregate files that fit into the zombie data criterion to help you quickly classify your data.

Data Classification and Komprise Deep Analytics

Komprise Deep Analytics gives data storage administrators and line of business users granular, flexible search capabilities and indexes data creating a Global File Index across file, object and cloud data storage spanning petabytes of unstructured data. Komprise Deep Analytics Actions uses these virtual datasets (see virtual data lake) for systematic, policy-driven data management actions that can feed your data pipelines.

Want To Learn More?

Data Hoarding

What is Data Hoarding?

Data hoarding is now being recognized as a growing challenge in the technology world. Many IT teams are caught in an endless cycle of buying more data storage. Unstructured data is growing at record rates and this data is increasingly being stored across hybrid cloud infrastructure. This massive data growth and increased data mobility has only created more disconnected data silos. Just like hoarding has been recognized as a real problem in the real-world (see reality TV shows like Hoarders and Storage Wars), data hoarding refers to the practice of retaining large amounts of data that is no longer needed or is rarely used, for extended periods of time. This is a common problem in many organizations, where employees tend to save data out of habit, fear of losing it, or simply because they don’t know what to do with it.

What is the impact of data hoarding?

The impact of data hoarding is more significant than most people / organizations realize, including:

• Increased costs: Storing large amounts of unnecessary data can be expensive, especially if the organization is using expensive storage solutions, such as high-end disk arrays or tape libraries.
• Reduced efficiency: Hoarded data can slow down systems and applications, as well as increase the time required to complete backups and other data management tasks.
• Compliance risks: Hoarded data can pose a risk to organizations in terms of compliance, as they may contain sensitive information that is subject to data privacy regulations.
• Cybersecurity risks: Hoarded data can also pose a security risk, as it may contain sensitive information that could be targeted by cybercriminals or hackers.

Stop Treating All Data the Same

Sound familiar?

• Cold data sits on expensive storage.
• Everything gets replicated.
• Everything gets backed up and backup windows are getting longer.
• Costs are spiraling out of control.

The IDC report, How to Manage Your Data Growth Smarter with Data Literacy noted:

• 60% of the storage budget is not really spent on storage. It’s spent on secondary copies of data for data protection – backups, backup software licenses, replication, and disaster recovery.
• 1/3 of IT organizations are spending most of their IT storage on secondary data.

And with ransomware attacks on the rise, which increasingly target unstructured data, it’s increasingly important to find ways to manage, tier, migrate, replicate file data within tight IT budgets. Read the blog post: How to Protect File Data from Ransomware at 80% Lower Cost.

Dealing with Data Hoarding

To address the data hoarding challenge and establish an Intelligent Data Management strategy, IDC recommends the following:

1. Focus less on finding alternatives to store data better/faster and focus more on finding intelligent alternatives to unstructured data management.
2. Use modern, next-generation cloud data management technologies that are lightweight and non-intrusive, and that demonstrate powerful return on investment.
3. Aim to deliver continuous insights as a service to business and achieve speed of intelligence for a competitive edge.

Establish a Cold Data Storage Strategy

One obvious strategy to deal with data hoarding is to define a cold data storage strategy and establish unstructured data management policies.

Read this post to learn how to quantify the business value impact of Komprise Intelligent Data Management.

Want To Learn More?

Data Management Policy

What is a Data Management Policy?

A data management policy addresses the operating policy that focuses on the management and governance of data assets, and is a cornerstone of governing enterprise data assets. This policy should be managed by a team within the organization that identifies how the policy is accessed and used, who enforces the data management policy, and how it is communicated to employees.

It is recommended that an effective data management policy team include top executives to lead in order for governance and accountability to be enforced. In many organizations, the Chief Information Officer (CIO) and other senior management can demonstrate their understanding of the importance of data management by either authoring or supporting directives that will be used to govern and enforce data standards.

Considerations to consider in a data management policy

• Enterprise data is not owned by any individual or business unit, but is owned by the enterprise
• Enterprise data must be safe
• Enterprise data must be accessible to individuals within the organization
• Metadata should be developed and utilized for all structured and unstructured data
• Data owners should be accountable for enterprise data
• Users should not have to worry about where data lives
• Data should be accessible to users no matter where it resides

Ultimately, a data management policy should guide your organization’s philosophy toward managing data as a valued enterprise asset. Watch the video: Intelligent Data Management: Policy-Based Automation

Developing an unstructured data management policy

It is important to develop enterprise-wide data management policies using a flexible governance framework that can adapt to unique business scenarios and requirements. Identify the right technologies following a proof of concept approach that supports specific risk management and compliance use cases. Tool proliferation is always a problem so look to consolidate and set standards that address end-to-end scenarios. Unstructured data management policies must address data storage, data migration, data tiering, data replication, data archiving and data lifecycle management of unstructured data (block, file, and object data stores) in addition to the semi-structured and structured data lakes, data warehouses and other so-called big-data repositories.

Read the VentureBeat article: How to create data management policies for unstructured data.

Want To Learn More?

Data Retention

Data retention is the term used for storing and keeping data for a specific period of time based on legal, regulatory, business, or operational requirements. While for many organizations there is overlap with the term data hoarding, data retention involves defining policies and procedures to determine how long different types of data (the majority of which is unstructured data) should be retained, as well as ensuring compliance with applicable laws and regulations regarding data storage and privacy.

Key points about data retention:

• Legal and Regulatory Requirements: Many industries and jurisdictions have specific regulations or laws that dictate how long certain types of data must be retained. These requirements aim to ensure compliance, support legal obligations, facilitate audits, or provide evidence in case of disputes or investigations. Examples include financial records, healthcare data, customer information, and communication records.
• Business and Operational Needs: Organizations establish data retention policies to address their internal needs, such as operational efficiency, historical analysis, reporting, or knowledge management. Retaining data for a certain period allows organizations to reference past information, track trends, support decision-making, or fulfill business requirements.
• Retention Periods: The duration for which data should be retained varies depending on factors such as data type, industry regulations, legal requirements, business practices, and risk considerations. Some data may only need to be retained for a short period, while other data, especially for compliance-related purposes, may need to be retained for several years or even indefinitely.
• Data Lifecycle: Data retention is part of the broader data lifecycle management process. It involves stages such as data creation, storage, usage, archival, and ultimately disposal. Retention policies define how long data should be kept at each stage and provide guidelines for when and how data should be archived or deleted.
• Data Security and Privacy: During the retention period, it is essential to ensure the security and privacy of the stored data. Adequate security measures, access controls, and data protection mechanisms should be in place to protect the data from unauthorized access, loss, or breach.
• Disposal and Data Destruction: At the end of the retention period, data should be disposed of properly. Secure data disposal methods, including data destruction techniques like shredding or data wiping, should be employed to ensure that sensitive or confidential information cannot be recovered or accessed.
• Legal Holds and Exceptions: In some cases, legal holds or litigation may require data retention beyond the initially defined periods. Legal holds suspend the regular data disposal practices to preserve relevant data for legal proceedings or investigations. Learn more about Smart Data Workflow use cases, including legal hold.

It is crucial for organizations to establish clear data retention policies, regularly review and update them to align with changing requirements, and ensure compliance with applicable laws and regulations. Consulting legal and compliance professionals can help organizations determine the appropriate retention periods and develop robust data retention practices. Policy-based unstructured data management and mobility should be a core component of your enterprise data retention strategy.

Want To Learn More?

Data Storage Costs

Data storage costs are the expenses associated with storing and maintaining data in various forms of storage media, such as hard drives, solid-state drives (SSDs), cloud storage, and tape storage. These costs can be influenced by a variety of factors, including the size of the data, the type of storage media used, the frequency of data access, and the level of redundancy required. As the amount of unstructured data generated continues to grow, the cost of storing it remains a significant consideration for many organizations. In fact, according to the Komprise 2023 State of Unstructured Data Management Report, the majority of enterprise IT organizations are spending over 30% of their budget on data storage, backups and disaster recovery. This is why shifting from storage management to storage-agnostic data management continues to be a topic of conversation for enterprise IT leaders.

Cloud Data Storage Costs

Cloud data storage costs refer to the expenses incurred for storing data on cloud storage platforms provided by companies like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). In addition to the points above about data storage costs (amount of data stored and frequency of data access) in the cloud the level of durability and availability required are also factors when it comes to cloud storage costs. Cloud data storage providers typically charge based on the amount of data stored per unit of time, and additional fees may be incurred for data retrieval, data transfer, and data processing. Many cloud storage providers offer different storage tiers with varying levels of performance and cost, allowing customers to choose the option that best fits their budget and performance needs. With the right cloud data management strategy, cloud storage can be more cost-effective than traditional hardware-centric on-premises storage, especially for organizations with large amounts of data and high storage needs.

Managing Data Storage Costs

Managing data storage costs involves making informed decisions (and the right investment strategies) about how to store, access, and use data in a cost-effective manner. Here are some strategies for managing data storage costs:

• Data archiving: Archiving infrequently accessed data to lower cost storage options, such as object storage or tape, can help reduce storage costs.
• Data tiering: Using different storage tiers for different types of data based on their access frequency and importance can help optimize costs.
• Compression and deduplication: A well known data storage technique, compressing data and deduplicating redundant data can help reduce the amount of storage needed and lower costs.
• Cloud file storage: Using cloud storage can be more cost-effective than traditional on-premises storage, especially for organizations with large amounts of data and high storage needs.
• Data lifecycle management (aka Information Lifecycle Management): Regularly reviewing and purging unneeded data can help control storage costs over time.
• Cost monitoring and optimization (see cloud cost optimization): Regularly monitoring and analyzing data storage costs and usage patterns can help identify opportunities for cost optimization.

By using a combination of these strategies, organizations can effectively manage their data storage costs and ensure that they are using their data storage resources efficiently. Additionally, organizations can negotiate with data storage providers to secure better pricing and take advantage of cost-saving opportunities like bulk purchasing or long-term contracts.

Stop Overspending on Data Storage with Komprise

The blog post How Storage Teams Use Komprise Deep Analytics summarizes a number of strategies storage teams use Komprise Intelligent Data Management to deliver greater data storage cost savings and unstructured data value to the business, including:

• Business unit metrics with interactive dashboards
• Business-unit data tiering, retention and deletion
• Identifying and deleting duplicates
• Mobilizing specific data sets for third-party tools
• Using data tags from on-premises sources in the cloud

In the blog post Quantifying the Business Value of Komprise Intelligent Data Management, we review a storage cost savings analysis that saves customers an average 57% of overall data storage costs and over $2.6M+ annually. In addition to cost savings, benefits include:

Plan Future Data Storage Purchases with Visibility and Insight

With an analytics-first approach, Komprise delivers visibility into how data is growing and being used across a customer’s data storage silos – on-premises and in the cloud. Data storage administrators no longer have to make critical storage capacity planning decisions in the dark and now can understand how much more storage will be needed, when and how to streamline purchases during planning.

Optimize Data Storage, Backup, and DR Footprint

Komprise reduces the amount of data stored on Tier 1 NAS, as well as the amount of actively managed data—so customers can shrink backups, reduce backup licensing costs, and reduce DR costs.

Faster Cloud Data Migrations

Auto parallelize at every level to maximize performance, minimize network usage to migrate efficiently over WANs, and migrate more than 25 times faster than generic tools across heterogeneous cloud and storage with Elastic Data Migration.

Reduced Datacenter Footprint

Komprise moves and copies data to secondary storage to help reduce on-premises data center costs, based on customizable data management policies.

Risk Mitigation

Since Komprise works across storage vendors and technologies to provide native access without lock-in, organizations reduce the risk of reliance on any one storage vendor.

Want To Learn More?

File Analysis (File Storage Analysis)

File analysis or file storage analysis is the process of evaluating and managing the storage of digital files within an organization or on a computer system. The goal of storage analysis is to optimize file storage resources, improve data accessibility, and ensure efficient use of data storage infrastructure.

Gartner Peer Insights defines File Analysis (FA) products this way:

“File analysis (FA) products analyze, index, search, track and report on file metadata and file content, enabling organizations to take action on files according to what was identified. FA provides detailed metadata and contextual information to enable better information governance and organizational efficiency for unstructured data management. FA is an emerging solution, made of disparate technologies, that assists organizations in understanding the ever-growing volume of unstructured data, including file shares, email databases, enterprise file sync and share, records management, enterprise content management, Microsoft SharePoint and data archives.”

Read: Komprise Names Top File Analysis Software Vendor by Gartner

Komprise Analysis: Make the Right File Data Storage Investments

Komprise Analysis allows customers with petabyte-scale unstructured data volumes to quickly gain visibility across storage silos and the cloud and make data-driven decisions. Plan what to migrate, what to tier, and understand the financial impact with an analytics-driven approach to unstructured data management and mobility. Komprise Analysis is available as a standalone SaaS solution included with Komprise Elastic Data Migration and the full Komprise Intelligent Data Management Platform. Read: What Can Komprise Analysis Do For You?

Why File Data Analysis?

File storage analysis is the process of evaluating and managing the storage of digital files within an organization. The goal of storage analysis is typically to optimize file storage resources and cost, improve data accessibility, and ensure efficient use of storage infrastructure. Some common file storage analysis use cases include:

• Storage Capacity Assessment: Determine the total storage capacity available, both in terms of physical storage devices (e.g., hard drives, SSDs) and cloud storage services (e.g., AWS S3, Azure Blob Storage). This assessment helps in understanding how much storage is currently being used and how much is available for future use.
• Storage Usage Analysis: Analyze how storage space is being utilized, including the types and sizes of files stored, the distribution of data across different file types, and the storage consumption patterns over time.
• File Data Lifecycle Management: Implement file lifecycle policies to identify and manage files based on their age, usage, and importance. This includes data archiving, data deletion (See: Data Hoarding), or file data migration to different storage tiers as they age or become less frequently accessed.
• Duplicate File Identification: Identify and eliminate duplicate files to free up storage space. Duplicate files are common in many organizations and can waste valuable storage resources. Watch a demonstration of the Komprise Potential Duplicates Report.
• Access and Permission Analysis: Review and audit access permissions to files and folders to ensure that only authorized users have access. This analysis helps enhance security and compliance with data privacy regulations.
• Performance Optimization: Analyze storage performance to ensure that data retrieval and storage operations meet performance expectations. This may involve optimizing file placement on storage devices, load balancing, and caching strategies.
• Cost Optimization (including Cloud Cost Optimization): Evaluate the costs associated with different storage solutions, including on-premises storage, cloud storage, and hybrid storage configurations. Optimize storage costs by selecting the most cost-effective storage options based on data usage patterns.
• Backup and Disaster Recovery Analysis: Ensure that files are properly backed up and that disaster recovery plans are in place. Regularly test data recovery processes to verify their effectiveness. It’s important to analyze your data before backup to optimize data storage and backup costs.
• Data Retention Policy Compliance: Ensure that data retention policies are adhered to, particularly in industries subject to strict data compliance regulations (e.g., healthcare, finance). This involves safely deleting files that are no longer needed and retaining data as required by law.
• Storage Tiering and Optimization: Implement data storage tiering strategies to allocate data to the most suitable storage class based on access frequency and performance requirements. This can include the use of high-performance SSDs for frequently accessed data and slower, less expensive storage for archival purposes. Read the white paper: File-level Tiering vs. Block Level Tiering.
• Forecasting and Capacity Planning: Predict future storage needs based on historical data and growth trends. This helps organizations prepare for increased storage requirements and avoid unexpected storage shortages. See FinOps.

The right approach to file storage analysis involves the use of specialized data management and storage management software and tools. Read more about the benefits of storage-agnostic unstructured data management. The goal is to deliver insights into storage usage, performance metrics, and compliance with storage policies in order to make informed decisions about storage investments and ensure that file storage is efficient, cost-effective, and aligned with business needs.

Want To Learn More?

File Data Management

File data management is the process of organizing, storing, and retrieving digital files in an efficient and secure manner. This can include tasks such as:

• Naming files in a consistent and descriptive manner
• Creating folders and sub-folders to categorize and store files
• Regularly backing up important files to prevent data loss
• Purging old or unnecessary files to free up storage space
• Using appropriate software tools to manage, search and retrieve files

Effective file data management helps improve productivity and organization, and reduces the risk of data loss or corruption. It is a critical aspect of overall data management, especially in businesses and organizations where large amounts of data are generated and stored on a regular basis.

File Data Management Challenges

Because we’re talking about unstructured data, file data management can present a number of challenges, including:

• Data Growth: As more and more data is generated and stored, it can become difficult to manage and organize effectively. The majority is unstructured data.
• Data Duplication: Duplicate files can lead to confusion, waste storage space and make it harder to find the most up-to-date version of a file.
• Data Security: Protecting sensitive information from unauthorized access or cyberattacks is a major concern in file data management. (Read about cyber resiliency and saving on ransomware production.)
• Data Loss: Accidentally deleting or losing files can result in significant data loss and potential productivity loss.
• Compliance: Certain industries and organizations may have regulatory requirements for file data management, such as retention policies and data privacy laws.
• Integration with Other Systems: Integrating file data management systems with other applications, such as email, CRM, and collaboration platforms, can be complex and time-consuming.
• Scalability: As the amount of data grows, the file data management system must be able to scale to meet the demands of the organization.
• Compatibility: Ensuring that files can be opened and used by multiple users and systems can be a challenge, especially with different file formats and software versions.

These challenges can be addressed through the use of appropriate software tools, best practices for file data management, and regular reviews and updates to the file data management policies.

Komprise File Data Management

Komprise Intelligent Data Management has been designed from the ground-up to simplify file data management and put customers in control of unstructured data, no matter where data lives. Analytics-first approach, Komprise works across file and object storage, across cloud and on-premises, and across data storage and data backup architectures to deliver a consistent way to manage data. With Komprise you get instant insight into all of your unstructured data—wherever it resides. See patterns, make decisions, make moves, and save money—all without compromising user access to any data. Komprise puts you in control of your data while simplifying file data management by creating a lightweight management plane across all your data storage silos without getting in the path of data access.

Block vs File Level Data Storage Tiering

A primary file data management technique is data tiering. Here is a summary of block-level versus file-level tiering and the impact. Also download the whitepaper and learn more about Komprise Transparent Move Technology (TMT).

Want To Learn More?

Hybrid Cloud File Data Services

In February 2023, Gartner industry analyst Julia Palmer published published a research article: Modernize Your File Storage and Data Services for the Hybrid Cloud Future. (Blocks & Files summary). According to Gartner, hybrid cloud file data services provide data access, data movement, life cycle management and data orchestration. Komprise is listed as top vendor in this category. The other two categories included in the note are:

• Next-generation file platforms: on-premises filers adding hybrid cloud capability and new software-only file services suppliers (VAST Data, NetApp, Qumulo)
• Hybrid cloud file platforms: providing public cloud-based distributed file services (Ctera, Nasuni, Panzura)

Hybrid cloud file data services refer to the use of file-based storage solutions in a hybrid cloud environment, which combines on-premises infrastructure with cloud resources, allowing organizations to leverage the benefits of both private and public clouds. File data services, in this context, involve the access, management, movement and even storage and retrieval of files and data within a hybrid cloud setup.

In the article: Unstructured Data Growth and AI Give Rise to Data Services, Komprise cofounder and COO Krishna Subramanian summarized the benefits of a data services approach as:

• Holistic visibility and granular search across multiple storage systems and clouds;
• Analytics and insights on data types and usage for more accurate storage decisions;
• Automated, policy-driven actions based on that analysis;
• Reduced security and compliance risks;
• Full use of data wherever it is stored, especially in the cloud;
• User self-service access to support departmental and research needs for data storage, management, and AI workflows;
• Greater flexibility to adopt new storage, backup, and DR technologies because data is managed independently of any vendor technology.

The article concludes:

Above all, data management and storage infrastructure experts will need to shift their thinking and practices from managing storage technologies to understanding and managing data for a variety of purposes. A data storage and data management infrastructure that supports flexibility and agility to shift with organizational data needs will allow IT to make the shift faster and with better outcomes for all.

What are some of the components and features of hybrid cloud file data services?

Still an emerging category, in the Gartner Top Trends in Enterprise Data Storage 2023 report from Gartner (subscription required), it is note that, “by 2027, 60% of Infrastructure and Operations leaders will implement hybrid cloud file deployments, up from 20% in early 2023.” Hybrid cloud file data services “provide data access and data management across edge, cloud and core data center locations through a single global namespace.” The report goes on to note: “Increasingly, enterprises are creating, ingesting and accessing data in edge locations, factories, field offices and retail locations. The data services to analyze or enhance data are typically present in the public cloud, but the workers who collaborate on the data are spread across many geographic locations, raising the demand for a single global namespace.”

Read the white paper: Global Namespace vs Global File System: What is the Difference and Why Does it Matter?

Some of the components and features of hybrid cloud file data services may include:

Data Storage and Management

• On-Premises Storage: Traditional file servers or network-attached storage (NAS) devices located within an organization’s physical premises.
• Cloud Storage: File storage services provided by public cloud providers (e.g., Amazon S3, Azure Blob Storage) for scalable and elastic storage options.

Data Synchronization and Sharing

• Bidirectional Sync: Ensures that data remains consistent across on-premises and cloud environments, allowing users to seamlessly access and update files from either location.
• Collaboration Tools: Integration with collaboration platforms to enable efficient sharing and collaboration on files among users in different locations.

In Gartner’s definition of Hybrid Cloud File Platforms, these features would require a Global File System.

Scalability and Flexibility

• Elastic Scaling: The ability to scale file storage both on-premises and in the cloud based on changing storage requirements.
• Multi-Cloud Support: Compatibility with multiple cloud providers, giving organizations flexibility in choosing the most suitable cloud services for their needs.

Data Security and Compliance

• Encryption: Secure transmission and storage of files through encryption mechanisms, ensuring data confidentiality.
• Compliance Features: Tools and features to help organizations comply with data protection regulations and industry-specific standards.

Data Access and Mobility

• Global Access: Enable users to access files from any location, promoting a mobile and distributed workforce.
• Data Mobility: Facilitate seamless movement of data between on-premises and cloud environments.

Learn more about Komprise Intelligent Data Management: Unified data control plane for file and object data analytics, mobility and management without creating a bottleneck and never being in the hot data path.

Backup and Disaster Recovery

• Snapshot and Backup: Regularly capture snapshots and backups of file data to protect against data loss or corruption.
• Disaster Recovery Planning: Implement strategies to quickly recover file data in the event of a disaster or data loss incident.

Integrated Management and Monitoring

• Unified Dashboard: A centralized management interface for overseeing file data services across on-premises and cloud environments.
• Monitoring Tools: Tools for tracking performance, usage, and potential issues in the hybrid cloud file storage infrastructure.

Implementing hybrid cloud file data services requires careful planning, integration, and management to ensure a seamless and efficient experience for users while maximizing the benefits of both on-premises and cloud-based data storage solutions.

Want To Learn More?

Intelligent Data Management

Intelligent Data Management is the process of managing unstructured data throughout its lifecycle with analytics and intelligence. It is also the name of the Komprise platform as a service: Intelligent Data Management.

The criteria for a solution to be considered as Intelligent Data Management includes:

Analytics-Driven Data Management

Is the solution able to leverage analysis of the data to inform its behavior? Is it able to deliver analysis of the data to guide the data management planning and policies? Learn more about Komprise Analysis.

Storage-Agnostic Data Management

Is the data management solution able to work across different vendor and different storage platforms?

Adaptive Data Management

Based on the network, storage, usage, and other conditions, is the data management solution able to intelligently adapt its behavior? For instance, does it throttle back when the load gets higher, does it move bigger files first, does it recognize when metadata does not translate properly across environments, does it retry when the network fails?

Closed Loop Unstructured Data Management

Analytics feeds the data management which in turn provides additional analytics. A closed loop system is a self-learning system that uses machine learning techniques to learn and adapt progressively in an environment.

Efficient and Cost Effective Data Management

An intelligent data management solution should be able to scale out efficiently to handle the load, and to be resilient and fault tolerant to errors.It should also ensure you’re able to achieve data storage cost savings.

Intelligent data management solutions typically address the following use cases:

• Analysis: Find the what, who, when of how data is growing and being used
• Planning: Understand the impact of different policies on costs, and on data footprint
• Data Tiering or Data Archiving: Support various forms of managing cold data and offloading it from primary storage and backups without impacting user access. Includes: Tier and archive data by policy – move data with links for seamless access, Archive project data – archive data that belongs to a project as a collection, Archive without links – move data without leaving a link behind when data needs to be moved out of an environment
• Data Replication: Create a copy of data on another location.
• Data Migration: Move data from one storage environment to another
• Deep Analytics: Search and query data at scale across storage

Want To Learn More?

Komprise Deep Analytics

Komprise Deep Analytics delivers granular, flexible search and indexes data in-place across file, object and cloud data storage to build a comprehensive Global File Index (GFI) spanning petabytes of unstructured data.

Komprise Deep Analytics Actions: Add Deep Analytics queries to a plan and operationalize your ability to search and find what you need and when you need it.

Smart Data Workflows: Leverage the GFI metadata catalog for systematic, policy-driven data management actions that can feed your data pipelines.

Want To Learn More?

Observer (Komprise Observer)

The Komprise Observer is a virtual appliance running at the customer site that analyzes data across NAS silos, moves and replicates data by data management policy, and provides transparent file access to data that’s stored in the cloud.

Want To Learn More?

OneFS FilePolicy

OneFS FilePolicy is a feature of Dell EMC’s PowerScale Isilon OneFS operating system. It enables organizations to automate and enforce policies for managing files within the Isilon cluster based on specified criteria. With OneFS FilePolicy, administrators can define rules and conditions that determine how files are organized, protected, and managed within the file system. These policies can be based on file attributes such as file type, file size, creation date, access patterns, or any other metadata associated with the files.

OneFS FilePolicy Features

• Automated File Management: FilePolicy allows administrators to automate file management tasks, such as moving, copying, or deleting files based on predefined policies within Isilon environments. For example, files older than a certain date can be automatically moved to a lower-tier storage tier or archived to a separate storage system.
• Storage Tiering: OneFS FilePolicy enables storage tiering by automatically moving files between different storage tiers based on policies. This helps optimize storage utilization and performance by placing frequently accessed or high-priority files on faster storage tiers, while less frequently accessed or lower-priority files can be moved to lower-cost, slower storage tiers.
• Data Protection and Replication: FilePolicy can be used to define policies for data protection and replication. For instance, it can automatically replicate critical files to remote locations or create snapshots at regular intervals to ensure data durability and availability.
• Data Retention and Compliance: FilePolicy enables organizations to enforce data retention and compliance requirements by automatically applying policies for file retention, archival, and deletion. This helps ensure that files are retained for the required period and are disposed of properly when no longer needed.
• Customizable Policies: OneFS FilePolicy offers flexibility in defining policies based on specific business requirements. Administrators can set up multiple policies with different criteria and actions to accommodate varying file management needs.

By leveraging OneFS FilePolicy, organizations can automate and streamline file data management tasks within their Isilon cluster. It helps optimize storage utilization, improve data protection and compliance, and reduce manual intervention for routine file management operations.

Komprise Intelligent Data Management integrates with FilePolicy across hybrid, multi-cloud and multi-storage environments and delivers analytics-driven unstructured data management. While OneFS FilePolicy enables Dell EMC customers to manage data inside an Isilon cluster, moving data between disks for Isilon tiering, with Komprise you are able to analyze across storage vendors formats, storage types and clouds

Know more. Move smart. Save more.

Komprise for Dell/EMC.

Isilon Migration

Want To Learn More?

Policy-Based Data Management

Policy-based data management is data management based on metrics such as data growth rates, data locations and file types, which data users regularly access and which they do not, which data has protection or not, and more.

The trend to place strict policies on the preservation and dissemination of data has been escalating in recent years. This allows rules to be defined for each property required for preservation and dissemination that ensure compliance over time. For instance, to ensure accurate, reliable, and authentic data, a policy-based data management system should generate a list of rules to be enforced, define the data storage locations, storage procedures that generate data tiering and archival information packages, and manage replication.

Policy-based data management is becoming critical as the amount of unstructured data continues to grow while IT budgets remain flat. By automating movement of data to cheaper storage such as cloud data storage or private object storage, IT organizations can rein in data sprawl and cut costs.

Other things to consider are how to secure data from loss and degradation by assigning an owner to each file, defining access controls, verifying the number of replicas to ensure integrity of the data, as well as tracking the chain of custody. In addition, rules help to ensure compliance with legal obligations, ethical responsibilities, generating reports, tracking staff expertise, and tracking management approval and enforcement of the rules.

As data footprint grows, managing billions and billions of files manually becomes untenable. Using analytics-driven data management to define governing policies for when data should move, to where and having data management solutions that automate based on these policies becomes critical. Policy-based data management systems rely on consensus. Validation of these policies is typically done through automatic execution – these should be periodically evaluated to ensure continued integrity of your data.

Want To Learn More?

Power Usage Effectiveness (PUE)

Power Usage Effectiveness (PUE) is the metric used to measure the energy efficiency of a data center or computing facility. It is calculated by dividing the total amount of energy consumed by the data center (including IT equipment and supporting infrastructure) by the energy consumed by the IT equipment alone. (See Data Center Consolidation.)

The formula for calculating PUE

PUE = Total Facility Energy Consumption / IT Equipment Energy Consumption

“Total Facility Energy Consumption” refers to the combined energy consumed by the entire data center, including cooling systems, lighting, power distribution, backup generators, and other supporting infrastructure.

“IT Equipment Energy Consumption” represents the energy used specifically by the IT servers, storage devices, networking equipment, and other computing hardware.

The purpose of the Power Usage Effectiveness metric

The purpose of PUE is to provide insight into the efficiency of a data center’s power usage. A lower PUE value indicates higher energy efficiency because it means a larger proportion of the total energy consumption is used directly by the IT equipment rather than being allocated to supporting infrastructure.

A PUE of 1.0 represents a hypothetical ideal state where all the energy consumed is used exclusively by the IT equipment, with no additional energy needed for cooling or other infrastructure. In practice, achieving a PUE of exactly 1.0 is extremely challenging, and most data centers typically have PUE values above 1.0.

Data center strategies to reduce PUE and improve efficiency

• Efficient Cooling Systems: Implementing energy-efficient cooling technologies, such as hot and cold aisle containment, precision cooling, or free cooling, to optimize cooling efficiency and reduce energy consumption.
• Virtualization and Consolidation: Using virtualization technologies to consolidate servers and optimize resource utilization, thereby reducing the overall power requirements of the IT equipment.
• Energy Management and Monitoring: Implementing energy management systems and monitoring tools to track and optimize energy usage, identify areas of inefficiency, and make data-driven decisions for improvement.
• Efficient Power Distribution: Employing efficient power distribution systems, such as uninterruptible power supplies (UPS) with high-efficiency ratings, to minimize power losses and increase energy efficiency.
• Renewable Energy Sources: Incorporating renewable energy sources, such as solar or wind power, into the data center’s energy mix to reduce reliance on fossil fuels and lower the environmental impact.

PUE is just one metric to evaluate data center energy efficiency. Additional factors like water usage efficiency (WUE) and carbon usage effectiveness (CUE) may also be considered for a comprehensive assessment of environmental impact and resource efficiency.

Data Management and Sustainability

In early 2022 supply chain challenges and sustainability were grabbing headlines: See this post for coverage. The focus on improving energy efficiency and reducing PUE has become increasingly important as pressures mount to consolidate data centers, accelerate cloud migration and reduce data storage costs, but to reduce the overall carbon footprint and contribute to sustainable IT operations.

Komprise cofounder and COO Krishna Subramanian published this article: Sustainable data management and the future of green business. Here is how she summarized the importance of unstructured data management to sustainability in the enterprise:

A lesser-known concept relates to managing data itself more efficiently. Most organizations have hundreds of terabytes of data, if not petabytes, which can be managed more efficiently and even deleted but are hidden and/or not understood well enough to manage appropriately. In most businesses, 70% of the cost of data is not in storage but in data protection and management. Creating multiple backup and DR copies of rarely used cold data is inefficient and costly, not to mention its environmental impact. Furthermore, storing obsolete “zombie data” on expensive on-premises hardware (or even, cloud file storage, which is the highest cost tier for cloud storage), doesn’t make sage economic sense and consumes the most energy resources.

The recommendations for achieving sustainable data management in the article are:

1. Understand your unstructured data
2. Automate data actions by policy
3. Work with data owners and key stakeholders

Read the article.

Want To Learn More?

Rehydration

What is rehydration?

Rehydration is the process to fully reconstitute files so the transferred data can be accessed and used. Block-level tiering requires rehydrating tiered archived data before it can be used, migrated or backed up. No rehydration is needed with Komprise, which uses file-based tiering.

Rehydration and the Cloud

In this post, Komprise CEO Kumar Goswami answers the question: “Will I lose storage efficiencies such as de-dupe by not using a storage tiering solution in the cloud?” He notes:

The overhead of keeping blocks in the cloud due to high egress costs, high data rehydration costs and high defragmentation costs significantly overshadows any potential de-dupe savings. When data is moved at the block level to the cloud, you are really not saving on any third-party backups and other applications because block tiering is a proprietary solution – read this white paper for more background on block-level vs file-based data tiering and cloud tiering. So if you consider all the additional backup licensing costs, cloud egress costs, cloud retrieval costs plus the fact that you are now locked-in and have to pay file system costs forever in the cloud to access your data (learn more about the benefits of cloud native unstructured data access), then the small savings you may get from dedupe are significantly overshadowed by overall costs and the loss of flexibility.

Komprise provides a custom data rehydration policy that the user can configure to meet their needs. Data need not be re-hydrated on the first access. Komprise also provides a bulk recall feature if needed. Learn more about file-based cloud tiering with Komprise.

Want To Learn More?

Scale-Out Storage

Scale-out storage is a type of storage architecture in which devices in connected arrays add to the storage architecture to expand disk storage space. This allows for the storage capacity to increase only as the need arises. Scale-out storage architectures adds flexibility to the overall data storage environment while simultaneously lowering the initial storage set up costs.

With data growing at exponential rates, enterprises will need to purchase additional storage space to keep up. This data growth comes largely from unstructured data, like photos, videos, PowerPoints, and Excel files. Another factor adding to the expansion of data is that the rate of data deletion is slowing, resulting in longer data retention policies. For example, many organizations are now implementing “delete nothing” data management policies for all kinds of data. With data storage demands skyrocketing and budgets shrinking, scale-out storage can help manage these growing costs.

Whether it’s NetApp, Pure Storage, Dell EMC, Qumulo or other enterprise scale-out storage technology, including cloud services from AWS, Azure or Google, Komprise Intelligent Data Management ensures you get maximum cost savings and value from your unstructured data.

Read the white paper: Why Data Growth is Not a Storage Problem

Want To Learn More?

Storage Assessment

A storage assessment is a process of evaluating an organization’s data storage infrastructure to gain insights into its performance, capacity, efficiency, and overall effectiveness. The goal of a storage assessment is typically to identify any bottlenecks, inefficiencies, or areas for improvement in the storage environment.

Whether delivered by a service provider or the storage vendor, traditional storage assessments have focused on:

• Storage Performance: The assessment examines the performance of the storage infrastructure, including storage arrays, network connectivity, and storage protocols. It measures factors such as IOPS (Input/Output Operations Per Second), latency, throughput, and response times to identify any performance limitations or areas for optimization.
• Capacity Planning: The assessment analyzes the current storage capacity utilization and predicts future storage requirements based on data growth trends and business needs. It helps identify potential capacity constraints and ensures adequate storage resources are available to meet future demands.
• Storage Efficiency: The assessment evaluates the efficiency of storage utilization and identifies opportunities for optimization. This may include analyzing data deduplication, compression, thin provisioning, and other techniques to reduce storage footprint and improve storage efficiency.
• Data Protection and Disaster Recovery: The assessment reviews the data protection and disaster recovery strategies in place, including backup and recovery processes, replication, snapshots, and data redundancy. It ensures that appropriate data protection measures are in place to minimize the risk of data loss and to achieve desired recovery objectives.
• Storage Management and Monitoring: The assessment examines the storage management practices, including storage provisioning, data lifecycle management, storage tiering, and data classification. It assesses the effectiveness of storage management tools and processes and identifies areas for improvement.
• Storage Security: The assessment assesses the security measures implemented within the storage infrastructure, including access controls, encryption, data privacy, and compliance with industry standards and regulations. It helps ensure the security of sensitive data stored in the infrastructure.
• Cost Optimization: The assessment examines the data storage costs and identifies opportunities for cost optimization. This may include evaluating storage utilization, identifying unused or underutilized storage resources, and recommending strategies to optimize storage spending.

Based on the findings of the storage assessment, organizations can develop a roadmap for improving their storage infrastructure, addressing performance bottlenecks, enhancing data protection, optimizing storage efficiency, and aligning storage resources with business requirements. This helps ensure a robust and well-managed data storage environment that supports the organization’s data storage and unstructured management needs effectively.

Analyzing Data Silos Across Vendors: Hybrid Cloud Storage Assessments

Komprise Intelligent Data Management is an unstructured data management solution that helps organizations gain visibility, control, and cost optimization over their file and object data across on-premises and cloud storage environments. It offers a range of features and capabilities to simplify data management processes and improve storage efficiency. Komprise is used by customers and partners to deliver a data-centric, storage agnostic assessment of unstructured data growth and potential data storage cost savings. It helps organizations optimize storage resources, reduce costs, and improve data management efficiency based on real-time analysis of data usage patterns.

Common Komprise Use Cases

In addition to storage assessments, common use cases for Komprise include:

Data Visibility and Analytics: Komprise Analysis provides comprehensive visibility into data usage, access patterns, and storage costs across heterogeneous storage systems. It offers detailed analytics and reporting, allowing organizations to understand their data landscape and make informed decisions.

Transparent File Archiving: Komprise identifies and archives infrequently accessed data to lower-cost storage tiers without disrupting user access thanks to patented Transparent Move Technology (TMT). It provides a transparent file system view, allowing users to access archived files seamlessly and retrieve them on-demand when needed. It identifies cold or inactive data and migrates it to more cost-effective storage, without disrupting user access or requiring changes to existing applications or file systems.

Cloud Data Management: Komprise extends its data management capabilities to cloud storage environments, including major cloud providers such as Amazon S3, Microsoft Azure Blob Storage, and Google Cloud Storage. It enables organizations to manage data across hybrid and multi-cloud environments with consistent policies and visibility.

Data Migration: Komprise Elastic Data Migration is a SaaS solution available with the Komprise Intelligent Data Management platform or standalone. Designed to be fast, easy and reliable with elastic scale-out parallelism and an analytics-driven
approach, it is the market leader in file and object data migrations, routinely migrating petabytes of data (SMB,
NFS, Dual) for customers in many complex scenarios. Komprise Elastic Data Migration ensures data integrity is fully
preserved by propagating access control and maintaining file-level data integrity checks such as SHA-1 and MD5
checks with audit logging. As outlined in the white paper How To Accelerate NAS and Cloud Data Migrations, Komprise Elastic Data Migration is a highly parallelized, multi-processing, multi-threaded approach that improves performance at many levels. And with Hypertransfer, Komprise Elastic Data Migration is 27x faster than other migration tools.

Data Lifecycle Management: Komprise helps organizations automate the movement and placement of data based on data management policies. It enables the seamless transition of data between storage tiers, such as high-performance storage and lower-cost archival storage, to optimize performance and reduce storage costs.

Komprise Intelligent Data Management helps organizations optimize their storage infrastructure, reduce storage costs, improve data management efficiency, and gain better control and insights into their unstructured data. It simplifies complex data management processes and empowers organizations to make informed decisions about their data storage and utilization.

Want To Learn More?

Sustainable Data Management

What is Sustainable Data Management?

Sustainable data management refers to the practice of collecting, storing, and using data in a way that is environmentally friendly, economically feasible, and socially responsible. This involves reducing the carbon footprint of data centers, leveraging renewable energy sources, and following ethical principles in the collection, storage, and use of data (regardless of source, structure or location). It also involves implementing data management strategies that ensure the long-term preservation and accessibility of valuable data, while reducing waste and avoiding data hoarding. The goal of sustainable data management is to balance the economic, environmental, and social impacts of data operations and ensure that data is managed in a way that supports the well-being of both current and future generations

In an an article for Sustainability magazine, Komprise co-founder and COO Krishna Subramanian noted:

Most organizations have hundreds of terabytes of data, if not petabytes, which can be managed more efficiently and even deleted but are hidden and/or not understood well enough to manage appropriately. In most businesses, 70% of the cost of data is not in storage but in data protection and management. Creating multiple backup and DR copies of rarely used cold data is inefficient and costly, not to mention its environmental impact. Furthermore, storing obsolete “zombie data” on expensive on-premises hardware (or even, cloud file storage, which is the highest cost tier for cloud storage), doesn’t make sage economic sense and consumes the most energy resources.

She summarized the steps to sustainable unstructured data management as:

1. Understand your unstructured data (analyze your unstructured data)
2. Automate data actions by policy (data management policy)
3. Work with data owners and key stakeholders (getting departments to care about data storage savings)

Her sustainable data management conclusion:

Sustainable data center and data management practices are no longer nice to have – but in many respects, a need to have. The world is storing too much data and without smart strategies for managing it, the price is becoming too high: significantly higher IT infrastructure costs, lack of opportunity to participate in government incentives, potential customer attrition, and long-term potential brand damage by ignoring the sustainability movement.

Read the full article here.

———-

Want To Learn More?

Tiering

What is Tiering?

In the context of data storage, tiering refers to the practice of organizing data into different tiers based on its value or frequency of access. Each tier is assigned a different level of performance, cost, and capacity, with the goal of optimizing the use of storage resources and reducing costs.

The most commonly used tiers are:

• Tier 1: This is the highest-performing and most expensive tier, typically using solid-state drives (SSDs) for fast access to critical data that is frequently accessed or requires low latency.
• Tier 2: This tier is less expensive than Tier 1 and is typically made up of hard disk drives (HDDs) or slower SSDs. It is used for data that is still frequently accessed but not as critical as Tier 1 data.
• Tier 3: This is a low-cost and high-capacity tier, typically using slower HDDs or object storage. It is used for infrequently accessed data or data that is older and less valuable.

Unstructured data is typically moved automatically between tiers based on predefined data management policies that consider factors such as data age, access frequency, and cost. This ensures that frequently accessed data is stored in the higher-performing and more expensive tiers, while infrequently accessed data is stored in the lower-cost tiers. The goal of tiering is to optimize storage utilization and reduce costs while ensuring that data is accessible when needed.

Storage Tiering, Data Archiving, and Transparent Archiving – What’s the Difference?

File Migration Isn’t File Archiving

Cloud Tiering: Storage-based vs. Gateway vs. File Based

Want To Learn More?

Treesize

What is Treesize?

Treesize is a free disk space analysis utility written by Jam Software. According to Wikipedia, the first version of TreeSize was programmed by Joachim Marder in 1996.

Treesize has become a term used to describe the amount of storage space that a storage directory or folder (and its contents) take up on a computer’s hard drive. Treesize is a measure of the overall size of all files and subfolders within a particular directory. Knowing this information his been useful for storage administrators and IT teams managing hard drive space and identifying which files or folders are taking up the most space.

Going Beyond Basic Disk Analysis

As enterprise data storage is increasingly hybrid and multi-cloud, IT operations and storage teams need visibility and analysis across all unstructured data so they always know what data they have, how much is hot, how much is cold, how fast data is growing and data storage costs across silos. This is why storage-agnostic unstructured data management solutions that provide aggregate metrics in addition to information by share or directory and the ability to move data by policy has become essential to enterprise IT and line of business teams. It’s also important to have a tool that can scale to the needs of the enterprise. That’s where Komprise comes in. Check out the Komprise Directory Explorer.

Learn more about Komprise Analysis. Learn more about Komprise Deep Analytics.

———-

Want To Learn More?

Unstructured Data Classification

Unstructured data classification involves the process of categorizing and organizing unstructured data based on its content, context, or other characteristics. Unstructured data typically refers to information that does not have a predefined data model or is not organized in a structured manner, such as text documents, images, audio files and videos. Classifying unstructured data is increasingly recognized as essential for efficient unstructured data management, search, and analysis.

Komprise Deep Analytics allows you to find the right data that fits specific criteria across all your data storage silos to answer questions, such what file types the top data owners are storing. Once you connect Komprise to your file and object storage, Komprise indexes the data and creates a Global File Index of all your data. You do not have to move the data anywhere; but you now have a single way to query and search across all file and object stores. For instance, say you have some NetApp, some Isilon, some Windows servers, some Pure Storage at different sites and you have some cloud file storage on Amazon, Azure, and Google. You get a single index via Komprise of all the data across all these environments you can search and find exactly the data you need across all these environments with a single console and API. Once you find the data you want to operate on, you can systematically move it using Komprise Intelligent Data Management. For example, if you want to tier files generated by certain instruments to the cloud, you can create a policy so that as new files are generated, they are continuously and automatically moved. This makes it easy to systematically leverage analytics to move and operate on unstructured data.

Unstructured Data Classification: A Top Enterprise Data Storage Trend

According to Gartner’s Top Trends in Enterprise Data Storage 2023 (subscription required):

By 2027, at least 40% of organizations will deploy data storage management solutions for classification, insights and optimization, up from 15% in early 2023.

The report goes on to note that:

Data classification or categorization helps improve IT and business outcomes such as storage optimization, data life cycle enforcement, security risk reduction and faster data workflows. Data classification and insights solutions are typically vendor storage agnostic, and work on any data that can be accessed over a file or object access protocols like NFS, SMB or S3.

What are some approaches and techniques for unstructured data classification?

Text-Based Classification

• Natural Language Processing (NLP): NLP techniques, including text tokenization, sentiment analysis, and named entity recognition, can be used to analyze the content of textual data.
• Keyword Matching: Classifying documents based on the presence of specific keywords or key phrases related to predefined categories.

Image-Based Classification

• Computer Vision: Utilizing computer vision techniques, such as image recognition and object detection, to classify and categorize images based on their visual content.
• Feature Extraction: Extracting features from images, such as color histograms or texture patterns, and using machine learning models for classification.

Audio and Speech-Based Classification

• Speech Recognition: Converting spoken language into text for further analysis and classification.
• Audio Analysis: Extracting features from audio files, such as pitch or frequency, and using machine learning algorithms for classification.

Metadata-Based Classification

• File Metadata: Utilizing metadata associated with files, such as creation date, author, or file type, for classification purposes.
• Exif Data: For images, extracting metadata embedded in the file, such as camera settings and location information. Exchangeable image file format (EXIF).

Pattern Recognition

• Machine Learning Algorithms: Training machine learning models, including supervised or unsupervised learning algorithms, to recognize patterns and classify unstructured data based on historical examples.
• Clustering: Grouping similar data points together using clustering algorithms to discover natural groupings within unstructured data.

Rule-Based Classification

• Predefined Rules: Establishing rules and criteria for classifying data based on certain characteristics or conditions.
• Expert Systems: Using expert systems that encode human expertise and rules for classification.

Content Analysis

• Topic Modeling: Identifying topics or themes within unstructured text data using techniques like Latent Dirichlet Allocation (LDA).
• Sentiment Analysis: Determining the sentiment expressed in textual content, such as positive, negative, or neutral sentiments.

Combination of Techniques

• Hybrid Approaches: Combining multiple techniques, such as text analysis, image recognition, and metadata examination, for a more comprehensive and accurate classification.

Deep Learning

• Neural Networks: Leveraging deep learning models, such as convolutional neural networks (CNNs) for images or recurrent neural networks (RNNs) for sequential data, to automatically learn features and patterns for classification.

Feedback Loop and Continuous Improvement

• Establishing a feedback loop where the classification system continuously learns and improves based on user feedback, corrections, and updates to the training data.

Unstructured data classification is a challenging task, but advancements in machine learning, deep learning, and natural language processing have significantly improved the accuracy and efficiency of these classification methods and modern unstructured data management software solutions have emerged to address elements of data classification and ongoing data lifecycle management.

Depending on the specific requirements and characteristics of the unstructured data, different techniques or a combination of approaches may be suitable for effective unstructured data classification.

Read the article: How to Control Unstructured Data

Komprise Use Case: Data Classification

Want To Learn More?

Unstructured Data Management

What is Unstructured Data Management?

Unstructured data management is a category of software that has emerged to address the explosive growth of unstructured data in the enterprise and the modern reality of hybrid cloud storage. In the Komprise 2023 Komprise State of Unstructured Data Management, 32% of organizations report that they are managing 10PB of data or more. That equates to 110,000 ultra-high-definition (UHD) movies, or half of the data stored by the U.S. Library of Congress. Most (73%) of organizations are spending more than 30% of their IT budget on data storage.

Data storage and data backup technology vendors are now recognizing the importance of unstructured data management as data outlives infrastructure and as data mobility is needed to leverage cloud data storage.

Unstructured data management must be independent and agnostic from data storage, backup, and cloud infrastructure technology platforms.

There are 5 requirements for unstructured data management solutions:

1. Goes Beyond Storage Efficiency
2. Must be Multi-Directional
3. Doesn’t Disrupt Users and Workflows
4. Should Create New Uses for Your Data
5. Puts Your Data First and Avoids Vendor Lock-In

An analytics-based unstructured data management solution brings value by analyzing all data in storage across on-premises and cloud environments to deliver deep insights. This knowledge helps IT managers make great decisions with users in mind, optimize costs and reduce security and regulatory compliance risks. These insights go beyond traditional storage metrics such as latency, IOPS and network throughput.

Here are some of the new metrics made possible with data management software:

• Top data owners/users: See trends in usage and and possible compliance issues, such as individual users storing excessive video files or PII files being stored in an insecure location.
• Common file types: The ability to see data by file extension eases the process of finding all files related to a project and can inform future research initiatives. This could be as simple as finding all the log files, trace files or extracts from a given application or instrument and moving them to a data lake for analysis.
• Storage costs for chargeback or showback: Whether for chargeback requirements or not, stakeholders should understand costs in their department and be able to view metrics. This will help identify areas where low-cost storage or data tiering to archival storage is a viable cost-reduction opportunity.
• Data growth rates: High level metrics on data growth keeps IT and business heads on the same page so they can collaborate on data management decisions. Understand which groups and projects are growing data the fastest and ensure that data creation/storage is appropriate according to its overall business priority.
• Age of data and access patterns. In most enterprises, 60-80% of data is  “cold” and hasn’t been accessed in a year or more. Metrics showing percentage of cold versus warm versus hot data are critical to ensure that data is living in the right place at the right time according to its business value and to optimize costs.

Read: File Data Metrics to Live By

Beyond cost optimization, unstructured data management tools and practices can help deliver new value from data.

Unstructured data is the fuel needed for AI, yet its difficult to leverage because unstructured data is hard to find, search across, and move due to its size and distribution across hybrid cloud environments. Tagging and automation can help prepare unstructured data for AI and big data analytics programs. Tactics include:

• Preprocess data at the edge so it can be analyzed and tagged with new metadata before moving it into a cloud data lake. This can drastically reduce the wasted cost and effort of moving and storing useless data and can minimize the occurrence of data swamps.
• Applying automation to facilitate data segmentation, cleansing, search and enrichment. You can do this with data tagging, deletion or tiering of cold data by policy and moving data into the optimal storage where it can be ingested by big data and ML tools. A leading new approach to is the ability to initiate and execute data workflows.
• Use a solution that persists metadata tags as data moves from one location to another. For instance, files tagged as containing key project keywords by a third-party AI service should retain those tags indefinitely so that a new research team doesn’t have to run the same analysis over again — at high cost. Komprise Intelligent Data Management has these capabilities.
• Plan appropriately for large-scale data migration efforts with thorough diligence and testing. This can prevent common networking and security issues that delay data migrations and introduce errors or data loss.

The State of Unstructured Data Management

In August 2021, Komprise published the first State of Unstructured Data Management Report:

Highlights of the 2021 Unstructured Data Management Report

Unstructured Data is Growing, as are its Costs

• 65.5% of organizations spend more than 30% of their IT budgets on data storage and data management.
• Most (62.5%) will spend more on storage in 2021 versus 2020.

Getting More Unstructured Data to the Cloud is a Key Priority

• 50% of enterprises have data stored in a mix of on-premises and cloud-based storage.
• Top priorities for cloud data management include: migrating data to the cloud (56%) cutting storage and data costs (46%) and governance and security of data in the cloud (41%).

IT Leaders Want Visibility First Before Investing in More Data Storage

• Investing in analytics tools was the highest priority (45%) over buying more cloud or on-premises storage or modernizing backups.
• One-third of enterprises acknowledge that over 50% of data is cold while 20% don’t know, suggesting a need to right-place data through its lifecycle.

Unstructured Data Management Goals & Challenges: Visibility, Cost Management and Data Lakes

• 44.9% wish to avoid rising costs.
• 44.5% want better visibility for planning.
• 42% are interested in tagging data for future use and enabling data lakes.

2022 State Unstructured Data Management Report

In August 2022, Komprise published the 2nd annual State of Unstructured Data Management Report: Komprise Survey Finds 65% of Enterprise IT Leaders are Investing in Unstructured Data Analytics. The Top 5 trends from the report are summarized here. They are:

1. User Self-Service: In data management, self-service typically refers to the ability for authorized users outside of storage disciplines to search, tag and enrich and act on data through automation—such as a research scientist wanting to continuously export project files to a cloud analytics service.
2. Moving Data to Analytics Platforms: A majority (65%) of organizations plan to or are already delivering unstructured data to their big data analytics platforms.
3. Cloud File Storage Gains Favor: Cloud NAS topped the list for storage investments in the next year (47%).
4. User Expectations Beg Attention: Organizations want to move data without disrupting users and applications (42%).
5. IT and Storage Directors want Flexibility: A top goal for unstructured data management (42%) is to adopt new storage and cloud technologies without incurring extra licensing penalties and costs, such as cloud egress fees.

State of Unstructured Data Management 2023

In September 2023, Komprise published the 3rd annual State of Unstructured Data Management report.

• Read the press release
• Download the report

The coverage focused on the fact that 66% of respondents said preparing data storage and data management for AI and GenerativeAI in general is a top priority and challenge.

• Data governance is top enterprise priority when introducing AI
• AI: Intelligent data needs intelligent solutions
• Nearly a third of enterprises are already prepping for AI
• Getting Data Governance Right Top AI Priority in 2023

Why you need to manage your unstructured data?

In a 2022 interview, Komprise co-founder and COO Krishna Subramanian defined unstructured data this way:

Unstructured data is any data that doesn’t fit neatly into a database, and isn’t really structured in rows and columns. So every photo on your phone, every X-ray, every MRI scan, every genome sequence, all the data generated by self-driving cars – all of that is unstructured data. And perhaps more relevant to more businesses, artificial intelligence (AI) and machine learning (ML) – they depend on, and usually output, unstructured data too.

Unstructured data is growing every day at a truly astonishing rate. Today, 85% of the world’s data is unstructured data.

And it’s more than doubling, every two years.

The importance of an unstructured data strategy for enterprise

In part two of the interview, Krishna Subramanian noted:

Unstructured data doesn’t have a common structure. But it does have something called metadata. So every time you take a picture on your phone, there’s certain information that the phone captures, like the time of day, the location where the picture was taken, and if you tag it as a favorite, it’ll have that metadata tag on it too. It might know who’s in the photo, there are certain metadata that are kept.

All filing systems store some metadata about the data. A product like Komprise Intelligent Data Management has a distributed way to search across all the different environments where you’ve stored data, and create a global index of all that metadata around the data. And that in itself is a difficult problem, because again, unstructured data is so huge. A petabyte of data might be a few billion files, and a lot of these customers are dealing with tens to hundreds of petabytes.

So you need a system that can create an efficient index of hundreds of billions of files that could be distributed in different places. You can’t use a database, you have to have a distributed index, and that’s the technology we use under the hood, but we optimize it for this use case. So you create a global index. Learn more about unstructured data tagging.

The Future of Unstructured Data Management

In an end of the year blog post, Komprise executives review unstructured data management and data storage predictions for 2023 and the implications of adopting data services, processing data at the edge, multi-cloud challenges, the importance of getting smart data migration strategies, and more.

Want To Learn More?

Unstructured Data Storage

Unstructured data storage is the storage of data that does not adhere to a predefined data model or schema. Unlike structured data, which fits neatly into tables with rows and columns, unstructured data lacks a specific organization and may include various file types, such as text documents, images, videos, audio files, emails, social media posts, and more.

Read the article: Here’s How to Take Control of Unstructured Data

Gartner on unstructured data storage

Each year Gartner publishes the Magic Quadrant for Distributed File Systems and Object Storage.

Gartner defines distributed file systems and object storage as software and hardware appliance products that offer object and distributed file system technologies for unstructured data. Their purpose is to store, secure, protect and scale unstructured data with access over the network using file and object protocols, such as Amazon Simple Storage Service (S3), Network File System (NFS) and Server Message Block (SMB).

Gartner also has a Primary Data Storage Magic Quadrant, as summarized in this Blocks & Files article.

Common requirements for unstructured data storage

• Flexibility: Unstructured data storage systems are flexible and can accommodate various types of data without requiring predefined schemas. This flexibility allows organizations to store and manage diverse data types efficiently.
• Scalability: Unstructured data storage solutions are often designed to scale easily, allowing organizations to handle massive volumes of data as their storage requirements grow over time.
• Indexing and Search: Effective management of unstructured data involves indexing and search capabilities to quickly locate and retrieve specific information within large datasets. This may involve metadata tagging, full-text search, and other techniques to facilitate data discovery. See unstructured data classification.
• Object Storage: Object storage is a common approach to storing unstructured data, where each piece of data is stored as an object with a unique identifier and metadata. Object storage systems provide scalability, durability, and accessibility for large-scale unstructured data environments.
• Cloud Storage: Many organizations leverage cloud storage services for unstructured data storage due to their scalability, reliability, and cost-effectiveness. Cloud providers offer a range of storage options, including object storage, file storage, and content delivery networks (CDNs), to accommodate different types of unstructured data.
• Data Governance and Security: Managing unstructured data requires robust data governance practices to ensure compliance, data security, and privacy protection. This may involve implementing access controls, encryption, data classification, and audit trails to safeguard sensitive information.

Effective storage and unstructured data management are essential for organizations to derive insights, make data-driven decisions, and unlock the value of their data assets.

Unstructured Data Storage Vendors

Many vendors offer solutions for storing unstructured data, each with its own set of features, capabilities, and pricing models. Here are some notable vendors in the unstructured data storage space:

• Amazon Web Services (AWS): Amazon Simple Storage Service (S3) (AWS S3) is a highly scalable object storage service designed for storing and retrieving any amount of data. It is commonly used for unstructured data storage and offers features such as versioning, lifecycle management, and security features. Learn more about Komprise for AWS.
• Microsoft Azure: Azure Blob Storage provides scalable, cost-effective storage for unstructured data. It offers tiered storage options, access controls, and integration with other Azure services for data analytics and processing. Learn more about Komprise for Azure.
• Google Cloud Platform (GCP): Google Cloud Storage is a scalable object storage solution suitable for storing unstructured data. It provides features such as versioning, lifecycle management, and integration with other GCP services. Learn more about Komprise for Google. 
• IBM: IBM Cloud Object Storage: IBM offers Cloud Object Storage, a scalable, secure, and durable object storage service. It is designed to support large-scale unstructured data storage and offers features such as encryption, access controls, and global data distribution. Learn more about Komprise for IBM.
• Dell: Dell EMC Isilon, now Dell PowerScale, is a scale-out network-attached storage (NAS) platform designed for storing and managing large volumes of unstructured data. It offers high performance, scalability, and multi-protocol support for various data types. Learn about Komprise Elastic Data Migration for Isilon.
• NetApp: NetApp StorageGRID is an object storage solution from NetApp that enables organizations to store, manage, and protect unstructured data at scale. It offers features such as geo-distribution, data tiering, and policy-based management. Learn more about Komprise for NetApp.
• Pure Storage: Pure Storage FlashBlade is a scalable, all-flash storage platform designed for unstructured data workloads. It offers high performance, simplicity, and native support for file, object, and analytics workloads.

HPE (Hewlett Packard Enterprise): For years it has been HPE Nimble Storage, which offers a range of storage solutions, including Nimble Storage dHCI and Nimble Storage All Flash Arrays, suitable for storing unstructured data. HPE now resells VAST Data solutions as HPE File Services.

Qumulo: Qumulo’s Scale Anywhere™ platform is a 100% software solution for hybrid enterprises to efficiently store and manage file & object data at the edge, in the core, and in the cloud

These are some examples of vendors providing solutions for unstructured data storage.

Optimize unstructured data storage with Komprise

Komprise Intelligent Data Management frees you to analyze, mobilize, and access the right file and object data across clouds without shackling your data to any unstructured data storage vendor. Komprise helps enterprise customers optimize data storage costs by right-sizing and right-placing
data, while making it easy for users to unlock data value with smart data workflows.

Want To Learn More?