Data Management Glossary
Move Smart: Ensure the right data is in the right place at the right time. Establish analytics-driven p
Take Control: Get back to the business at hand while reducing your storage, backup, and cloud costs and get the fastest, easiest 
Why Archival Storage?
Komprise Cloud Data Management
Cloud data analytics refers to the use of cloud computing resources to process, analyze, and extract insights from large amounts of data. These solutions can include data warehousing, big data processing, machine learning, and business intelligence and can ingest a wide range of data, including structured, semi-structured, and 
-
- Data Analytics
Data analytics refers to the process used to enhance productivity and business improvement by extracting and categorizing data to identify and analyze behavioral patterns. Techniques vary according to organizational requirements.
The primary goal of data analytics is to help organizations make more informed business decisions by enabling analytics professionals to evaluate large volumes of transactional and other forms of data. Data analytics can be pulled from anything from Web server logs to social media comments.
Potential issues with data analytics initiatives include a lack of analytics professionals and the cost of hiring qualified candidates. The amount of information that can be involved and the variety of data analytics data can also cause data analytics issues, including the quality and consistency of the data. In addition, integrating technologies and data warehouses can be a challenge, although various vendors offer data integration tools with big data capabilities.
Big data has drastically changed the requirements for extracting data analytics from business data. With relational databases, administrators can easily generate reports for business use, but they lack the broader intelligence data warehouses can provide. However, the challenge for data analytics from data warehouses is the costs associated.
Unstructured Data Analytics
There is also the challenge of pulling the relevant data sets to enable data analytics from cold data. This requires intelligent data management solutions that track what unstructured data is kept and where, and enable you to easily search and find relevant data sets for big-data analytics.
Deliver the right data to the right place at right time with Komprise and bring unstructured data to you your analytics projects. - Data Archiving
What is Data Archiving?
Data Archiving, often referred to as Data Tiering, protects older data that is not needed for everyday operations of an organization. A data archiving strategy reduces primary storage and allows an organization to maintain data that may be required for regulatory or other needs.
Benefits of a Data Archiving SolutionData archiving is intended to protect older information that is not needed for everyday operations but may have to be accessed occasionally. Data archiving tools deliver the most value by reducing primary storage and the related costs, rather than acting as a data recovery tool. Unstructured data archive tools are in high demand because of the opportunity to drastically reduce overall storage costs since most data is now unstructured—and often residing on expensive, high-performance storage devices. Archive data storage, meanwhile, is typically on a low-performance, lost-cost, high-capacity data storage medium.
Types of Data Archiving
Some data archiving products enable read-only access to protect data from modification, while other data tiering and archiving products allow users to make changes.
Data archiving take a few different forms. Options include online data storage, which places archive data onto disk systems where it is readily accessible. Archives are frequently file-based, but object storage is also growing in popularity. A key challenge when using object storage to archive file-based data is the impact it can have on users and applications. To avoid changing paradigms from file to object and breaking user and application access, use data management solutions that provide a file interface to data that is archived as objects.
Another archival system uses offline data storage where archive data is written to tape or other removable media using data archiving software rather than being kept online. Data archiving on tape consumes less power than disk systems, translating to lower costs.
A third option is using cloud data storage, such as those offered by Amazon and Azure. Cloud object storage is a smart choice for cloud tiering and data archiving because of its low-cost, immutable nature.This is inexpensive but requires ongoing investment.
With the ongoing threats from ransomware and other sophisticated cybersecurity actors, creating a secure data archiving and data tiering strategy is imperative. Encryption of sensitive archives, use of multi-factor authentication for access and object lock storage (such as AWS S3) are a few ways to protect archival data from modification, corruption and theft.The data archiving process typically uses automated software, which will automatically move cold data via policies set by an administrator. Today, a popular approach to data archiving is to make the archive “transparent.” By this, archived data is not only online but is fully accessed exactly as before by users and applications, so they experience no change in behavior. (See Native Access)
Learn more about Komprise Transparent Move Technology (TMT).
- Data Backup
Why Data Backup?
Data loss can occur from a variety of causes, including computer viruses, hardware failure, file corruption, fire, flood, or theft, etc. Data loss may involve critical financial, customer, and company data, so a solid data backup plan is critical for every organization.
Data backup plan considerations:
- What data (files and folders) to backup
- How often to run your backups
- Where to store the backup data
- What compression method to use
- What type of backups to run
- What kind of media on which to store the backups
In general, you should back up any data that can’t be replaced easily. Some examples are structured data like databases, and unstructured data such as word processing documents, spreadsheets, photos, videos, emails, etc. Typically, programs or system folders are not part of a data backup program. Installation discs, operating system discs, and registration information should be stored in a safe place.
Data backup frequency depends on how often your organizational data changes.
- Frequently changing data may need daily or hourly backups
- Data that changes every few days might require a weekly or even monthly backup
- For some data, a backup may need to be created each time it changes
The challenge with unstructured data is that backing up unstructured data is not only time consuming but also very complex, with millions to billions of files of various sizes and types and growing at an astronomical rate, leaving enterprises to struggle with long backup windows, overlapping backup cycles, backup footprint sprawl, spiraling costs, and above all, vulnerable in the case of a disaster.
Read the white paper: Rein in Storage and Backup Costs.
Read the post: 5 Ways to Get to the Cloud Smarter and Faster
Backing Up Unstructured Data First (Before Analysis) is Backwards
Don’t backup data first. Know your data first to make smarter, cost-saving decisions. Start with the Komprise TCO calculator.
Learn more about Komprise Analysis.
- Data Center Consolidation
Data center consolidation is the process of merging or reducing the number of data centers that an organization operates. The consolidation is typically done in order to reduce costs, increase efficiency, and simplify management of the data center infrastructure.
There are several steps involved in data center consolidation, including:
- Assessing the current state of the data center environment, including the number and locations of data centers, the types of systems and applications being used, and the costs associated with operating and maintaining the infrastructure.
- Developing a consolidation plan that outlines the goals, timelines, and resources needed for the project. This plan should include an analysis of the potential benefits and risks of consolidation, as well as a detailed roadmap for migrating applications and data to the new infrastructure.
- Migrating applications and data migration to the consolidated data center(s). This may involve re-architecting applications to run in a virtualized environment or on cloud infrastructure.
- Decommissioning or repurposing the legacy data center(s), including disposing of any equipment that is no longer needed.
- Continuously monitoring and optimizing the consolidated data center infrastructure to ensure it remains efficient and cost-effective.
Overall, data center consolidation can be a complex process that requires careful planning and execution. However, the benefits of consolidation can be significant, including lower costs, improved performance, and increased agility and flexibility for the organization.
In 2023, Komprise summarized the following customers trends in unstructured data management and storage.
- Simplifying infrastructure, getting rid of legacy apps and software and data center consolidation to support business growth and IT modernization.
- Reducing IT spending by pivoting to more of an OPEX environment and by deleting data that is no longer needed to reduce data storage costs and complexity.
- Managing research workflows and the full lifecycle of data: Examples include, from a major university: enabling users to share data between labs and send some data to the cloud for processing, then bring it back on-premises.
- Using industry standards to move data easily between platforms.
- Externalize (tier) data off NAS: IT and storage managers want to tier cold data from across the business to cheaper, secondary storage to save money and free up primary storage capacity.
Learn more about Komprise Elastic Data Migration and read 5 Industry Case Studies.
- Data Classification
Data classification is the process of organizing data into tiers of information for data organizational purposes.
Data classification is essential to make data easy to find and retrieve so that your organization can optimize risk management, compliance, and legal requirements. Written guidelines are essential in order to define the categories and criteria to classify your organization’s data. It is also important to define the roles and responsibilities of employees in the data organization structure.
When data classification procedures are established, security standards should also be established to address data lifecycle requirements. Classification should be simple so employees can easily comply with the standard.
Examples of types of data classifications:
- 1st Classification: Data that is free to share with the public
- 2nd Classification: Internal data not intended for the public
- 3rd Classification: Sensitive internal data that would negatively impact the organization if disclosed
- 4th Classification: Highly sensitive data that could put an organization at risk
Data classification is a complex process, but automated systems can help streamline this process. The enterprise must create the criteria for classification, outline the roles and responsibilities of employees to maintain the protocols, and implement proper security standards. Properly executed, data classification will provide a framework for the data storage, transmission and retrieval of data.
Automation simplifies data classification by enabling you to dynamically set different filters and classification criteria when viewing data across your storage. For instance, if you wanted to classify all data belonging to users who are no longer at the company as “zombie data,” the Komprise Intelligent Data Management solution will aggregate files that fit into the zombie data criterion to help you quickly classify your data.
Data Classification and Komprise Deep Analytics
Komprise Deep Analytics gives data storage administrators and line of business users granular, flexible search capabilities and indexes data creating a Global File Index across file, object and cloud data storage spanning petabytes of unstructured data. Komprise Deep Analytics Actions uses these virtual datasets (see virtual data lake) for systematic, policy-driven data management actions that can feed your data pipelines.
- Data Governance
What is data governance?
Data governance refers to the management of the availability, security, usability, and integrity of data used in an enterprise. Data governance in an organization typically includes a governing council, a defined set of procedures, and a plan to execute those procedures.
Data governance is not about allowing access to a few privileged users; instead, it should allow broad groups of users access with appropriate controls. Business and IT users have different needs; business users need secure access to shared data and IT needs to set policies around security and business practices. When done right, data governance allows any user access to data anytime, so the organization can run more efficiently, and users can manage their workload in a self-service manner.
3 things to consider when developing a data governance strategy:
Selecting a Data Governance Team
- Balance IT and business leaders to get a broad view of the data and service needs
- Start small – choose a small group to review existing data analytics
Data Quality Strategy
- Audit existing data to discover data types and how they are used
- Define a process for new data sources to ensure quality and availability standards are met
Data Security
- Make sure data is classified so data requiring protection for legal or regulatory reasons meets those requirements
- Implement policies that allow for different levels of access based on user privileges
Komprise is not a data governance solution but we are part of an overall governance strategy as it relates to unstructured data management. With the Deep Analytics user profile, you can provide secure data access to specific users to search and tag file and object data so that it can then be incorporated into smart data migration and data mobility use cases, including Smart Data Workflows.
- Data Hoarding
What is Data Hoarding?
Data hoarding is now being recognized as a growing challenge in the technology world. Many IT teams are caught in an endless cycle of buying more data storage. Unstructured data is growing at record rates and this data is increasingly being stored across hybrid cloud infrastructure. This massive data growth and increased data mobility has only created more disconnected data silos. Just like hoarding has been recognized as a real problem in the real-world (see reality TV shows like Hoarders and Storage Wars), data hoarding refers to the practice of retaining large amounts of data that is no longer needed or is rarely used, for extended periods of time. This is a common problem in many organizations, where employees tend to save data out of habit, fear of losing it, or simply because they don’t know what to do with it.
What is the impact of data hoarding?
The impact of data hoarding is more significant than most people / organizations realize, including:
- Increased costs: Storing large amounts of unnecessary data can be expensive, especially if the organization is using expensive storage solutions, such as high-end disk arrays or tape libraries.
- Reduced efficiency: Hoarded data can slow down systems and applications, as well as increase the time required to complete backups and other data management tasks.
- Compliance risks: Hoarded data can pose a risk to organizations in terms of compliance, as they may contain sensitive information that is subject to data privacy regulations.
- Cybersecurity risks: Hoarded data can also pose a security risk, as it may contain sensitive information that could be targeted by cybercriminals or hackers.
Stop Treating All Data the Same
Sound familiar?
- Cold data sits on expensive storage.
- Everything gets replicated.
- Everything gets backed up and backup windows are getting longer.
- Costs are spiraling out of control.
The IDC report, How to Manage Your Data Growth Smarter with Data Literacy noted:
- 60% of the storage budget is not really spent on storage. It’s spent on secondary copies of data for data protection – backups, backup software licenses, replication, and disaster recovery.
- 1/3 of IT organizations are spending most of their IT storage on secondary data.
And with ransomware attacks on the rise, which increasingly target unstructured data, it’s increasingly important to find ways to manage, tier, migrate, replicate file data within tight IT budgets. Read the blog post: How to Protect File Data from Ransomware at 80% Lower Cost.
Dealing with Data Hoarding
To address the data hoarding challenge and establish an Intelligent Data Management strategy, IDC recommends the following:
- Focus less on finding alternatives to store data better/faster and focus more on finding intelligent alternatives to unstructured data management.
- Use modern, next-generation cloud data management technologies that are lightweight and non-intrusive, and that demonstrate powerful return on investment.
- Aim to deliver continuous insights as a service to business and achieve speed of intelligence for a competitive edge.
Establish a Cold Data Storage Strategy
One obvious strategy to deal with data hoarding is to define a cold data storage strategy and establish unstructured data management policies.
Read this post to learn how to quantify the business value impact of Komprise Intelligent Data Management.
- Data Lake
A data lake is data stored in its natural state. The term typically refers to unstructured data that is sitting on different storage environments and clouds. The data lake supports data of all types – for example, you may have videos, blogs, log files, seismic files and genomics data in a single data lake. You can think of each of your Network Attached Storage (NAS) devices as a data lake.
One big challenge with data lakes is to comb through them and find the relevant data you need. With unstructured data, you may have billions of files strewn across different data lakes, and finding data that fits specific criteria can be like finding a needle in a haystack
A virtual data lake is a collection of data that fits certain criteria – and as the name implies, it is virtual because the data is not moved. The data continues to reside in its original location, but the virtual data lake gives a discrete handle to manipulate that entire data set. The Komprise Global File Index can be considered to be a virtual data lake for file and object metadata.
Some key aspects of data lakes – both physical and virtual:
- Data Lakes Support a Variety of Data Formats: Data lakes are not restricted to data of any particular type.
- Data Lakes Retain All Data: Even if you do a search and find some data that does not fit your criteria, the data is not deleted from the data lake. A virtual data lake provides a discrete handle to the subset of data across different storage silos that fits specific criteria, but nothing is moved or deleted.
- Virtual Data Lakes Do Not Physically Move Data: Virtual data lakes do not physically move the data, but provide a virtual aggregation of all data that fits certain criteria. Deep Analytics can be used to specify criteria.
- Data Lakehouse
Data Lakehouse is a term first coined by the co-founder and then CTO of Pentaho, James Dixon. And while both Amazon and Snowflake had already started using the term “lakehouse,” it wasn’t until Databricks really endorsed it in a January 30, 2020 blog post entitled “What is a Data Lakehouse?” that it received more mainstream attention (amongst data practitioners at least).
You’ve heard of a Data Lake. You’ve heard of a Data Warehouse. Enter the Data Lakehouse.
A data lakehouse is a modern data architecture that combines the benefits of data lakes and data warehouses. A data lake is a centralized repository that stores vast amounts of raw, unstructured, and semi-structured data, making it ideal for big data analytics and machine learning. A data warehouse, on the other hand, is designed to store structured data that has been organized for querying and analysis.
A data lakehouse builds on key elements of these two approaches by providing a centralized platform for storing and processing large volumes of structured and unstructured data, while supporting real-time data analytics. It allows organizations to store all of their data in one place and perform interactive and ad-hoc analysis at scale, making it easier to derive insights from complex data sets. A data lakehouse typically uses modern (and often open source) technologies such as Apache Spark, Apache Arrow, to provide high-performance, scalable data processing.
Who are the data lakehouse vendors?
There are several vendors that offer data lakehouse solutions, including:
- Amazon Web Services (AWS) with Amazon Lake Formation
- Microsoft with Azure Synapse Analytics
- Google with Google BigQuery Omni
- Snowflake
- Databricks
- Cloudera with Cloudera Data Platform
- Oracle with Oracle Autonomous Data Warehouse Cloud
- IBM with IBM Cloud Pak for Data
These vendors provide a range of services, from cloud-based data lakehouse solutions to on-premises solutions that can be deployed in an organization’s own data center. The choice of vendor will depend on the specific needs and requirements of the organization, such as: the size of the data sets, the required performance and scalability, the level of security and compliance needed and the overall budget.
Komprise Smart Data Workflows is an automated process for all the steps required to find the right unstructured data across your data storage assets, tag and enrich the data, and send it to external tools such as a data lakehouse for analysis. Komprise makes it easier and more streamlined to find and prepare the right file and object data for analytics, AI, ML projects.
- Data Literacy
The ability to derive meaningful information from data. Komprise Data Analytics provides data literacy by showing how much data, what kind, who’s using it, how often—across all storage silos.
Read the IDC InfoBrief: How to Manage Your Data Growth Smarter with Data Literacy.
- Data Management
Data management is officially defined by DAMA International, the professional organization data management professionals, is:
“Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise.”
Data management is the process of developing policies and procedures in order to effectively manage the information lifecycle needs of an enterprise. This includes identifying how data is acquired, validated, stored, protected, and processed. Data management policies should cover the entire lifecycle of the data, from creation to deletion.
Due to the sheer volume of unstructured data, an unstructured data management plan is necessary for every organization. The numbers are staggering – for example, more data has been created in the past two years than in the entire previous history of the human race. Cloud data management is also a growing area of investment in the enterprise.
Unstructured Data Management Report - Data Management Policy
What is a Data Management Policy?
A data management policy addresses the operating policy that focuses on the management and governance of data assets, and is a cornerstone of governing enterprise data assets. This policy should be managed by a team within the organization that identifies how the policy is accessed and used, who enforces the data management policy, and how it is communicated to employees.
It is recommended that an effective data management policy team include top executives to lead in order for governance and accountability to be enforced. In many organizations, the Chief Information Officer (CIO) and other senior management can demonstrate their understanding of the importance of data management by either authoring or supporting directives that will be used to govern and enforce data standards.
Considerations to consider in a data management policy
- Enterprise data is not owned by any individual or business unit, but is owned by the enterprise
- Enterprise data must be safe
- Enterprise data must be accessible to individuals within the organization
- Metadata should be developed and utilized for all structured and unstructured data
- Data owners should be accountable for enterprise data
- Users should not have to worry about where data lives
- Data should be accessible to users no matter where it resides
Ultimately, a data management policy should guide your organization’s philosophy toward managing data as a valued enterprise asset. Watch the video: Intelligent Data Management: Policy-Based Automation
Developing an unstructured data management policy
It is important to develop enterprise-wide data management policies using a flexible governance framework that can adapt to unique business scenarios and requirements. Identify the right technologies following a proof of concept approach that supports specific risk management and compliance use cases. Tool proliferation is always a problem so look to consolidate and set standards that address end-to-end scenarios. Unstructured data management policies must address data storage, data migration, data tiering, data replication, data archiving and data lifecycle management of unstructured data (block, file, and object data stores) in addition to the semi-structured and structured data lakes, data warehouses and other so-called big-data repositories.
Read the VentureBeat article: How to create data management policies for unstructured data.
What is a Data Management Policy?
A data management policy addresses the operating policy that focuses on the management and governance of data assets. The data management policy should contain all the guidelines and information necessary for governing enterprise data assets and should address the management of structured, semi-structured and unstructured data.
What does a Data Management Policy contain?
A comprehensive Data Management Policy should contain the following:
- An inventory of the organization’s data assets
- A strategy of effective management of the organization’s data assets
- An appropriate level of security and protection for the data including details of which roles can access with data elements
- Categorization of the different sensitivity and confidentiality levels of the data
- The objectives for measuring expectations and success
- Details of the laws and regulations that must be adhered to regarding the data program
Data Management policy and procedures
Firstly the business much select who should be part of the policy-making process. This should include legal, compliance and risk executives, security and IT leaders, business unit heads and the chief data officer or relevant alternative. Once the committee is selected, they should identify the risks associated with the organizations data and create a data management policy. - Data Migration
Data migration means many different things. At it’s core, it is the process of selecting and moving data from one location to another. For this Glossary, we’re focused on Unstructured Data Migration, specifically file and object data, which may involve moving data across different data storage vendors, and across different formats and protocols (SMB, NFS, S3, etc.)
Data migrations are often done in the context of retiring a system and moving to a new system, or in the context of a cloud migration, or in the context of a modernization or upgrade strategy.
When it comes to unstructured data migrations and migrating enterprise file data workloads to the cloud, data migrations can be laborious, error prone, manual, and time consuming. Migrating data may involve finding and moving billions of files (large and small), which can succumb to storage and network slowdowns or outages. Also, different file systems do not often preserve metadata in exactly the same way, so migrating data without loss of fidelity and integrity can be a challenge.
Two Data Migration Approaches
Lift-and-Shift
Many organizations start here, thinking they’ll just migrate entire file shares and directories to the cloud. If this is your plan, it’s important to use analytics to plan and migrate to reduce errors, ensure alignment and multi-storage visibility while minimizing cutover. With Komprise Elastic Data Migration, you can readily migrate from one primary vendor to another without rehydrating all the archived data, so migrations are cheaper and faster.
Cloud Data Tiering as a First Step: Smart Data MigrationSince a large percentage of file data is cold and has not been used in a year or more, tiering and archiving cold data is a smart first step – especially if you use Transparent Move Technology so users can access the files exactly as before. You can follow this up by migrating the remaining hot data to a performance cloud tier.
Data Migration Questions
Here are some questions that will help you determine the best file and object data migration strategy:
- What data storage do we have and where? (primary storage, secondary storage)
- What data sets are accessed most frequently (hot) and less frequently (cold)?
- What types of files do we have and which are taking up the most storage (image files, video, audio files, sensor data, etc.)?
- What is the cost of storing these different file types today? How does this align with the budget and projected growth?
- Which types of files should be stored at a higher security level? (PII or IP data? Mission-critical projects?)
- Are we complying with regulations and internal policies with our unstructured data management practices?
- What constraints do my network and environment pose and how do I avoid surprises during migrations?
- Do we have the best possible strategy in place for WAN acceleration, such as Komprise Hypertransfer for Elastic Data Migration.
- Data Protection
Data protection is used to describe both data backup and disaster recovery. A quality data protection strategy should automate the movement of critical data to online and offline storage and include a comprehensive strategy for valuing, classifying, and protecting data as to protect these assets from user errors, malware and viruses, machine failure, or facility outages/disruptions.
Data protection storage technologies include tape backup, which copies data to a physical tape cartridge, or cloud backup, which copies data to the cloud, and mirroring, which replicates a website or files to a secondary location. These processes can be automated and policies assigned to the data, allowing for accurate, faster data recovery.
Data protection should always be applied to all forms of data within an organization, in order to protect the integrity of the data, protect from corruption or errors, and ensuring privacy of the data. When classifying data, policies should be established to identify different levels of security, from least secure (data that anyone can see) to most secure (data that if released, would put the organization at risk).
- Data Retrieval
Data retrieval refers to the process of accessing and retrieving data from a database or data storage system. Data retrieval is possible using various techniques and tools, such as database querying, data mining, and data warehousing. The specific techniques and tools used will depend on the type of data being retrieved, along with the requirements and goals of the organization.
Some benefits of effective data retrieval include:
- Improved data access: By providing quick and easy access to data, organizations can improve their overall data management processes and make better use of their existing data.
- Better decision making: By providing access to up-to-date and accurate information, data retrieval can help organizations to make better decisions and improve their overall performance.
- Better customer insights: By retrieving and analyzing customer data, organizations can gain valuable insights into customer behavior and preferences, so they can improve customer relationships and drive business growth.
Cloud Data Retrieval
There are several challenges associated with retrieving data from the cloud, including:
- Network Latency: Retrieving data from a remote server can result in significant latency, especially if the data is large or the network is congested.
- Bandwidth Limitations: Bandwidth limitations can limit the speed at which data can be retrieved from the cloud.
- Data Security: Ensuring the security and privacy of data stored in the cloud can be challenging, especially for sensitive data.
- Data Compliance: Organizations must ensure that their data retrieval practices comply with relevant regulations and standards, such as data privacy laws and industry standards.
- Data Availability: In some cases, cloud data may not be available due to network outages, server downtime, or other technical issues.
- Cloud Costs: Retrieving large amounts of data from the cloud can be expensive, especially if the data is stored in a high-performance tier.
- Complexity: Interacting with cloud data storage systems can be complex and requires a certain level of technical expertise.
Cloud Data Retrieval and Egress Costs
Egress fees refer to the costs associated with transferring data from a cloud storage service to an external location or to another cloud provider. Many cloud service providers charge fees for data egress, as transferring large amounts of data can put a strain on their network and infrastructure. The cost of egress is usually based on the amount of data transferred, the distance of the transfer, and the speed of the transfer.
It is important for organizations to understand their cloud service provider’s data egress policies and fees, as well as their data transfer needs, to avoid unexpected costs. Organizations can minimize egress costs by compressing data, reducing the amount of data transferred, or storing data in the same geographic region as their computing resources.
The Benefits of Smart File Data Migration
A smart data migration strategy for enterprise file data means an analytics-first approach ensuring you know which data can migrate, to which class and tier, and which data should stay on-premises in your hybrid cloud storage infrastructure. With Komprise, you always have native data access, which not only removes end-user disruption, but also reduces egress costs and the need for rehydration and accelerates innovation in the cloud.
- Data Services
Data services is a term that is increasingly being used to describe the range of services typically provided by enterprise IT operations teams or shared services teams that involve data processing, data integration, data security, data reduction, data protection, data storage, and unstructured data management. Data services is a broad term that many enterprise IT teams, industry analysts and technology vendors and service providers use differently today, often overlapping, but not to be confused with analytics services, cloud services or professional services. Data services should be tied to financial operations (FinOps) goals to drive efficiency, cost savings and/or specific business objectives. Examples of data services include:
Data storage services: The storage of data in various forms, including files, databases, and cloud storage. The shift to Storage as a Service (STaaS) can be considered to part of a data services strategy. Read the white paper: Getting Departments to Care About Storage Savings.- Data management services: The management of data throughout its lifecycle, including data quality management, data governance, data classification. Data management services include analysis and line of business reporting into data storage usage and costs for showback as well as data mobility use cases (data migration, data tiering, data replication, deletion, etc.)
- Data processing services: The processing of data through various algorithms and techniques, including data analytics, machine learning, and artificial intelligence. Data engineering services can also be considered to be a data service
- Data integration services: The integration of data from multiple sources (ETL/ELT) to create a single, unified view of the data (usually for analytics) as well as real-time application of data between systems (EAI, ESB, streaming).
Data services are essential for data-heavy enterprise organizations that need to manage, process, and analyze large amounts of (mostly unstructured) data to gain insights and make better business decisions.
From Storage Services to Data Services
In VMblog predictions post: Unstructured Data Management Predictions for 2023: Data Insights and Automation take Center Stage, Komprise cofounder and COO Krishna Subramanian noted that enterprises are moving away from managing storage to managing data services:
“Storage teams have traditionally measured infrastructure metrics for capacity and performance such as latency, I/O operations per second (IOPS) and throughput. But given the massive data growth of unstructured data, data focused metrics are becoming paramount as enterprises move away from managing storage to managing data services in hybrid cloud infrastructure. New data management metrics look at usage indicators such as top data owners, percentage of “cold” files which haven’t been accessed in over a year, most common file size and type, and financial operations metrics such as storage costs per department, storage costs per vendor per TB, percentage of backups reduced, rate of data growth, chargeback metrics and more.”
In the same post she highlighted the changing role of storage administrators:
The storage architect/engineer will evolve to incorporate data services
We’ll see more experienced individuals in these roles move on to cloud architect and other engineering roles while IT generalists/junior cloud engineers inherit their responsibilities. This is a challenging time for IT organizations in a hybrid model as there is still significant NAS expertise needed. Either way, the IT employees managing the storage function will need new skills beyond managing the storage hardware. These individuals must understand the concept of data services-including facilitating secure, reliable governance and access to data and making data searchable and available to business stakeholders for applications such as cloud-based machine learning and data lakes. The new storage architect will frequently analyze and interpret data characteristics, developing data management plans which factor in cost savings strategies and business demands to create new value from data. This individual will interact regularly with departments to create and execute ongoing data management processes and plans.
In a Solutions Review post: 2023 Expert Data Management Best Practices & Predictions, Komprise cofounder and CEO Kumar Goswami noted:
“IT organizations must better understand data to improve migrations and gain maximum ROI from cloud, meet compliance requirements, deliver data services to departments, and to facilitate new value generation from data.”
He went on to say:
To keep up with ever-changing data services demands from the business, IT will implement collaborative processes with stakeholders across many different departments such as finance, marketing, legal, research, HR. Data workflow automation will support a variety of use cases from governance and compliance to cost savings to big data analytics.
In the 2022 Strategic Roadmap for Storage, Gartner noted (subscription required):
I&O leaders must implement intelligent data services infrastructure powered by software-defined storage and hybrid cloud IT operations….Integration of data services to the hybrid cloud platform is among the top enterprise challenges to address the need for seamless data services across the edge, the core data center and public clouds.
- Data Sprawl
What is Data Sprawl?
Data sprawl describes the staggering amount of unstructured data produced by enterprises worldwide every day; with new devices, including enterprise and mobile applications added to a network, it is estimated data sprawl to be 40% year over year, into the next decade.
Given this growth in data sprawl, data security is imperative, as it can lead to enormous problems for organizations, as well as its employees and customers. In today’s fast-paced world, organizations must carefully consider how to best manage the precious information it holds.
Organizations experiencing unstructured data sprawl need to secure all of their endpoints. Security is critical. Addressing data security as well as remote physical devices ensure organizations are in compliance with internal and external regulations.
As the amount of security threats mount, it is critical that data sprawl is addressed. Taking the right steps to ensure data sprawl is controlled, via policies and procedures within an organization, means safeguarding not only internal data, but also critical customer data.
Organizations should develop solid practices that may have been dismissed in the past. Left unchecked, control of an organization’s unstructured data will continue to manifest itself in hidden costs and limited options. With a little evaluation and planning, it is an aspect of your network that can be improved significantly and will pay off long term.
Analyzing and Managing Unstructured Data: Getting Sprawl (and Costs) Under Control
According to this Geekwire article, Gartner estimates that unstructured data represents an astounding 80 to 90% of all new enterprise data, and it’s growing 3X faster than structured data. Komprise Intelligent Data Management rapidly analyzes file ad object unstructured data in-place across multi-vendor storage to provide aggregate analytics (e.g., how much data, how much is hot, how much is cold, what types, top users, etc.) as well as a Global File Index across cloud and on-prem environments. The Komprise Global File Index is highly efficient and scalable to handle billions of files, exabytes of data without the scalability issues of using a central database or any other centralized architectures. Customers can build queries using Komprise Deep Analytics to find the precise subset of data they need through any combination of metadata and tags, and then move, copy and tier that data using Deep Analytics Actions. Komprise combines in-place analytics with data movement and on-going data management to provide a closed-loop system that is intelligent and adapts to a customer’s unique needs. The functionality is also available via API.
Tackling Data Sprawl with Komprise Analysis
Komprise Analysis provides consistent unified insights into unstructured data across many vendors’ storage and cloud platforms. Key metrics include data volume, data growth rates, where data is stored, top owners, top file types/sizes and time of last access. Komprise can create cost models based on different storage targets and tiering plan that will show.
- Data Storage
What is Data Storage?
Data storage refers both to the methods of transferring digital information from the source (users, applications, sensors) via protocols or APIs and to the destination; physical storage media such as magnetic or solid-state disks, tape, or optical. Data storage is pervasive: implemented in enterprise data centers, cloud providers, and consumer technology such as laptops, and phones.
From genomics and medical imaging to streaming video, electric cars, IoT at the edge and user generated data, unstructured data growth is exploding. Enterprise IT organizations are looking to new cloud and hybrid cloud strategies to manage costs and investing in unstructured data management and cloud data migration and cloud data management technologies and strategies to reduce data storage costs and while maximizing data value.
What are the different types of data storage protocols?
File Data Storage: File storage records data to files that are organized in folders, and the folders are organized under a hierarchy of directories and subdirectories. For example, a text file stored to your home directory on your laptop. File data is typically used for collaboration and shared access.
Examples of File Storage
NAS Network Attached Storage, Network File System NFS, and Server Message Block SMB
File Storage Vendor Solutions
NetApp ONTAP, Dell/EMC PowerScale (Isilon), Qumulo, Microsoft Windows Server, Pure FlashBlade, Amazon FSx, Azure Files
Block Data Storage
Typically used in servers and workstations where data is being written directly to physical media (HDD or SSD) in chunks or blocks. In contrast to file, block data is typically dedicated for access by a single application. Block storage is often used for the most performance intensive applications.
- Examples of Block Data Storage: Direct Attached Storage DAS, Storage Attached Network SAN, iSCSI, NVME
- Block Storage Vendor Solutions: Pure FlashArray, Dell/EMC VMAX, NetApp ONTAP and E-series, HDS
Object Storage
Also known as object-based storage or cloud storage, is a way of addressing and manipulating data storage as objects. In contrast to file storage, object data is stored in a flat namespace. Object storage was designed for use in massive repositories and is accessed over the HTTP protocol as a REST API.
- Examples of Object Storage: AWS S3, Azure Blob, Google Cloud Storage, Cloud Data Management Interface (CDMI)
- Object Storage Vendors: AWS, Azure, Google, Wasabi, Cloudian, NetApp, Dell/EMC, Scality
NDMP (Network Data Management Protocol)
Storage protocol that allows file servers and backup applications to communicate directly to a network-attached tape device for backup or recovery operations.
What are types of physical storage media?
- Hard Disk Drive (HDD): Disk based storage, used for high density data storage. Data is written to a magnetic layer of spinning disk.
- Solid State Drive (SSD): Also known as flash. Silicone replaces the spinning disk component of HDD to achieve higher performance and smaller form factor.
- Tape: Data is written to a ribbon of magnetic material in a cartridge. Used strictly for backup and archive, tape’s slow performance is off set by low cost, high levels of density, and the ability to be stored offline.
- Optical Storage: In contrast to magnetic storage data is recorded optically to media such as CD and DVD disks. Optical storage is used for durable, long term, off-line, archival storage.
What is Primary Storage?
Primary storage is used for active read and write data sets where high performance is critical. SSD or flash media with the highest level of performance is the ideal storage media for primary storage. While less typical HDD is also used as primary storage where lower cost and storage density is the key factor.
What is Secondary Storage?
Also referred to as active archive, secondary storage is used for less frequently accessed data sets. While any protocol and media can be used for secondary storage HDD with NAS and Object are the most common choices. Use cases for secondary storage is data tiering and backup / data protection applications.
Read the white paper: Block-Level vs. File Level Tiering
What is Data Storage?
Data storage refers both to the methods of transferring digital information from the source (users, applications, sensors) via protocols or APIs and to the destination; physical storage media such as magnetic or solid-state disks, tape, or optical.
What is Block Level Data Storage?
Mainly used in servers and workstations where data is being written directly to physical media (HDD or SSD) in chunks or blocks. As opposed to file level data storage, block level data storage is mostly dedicated for access by a single application. Block storage uses either direct attached storage (DAS), or data transfer protocols Fiber Channel (FC) or iSCSI (Internet Small Computer Systems Interface) via a storage area network (SAN).
What is Data Lake Storage in Azure?
Data Lake Storage in Azure from Microsoft is a fully managed scalable system based on a secure cloud platform that provides industry-standard, cost-effective storage for big data analytics.
- Data Storage Costs
Data storage costs are the expenses associated with storing and maintaining data in various forms of storage media, such as hard drives, solid-state drives (SSDs), cloud storage, and tape storage. These costs can be influenced by a variety of factors, including the size of the data, the type of storage media used, the frequency of data access, and the level of redundancy required. As the amount of unstructured data generated continues to grow, the cost of storing it remains a significant consideration for many organizations. In fact, according to the Komprise 2022 State of Unstructured Data Management Report, the majority of enterprise IT organizations are spending over 30% of their budget on data storage, backups and disaster recovery—similar to 2021. This is why shifting from storage management to storage-agnostic data management continues to be a topic of conversation for enterprise IT leaders.
Unstructured Data Management Cloud Data Storage Costs
Cloud data storage costs refer to the expenses incurred for storing data on cloud storage platforms provided by companies like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). In addition to the points above about data storage costs (amount of data stored and frequency of data access) in the cloud the level of durability and availability required are also factors when it comes to cloud storage costs. Cloud data storage providers typically charge based on the amount of data stored per unit of time, and additional fees may be incurred for data retrieval, data transfer, and data processing. Many cloud storage providers offer different storage tiers with varying levels of performance and cost, allowing customers to choose the option that best fits their budget and performance needs. With the right cloud data management strategy, cloud storage can be more cost-effective than traditional hardware-centric on-premises storage, especially for organizations with large amounts of data and high storage needs.
Managing Data Storage Costs
Managing data storage costs involves making informed decisions (and the right investment strategies) about how to store, access, and use data in a cost-effective manner. Here are some strategies for managing data storage costs:
- Data archiving: Archiving infrequently accessed data to lower cost storage options, such as object storage or tape, can help reduce storage costs.
- Data tiering: Using different storage tiers for different types of data based on their access frequency and importance can help optimize costs.
- Compression and deduplication: A well known data storage technique, compressing data and deduplicating redundant data can help reduce the amount of storage needed and lower costs.
- Cloud file storage: Using cloud storage can be more cost-effective than traditional on-premises storage, especially for organizations with large amounts of data and high storage needs.
- Data lifecycle management (aka Information Lifecycle Management): Regularly reviewing and purging unneeded data can help control storage costs over time.
- Cost monitoring and optimization (see cloud cost optimization): Regularly monitoring and analyzing data storage costs and usage patterns can help identify opportunities for cost optimization.
By using a combination of these strategies, organizations can effectively manage their data storage costs and ensure that they are using their data storage resources efficiently. Additionally, organizations can negotiate with data storage providers to secure better pricing and take advantage of cost-saving opportunities like bulk purchasing or long-term contracts.
Stop Overspending on Data Storage with Komprise
The blog post How Storage Teams Use Komprise Deep Analytics summarizes a number of strategies storage teams use Komprise Intelligent Data Management to deliver greater data storage cost savings and unstructured data value to the business, including:
- Business unit metrics with interactive dashboards
- Business-unit data tiering, retention and deletion
- Identifying and deleting duplicates
- Mobilizing specific data sets for third-party tools
- Using data tags from on-premises sources in the cloud
In the blog post Quantifying the Business Value of Komprise Intelligent Data Management, we review a storage cost savings analysis that saves customers an average 57% of overall data storage costs and over $2.6M+ annually. In addition to cost savings, benefits include:
Plan Future Data Storage Purchases with Visibility and Insight
With an analytics-first approach, Komprise delivers visibility into how data is growing and being used across a customer’s data storage silos – on-premises and in the cloud. Data storage administrators no longer have to make critical storage capacity planning decisions in the dark and now can understand how much more storage will be needed, when and how to streamline purchases during planning.
Optimize Data Storage, Backup, and DR Footprint
Komprise reduces the amount of data stored on Tier 1 NAS, as well as the amount of actively managed data—so customers can shrink backups, reduce backup licensing costs, and reduce DR costs.
Faster Cloud Data Migrations
Auto parallelize at every level to maximize performance, minimize network usage to migrate efficiently over WANs, and migrate more than 25 times faster than generic tools across heterogeneous cloud and storage with Elastic Data Migration.
Reduced Datacenter Footprint
Komprise moves and copies data to secondary storage to help reduce on-premises data center costs, based on customizable data management policies.
Risk Mitigation
Since Komprise works across storage vendors and technologies to provide native access without lock-in, organizations reduce the risk of reliance on any one storage vendor.
- Data Tagging
What is data tagging?
Data tagging is the process of adding metadata to your file data in the form of key value pairs. These values give context to your data, so that others can easily find it in search and execute actions on it, such as move to confinement or a cloud-based data lake. Data tagging is valuable for research queries and analytics projects or to comply with regulations and policies.
How does Komprise data tagging work?
Users, such as data owners, can apply tags to groups of files and tags can also be applied programmatically by analytics applications via API. In the Komprise Deep Analytics interface, users can query the Global File Index and find the data for tagging. This is done by creating a Komprise Plan that will invoke the text search function to inspect and tag the selected files. The ability to use Komprise Intelligent Data Management to search, find, apply tags and then take action makes it possible for customers to get faster value from enriched data sets.Tagging and Smart Data Workflows
Komprise Smart Data Workflows automate unstructured data discovery, data mobility and the delivery of data services.
- Define custom query to find specific data set.
- Analyze and tag data sets with additional metadata
- Move only the tagged data for analytics, AI/ML, etc.
- Move to a lower-cost data storage tier after analysis
———-
- Data Tiering
Data Tiering refers to a technique of moving less frequently used data, also known as cold data, to cheaper levels of storage or tiers. The term “data tiering” arose from moving data around different tiers or classes of storage within a storage system, but has expanded now to mean tiering or archiving data from a storage system to other clouds and storage systems. See also cloud tiering and choices for cloud data tiering.
Data Tiering Cuts Costs Because 70%+ of Data is Cold
As data grows, storage costs are escalating. It is easy to think the solution is more efficient storage. But the real cause of storage costs is poor data management. Over 70% of data is cold and has not been accessed in months, yet it sits on expensive storage and consumes the same backup resources as hot data. As a result, data storage costs are rising, backups are slow, recovery is unreliable, and the sheer bulk of this data makes it difficult to leverage new options like Flash and Cloud.
Data Tiering Was Initially Used within a Storage Array
Data Tiering was initially a technique used by storage systems to reduce the cost of data storage by tiering cold data within the storage array to cheaper but less performant options – for example, moving data that has not been touched in a year or more from an expensive Flash tier to a low-cost SATA disk tier.
Typical storage tiers within a storage array include:
- Flash or SSD: A high-performance storage class but also very expensive. Flash is usually used on smaller data sets that are being actively used and require the highest performance.
- SAS Disks: Usually the workhorse of a storage system, they are moderately good at performance but more expensive than SATA disks.
- SATA Disks: Usually the lowest price-point for disks but not as performant as SAS disks.
- Secondary Storage, often Object Storage: Usually a good choice for capacity storage – to store large volumes of cool data that is not as frequently accessed, at a much lower cost.
Cloud Data Tiering is now Popular
Increasingly, customers are looking at another option – tiering or archiving data to a public cloud.
- Public Cloud Storage: Public clouds currently have a mix of object and file storage options. The object storage classes such as Amazon S3 and Azure Blob (Azure Storage) provide tremendous cost efficiency and all the benefits of object storage without the headaches of setup and management.
Tiering and archiving less frequently used data or cold data to public cloud storage classes is now more popular. This is because customers can leverage the lower cost storage classes within the cloud to keep the cold data and promote them to the higher cost storage classes when needed. For example, data can be archived or tiered from on-premises NAS to Amazon S3 Infrequent Access or Amazon Glacier for low ongoing costs, and then promoted to Amazon EFS or FSX when you want to operate on it and need performance.
But in order to get this level of flexibility, and to ensure you’re not treating the cloud as just a cheap storage locker, data that is tiered to the cloud needs to be accessible natively in the cloud without requiring third-party software. This requires file-tiering, not block-tiering.
Block Tiering Creates Unnecessary Costs and Lock-In
Block-level tiering was first introduced as a technique within a storage array to make the storage box more efficient by leveraging a mix of technologies such as more expensive SAS disks as well as cheaper SATA disks.
Block tiering breaks a file into various blocks – metadata blocks that contain information about the file, and data blocks that are chunks of the original file. Block-tiering or Block-level tiering moves less used cold blocks to lower, less expensive tiers, while hot blocks and metadata are typically retained in the higher, faster, and more expensive storage tiers.
Block tiering is a technique used within the storage operating system or filesystem and is proprietary. Storage vendors offer block tiering as a way to reduce the cost of their storage environment. Many storage vendors are now expanding block tiering to move data to the public cloud or on-premises object storage.
But, since block tiering (often called CloudPools – examples are NetApp FabricPool and Dell EMC Isilon CloudPools) is done inside the storage operating system as a proprietary solution, it has several limitations when it comes to efficiency of reuse and efficiency of storage savings. Firstly, with block tiering, the proprietary storage filesystem must be involved in all data access since it retains the metadata and has the “map” to putting the file together from the various blocks. This also means that the cold blocks that are moved to a lower tier or the cloud cannot be directly accessed from the new location without involving the proprietary filesystem because the cloud does not have the metadata map and the other data blocks and the file context and attributes to put the file together. So, block tiering is a proprietary approach that often results in unnecessary rehydration of the data and treats the cloud as a cheap storage locker rather than as a powerful way to use data when needed.
The only way to access data in the cloud is to run the proprietary storage filesystem in the cloud which adds to costs. Also, many third-party applications such as backup software that operate at a file level require the cold blocks to be brought back or rehydrated, which defeats the purpose of tiering to a lower cost storage and erodes the potential savings. For more details, read the white paper: Block vs. File-Level Tiering and Archiving.
Know Your Cloud Tiering Choices
File Tiering Maximizes Savings and Eliminates Lock-In
File-tiering is an advanced modern technology that uses standard protocols to move the entire file along with its metadata in a non-proprietary fashion to the secondary tier or cloud. File tiering is harder to build but better for customers because it eliminates vendor lock-in and maximizes savings. Whether files have POSIX-based Access Control Lists (ACLs) or NTFS extended attributes, all this metadata along with the file itself is fully tiered or archived to the secondary tier and stored in a non-proprietary format. This ensures that the entire data can be brought back as a file when needed. File tiering does not just move the file, but it also moves the attributes and security permissions and ACLS along with the file and maintains full file fidelity even when you are moving a file to a different storage architecture such as object storage or cloud. This ensures that applications and users can use the moved file from the original location, and they can directly open the file natively in the secondary location or cloud without requiring any third-party software or storage operating system.
Since file tiering maintains full file fidelity and native access based on standards at every tier, it also means that third party applications can access the moved data without requiring any agents or proprietary software. This ensures that savings are maximized since backup software and other third -arty applications can access moved data without rehydrating or bringing the file back to the original location. It also ensures that the cloud can be used to run valuable applications such as compliance search or big data analytics on the trove of tiered and archived data without requiring any third-party software or additional costs.
File-tiering is an advanced technique for archiving and cloud tiering that maximizes savings and breaks vendor lock-in.
Data Tiering Can Cut 70%+ Storage and Backup Costs When Done Right
In summary, data tiering is an efficient solution to cut storage and backup costs because it tiers or archives cold, unused files to a lower-cost storage class, either on-premises or in the cloud. However, to maximize the savings, data tiering needs to be done at the file level, not block level. Block-level tiering creates lock-in and erodes much of the cost savings because it requires unnecessary rehydration of the data. File tiering maximizes savings and preserves flexibility by enabling data to be used directly in the cloud without lock-in.
Why Komprise is the easy, fast, no lock-in path to the cloud for file and object data.
- Data Virtualization
Data virtualization delivers a unified, simplified view of an organization’s data that can be accessed anytime. It integrates data from multiple sources, to create a single data layer to support multiple layers and users. The result is faster access to this data, providing instant access, any way you want it.
Data virtualization involves abstracting, transforming, federating and delivering data from disparate sources. This allows users to access the applications without having to know their exact location.
Advantages to data virtualization:
- An organization can gain business insights by leveraging all data
- They can become aware of analytics and business intelligence
- Data virtualization can streamline an organization’s data management approach, which reduces complexity and saves money
Data virtualization involves three key steps. First, data virtualization software is installed on-premise or in the cloud, which collects data from production sources and stays synchronized as those sources change over time. Next, administrators are able to secure, archive, replicate, and transform data using the data virtualization platform as a single point of control. Last, it allows users to provision virtual copies of the data that consume significantly less storage than physical copies.
Data virtualization use cases:
- Application development
- Backup and disaster recovery
- Datacenter migration
- Test data management
- Packaged application projects
- Deep Analytics
What is Deep Analytics?
Deep analytics is the process of applying data mining and data processing techniques to analyze and find large amounts of data in a form that is useful and beneficial for new applications. Deep analytics can apply to both structured and unstructured data.
In the context of unstructured data, deep analytics is the process of examining file and object metadata (both standard and extended) across billions of files to find data that fits specific criteria. A petabyte of unstructured data can be a few billion files. Analyzing petabytes of data typically involves analyzing tens to hundreds of billions of files. Because analysis of such large workloads can require distribution over a farm of processing units, deep analytics is often associated with scale-out distributed computing, cloud computing, distributed search, and metadata analytics.
Deep analytics of unstructured file and object data requires efficient indexing and search of files and objects across a distributed farm. Financial services, genomics, research and exploration, biomedical, and pharmaceutical are some of the early adopters of Komprise Deep Analytics, which is powered by a Global File Index. In recent years, enterprises have started to show interest in deep analytics as the amount of corporate unstructured data has increased, and with it, the desire to extract value from the data.
Deep analytics enables additional use cases such as Big Data Analytics, Artificial Intelligence and Machine Learning.
When the result of a deep analytics query is a virtual data lake, which we call the Global File Index, data does not have to be moved or disrupted from its original destination to enable reuse. This is an ideal scenario to rapidly leverage deep analytics without disruption since data can be pretty heavy to move.
Learn more about Komprise Deep Analytics.
Read the blog post: How Storage Teams Use Deep Analytics
- Digital Business
A digital business is one that uses technology as an advantage in its internal and external operations.
Information technology has changed the infrastructure and operation of businesses from the time the Internet became widely available to businesses and individuals. This transformation has profoundly changed the way businesses conduct their day-to-day operations. This has maximized the benefits of data assets and technology-focused initiatives.
This digital transformation has had a profound impact on businesses; accelerating business activities and processes to fully leverage opportunities in a strategic way. A digital business takes advantage of this fully so to not be disrupted and to thrive in this era. C-Level staff needs to help their organizations seize opportunities while mitigating risks.
This technology mindset has become standard in even the most traditional of industries, making a digital business strategy imperative for storing and analyzing data to gain a competitive advantage over the competition. The introduction of cloud computing and SaaS delivery models means that internal processes can be easily managed through a wide choice of applications, giving organizations the flexibility to chose, and change software as the businesses grows and changes.
A digital business also has seen a shift in purchasing power; individual departments now push for the applications that will best suit their needs, rather than relying on IT to drive change.
Unstructured Data Management is a Digital Business Priority
The Komprise 2022 State of Unstructured Data Management Report found that data storage costs comprise over 30% of enterprise IT budgets. This is why the right unstructured data management strategy has become an essential component of a digital business strategy.
Unstructured Data Management Unstructured data management is about being able to realize business outcomes from analytics through data movement, extraction, and value. Komprise provides a storage-independent way to manage data no matter where it lives so a digital business can get value from unstructured data from every tier. Unlike storage tiering or data backup solutions that move blocks of data and lock customers into proprietary file systems, Komprise Intelligent Data Management moves the entire file intact and enables customers to directly leverage native services at every tier without going through Komprise or their primary file system. This is key to a seamless user experience because users in a modern digital enterprise transparently access data from their original file system while also being able to build new applications in the cloud. Furthermore, user transparency, powered by patented Transparent Move Technology, makes it possible for IT teams to deploy transparent tiering company wide. Without this transparency, IT would need user and/or departmental approval and this essentially is a major roadblock that prevents any large scale tiering.
- Digital Pathology Data Management
According to the Digital Pathology Association:
“Digital pathology is a dynamic, image-based environment that enables the acquisition, management and interpretation of pathology information generated from a digitized glass slide.”
Healthcare organizations have shifted to digital media for medical imaging. Digital pathology, digital PACS and VNA systems are all generating and now storing petabytes of medical imaging data—lab slides, X-rays, MRIs, CT scans and more. These ever-expanding datasets are pushing the limitations of data storage systems and challenging IT department’s ability to effectively manage data. And with increasing regulations, healthcare providers typically must retain medical imaging files for many years. In addition to compliance requirements, clinical researchers may also need access to the data indefinitely. They also typically need access to the unstructured data immediately. The potential future value of this ever-expanding data repository must be weighed against the growing financial and overall unstructured data management costs.
The Digital Pathology Data Management Challenge
Data center storage for large image files is expensive – typically costing millions a year for some organizations on expensive NAS devices. Not only is NAS expensive, but its data must also be secured, replicated and backed up, which typically triples the costs. Meanwhile, in most cases, imaging data is rarely accessed after a few days or weeks. To get greater flexibility and manage data storage costs, healthcare organizations are adopting unstructured data management software to tier cold medical imaging data out of expensive storage to cost-effective environments such as the cloud. Data management decisions can be difficult internally with politics, vendor relationships and long-standing institutional perspectives. Health systems are handling sensitive patient information and tolerance for downtime is usually quite low.There are many benefits from augmenting medical imaging solutions with data management software that transparently tiers cold data from your data storage and backups.
Komprise has many customers in the healthcare industry dealing with multiple petabytes of file and object data.
- Direct Data Access
Direct data access is the ability to directly access your data whether on-premises, in the cloud, or a hybrid environment without needing to rehydrate.
The patented Komprise Transparent Move Technology™ (TMT) tiers file data workloads to a target without using any agents or stubs, allowing users to still access files natively from the original source as if they had never moved. Known as file and object duality, with Komprise users access files as native objects without getting in front of hot, mission-critical data.
Native Data Access definition.
- Director (Komprise Director)
The Komprise Director is the administrative console of the Komprise distributed architecture that runs as a cloud service or on-premises. Read the white paper: Komprise Intelligent Data Management Architecture Overview or one of the Komprise TechKrunch videos to learn more.
Learn more about the Komprise architecture.
- Disaster Recovery
Disaster recovery refers to security planning to protect an organization from the effects of a disaster – such as a cyber attack or equipment failure. A properly constructed disaster recovery plan will allow an organization to maintain or quickly resume mission critical functions following a disaster.
The disaster recovery plan includes policies and testing, and may involve a separate physical site for restoring operations. This preparation needs to be taken very seriously, and will involve a significant investment of time and money to ensure minimal losses in the event of a disaster.
Control measures are steps that can reduce or eliminate various threats for organizations. Different types of measures can be included in disaster recovery plan. There are three types of disaster recovery control measures that should be considered:
- Preventive measures – Intended to prevent a disaster from occurring
- Detective measures – Intended to detect unwanted events
- Corrective measures – The plan to restore systems after a disaster has occurred.
A quality disaster recovery plan requires these policies be documented and tested regularly. In some cases, organizations outsource disaster recovery to an outsourced provider instead of using their own remote facility, which can save time and money. This solution has become increasingly more popular with the rise in cloud computing.
Read the case study:
Leading Idaho Health System Selects Komprise to Right-Place Data and Bolster Disaster Recovery
- Dynamic Data Analytics
Komprise unstructured data analytics allows organizations to analyze data across all storage to know how much exists, what kind, who’s using it, and how fast it’s growing. “What if” data scenarios can be run based on various policies to instantly see capacity and data storage cost savings, enabling informed, optimal unstructured data management planning decisions without risk.
Learn more about Komprise Analysis.
Learn more about Komprise Deep Analytics.
- Data Analytics
Benefits of a Data Archiving Solution
The S3 protocol is used in a URL that specifies the location of an 
What is Unstructured Data Migration?
Data Tiering
Data Tiering refers to a technique of moving less frequently used data, also known as cold data, to cheaper levels of storage or tiers. The term “data tiering” arose from moving data around different tiers or classes of storage within a storage system, but has expanded now to mean tiering or archiving data from a storage system to other clouds and storage systems. See also cloud tiering and choices for cloud data tiering.
Data Tiering Cuts Costs Because 70%+ of Data is Cold
As data grows, storage costs are escalating. It is easy to think the solution is more efficient storage. But the real cause of storage costs is poor data management. Over 70% of data is cold and has not been accessed in months, yet it sits on expensive storage and consumes the same backup resources as hot data. As a result, data storage costs are rising, backups are slow, recovery is unreliable, and the sheer bulk of this data makes it difficult to leverage new options like Flash and Cloud.
Data Tiering Was Initially Used within a Storage Array
Data Tiering was initially a technique used by storage systems to reduce the cost of data storage by tiering cold data within the storage array to cheaper but less performant options – for example, moving data that has not been touched in a year or more from an expensive Flash tier to a low-cost SATA disk tier.
Typical storage tiers within a storage array include:
- Flash or SSD: A high-performance storage class but also very expensive. Flash is usually used on smaller data sets that are being actively used and require the highest performance.
- SAS Disks: Usually the workhorse of a storage system, they are moderately good at performance but more expensive than SATA disks.
- SATA Disks: Usually the lowest price-point for disks but not as performant as SAS disks.
- Secondary Storage, often Object Storage: Usually a good choice for capacity storage – to store large volumes of cool data that is not as frequently accessed, at a much lower cost.
Cloud Data Tiering is now Popular
Increasingly, customers are looking at another option – tiering or archiving data to a public cloud.
- Public Cloud Storage: Public clouds currently have a mix of object and file storage options. The object storage classes such as Amazon S3 and Azure Blob (Azure Storage) provide tremendous cost efficiency and all the benefits of object storage without the headaches of setup and management.
Tiering and archiving less frequently used data or cold data to public cloud storage classes is now more popular. This is because customers can leverage the lower cost storage classes within the cloud to keep the cold data and promote them to the higher cost storage classes when needed. For example, data can be archived or tiered from on-premises NAS to Amazon S3 Infrequent Access or Amazon Glacier for low ongoing costs, and then promoted to Amazon EFS or FSX when you want to operate on it and need performance.
But in order to get this level of flexibility, and to ensure you’re not treating the cloud as just a cheap storage locker, data that is tiered to the cloud needs to be accessible natively in the cloud without requiring third-party software. This requires file-tiering, not block-tiering.
Block Tiering Creates Unnecessary Costs and Lock-In
Block-level tiering was first introduced as a technique within a storage array to make the storage box more efficient by leveraging a mix of technologies such as more expensive SAS disks as well as cheaper SATA disks.
Block tiering breaks a file into various blocks – metadata blocks that contain information about the file, and data blocks that are chunks of the original file. Block-tiering or Block-level tiering moves less used cold blocks to lower, less expensive tiers, while hot blocks and metadata are typically retained in the higher, faster, and more expensive storage tiers.
Block tiering is a technique used within the storage operating system or filesystem and is proprietary. Storage vendors offer block tiering as a way to reduce the cost of their storage environment. Many storage vendors are now expanding block tiering to move data to the public cloud or on-premises object storage.
But, since block tiering (often called CloudPools – examples are NetApp FabricPool and Dell EMC Isilon CloudPools) is done inside the storage operating system as a proprietary solution, it has several limitations when it comes to efficiency of reuse and efficiency of storage savings. Firstly, with block tiering, the proprietary storage filesystem must be involved in all data access since it retains the metadata and has the “map” to putting the file together from the various blocks. This also means that the cold blocks that are moved to a lower tier or the cloud cannot be directly accessed from the new location without involving the proprietary filesystem because the cloud does not have the metadata map and the other data blocks and the file context and attributes to put the file together. So, block tiering is a proprietary approach that often results in unnecessary rehydration of the data and treats the cloud as a cheap storage locker rather than as a powerful way to use data when needed.
The only way to access data in the cloud is to run the proprietary storage filesystem in the cloud which adds to costs. Also, many third-party applications such as backup software that operate at a file level require the cold blocks to be brought back or rehydrated, which defeats the purpose of tiering to a lower cost storage and erodes the potential savings. For more details, read the white paper: Block vs. File-Level Tiering and Archiving.
Know Your Cloud Tiering Choices
File Tiering Maximizes Savings and Eliminates Lock-In
File-tiering is an advanced modern technology that uses standard protocols to move the entire file along with its metadata in a non-proprietary fashion to the secondary tier or cloud. File tiering is harder to build but better for customers because it eliminates vendor lock-in and maximizes savings. Whether files have POSIX-based Access Control Lists (ACLs) or NTFS extended attributes, all this metadata along with the file itself is fully tiered or archived to the secondary tier and stored in a non-proprietary format. This ensures that the entire data can be brought back as a file when needed. File tiering does not just move the file, but it also moves the attributes and security permissions and ACLS along with the file and maintains full file fidelity even when you are moving a file to a different storage architecture such as object storage or cloud. This ensures that applications and users can use the moved file from the original location, and they can directly open the file natively in the secondary location or cloud without requiring any third-party software or storage operating system.
Since file tiering maintains full file fidelity and native access based on standards at every tier, it also means that third party applications can access the moved data without requiring any agents or proprietary software. This ensures that savings are maximized since backup software and other third -arty applications can access moved data without rehydrating or bringing the file back to the original location. It also ensures that the cloud can be used to run valuable applications such as compliance search or big data analytics on the trove of tiered and archived data without requiring any third-party software or additional costs.
File-tiering is an advanced technique for archiving and cloud tiering that maximizes savings and breaks vendor lock-in.
Data Tiering Can Cut 70%+ Storage and Backup Costs When Done Right
In summary, data tiering is an efficient solution to cut storage and backup costs because it tiers or archives cold, unused files to a lower-cost storage class, either on-premises or in the cloud. However, to maximize the savings, data tiering needs to be done at the file level, not block level. Block-level tiering creates lock-in and erodes much of the cost savings because it requires unnecessary rehydration of the data. File tiering maximizes savings and preserves flexibility by enabling data to be used directly in the cloud without lock-in.
Why Komprise is the easy, fast, no lock-in path to the cloud for file and object data.
