Data Management Glossary
-
- Data Analytics
Data analytics refers to the process used to enhance productivity and business improvement by extracting and categorizing data to identify and analyze behavioral patterns. Techniques vary according to organizational requirements.
The primary goal of data analytics is to help organizations make more informed business decisions by enabling analytics professionals to evaluate large volumes of transactional and other forms of data. Data analytics can be pulled from anything from Web server logs to social media comments.
Potential issues with data analytics initiatives include a lack of analytics professionals and the cost of hiring qualified candidates. The amount of information that can be involved and the variety of data analytics data can also cause data analytics issues, including the quality and consistency of the data. In addition, integrating technologies and data warehouses can be a challenge, although various vendors offer data integration tools with big data capabilities.
Big data has drastically changed the requirements for extracting data analytics from business data. With relational databases, administrators can easily generate reports for business use, but they lack the broader intelligence data warehouses can provide. However, the challenge for data analytics from data warehouses is the costs associated.
There is also the challenge of pulling the relevant data sets to enable data analytics from cold data. This requires intelligent data management solutions that track what unstructured data is kept and where, and enable you to easily search and find relevant data sets for big-data analytics.
- Data Archiving
Data Archiving, often referred to as Data Tiering, protects older data that is not needed for everyday operations of an organization that is no longer needed for everyday access. Data Archiving reduces primary storage required, and allows an organization to maintain data that may be required for regulatory or other requirements.
Data archiving is intended to protect older information that is not needed for everyday operations but may have to be accessed occasionally. Data archives serve as a way of reducing primary storage and the related costs, rather than acting as a data recovery tool.
Some data archives allow data to be read-only to protect it from modification, while other data archiving products treat data as to allow users to modify it.
The benefit of data archiving is that it reduces the cost of primary storage. Alternatively, archive storage costs less because it is typically based on a low-performance, high-capacity storage medium.
Data archiving take a number of different forms. Options can be online data storage, which places archive data onto disk systems where it is readily accessible. Archives are frequently file-based, but object storage is also growing in popularity. A key challenge when using object storage to archive file-based data is the impact it can have on users and applications. To avoid changing paradigms from file to object and breaking user and application access, use data management solutions that provide a file interface to data that is archived as objects.
Another archival system uses offline data storage where archive data is written to tape or other removable media using data archiving software rather than being kept online. Data archiving on tape consumes less power than disk systems, translating to lower costs.
A third option is using cloud storage, such as those offered by Amazon – this is inexpensive but requires ongoing investment.
The data archiving process typically uses automated software, which will automatically move “cold” data via policies set by an administrator. Today, a popular approach to data archiving is to make the archive “transparent” – so the archived data is not only online but the archived data is fully accessed exactly as before by users and applications, so they experience no change in behavior. (See Native Access)
- Data Backup
Data loss can occur from a variety of causes, including computer viruses, hardware failure, file corruption, fire, flood, or theft, etc. Data loss may involve critical financial, customer, and company data, so a solid data backup plan is critical for every organization.
As part of a data backup plan, consider the following:
- What data (files and folders) to backup
- How often to run your backups
- Where to store the backup data
- What compression method to use
- What type of backups to run
- What kind of media on which to store the backups
In general, you should back up any data that can’t be replaced easily. Some examples are structured data like databases, and unstructured data such as word processing documents, spreadsheets, photos, videos, emails, etc. Typically, programs or system folders are not part of a data backup program. Installation discs, operating system discs, and registration information should be stored in a safe place.
Data backup frequency depends on how often your organizational data changes.
- Frequently changing data may need daily or hourly backups
- Data that changes every few days might require a weekly or even monthly backup
- For some data, a backup may need to be created each time it changes
The challenge with unstructured data is that backing up unstructured data is not only time consuming but also very complex, with millions to billions of files of various sizes and types and growing at an astronomical rate, leaving enterprises to struggle with long backup windows, overlapping backup cycles, backup footprint sprawl, spiraling costs, and above all, vulnerable in the case of a disaster.
Read the white paper: Rein in Storage and Backup Costs.
Read the post: 5 Ways to Get to the Cloud Smarter and Faster
Don’t backup data first. Know your data first to make smarter, cost-saving decisions. Start with the Komprise TCO calculator.
- Data Classification
Data classification is the process of organizing data into tiers of information for data organizational purposes.
Data classification is essential to make data easy to find and retrieve so that your organization can optimize risk management, compliance, and legal requirements. Written guidelines are essential in order to define the categories and criteria to classify your organization’s data. It is also important to define the roles and responsibilities of employees in the data organization structure.
When data classification procedures are established, security standards should also be established to address data life-cycle requirements. Classification should be simple so employees can easily comply with the standard.
Examples of types of data classifications are:
- 1st Classification: Data that is free to share with the public
- 2nd Classification: Internal data not intended for the public
- 3rd Classification: Sensitive internal data that would negatively impact the organization if disclosed
- 4th Classification: Highly sensitive data that could put an organization at risk
Data classification is a complex process, but automated systems can help streamline this process. The enterprise must create the criteria for classification, outline the roles and responsibilities of employees to maintain the protocols, and implement proper security standards. Properly executed, data classification will provide a framework for the storage, transmission and retrieval of data.
Automation simplifies data classification by enabling you to dynamically set different filters and classification criteria when viewing data across your storage. For instance, if you wanted to classify all data belonging to users who are no longer at the company as “zombie data,” the Komprise solution will aggregate files that fit into the zombie data criterion to help you quickly classify your data.
- Data Governance
Data governance refers to the management of the availability, security, usability, and integrity of data used in an enterprise. Data governance in an organization typically includes a governing council, a defined set of procedures, and a plan to execute those procedures.
Data governance is not about allowing access to a few privileged users; instead, it should allow broad groups of users access with appropriate controls. Business and IT users have different needs; business users need secure access to shared data and IT needs to set policies around security and business practices. When done right, data governance allows any user access to data anytime, so the organization can run more efficiently, and users can manage their workload in a self-service manner.
Here are four things to consider when developing a data governance strategy:
Selecting a Team:
- Balance IT and business leaders to get a broad view of the data and service needs
- Start small – choose a small group to review existing data analytics
Data Quality:
- Audit existing data to discover data types and how they are used
- Define a process for new data sources to ensure quality and availability standards are met
Data Security:
- Make sure data is classified so data requiring protection for legal or regulatory reasons meets those requirements
- Implement policies that allow for different levels of access based on user privileges
- Data Lake
A data lake is data stored in its natural state. The term typically refers to unstructured data that is sitting on different storage environments and clouds. The data lake supports data of all types – for example, you may have videos, blogs, log files, seismic files and genomics data in a single data lake. You can think of each of your Network Attached Storage (NAS) devices as a data lake.
One big challenge with data lakes is to comb through them and find the relevant data you need. With unstructured data, you may have billions of files strewn across different data lakes, and finding data that fits specific criteria can be like finding a needle in a haystack
A virtual data lake is a collection of data that fits certain criteria – and as the name implies, it is virtual because the data is not moved. The data continues to reside in its original location, but the virtual data lake gives a discrete handle to manipulate that entire data set.
Some key aspects of data lakes – both physical and virtual:
- Data Lakes Support a Variety of Data Formats: Data lakes are not restricted to data of any particular type.
- Data Lakes Retain All Data: Even if you do a search and find some data that does not fit your criteria, the data is not deleted from the data lake. A virtual data lake provides a discrete handle to the subset of data across different storage silos that fits specific criteria, but nothing is moved or deleted.
- Virtual Data Lakes Do Not Physically Move Data: Virtual data lakes do not physically move the data, but provide a virtual aggregation of all data that fits certain criteria. Deep Analytics can be used to specify criteria.
- Data Literacy
The ability to derive meaningful information from data. Komprise Data Analytics provides data literacy by showing how much data, what kind, who’s using it, how often—across all storage silos. Read the IDC InfoBrief: How to Manage Your Data Growth Smarter with Data Literacy.
- Data Management
Data management is officially defined by DAMA International, the professional organization data management professionals, is:
“Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise.”
Data management is the process of developing policies and procedures in order to effectively manage the information lifecycle needs of an enterprise. This includes identifying how data is acquired, validated, stored, protected, and processed. Data management policies should cover the entire lifecycle of the data, from creation to deletion.
Due to the sheer volume of unstructured data, a data management plan is necessary for every organization. The numbers are staggering – for example, more data has been created in the past two years than in the entire previous history of the human race.
- Data Management Policy
A data management policy addresses the operating policy that focuses on the management and governance of data assets, and is a cornerstone of governing enterprise data assets. This policy should be managed by a team within the organization that identifies how the policy is accessed and used, who enforces the data management policy, and how it is communicated to employees.
It is recommended that an effective data management policy team include top executives to lead in order for governance and accountability to be enforced. In many organizations, the Chief Information Officer (CIO) and other senior management can demonstrate their understanding of the importance of data management by either authoring or supporting directives that will be used to govern and enforce data standards.
The following are some of the considerations to consider in a data management policy:
- Enterprise data is not owned by any individual or business unit, but is owned by the enterprise
- Enterprise data must be safe and
- Enterprise Data Must Be Accessible to individuals within the organization
- Meta data should be developed and utilized for all structured and unstructured data
- Data owners should be accountable for enterprise data
- Users should not have to worry about where data lives. Data should be accessible to users no matter where it resides.
Ultimately, a data management policy should guide your organization’s philosophy toward managing data as a valued enterprise asset. Watch the video: Intelligent Data Management: Policy-Based Automation
- Data Migration
Data Migration is the process of selecting and moving data from one location to another – this may involve moving data across different storage vendors, and across different formats.Data migrations are often done in the context of retiring a system and moving to a new system, or in the context of a cloud migration, or in the context of a modernization or upgrade strategy.
Data migrations can be laborious, error prone, manual, and time consuming. Migrating data may involve finding and moving billions of files, which can succumb to storage and network slowdowns or outages. Also, different file systems do not often preserve metadata in exactly the same way, so migrating data without loss of fidelity and integrity can be a challenge.
Network Attached Storage (NAS) migration is the process of migrating from one NAS storage environment to another. This may involve migrations within a vendor’s ecosystem such as NetApp to NetApp or across vendors such as NetApp to Isilon or EMC to NetApp or EMC to Pure FlashBlade. A high-fidelity NAS migration solution should preserve not only the file itself but all of its associated metadata and access controls.
Network Attached Storage (NAS) to Cloud data migration is the process of moving data from an on-premises data center to a cloud. It requires data to be moved from a file format (NFS or SMB) to an Object/Cloud format such as S3. A high-fidelity NAS-to-Cloud migration solution preserves all the file metadata including access control and privileges in the cloud. This enables data to be used either as objects or as files in the cloud.
Storage migration is a general-purpose term that applies to moving data across storage arrays.
Data migrations typically involve four phases:
- Planning – Deciding what data should be migrated. Planning may often involve analyzing various sources to find the right data sets. For example, several customers today are interested in upgrading some data to Flash – finding hot, active data to migrate to Flash can be a useful planning exercise.
- Initial Migration – Do a first migration of all the data. This should involve migrating the files, the directories and the shares.
- Iterative Migrations – Look for any changes that may have occurred during the initial migration and copy those over.
- Final Cutoff – A final cutoff involves deleting data at the original storage and managing the mounts, etc., so data can be accessed from the new location going forward.
Resilient data migration refers to an approach that automatically adjusts for failures and slowdowns and retries as needed. It also checks the integrity of the data at the destination to ensure full fidelity.
- Data Protection
Data protection is used to describe both data backup and disaster recovery. A quality data protection strategy should automate the movement of critical data to online and offline storage and include a comprehensive strategy for valuing, classifying, and protecting data as to protect these assets from user errors, malware and viruses, machine failure, or facility outages/disruptions.
Data protection storage technologies include tape backup, which copies data to a physical tape cartridge, or cloud backup, which copies data to the cloud, and mirroring, which replicates a website or files to a secondary location. These processes can be automated and policies assigned to the data, allowing for accurate, faster data recovery.
Data protection should always be applied to all forms of data within an organization, in order to protect the integrity of the data, protect from corruption or errors, and ensuring privacy of the data. When classifying data, policies should be established to identify different levels of security, from least secure (data that anyone can see) to most secure (data that if released, would put the organization at risk).
- Data Sprawl
Data sprawl describes the staggering amount of data produced by enterprises worldwide every day; with new devices, including enterprise and mobile applications added to a network, it is estimated data sprawl to be 40% year over year, into the next decade.
Given this growth in data sprawl, data security is imperative, as it can lead to enormous problems for organizations, as well as its employees and customers. In today’s fast-paced world, organizations must carefully consider how to best manage the precious information it holds.
Organizations experiencing data sprawl need to secure all of their endpoints. Security is critical. Addressing data security as well as remote physical devices ensure organizations are in compliance with internal and external regulations.
As the amount of security threats mount, it is critical that data sprawl is addressed. Taking the right steps to ensure data sprawl is controlled, via policies and procedures within an organization, means safeguarding not only internal data, but also critical customer data.
Organizations should develop solid practices that may have been dismissed in the past. Left unchecked, control of an organization’s unstructured data will continue to manifest itself in hidden costs and limited options. With a little evaluation and planning, it is an aspect of your network that can be improved significantly and will pay off long term.
- Data Tiering
Data Tiering refers to a technique of moving less frequently used data, also known as cold data, to cheaper levels of storage or tiers. The term “data tiering” arose from moving data around different tiers or classes of storage within a storage system, but has expanded now to mean tiering or archiving data from a storage system to other clouds and storage systems.
Data Tiering Cuts Costs Because 70%+ of Data is Cold
As data grows, storage costs are escalating. It is easy to think the solution is more efficient storage. But the real cause of storage costs is poor data management. Over 70% of data is cold and has not been accessed in months, yet it sits on expensive storage and consumes the same backup resources as hot data. As a result, storage costs are rising, backups are slow, recovery is unreliable, and the sheer bulk of this data makes it difficult to leverage new options like Flash and Cloud.Data Tiering Was Initially Used within a Storage Array
Data Tiering was initially a technique used by storage systems to reduce the cost of data storage by tiering cold data within the storage array to cheaper but less performant options – for example, moving data that has not been touched in a year or more from an expensive Flash tier to a low-cost SATA disk tier.Typical storage tiers within a storage array include:
- Flash or SSD: A high-performance storage class but also very expensive. Flash is usually used on smaller data sets that are being actively used and require the highest performance.
- SAS Disks: Usually the workhorse of a storage system, they are moderately good at performance but more expensive than SATA disks.
- SATA Disks: Usually the lowest price-point for disks but not as performant as SAS disks.
- Secondary Storage, often Object Storage: Usually a good choice for capacity storage – to store large volumes of cool data that is not as frequently accessed, at a much lower cost.
Increasingly, customers are looking at another option – tiering or archiving data to a public cloud.
- Public Cloud Storage: Public clouds currently have a mix of object and file storage options. The object storage classes such as Amazon S3 and Azure Blob provide tremendous cost efficiency and all the benefits of object storage without the headaches of setup and management.
Cloud Data Tiering is now Popular
Tiering and archiving less frequently used data or cold data to public cloud storage classes is now more popular. This is because customers can leverage the lower cost storage classes within the cloud to keep the cold data and promote them to the higher cost storage classes when needed. For example, data can be archived or tiered from on-premises NAS to Amazon S3 Infrequent Access or Amazon Glacier for low ongoing costs, and then promoted to Amazon EFS or FSX when you want to operate on it and need performance.But in order to get this level of flexibility, and to ensure you’re not treating the cloud as just a cheap storage locker, data that is tiered to the cloud needs to be accessible natively in the cloud without requiring third-party software. This requires file-tiering, not block-tiering.
Block Tiering Creates Unnecessary Costs and Lock-In
Block-level tiering was first introduced as a technique within a storage array to make the storage box more efficient by leveraging a mix of technologies such as more expensive SAS disks as well as cheaper SATA disks.Block tiering breaks a file into various blocks – metadata blocks that contain information about the file, and data blocks that are chunks of the original file. Block-tiering or Block-level tiering moves less used cold blocks to lower, less expensive tiers, while hot blocks and metadata are typically retained in the higher, faster, and more expensive storage tiers.
Block tiering is a technique used within the storage operating system or filesystem and is proprietary. Storage vendors offer block tiering as a way to reduce the cost of their storage environment. Many storage vendors are now expanding block tiering to move data to the public cloud or on-premises object storage.
But, since block tiering is done inside the storage operating system as a proprietary solution, it has several limitations when it comes to efficiency of reuse and efficiency of storage savings. Firstly, with block tiering, the proprietary storage filesystem must be involved in all data access since it retains the metadata and has the “map” to putting the file together from the various blocks. This also means that the cold blocks that are moved to a lower tier or the cloud cannot be directly accessed from the new location without involving the proprietary filesystem because the cloud does not have the metadata map and the other data blocks and the file context and attributes to put the file together. So, block tiering is a proprietary approach that often results in unnecessary rehydration of the data and treats the cloud as a cheap storage locker rather than as a powerful way to use data when needed.The only way to access data in the cloud is to run the proprietary storage filesystem in the cloud which adds to costs. Also, many third-party applications such as backup software that operate at a file level require the cold blocks to be brought back or rehydrated, which defeats the purpose of tiering to a lower cost storage and erodes the potential savings. For more details, read the white paper: Block vs. File-Level Tiering and Archiving.
File Tiering Maximizes Savings and Eliminates Lock-In
File-tiering is an advanced modern technology that uses standard protocols to move the entire file along with its metadata in a non-proprietary fashion to the secondary tier or cloud. File tiering is harder to build but better for customers because it eliminates vendor lock-in and maximizes savings. Whether files have POSIX-based Access Control Lists (ACLs) or NTFS extended attributes, all this metadata along with the file itself is fully tiered or archived to the secondary tier and stored in a non-proprietary format. This ensures that the entire data can be brought back as a file when needed. File tiering does not just move the file, but it also moves the attributes and security permissions and ACLS along with the file and maintains full file fidelity even when you are moving a file to a different storage architecture such as object storage or cloud. This ensures that applications and users can use the moved file from the original location, and they can directly open the file natively in the secondary location or cloud without requiring any third-party software or storage operating system.Since file tiering maintains full file fidelity and native access based on standards at every tier, it also means that third party applications can access the moved data without requiring any agents or proprietary software. This ensures that savings are maximized since backup software and other third -arty applications can access moved data without rehydrating or bringing the file back to the original location. It also ensures that the cloud can be used to run valuable applications such as compliance search or big data analytics on the trove of tiered and archived data without requiring any third-party software or additional costs.
File-tiering is an advanced technique for archiving and tiering that maximizes savings and breaks vendor lock-in.
Data Tiering Can Cut 70%+ Storage and Backup Costs When Done Right
In summary, data tiering is an efficient solution to cut storage and backup costs because it tiers or archives cold, unused files to a lower-cost storage class, either on-premises or in the cloud. However, to maximize the savings, data tiering needs to be done at the file level, not block level. Block-level tiering creates lock-in and erodes much of the cost savings because it requires unnecessary rehydration of the data. File tiering maximizes savings and preserves flexibility by enabling data to be used directly in the cloud without lock-in. - Data Virtualization
Data virtualization delivers a unified, simplified view of an organization’s data that can be accessed anytime. It integrates data from multiple sources, to create a single data layer to support multiple layers and users. The result is faster access to this data, providing instant access, any way you want it.
Data virtualization involves abstracting, transforming, federating and delivering data from disparate sources. This allows users to access the applications without having to know their exact location.
There are some important advantages to data virtualization:
- An organization can gain business insights by leveraging all data
- They can become aware of analytics and business intelligence
- Data virtualization can streamline an organization’s data management approach, which reduces complexity and saves money
Data virtualization involves three key steps. First, data virtualization software is installed on-premise or in the cloud, which collects data from production sources and stays synchronized as those sources change over time. Next, administrators are able to secure, archive, replicate, and transform data using the data virtualization platform as a single point of control. Last, it allows users to provision virtual copies of the data that consume significantly less storage than physical copies.
Some use cases for data virtualization are:
- Application development
- Backup and disaster recovery
- Datacenter migration
- Test data management
- Packaged application projects
- Deep Analytics
Deep analytics is the process of applying data mining and data processing techniques to analyze and find large amounts of data in a form that is useful and beneficial for new applications. Deep analytics can apply to both structured and unstructured data.
In the context of unstructured data, deep analytics is the process of examining file metadata (both standard and extended) across billions of files to find data that fits specific criteria. A petabyte of unstructured data can be a few billion files. Analyzing petabytes of data typically involves analyzing tens to hundreds of billions of files. Because analysis of such large workloads can require distribution over a farm of processing units, deep analytics is often associated with scale-out distributed computing, cloud computing, distributed search, and metadata analytics.
Deep analytics of unstructured file data requires efficient indexing and search of files and objects across a distributed farm. Financial services, genomics, research and exploration, biomedical, and pharmaceutical are some of the early adopters of deep analytics. In recent years, enterprises have started to show interest in deep analytics as the amount of corporate data has increased, and with it, the desire to extract value from the data.
Deep analytics enables additional use cases such as Big Data Analytics, Artificial Intelligence and Machine Learning.
When the result of a deep analytics query is a virtual data lake, data does not have to be moved or disrupted from its original destination to enable reuse. This is an ideal scenario to rapidly leverage deep analytics without disruption since data can be pretty heavy to move.
- Digital Business
A digital business is one that uses technology as an advantage in its internal and external operations.
Information technology has changed the infrastructure and operation of businesses from the time the Internet became widely available to businesses and individuals. This transformation has profoundly changed the way businesses conduct their day-to-day operations. This has maximized the benefits of data assets and technology-focused initiatives.
This digital transformation has had a profound impact on businesses; accelerating business activities and processes to fully leverage opportunities in a strategic way. A digital business takes advantage of this fully so to not be disrupted and to thrive in this era. C-Level staff needs to help their organizations seize opportunities while mitigating risks.
This technology mindset has become standard in even the most traditional of industries, making a digital business strategy imperative for storing and analyzing data to gain a competitive advantage over the competition. The introduction of cloud computing and SaaS delivery models means that internal processes can be easily managed through a wide choice of applications, giving organizations the flexibility to chose, and change software as the businesses grows and changes.
A digital business also has seen a shift in purchasing power; individual departments now push for the applications that will best suit their needs, rather than relying on IT to drive change.
- Director (Komprise Director)
The administrative console of the Komprise distributed architecture that runs as a cloud service or on-premises. Read the white paper: Komprise Intelligent Data Management Architecture Overview or one of the Komprise Chalk-Talk videos to learn more.
- Disaster Recovery
Disaster recovery refers to security planning to protect an organization from the effects of a disaster – such as a cyber attack or equipment failure. A properly constructed disaster recovery plan will allow an organization to maintain or quickly resume mission critical functions following a disaster.
The disaster recovery plan includes policies and testing, and may involve a separate physical site for restoring operations. This preparation needs to be taken very seriously, and will involve a significant investment of time and money to ensure minimal losses in the event of a disaster.
Control measures are steps that can reduce or eliminate various threats for organizations. Different types of measures can be included in disaster recovery plan. There are three types of disaster recovery control measures that should be considered:
- Preventive measures – Intended to prevent a disaster from occurring
- Detective measures – Intended to detect unwanted events
- Corrective measures – The plan to restore systems after a disaster has occurred.
A quality disaster recovery plan requires these policies be documented and tested regularly. In some cases, organizations outsource disaster recovery to an outsourced provider instead of using their own remote facility, which can save time and money. This solution has become increasingly more popular with the rise in cloud computing.
- Dynamic Data Analytics
The Komprise feature that allows organizations to analyze data across all storage to know how much exists, what kind, who’s using it, and how fast it’s growing. “What if” data scenarios can be run based on various policies to instantly see capacity and cost savings, enabling informed, optimal data management planning decisions without risk.
- Data Analytics
Direct Data Access
The ability to directly access your data whether on-premises, in the cloud, or a hybrid environment without needing to rehydrate