Data Management Glossary
Data Migration is the process of selecting and moving data from one location to another – this may involve moving data across different storage vendors, and across different formats.
-
- S3
The S3 protocol is used in a URL that specifies the location of an Amazon S3 (Simple Storage Service) bucket and a prefix to use for reading or writing files in the bucket. See S3 Intelligent Tiering.
- S3 Intelligent Tiering
S3 Intelligent Tiering is an Amazon storage class. Amazon S3 offers a range of storage classes for different uses. S3 Intelligent Tiering is a storage class aimed at data with unknown or unpredictable data access patterns. It was introduced in 2018 by AWS as a solution for customers who want to optimize storage costs automatically when their data access patterns change.
Instead of utilizing the other Amazon S3 storage classes and moving data across them based on the needs of the data, Amazon S3 Intelligent Tiering is a distinct storage class that has embedded tiers within it and data can automatically move across the four access tiers when access patterns change.
To fully understand what S3 Intelligent Tiering offers it is important to have an overview of all the classes available through S3:
Classes of AWS S3 Storage
- Standard (S3) – Used for frequently accessed data (hot data)
- Standard-Infrequent Access (S3-IA) – Used for infrequently accessed, long-lived data that needs to be retained but is not being actively used
- One Zone Infrequent Access – Used for infrequently accessed data that’s long-lived but not critical enough to be covered by storage redundancies across multiple locations
- Intelligent Tiering – Used for data with changing access patterns or uncertain need of access
- Glacier – Used to archive infrequently accessed, long-lived data (cold data) Glacier has a latency of a few hours to retrieve
- Glacier Deep Archive – Used for data that is hardly ever or never accessed and for digital preservation purposes for regulatory compliance
What is S3 Intelligent Tiering?
S3 Intelligent Tiering is a storage class that has multiple tiers embedded within it, each with its own access latencies and costs – it is an automated service that monitors your data access behavior and then moves your data on a per-object basis to the appropriate level of tier within the S3 Intelligent Tiering storage class. If your object has not been accessed for 30 consecutive days it will automatically move to the infrequent access tier within S3 Intelligent Tiering, and if the object is not accessed for 90 consecutive days it will automatically move the object to the Archive Access tier and then after 190 consecutive days to the Deep Archive access tier. If an object is moved to the archive tier, the retrieval can take 3 to 5 hours and if it is in the deep archive tier it can take 12 hours. and if it is then subsequently accessed it will move it into the frequently accessed storage class.
What are the costs of AWS S3 Intelligent Tiering?
You pay for monthly storage, request and data transfer. When using Intelligent-Tiering you also pay for a monthly per-object fee for monitoring and automation. While there is no retrieval fee in S3 Intelligent-Tiering and no fee for moving data between tiers, you do not manipulate each tier directly. S3 Intelligent Tier is a bucket, and it has tiers within it that objects move through. Objects in the Frequent Access tier are billed at the same rate as S3 Standard, objects stored in the Infrequent Access tier are billed at the same rate as S3 Standard Infrequent Access, objects stored in the Archive Access tier are billed at the same rate as S3 Glacier and objects stored in the Deep Archive access tier are billed at the same rate as S3 Deep Glacier.
What are the advantages of S3 Intelligent tiering?
The advantages of S3 Intelligent tiering are that savings can be made. There is no operational overhead, and there are no retrieval costs. Objects can be assigned a tier upon upload and then move between tiers based on access patterns. There is no impact on performance and it is designed for 99.999999999% durability and 99.9% availability over annual average.
What are the disadvantages of S3 Intelligent tiering?
The main disadvantage of S3 Intelligent Tiering is that it acts as a black-box – you move objects into it and cannot transparently access different tiers or set different versioning policies for the different tiers. You have to manipulate the whole of S3 Intelligent Tier as a single bucket. For example, if you want to transition an object that has versioning enabled, then you have to transition all the versions. Also, when objects move to the archive tiers, the latency of access is much higher than the access tiers. Not all applications may be able to deal with the high latency.
S3 Intelligent tiering is not suitable for companies with predictable data access behavior or companies that want to control data access, versioning, etc with transparency . Other disadvantages are that it is limited to objects, and cannot tier from files to objects, the minimum object storage requirement is 30 days, objects smaller than 128kb are never moved from the frequent access tier and lastly, because it is an automated system, you cannot configure different policies for different groups.
S3 Data Management with Komprise
Komprise is an AWS Advance Tier partner and can offer intelligent data management with visibility, transparency and cost savings on AWS file and object data. How is this done? Komprise enables analytics-driven intelligent tiering across EFS, FSX, S3 and Glacier storage classes in AWS so you can maximize price performance across all your data on Amazon. The Komprise mission is to radically simplify data management through intelligent automation.
- Scale-Out Grid
Traditional approaches to managing data have relied on a centralized architecture – using either a central database to store information, or requiring a primary-replica architecture with a central primary server to manage the system. These approaches do not scale to address the modern scale of data because they have a central bottleneck that limits scaling. A scale-out architecture delivers unprecedented scale because it has no central bottlenecks. Instead, multiple servers work together as a grid without any central database or master and more servers can be added or removed on-demand.
Scale-out grid architectures are harder to build because they need to be designed from the ground up to not only distribute the workload across a set of processes but also need to provide fault-tolerance so if any of the processes fails the overall system is not impaired. Below is a screenshot of the Komprise elastic grid architecture. Read the Komprise Architecture Overview white paper to learn more.
- Scale-Out Storage
Scale-out storage is a type of storage architecture in which devices in connected arrays add to the storage architecture to expand disk storage space. This allows for the storage capacity to increase only as the need arises. Scale-out storage architectures adds flexibility to the overall storage environment while simultaneously lowering the initial storage set up costs.
With data growing at exponential rates, enterprises will need to purchase additional storage space to keep up. This data growth comes largely from unstructured data, like photos, videos, PowerPoints, and Excel files. Another factor adding to the expansion of data is that the rate of data deletion is slowing, resulting in longer data retention policies. For example, many organizations are now implementing “delete nothing” data policies for all kinds of data. With storage demands skyrocketing and budgets shrinking, scale-out storage can help manage these growing costs.
- Secondary Storage
Secondary storage devices are storage devices that operate alongside the computer’s primary storage, RAM, and cache memory. Secondary storage is for any amount of data, from a few megabytes to petabytes. These devices store almost all types of programs and applications. This can consist of items like the operating system, device drivers, applications, and user data. For example, internal secondary storage devices include the hard disk drive, the tape disk drive, and compact disk drive.
Some key facts about secondary storage:
- It is typically designed for long-term storage – its non-volatile media such as solid state devices, optical or magnetic storage devices such as tape.
- It is typically orders of magnitude cheaper than primary storage – it is designed for more capacity storage than performance.
- It can either be hosted on-premises at data centers or in the cloud.
- It can use file (Network Attached Storage NAS via NFS and SMB/CIFS protocols) or block-based storage-area-network (SAN) or object formats. Object based secondary storage is extremely popular today especially in the cloud.
- Examples include: Amazon Simple Storage Service (S3), Amazon Glacier, Azure Blob, Google Cloud ColdLine Storage, and on-premises object stores such as IBM Cloud Object Storage.
- Use cases include: Cold data storage, Cold data archiving, backup and Disaster Recovery storage.
- Data management software is used to find the right data to place on secondary storage and move data to secondary storage without user disruption.
Secondary storage typically archives inactive cold data and backs up primary storage through data replication or other data backup methods. This replication or data backup process, ensures there is a second copy of the data. In an enterprise environment, the storage of secondary data can be in the form of a network-attached storage (NAS) box, storage-area network (SAN), or tape. In addition, to lessen the demand on primary storage, object storage devices may also be used for secondary storage. The growth of organizational data has prompted storage managers to move data to lower tiers of storage to reduce the impact on primary storage systems. Furthermore, in moving data from more expensive primary storage to less expensive tiers of storage, storage managers are able to save money. This keeps the data easily accessible in order to satisfy both business and compliance requirements.
When archiving cold data to secondary storage, it is important that the archiving / tiering solution does not disrupt users by requiring them to rewrite applications to find the data on the secondary storage. Transparent archiving is key to ensuring that data moved to secondary storage still appears to reside on the primary storage and continues to be accessed from the primary storage without any changes to users or applications. Transparent move technology solutions that use file-level tiering to accomplish this.
- Shadow IT
Shadow IT is a term used in information technology describing systems and solutions not compliant with internal organizational approval. This can mean typical internal complacence is not followed, such as documentation, security, reliability, etc.
However, shadow IT can be an important source of innovation, and can also be in compliance, even when not under the control of an IT organization.
An example of shadow IT is when business subject matter experts can use shadow IT systems and the cloud to manipulate complex datasets without having to request work from the IT department. IT departments must recognize this in order to improve the technical control environment, or select enterprise-class data analysis and management tools that can be implemented across the organization, while not stifling business experts from innovation.
Ways to IT teams can cope with shadow IT are:
- Reducing IT evaluation times for new applications
- Consider cloud applications
- Provide ways to safely identify and move relevant data to the cloud
- Clearly document and inform business controls
- Approve Shadow IT in the short term
- Get involved with teams across your organization to help stay informed of upcoming needs
- Showback (“Shameback”)
A method of tracking data center utilization rates of an organization’s business units or end users. Similar to IT chargeback, the metrics for showback are for informational purposes only; no one is billed.
- SMB format (Server Message Block)
A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network. (also known as Common Internet File Systems (CIFS)).
- Storage Pool
What is a storage pool?
Storage pools are collections of storage volumes exported to a shared storage environment. Traditionally, storage pools were limited to storage volumes from a single vendor – for instance, you may have Flash and Disk storage volumes in a storage pool.
Storage pools may be homogeneous – that is, all the storage volumes are SSD/Flash, or all the storage volumes are disk, etc. or they may be heterogeneous – the storage volumes are different classes of storage e.g. Flash, Disk, etc. Storage tiering is an integral solution to handling heterogeneous storage pools.
What is storage tiering within a storage pool and why is it needed?
Storage tiering is a technique whereby the file metadata and the frequently-accessed blocks are stored in the highest tier and less-accessed blocks are downgraded to lower, cheaper tiers within a storage pool. This automated storage tiering approach allows the vendor to reduce costs by using smaller, faster tiers while still providing good performance. Storage tiering is often touted as a storage efficiency technique for customers to save on storage costs. But a key thing to remember is that the bulk of the cost of data is not in the storage but in the active management and backups of the data. Storage efficiency impacts the storage cost but not the active data management costs.
What is cloud tiering and how does it relate to storage pools?
Storage array vendors are now using their tiering technologies to tier data to the cloud. This is not what the technology was originally designed for, since the storage pool is no longer under a single vendor’s control and no longer local to a network.
Storage array vendors like NetApp and Dell EMC have created “Pool” solutions to externally tier data to less expensive storage such as in the cloud. Storage pools can reduce the cost of fast, expensive flash-based storage by migrating non-critical data sets to lower-cost storage for archiving and compliance and for “cold” data which hasn’t been accessed for a designated period of time to the cloud.
Read the blog post: What you need to know before jumping into the cloud tiering pool
What are the challenges and considerations for cloud storage pools?
While these solutions work well for tiering secondary data such as snapshot copies to the cloud, they result in unnecessary costs and lock-in when tiering and archiving files. As well, the pool approach tiers data in proprietary blocks versus files that all applications can understand. This presents the following challenges:
- Policies to specify the blocks to be tiered are limited, resulting in much higher access rate to the cloud, and higher egress costs.
- Block tiering to the cloud can reduce the performance of the storage array. Given the vast quantities of data most enterprises are dealing with today, block tiering is not suited for general data tiering to a public cloud across high latency channels.
- Block tiering locks you into your storage vendor. Since the cold data is tiered to the cloud in a proprietary format, when it is time to decommission your storage array and replace it with a new one you must stay with the same vendor.
- Proprietary lock-in. You cannot directly use native cloud services to access your data in the cloud. It has to be through the proprietary storage filesystem itself. This creates unnecessary licensing costs that customers must pay forever to access their data., resulting in much higher access rate to the cloud, and higher egress costs.
- Stubs
Placeholders of the original data after it has been migrated to the secondary storage. Stubs replace the archived files in the location selected by the user during the archive. Because stubs are proprietary and static, if the stub file is corrupted or deleted, the moved data gets orphaned. Komprise does not use stubs, which eliminates this risk of disruption to users, applications, or data protection workflows.
- Symbolic Link
What is a Symbolic Link?
Symbolic Links, also known as symlinks, are file-system objects that point toward another file or folder. These links act as shortcuts with advanced properties that allow access to files from locations other than their original place in the folder hierarchy by providing operating systems with instructions on where the “target” file can be found.
For the operating system, the symlink is transparent for many operations and functions in the same manner as the target file or folder would even though it’s only a link that points to the original. For example, if a program needs to be in folder A to run, but you want to store it in folder B instead, the entire A folder could be moved into the B folder with a symbolic link created in folder A which points to folder B. When the program is launched, the operating system would refer to folder A, find the symbolic link to folder B, and run the program from folder B as if it was still in its original place in folder A.
This method is widely used in the storage industry in programs such as OneDrive, Google Drive, and Dropbox to sync files and folders across different platforms of storage or in the cloud.
These types of links began to appear in operating systems in the late 70’s such as RDOS. In modern computing, symbolic links are present in most Unix-like operating systems which are supported by the POSIX standard such as Linux, macOS, and Tru64. This feature was also added to Microsoft Windows starting with Windows Vista.Symbolic Links vs Hard Links
Both types of symbolic links allow seamless and mostly transparent targeting of a file, but they do so in different ways.
Soft links, also referred to as symbolic links by Microsoft, work similarly to a normal shortcut in the sense that they point directly to file or folder itself. These types of links also use less memory overall.
On the other hand, hard links point to the storage space designated to hold the contents of the file or folder.
In this sense, if the location or the name of the file changes, then a soft link would no longer work since it was pointing to the original file itself, but with a hard link, any changes made to the original file or the hard link contents are mirrored by the other because both are pointing to the same location on the storage.
Hard links act as a secondary entrance to the same file or folder which they are linked to, but they can only be used to connect two entities within the same file system, whereas soft links can bridge the gap between different storage devices and file systems.
Hard symbolic links also have more restrictive requirements than soft links:- Hard links may not be able to link to directories.
- The target file or folder for a hard link must exist.
- Hard links cannot point to targets that are located on different partitions, volumes, or file systems.
Junctions
A Junction is a lesser-used, third type of symbolic link that combines aspects from both hard and soft links. The target file must exist for the junction to be created, but if the target file or folder is erased afterward, the link will still be there but will no longer be functional.
How are Soft and Hard Symbolic Links Commonly Used?
Hard links are used to create “backups” on filesystems without using any additional storage space. This is a benefit as it is often easier to manage a single directory with multiple references pointing to it rather than managing multiple instances of the same directory. If the file or folder is no longer accessible from its original location, then the hard link can be used as a backup to regain access to those files.
The Time Machine feature on macOS uses hard symbolic links to create images to be used for backup.
Soft links are used more heavily to enable access for files and folders on different devices or filesystems. These types of symbolic links are also used in situations where multiple names are being used to link to the same location.Types of Businesses that Make Use of Symbolic Links
Symbolic links are leveraged in nearly every industry that uses computers, but some industries make use of these links more than others. Below are industries where symbolic links are most commonly used.
- Firms that offer big data management or analytics services
- Engineering & Semiconductor companies
- Financial service firms
- Genomics & healthcare companies
- Higher education institutions
- Media & entertainment organizations
- Oil & gas producers
- Agencies in the public sector
Creating Symbolic Links
The process used to create symbolic links is different on each type of operating system. Below are brief instructions on how a soft or hard link can be set up in Linux and Windows.
How to Create a Soft Link in Linux
To create a soft symbolic link in Linux, the ln command-line utility can be used as such:
ln -s [OPTIONS] FILE LINK
The FILE argument represents the origin of the link. The LINK argument represents the target destination for the soft link.
When the command is successful, there is no output and the command-line will return zero.How to Create a Hard Link in Linux
For creating hard links in Linux, a similar version of the ln command is used but without the -s:
ln [OPTIONS] FILE LINK
The FILE argument is still the origin location and the LINK argument is still the destination file or directory.Creating a Windows Soft Link
The mklink command can be used to create soft links in Windows Vista & later through a command prompt or powershell with elevated permissions. By default, this command with no options will produce a soft link.
mklink command:
mklink Link TargetThe Link argument is the origin file/directory location and the Target argument represents the intended destination file.
For creating a soft link pointing to a directory, this command is used instead:
mklink /D Link TargetCreating a Windows Hard Link
Similarly to creating a soft link in Windows, the mklink can also be used to create hard links when /H is included as an option as such:
mklink /H Link Target
For creating a junction, the /J option is used instead of /H:
mklink /J Link Target
- S3
Shared-Nothing Architecture
A distributed-computing architecture in which each update request is handled by a single node, which eliminates single points of failure, allowing continuous overall system operation despite individual node failure. Komprise Intelligent Data Management is based on a shared-nothing architecture.