In Part One of my interview with Komprise co-founder and CEO Kumar Goswami, we talked about the power of Komprise Transparent Move Technology (TMT) for right-placing data and the importance of direct data access and staying out of the data path. In Part Two, we dig deeper into the barriers with legacy unstructured data management approaches and how TMT delivers a cleaner, simpler approach.
How is TMT different from stubs?
KG: Traditionally, solutions that move data have relied on one of two approaches:
- They either move the data entirely out of the primary storage, which is undesirable in most scenarios because it creates user friction as users think their files have disappeared.
- If they try to move data transparently, they then leave behind a proprietary “stub” file that points to the moved file.
Stubs are problematic for two reasons:
- A stub is proprietary and you need an agent installed on the file storage to detect when the stub is opened. This puts the data movement tool in the hot data path, which is undesirable, as it impacts performance and can become a bottleneck.
- Worse, since a stub is statically pointing to the new location of the file, if a stub is accidentally deleted, then data can get orphaned. Also, a stub can only go to another similar file system, so you cannot bridge file and object for example.
The bottom line is stubs are hard to manage. They need to be backed up, they’re in the data path, they’re limiting, and they are risky because they are a single point of failure. Komprise eliminates these issues by using a patented mechanism called Dynamic Links that uses standard protocol constructs and therefore eliminates proprietary agents and doesn’t get in the hot data path.
What are the advantages of Dynamic Links?
KG: TMT uses the standard, built-in feature of Windows, Linux, and Mac called symbolic links which replace a file with a tiny pointer to another location. By using our Dynamic Link inside the standard symbolic link, we can extend the file system to call these files from the cloud or other storage systems. Dynamic Links dynamically bind a request to the actual data so it can move a file from NFS or SMB to a native cloud object and still provide transparent access from the source.
As a user if you want to see how this works, click a file that Komprise tiered to the cloud and you get it back instantly. Simply right click the file and you will see that the path points to the Komprise Dynamic Link instead of a file on your computer.
Komprise TMT is a scalable, storage-agnostic data movement solution that maintains file and object duality to give you the best of both worlds – transparent data access from the source and native data access outside the data path.
With our patented Dynamic Links, Komprise stays outside the hot data path to deliver a standards-based open data management solution that is resilient and avoids the pitfalls of static stubs or symlinks.
What is the response to TMT from non-technical users?
KG: Asking users to change how they access data prevents adoption of tiering and requires significant IT involvement. Komprise developed TMT to ensure that data can be moved without disrupting users. Users don’t have to go to a separate access point or know where their data is after movement. They can see their moved data when they run the “ls” command in a Unix shell or use Explorer as if it is still on the source. This approach keeps users happy and enables cost-effective data management.
You like to say direct data access puts you in control of your data, not your storage vendor. Why is that so important today?
KG: With the advent of affordable big data analytics, the ability to index, search, and operate on all your data is a game changer. It’s also a fundamental requirement for ever-increasing compliance and legal data hold requirements. Maintaining direct data access to all your data without going through the original source is critical to meet these objectives.
Komprise makes this possible because it transfers your company’s NAS data to a cloud or secondary storage target as a file with its complete metadata, whether that target is on-premises NAS or object storage or cloud storage. The file is stored in the format native to the target storage device, and the metadata is stored in NFS or SMB format, depending on the protocol used to archive or copy the data.
You can get more details on the benefits of TMT in the white paper.
In summary, what are three takeaways about Komprise TMT?
KG: I would emphasize the following:
1. Komprise Transparent Move Technology moves data without proprietary interfaces such as stubs or agents. Stubs and agents are proprietary interfaces that are problematic to manage and brittle. Stubs can be problematic: if deleted, they can leave data orphaned, creating myriad problems for users and applications. Instead, Komprise has improved upon industry-standard symbolic links and invented a superior system to point to the moved data using dynamic links that are resilient and work across multi-vendor storage, with no user impact.
2. Komprise also provides the ability to “lift and shift” data to the target. In this case, a dynamic link is not left behind. The files are completely moved from the source file system to the target in a format native to the target. This approach makes sense to move data-heavy projects that are completed. In this approach, the data is accessible either directly from the target or by mounting Komprise as an NFS / SMB drive to view, list, drag-and-drop and operate on just the data that has been moved. Users/admins will need to know to go to this mounted drive. Note that even if the data is moved to an object store or the cloud, it can be accessed as objects from the target but also still accessed as files (with all the proper permissions) from the Komprise mounted drive. Other solutions are unable to provide this critical feature that requires less training and less disruption to the users even when using a “lift and shift” approach.
3. TMT = insight + action. Komprise policies can be set up to take automatic actions on a set of data (e.g., data by age, or a custom query of owner, location, etc.). As new data fits the policy, Komprise will systematically find and move data based on the policy without manual intervention. With Komprise you don’t need to manually run scripts and manage the transfers, thus cutting IT labor and overhead. Komprise provides reports of transfer errors and it automatically retries to address transient errors. Komprise provides a powerful and simple interface to administer policies, and it makes APIs available for scripting where custom actions are required.
I get most excited about Komprise customer success, so check out some of our great customer stories or set up a custom demo with our team to learn more. Be sure to also watch the Storage Field Day video below with our CTO and co-founder Mike Peercy to learn more about TMT and the Komprise architecture.
Read Part One: Transparent Move Technology Deep Dive with Kumar Goswami