Getting to the Heart of your Unstructured Data

AI is a pervasive and controversial topic today. It dominates the headlines but has also given ample cause for concern. In my view, we are on the cusp of a transformational stage of existence with AI. However, in addition to developing AI strategies enterprise IT execs have bottom-line pressures to face in a still-difficult economy, security threats that are becoming harder to thwart and increased pressure to contain costs. Even so, business leaders don’t want to be left behind in the AI age. AI initiatives require large volumes of unstructured data. If you can’t find it and move it safely and efficiently to the right cloud services and tools, you’re missing out. That’s why getting your arms around your growing volumes of unstructured data and harnessing this data to deliver value will be a top priority in 2024.

There’s a lot at stake, and it points to the need to understand your unstructured data at a deeper level so you can do things you could never do before.

  • A core feature of the Komprise Intelligent Data Management platform is Deep Analytics. As Komprise analyzes your data across data silos, all the metadata is stored in a single, highly distributed Global File Index (GFI). Deep Analytics provides a simple UI that leverages the GFI so users can search across silos with a single global search.
  • With Smart Data Workflows (SDW), we take it one giant step further. SDW allows you to create custom workflows that can use your AI application or any third-party AI service to scan your files and store the findings in the GFI. You can find protected data quickly and create governance and compliance applications with Komprise, all using our simple UI.

This ability to understand your unstructured data at a granular level, regardless of which storage or cloud it lives in, and the ability to then move and manage it based on those insights is a game changer.

Below I’ve laid out three core areas where Komprise Deep Analytics is helping our customers answer questions they never could before and deliver meaningful benefits to their organizations.

1. Cost savings with unstructured data analysis

Komprise delivers core metrics to understand costs and create plans to save money, such as: rate of data growth, storage usage per department, cost savings opportunities by tiering cold data to archival storage and information such as top file types, top file sizes and top data owners. Here are two examples:

  • Searching across shares for cold data tiering opportunities. A multi-billion-dollar biomedical company generates huge amounts of instrument data every day. In the past, IT deleted some of the data over time. But due to the cost of regenerating the data if needed and regulations which require data retention for seven to 25 years, the company ceased deleting data. This resulted in hefty on-premises storage purchases, so IT decided to shift to the cloud. Using Komprise Deep Analytics to analyze access patterns and cold data by share and department, they were able to tier petabytes of cold data to cheaper tiers in the cloud– saving money and reducing the need to request approvals to buy more storage. They also love that Komprise leaves behind links so users can still access the data if needed and users can access the data in the cloud natively without requiring Komprise–supporting several AI efforts underway.
  • Identifying and deleting duplicates is another great use case for Deep Analytics. For instance, the marketing department copies 50,000 images for use in a global marketing campaign. Once the program ends, the images are no longer needed. You can run a query to search for duplicate image files and then move them off your expensive storage quickly; you’re cutting costs and freeing essential space on your primary storage.

2. Answering questions about unstructured data on the fly

When you have questions about your data, it can be difficult or even impossible to get the answers when managing petabytes of data across many different silos. Yet Komprise Deep Analytics allows you to ask all sorts of interesting questions, some of which are critical to daily operations. For security compliance, Deep Analytics allows you to investigate your data estate to ensure there are no files that should be removed or if certain files are not located in secure locations where they belong. Deep Analytics is a simple way to ensure your entire data estate is compliant. Here are some examples:

  • Discovering and correcting compliance issues. A data center manager at an oil and gas company who was asked to clean up several data silos leveraged the global search provided by Deep Analytics. Instead of repeating the effort for each silo, he leveraged Deep Analytics and found many old .PST files spread across the organization. He showed the report to the department manager to get buy-in to delete the files. In another case, after an acquisition, Deep Analytics found a large volume of files from an unsupported productivity application sitting on a file server. They posed a security risk and wasted precious storage space, so IT promptly deleted them.
  • Finding critical data across multiple data silos. A large European engineering construction company which has grown by acquisition had to find critical soil data regarding a project which was affected by an earthquake. An IT manager located the data across the silos of its many small engineering firms in minutes, using Komprise Deep Analytics. Without it, they used to make calls, send emails and wait for responses from the various firms.

3. Leveraging AI to find the precise data sets.

AI is punitively time-consuming and expensive if you must copy massive buckets of files to an AI application. For AI to be viable, it’s important to find and send the precise data sets to AI. In some cases, such as when you have lots of data at the edges, it is better to bring the AI application to the data. Komprise makes this possible. After processing the data with AI, you want to save the results so that you can use them anytime without having to repeat the analysis. Here are a few real-life examples:

  • A university is deploying Komprise for AI-aided image recognition and tagging to support marketing and fundraising projects. With millions of files, the marketing team surmised it would take several months just to find the images they needed for a big campaign. Rather than an arduous manual process of looking through thousands of images in storage, they used Komprise SDW to feed only image files to Amazon Rekognition. The image analysis tool tagged the files in the GFI with the required metadata ( e.g. tag images containing university buildings with GPS coordinates). The process took under two hours to complete and allowed the marketing team to get their campaign launched in time.
  • Finding and segmenting sensitive data based on PII. Smart Data Workflows (SDW) can create a continuous process to scan all new files through a personal identifiable information (PII) scanner. This could be a third-party scanner or a cloud service like Amazon Macie, which then tags files containing PII information in the Komprise GFI.  You could use Komprise to tier cold data with PII to AWS Gov Cloud and those without PII to the cheaper regular AWS cloud. You can set this up as something that happens all the time – ensuring continuous cost savings that comply with regulations. You can bet your security team will love this capability!

I’m very excited about our approach of providing one global search across data silos and the ability to enrich the metadata it contains through SDW and AI. It opens a host of data-driven management activities which were never possible until now. That’s the power of Komprise Intelligent Data Management and Deep Analytics. I look forward to hearing about your requirements, sharing more use cases with you and helping you bring structure to your unstructured data in 2024.

Happy Holidays!


Getting Started with Komprise:

Contact | Data Assessment