GUIDE TO UNSTRUCTURED DATA PREPARATION FOR AI
“Building an effective data management value chain can lead to powerful and game-changing benefits. Forward-looking data-driven companies are bringing in a product mindset, managing the data like a product across its entire life cycle.”– Deloitte
NewYork-Presbyterian Achieves 96% Savings and 10x Faster AI Data Ingestion with Komprise
Healthcare IT infrastructure team reduces cloud costs, automates AI workflows and delivers the right information to digital pathology teams at the right time with Komprise.
Overview: Is your data prepared for AI?
CIOs and other IT leaders are embroiled in the most disruptive wave of technological change of their careers as AI continues its unstoppable impact on daily work, life and society at large. The days of thinking that AI might settle out and wind down as just the latest overhyped trend are over.
There is quite a lot to consider: from building out the proper hybrid IT infrastructure, to reskilling IT staff, training employees, selecting the best tools and determining viable use cases for Generative AI and AI agents. At the heart of AI, of course, is the data. Most of today’s data is unstructured data: user files, chats and texts, images, video, sensor data, instrument data, and much more.
In this guide, we delve into the data challenges and requirements of deploying AI in the enterprise. For AI initiatives to scale and avoid negative outcomes, IT must lead with systematic processes to classify, govern, and manage unstructured data efficiently and securely.
Blog: Is Your Data Ready for AI Inferencing?
Understanding the risks and challenges of unstructured data for AI
GenAI, for all its transformative qualities in the workforce, has become a massive headache for CIOs. The security, liability and credibility risks of inappropriate, ungoverned use of these tools are no joke.
In the Komprise IT Survey: AI, Data & Enterprise Risk, IT leaders reported “extreme worry” about shadow AI, and 80% say their organizations have experienced negative outcomes from generative AI. Avoiding sensitive data leakage to AI and protecting intellectual property and PII is the CIO’s top priority,
Preventing sensitive data from ingestion into AI pipelines is one issue. Culling only the right data for AI is equally imperative because AI is only as good as the data you feed it to achieve a particular goal. While it may seem simpler and easier to send large volumes of data into a data lakehouse to filter through and process later, this creates exorbitant storage and compute costs plus complexity.
Organizations are storing petabytes of data and billions of files, which includes rogue and irrelevant data. Further, you may have duplicate and near-duplicate copies of files that have been created over the years, adding to the cost and the sorting-out mess. Sending too much and/or the wrong unstructured data to locations for AI won’t deliver the results that data stakeholders want, either.
Watch the video: Komprise Data on the Move: Agentic AI and Unstructured Data
The truth is, most enterprise AI pilots don’t make it to production. Gartner estimates up to 60% will fail, often due to inadequate data readiness. IT leaders need to focus on addressing the following issues to prepare their unstructured data for AI.
What are the top barriers for AI data preparation?
1) Too many data silos with no central visibility and insights. Given that most organizations are storing data across multiple vendor systems from on premises to the cloud, it is difficult to understand, locate and access all the data needed for AI training and inference. This fragmentation can lead to incomplete or biased datasets.
Silos may also result in the same data being copied and stored multiple times across different systems, which increases storage costs and adds confusion about which dataset is the “single source of truth” for AI usage. Preparing unstructured data for AI requires efficient tagging and movement to compute-ready platforms. Silos make it harder to automate or scale these processes across the enterprise. Understanding your file and object data is a foundational first step.
What can Komprise Analysis do for you?
2) Lack of unstructured data classification.
System-generated metadata for unstructured data is too basic to be useful when searching for and curating precise data sets for analytics projects. To make this data useful, it needs additional structure and context to aid rapid, precise data curation.
Departmental users need easier ways to find the data they need, eliminating the need to dump large, irrelevant data sets into AI to process and filter. That adds unnecessary storage and compute costs and time. Yet classifying unstructured data by enriching metadata is often a manual, incomplete process that doesn’t scale. Large percentages of an organization’s data estate is therefore not discoverable and unavailable for AI, depleting competitive advantage.
Cracking the Code for Unstructured Data Classification
3) Incomplete AI data governance.
IT organizations need new policies and technologies for AI data governance. AI has introduced an entirely new set of risks and liabilities to organizations. AI is innovating quickly and IT leaders are struggling to keep up with the latest requirements to keep data safe and to avoid negative outcomes from AI projects.
AI data governance is the framework, policies, and procedures organizations put in place to ensure that data used in AI systems is managed and used in a responsible, ethical, and compliant manner. Comprehensive AI data governance programs and tools cover:
- Sensitive data detection to avoid IP, PII and other private data leakage into commercial models;
- Provenance and transparency of training data;
- Data labeling or tagging for accuracy and consistency;
- Bias detection and mitigation in datasets;
- Auditing of AI model inputs and outputs;
- Human verification of AI derived works and decisions.
4) Achieving high unstructured data quality for AI is elusive.
With structured and semi-structured data, a common practice of the past has been to send files en masse to a data warehouse, data lake or data lakehouse where data engineers, data scientists and analysts can access it over and again for different projects. This model does not work for unstructured data which is much larger, expensive to store, heavier and difficult to move.
A data lake with petabytes of unstructured data becomes unwieldy data swamps that are hard to search. You’re also copying a healthy percentage of junk that will deliver poor and even dangerous results from AI, especially if it’s not filtered and classified before someone gives it to an AI prompt.
Getting data quality from AI demands a different approach. Your users need simpler ways to search and cull the right data in place before moving any data to a data lakehouse or AI engine.
TechVoices: Komprise’s Krishna Subramanian: AI and Data Management
5) Slow, difficult, costly process for feeding data to AI pipelines.
Per above, copying petabytes of data into other platforms and tools for AI is also financially untenable since this requires ample high-performance storage and AI compute processing. Even in the cloud, you could see annual IT infrastructure costs doubling or even tripling from AI. The iterative nature of AI workflows means that IT will need to move data to different processors repeatedly, multiplying your costs, especially if the data is retained after the processing is complete.
Duquesne University Finds and Tags Digital Images 99% Faster
The Komprise AI Data Readiness Model
Based on working with enterprises managing hundreds of petabytes of unstructured data, Komprise has identified five stages that determine whether an organization’s unstructured data is genuinely ready for AI use. Most enterprises begin at Stage 1 and progress through the stages as they implement systematic data management practices. AI projects launched without reaching at least Stage 3 have a significantly higher failure rate, consistent with Gartner’s finding that up to 60% of enterprise AI projects fail due to inadequate data readiness.
Stage 1: Visibility
The organization has no unified view of what unstructured data exists, where it lives, or how it is growing. Data is fragmented across multi-vendor NAS and cloud storage environments with no central index. AI projects at this stage rely on whatever data developers can manually locate, which produces biased, incomplete, and ungoverned datasets.
Komprise entry point: Komprise Analysis and the Global Metadatabase index all file and object data across the entire storage estate, providing unified visibility as a starting point for all subsequent stages.
Stage 2: Classification
Data is indexed but lacks the business context needed to distinguish valuable files from noise. System metadata covers basic file properties but cannot identify project relevance, sensitivity, domain context, or AI suitability. AI pipelines at this stage ingest large volumes of data and filter downstream, which increases compute costs and degrades model accuracy.
Komprise entry point: Komprise Deep Analytics queries the Global Metadatabase using metadata and custom tags to classify data precisely by any business criteria. KAPPA data services extract domain-specific custom metadata from file content and store it as first-class searchable attributes.
Stage 3: Governance
Data is classified but sensitive content, regulated data, and unauthorized files are not systematically excluded from AI pipelines. Organizations at this stage face compliance risk, IP exposure, and the possibility of AI models being trained on or reasoning from data they were never authorized to access.
Komprise entry point: Komprise Smart Data Workflows include sensitive data detection covering PII and regex-based classification, automatically excluding governed data from AI ingestion before it reaches any model or agent.
Stage 4: Curation
Governed data is available but AI teams must manually identify, request, and stage datasets for each new project. This creates a bottleneck at IT and slows AI program velocity. Data quality varies by project because curation is not systematic or repeatable.
Komprise entry point: Smart Data Workflows automate ongoing curation by applying policy-based rules that continuously identify new or updated files matching defined AI use case criteria and deliver them to the right AI destination in native format on a defined schedule. Deep Analytics user profiles allow authorized data owners and researchers to self-curate datasets within their own directories without IT involvement for each request.
Stage 5: Optimization
Curation is automated but AI infrastructure costs are not optimized. Cold data, dark data, and ROT data occupy expensive primary storage alongside active AI datasets, increasing infrastructure costs unnecessarily. AI pipelines run on more data than they need because the boundary between valuable and irrelevant data has not been continuously maintained.
Komprise entry point: Komprise Intelligent Tiering continuously right-places cold and inactive data off primary storage, keeping it accessible for any AI workflow that needs it via Dynamic Links and the Global Metadatabase while freeing premium capacity for active AI workloads. The Flash Stretch Assessment quantifies exactly how much primary storage capacity can be reclaimed before the next infrastructure investment decision.
Where does your organization sit on the Komprise AI Data Readiness Model? Contact Komprise to find out.
5 tactics to manage and prepare unstructured data for AI
As organizations ramp up their use of AI, IT infrastructure teams are playing a greater role in preparing data for smarter, safer use. This means gaining clear visibility into file and object data across all systems, tagging and organizing it for AI workflows, and making sure sensitive information is not jeopardized.
The old ways of moving and preparing data don’t work well for unstructured data nor for AI’s complex needs. To succeed, teams need modern tools to classify, manage, and move only the data that matters—saving money, improving outcomes, and lowering security and privacy risks.
Check our blog channel on AI-Ready Data.
- Get unified visibility across data silos: Independent unstructured data management solutions can work across all your silos, index metadata to deliver insights on data growth, file types and sizes and user access trends and move data wherever you wish without lock-in. This saves money and time and ensures that you are managing data appropriately for its use case and value in the moment. You can integrate storage-agnostic unstructured data management with any desired tools for additional analytics or specialized functions such as metadata enrichment and PII protection. Komprise is built on a global metadatabase which can be the hub for all of your unstructured data management actions, including preparing data for AI via metadata enrichment and automated Smart Data Workflows.
Adopt the appropriate data preparation modality for AI. The traditional extract, transform, load (ETL) model falls short for unstructured data used in AI because AI workflows are iterative, multistage, and nonlinear. Using a global metadatabase that indexes data with metadata tagging across all storage environments supports intelligent data curation. AI requires metadata indexing, user-driven data tagging, and built-in governance with sensitive data detection and lineage tracking. Komprise Smart Data Workflows deliver an easy UI to discover, enrich and classify data, confine sensitive data, move the right data to AI and even integrate third-party processors for specialized actions such as image identification. Read: Preparing unstructured data for AI? Forget ETL.-
Komprise Intelligent AI Ingest Power AI with the right data at the right time. With full visibility, analysis and a system to query across all data, your departments can create repeatable, curated unstructured data pipelines to AI. Use an unstructured data management solution that supports user-based tagging, such as clinical researchers tagging files by demographics and diagnostic codes. AI-based content indexing tools can inspect files and tag them rapidly and accurately. By bringing specificity to AI data workflows, employees can send the right files and no more to AI. Read about Komprise Intelligent AI Ingest, a 2025 update that delivers precise curation for RAG with 2X faster ingestion speeds than leading cloud sync tools.
Deliver trusted data for AI (PII) Komprise Smart Data Workflows delivers both standard PII detection and custom (regex and keyword) sensitive data detection. After detection, the solution automatically tags it in the metadatabase (global file index) and IT can set policies to confine or move data to a safe location. You can set up automated workflows to identify and exclude sensitive data from the data that is searchable and available for AI ingestion. Read more about sensitive data management.- Revisit skills/staff requirements for AI. Storage IT professionals are increasingly managing data movement and access across complex hybrid cloud and multi-vendor environments while addressing security threats from AI and cyberattacks. They need new tools and tactics to manage infrastructure and govern data workflows for AI. Key strategies include:
- Establish processes for departmental collaboration on AI and analytics initiatives to understand new requirements.
- Track metrics like data volume, growth, cold and hot data, data access trends and more;
- Use FinOps capabilities in unstructured data management to optimize storage and move cold data to cost-effective tiers;
- Mitigate risks of corporate data from ransomware by using immutable storage in the cloud for inactive data.
- Deliver AI-ready storage and compute resources (CPUs, GPUs, TPUs) to support model training and deployment.
- Prepare data for analytics and AI use with automated workflows and data classification techniques and deliver rapid search and tagging capabilities for department managers.
- Protect sensitive data from leaks by segregating private data, implementing audit trails, and establishing governance frameworks.
Learn more about Smart Data Workflows and AI-Ready Data from Komprise.
What is unstructured data management?
What is AI data preparation?
What is AI data management?
AI Data Preparation FAQs
What is AI data preparation for unstructured data?
AI data preparation for unstructured data is the process of discovering, classifying, enriching, governing, and curating file and object data (documents, images, video, medical scans, sensor data) so it is accurate, relevant, and safe for AI pipelines. Unlike structured data, unstructured data has no inherent schema, making it hard to find, filter, and move at scale. Key steps include:
- Building a global metadata index across all storage silos
- Enriching metadata to make files discoverable and AI-ready
- Detecting and excluding sensitive data (PII, IP, PHI) before ingestion
- Filtering out duplicate, outdated, and irrelevant files
- Automating governance and audit trails for compliance
According to Gartner, up to 60% of enterprise AI projects fail due to inadequate data readiness, making AI data preparation a critical first step.
Why is unstructured data so difficult to prepare for AI?
Unstructured data is harder to prepare for AI than structured data because it lacks schema, is scattered across silos, and is too large to process manually. The four core challenges:
- No central visibility: data is fragmented across on-premises NAS, cloud object stores, and hybrid environments with no unified namespace
- Metadata is too shallow: system-generated metadata (timestamps, file size) is insufficient for precise AI curation; useful context must be extracted from file content
- Massive noise: enterprise data estates contain billions of files including duplicates, outdated content, irrelevant data, and sensitive information that degrades AI accuracy
- Governance gaps: tracking what data was ingested, by whom, and when requires capabilities that traditional ETL tools were not designed to provide
The Komprise 2026 State of Unstructured Data Management report found that classifying and tagging unstructured data is the #1 challenge in AI data preparation, cited by 56% of IT leaders.
What techniques improve AI performance with noisy enterprise unstructured data?
AI models, RAG pipelines, and agentic AI systems perform significantly better when the data they operate on is relevant, current, well-classified, and free of noise. Noisy enterprise data, including cold archives, duplicate files, stale research, sensitive content, and files with no business context metadata, degrades AI accuracy, increases inferencing costs, and creates compliance risk. The following techniques address this systematically:
- Eliminate ROT data before ingestion. Redundant, obsolete, and trivial data is one of the leading sources of AI noise. Duplicate files, abandoned project folders, zero-byte files, and superseded datasets consume context window capacity and cause models to reason from outdated or irrelevant information. Identifying and removing ROT data before it reaches an AI pipeline using precision queries, rather than broad deletion policies, reduces noise at the source without risking accidental deletion of valuable data.
- Classify and tag data with business context before ingestion. Standard file system metadata covering file name, size, and timestamp is insufficient for AI systems to evaluate data relevance. Custom metadata extracted from file content using KAPPA data services, such as project codes, clinical parameters, document types, or sensitivity classifications, gives AI systems the context they need to filter and prioritize data effectively. Tags are first-class searchable attributes in the Komprise Global Metadatabase and perform at the same query speed as standard metadata across billions of files spanning multiple file and object data stores and sites.
- Exclude sensitive data before it reaches a model. Data pipelines that do not detect and exclude PII, regulated content, and confidential IP before ingestion create compliance risk and can cause AI outputs to expose protected information. Sensitive data detection built into Komprise Smart Data Workflows identifies PII and content matching regex-based classification patterns automatically, excluding governed files before they enter any AI pipeline.
- Tier cold and inactive data off primary storage before building AI datasets. When AI pipelines ingest from primary storage without filtering inactive files, they process cold data alongside active working data, increasing token usage and reducing the signal-to-noise ratio of every inference. Moving cold data to lower-cost storage via Intelligent Data Tiering before AI dataset construction ensures pipelines operate on current, active data. With Komprise, cold data remains accessible in native format via Dynamic Links for any AI workflow that legitimately needs it.
- Use precision queries to curate datasets rather than ingesting broad file shares. Komprise Deep Analytics searches the Global Metadatabase using any combination of metadata and custom tag criteria to find exactly the files relevant to a specific AI use case, rather than ingesting entire directories or file shares. A precision query like “all research files tagged with project code X, owned by department Y, modified in the past 18 months, not classified as sensitive” delivers a highly targeted dataset. Deep Analytics Actions turns that query into the direct input to a data mobility policy or Smart Data Workflow, automating curation at scale.
- Maintain ongoing AI data curation rather than one-time preparation. AI pipelines degrade over time if the underlying data estate is not continuously maintained. New files arrive, existing files become stale, and business context changes. Komprise Smart Data Workflows run on a schedule to continuously identify new or updated files matching defined curation criteria, apply tags, exclude newly identified sensitive content, and deliver refreshed datasets to AI platforms automatically. This keeps AI pipelines current without requiring manual intervention for each update cycle.
Applying these six techniques together addresses the full spectrum of enterprise data noise: structural noise from ROT and cold data, contextual noise from missing or inconsistent metadata, and governance noise from undetected sensitive content mixed with authorized data.
How do Komprise Smart Data Workflows and Intelligent AI Ingest automate AI data preparation?
Komprise Smart Data Workflows automate the full process of finding, classifying, curating, and ingesting the right unstructured data to any AI service, without manual effort. Key capabilities:
- Policy-driven curation: rich queries across all storage silos via the Global Metadatabase find exactly the right files for each AI use case
- Noise elimination: filters out 70%+ of unstructured data that would erode AI accuracy, including duplicates, outdated files, and sensitive content
- Intelligent data ingestion: copies only curated, governed data to the AI destination at 2x faster transfer speeds, leaving originals in place
- Full data lineage: maintains complete audit trails of what was ingested, when, and by whom for governance and compliance
- Proven at scale: NewYork-Presbyterian achieved 10x faster AI ingestion and 96% lower cloud costs for its digital pathology AI program using Komprise
How does Komprise protect sensitive data and maintain governance during AI data preparation?
Komprise combines a unified metadata layer with built-in sensitive data detection to ensure only safe, governed data reaches AI pipelines. How it works:
- Global Metadatabase: continuously indexes metadata across all NAS, cloud, and object storage, including PII status, sensitivity tags, and custom labels, without moving the data
- PII and PHI detection: built-in scanners plus custom regex and keyword search find sensitive data before it reaches AI tools
- Automated remediation: sensitive files can be confined, excluded, or moved to secure storage by policy
- Audit trails: every ingestion workflow logs who ingested what, from where, and when for GDPR, HIPAA, and IP compliance
- Attack surface reduction: removing sensitive and cold data from primary storage shrinks the ransomware attack surface by up to 80%
80% of IT leaders in the Komprise AI Data and Enterprise Risk survey cited sensitive data leakage into AI tools as a top concern.
What is Komprise KAPPA and how does it enable custom metadata enrichment for AI?
KAPPA (Komprise AI Preparation & Process Automation) is a serverless metadata enrichment platform that lets IT teams create custom metadata extraction functions using a few lines of Python, no infrastructure to provision or manage. What KAPPA enables:
- Industry-specific extraction: reads custom headers from medical DICOM files, genomics BAM files, or any proprietary format
- Enterprise context tagging: applies ERP project codes to R&D files, invoice status to media assets, or AD security labels to classify data by ownership
- Sensitive data handling: masks PII, imports sensitivity labels into Microsoft Purview, and flags regulated content before AI ingestion
- Global Metadatabase integration: all enriched metadata is stored and searchable in the Komprise Global Metadatabase, reusable across future AI workflows
- Agentic AI ready: KAPPA functions can be invoked directly by AI agents at runtime, enabling dynamic data preparation on demand
KAPPA goes beyond traditional ETL approaches of building custom connectors that can take months to build and maintain, delivering custom, governed metadata at petabyte scale in a fraction of the time.
