Big data analytics and AI are making an ever-greater impact in the healthcare industry. There are opportunities across the board, from speeding up genomics research and drug discovery to improving medical image analysis, clinical decision making, prevention and wellness programs, chronic disease management, clinician productivity and beyond. The interplay of unstructured data, AI and high-performance computing (HPC) is making all this possible.
HPC, in focus at SC24 this week in Atlanta, provides the computing infrastructure required to efficiently analyze large and complex data sets. HPC made headlines in healthcare during the pandemic with the COVID-19 HPC Consortium, a private-public collaboration that supported dozens of research projects to accelerate early-stage drug development.
The HPC market is expected to reach $49.9 billion in 2027, up from $36 billion in 2022, according to MarketsandMarkets. Healthcare, which creates an estimated 30% of the world’s data, will undoubtedly make up a healthy slice of this pie.
Today, leading healthcare organizations like NYU Langone Health are developing ambitious AI projects running on HPC. “Our eventual goal is that all text generated by or for us will be touched by a large language model, whether in our education, clinical, research, or operations missions,” said Yin Aphinyanaphongs, MD, PhD, director of operational data science and machine learning at NYU Langone Health. “AI will transform how we care for patients, run hospitals, write research grants, and train medical students.” Underneath the covers at the institution is an HPC cluster called Ultraviolet, which supports a range of programs, including training sophisticated AI models.
Komprise for HPC: Scalable, Indexed, Open, SaaS
Komprise has been helping enterprise organizations manage petabyte-scale data environments for more than eight years. The core tenets of our architecture include:
- Built on a distributed, scalable, fault-tolerant architecture of stateless observers placed near the storage where they are most effective at analysis and mobilization.
- Global File Index and a centralized management console through which you can view and manage all unstructured data across storage silos.
- The platform is standards based (NFS, SMB, S3), with no agents or stubs and it is never in the hot data path.
- Delivered as a cloud service and is easy to set up and easy to use.
Komprise in Healthcare and Life sciences
Healthcare and life sciences organizations represent one of our company’s top sectors. Learn more here. One customer, a U.S.-based academic healthcare system with several hospitals and a medical school faces data challenges like many other healthcare groups today: rapid growth of large medical image files and research data, which is constraining IT resources to store, protect and manage data assets.
The healthcare system adopted Komprise Intelligent Data Management for unstructured data tiering, migration and visibility across their data storage silos. Komprise indexes all the file and object data across storage to show insights which help IT teams and departmental research groups make better decisions.
The Komprise dashboard shows metrics such as data growth rates, volume and most common file types, and then gives IT users options to model different plans for cost savings. Storage engineers can also use Komprise to classify data through metadata tagging and move it to new storage as needed.
- The storage team has moved more than 2PB of medical files over one year old to cloud object storage—freeing significant space on its NAS storage arrays and saving the organization 70% on its annual storage and backup budget.
- With these savings, the healthcare system is freeing up funds for AI projects that rely upon clinical documents, images and research files from HPC applications.
- The organization also plans to use Komprise Smart Data Workflows to automate the search, tagging, and curation of specific data sets needed by data scientists for research projects.
Komprise at SC24
Komprise is a sponsor at the SC24 trade show and conference in Atlanta this week. Please stop by our booth, #414, to meet the team and learn more about how we’re helping HPC teams and researchers get more value from unstructured data, gain maximum storage efficiencies and achieve more protection from ransomware.
Read the previous blog about how unstructured data management is relevant for petabyte-scale HPC IT environments.
