Australian Genome Research Facility simplifies data management for vital clinical research projects with S3-compatible object storage from Cloudian.
Cloudian has announced that the Australian Genome Research Facility (AGRF) has deployed Cloudian’s S3-compatible object storage solution, enabling AGRF to manage and store its rapidly growing volumes of genome sequencing data more efficiently and cost effectively at a vital time for the industry.
With facilities across Australia, AGRF empowers world-class genomic science in the biomedical, clinical, agricultural and environmental sectors, with over 15,000 clients worldwide. The organization has been at the center of essential COVID-19 research, from assisting researchers with vaccine development to detecting the virus in wastewater. AGRF is also distinguishing itself by playing a role in sequencing genomes of Australian threatened species to aid conservation efforts.
In an average week, AGRF processes at least 10 terabytes (TB) of data – equivalent to one year’s worth of data from the Hubble Space Telescope. Because this data is critical for driving some of the world’s most important research, AGRF needs a storage system that enables it to easily manage, protect and access the data at the lowest possible cost.
“We’re a data organization in every sense of the word,” said Douglas Morrison, AGRF ICT Manager. “Mapping the genome of one patient involves 100-125 gigabases, and easily three times that if we’re analyzing specific tumor samples. Multiply that by 1,500 patients or so in just one typical study, and the data adds up fast.
“Some of our clinical clients require urgent turnaround times for decisions on how to treat a patient – particularly in the case of cancer screening. We’re working on a project now where we’re aiming to significantly decrease our turnaround times to enable faster decisions. That accelerated process is possible because of how quickly we can handle all the data.”
AGRF had been using a tape backup system which involved 16 hours of manually swapping tapes in and out every week. This time-consuming process was a huge burden for its small IT team and had begun to impact operations.
“Because we had to spend so much time on backups, we weren’t always available to assist staff if they had an issue arise,” said Morrison. “In addition, if we needed to retrieve backup data for a project, they had to wait for us to locate the right backup tape and restore it.”
As its data volumes continued to grow, AGRF realized it needed a better solution. After evaluating alternatives including tape systems and public cloud, the facility chose the Cloudian HyperStore solution for its scalability, ease of management and cost-effectiveness.
“We liked that Cloudian provided the scalable capacity to meet our future growth needs without the hassle and time required in managing tapes and accessing data,” said Morrison. “In addition, the Cloudian HyperStore solution was half the cost of public cloud storage, a huge consideration for AGRF as a not-for-profit organization.”
AGRF deployed HyperStore appliances with a total capacity of 1.8 petabytes, using Cloudian’s HyperFile software to move data to HyperStore. The adoption of the Cloudian solution has brought huge benefits to the leading sequencing provider.
Cloudian has also enabled AGRF to increase its footprint in fast-growing applied markets and commercial diagnostic industries. In addition, the organization will leverage HyperStore in an upcoming project to map the genome of the infamous Australian box jellyfish, helping researchers to develop improved antivenoms and possibly therapeutic adjuncts.
“The importance of genome sequencing and mapping has become even more apparent over the past year in the effort to combat the pandemic,” said James Wright, Cloudian’s Regional Director, A/NZ and Oceania. “This industry is driving medical advancements all over the world and is key to beating the virus.
Reflecting AGRF’s leading role in the industry, the facility is leveraging its enhanced storage infrastructure to achieve better, faster research results that will enable continued innovation and ultimately improve people’s lives.”
We asked Douglas Morrison, AGRF ICT Manager, further questions to find out more.
Can you explain how the storage system is enabling your organization to: (1) manage (2) protect and (3) access data?
- HyperStore allows AGRF to use S3 object tags to easily categorize the data we generate and analyze. The auto-tiering feature grants AGRF flexibility when managing our data lifecycle. Coupled with HyperFile, legacy applications can still access data in a logical directory structure, without ever needing to know what is happening under the hood.
- Per-bucket storage policies allow differing levels of erasure-coding so AGRF can balance capacity utilization and data protection. S3 object versioning and data immutability ensures protection against both accidental deletion and malicious intent.
- Access via file and object allows AGRF’s internal applications native access to data. Using Cloudian for archival purposes ensures all data is accessible all the time. AGRF staff can now instantly access the data they require – reducing ICT overhead as there is no need to physically locate and load tapes.
What are the main advantages over your legacy system?
The most significant benefit for AGRF (and our valued clients) is a reduction in time taken to access and process data. This reduces the turnaround time for AGRF clients which can have life-changing consequences in cases like cancer screening. Our legacy archival system required the ICT team to first locate, then load LTO tapes so archives could be restored and processed. Efficiency has increased as AGRF staff have ready-access to the data they require and ICT staff can direct their efforts elsewhere.
Cloudian bought with it more performance, redundancy, resiliency and scalability over our legacy system coupled with an attractive cost when compared to other offerings. AGRF ICT staff do not need to be storage experts as installation and on-going management is simple. The entire solution was provisioned within days.
Will the Cloudian HyperStore solution be able to cope with your future requirements?
Absolutely. Cloudian HyperStore will scale to exabytes of object storage, ensuring continued use of the solution for years to come. The scale-out architecture means AGRF can add capacity without suffering the same performance issues as our legacy solution, while adding HyperFile nodes increases data access for legacy systems within AGRF. The multi-data center, multi-region, multi-tenancy and multi-controller features of the HyperStore and HyperFile products gives AGRF enormous flexibility for deployment in the future. Coupled with HyperIQ, AGRF has a single pane of glass for monitoring, reporting and analytics across the entire solution.
Why did you choose to work with the vendor?
It quickly became clear that AGRF and Layer 8 Networks (a Cloudian partner) were alike – going above and beyond for each of our respective clients. Layer 8 Networks took the time to understand the challenges AGRF faced as a non-profit organization, with five sites across Australia, dealing with substantial amounts of data. Their customer service and engagement were fantastic, asking probing questions we hadn’t previously considered. Subsequently, the Cloudian solution they presented ticked all the boxes – not only meeting AGRF’s current requirements but providing effortless scalability and management as AGRF continues to grow.Click below to share this article