SLAC Researchers Develop AI Tool for Data Compression Without Losing Critical Scientific Details

SLAC Researchers Tackle Data Deluge with AI Tool

SLAC researchers at the Department of Energy's SLAC National Accelerator Laboratory developed an AI tool to address a critical challenge facing next-generation science experiments: managing unprecedented volumes of data without sacrificing scientific value1

1

. The AI-based method uses neural networks to compress large datasets while preserving subtle details that conventional compression techniques would erase. Published in Nature Machine Intelligence, this work responds to the reality that upcoming experiments will surpass current data storage and analysis capabilities2

2

.

"There is going to be such a flood of data that there's really no way to handle it in the way we've done before," said Joshua Turner, a lead scientist at SLAC and the Stanford Institute for Materials and Energy Sciences, who serves as principal investigator of this work1

1

. The urgency is real. SLAC's Linac Coherent Light Source (LCLS), an ultrafast X-ray free-electron laser, will eventually generate up to a million X-ray pulses per second, producing nearly one terabyte of data per second1

1

.

Preserving Science-Rich Details Through Data Compression

Conventional data compression methods strip away fine details that often contain the most valuable scientific insights. Yuan Ni, research associate at SLAC and lead author, explained that tiny speckles in X-ray images carry crucial information about material transformations. "Those speckles often reflect the underlying arrangement, disorder, or dynamics of a material," Ni said. "If we lose them, we would lose unique scientific insights, like how a material is structured and how that changes over time"1

1

.

The AI-based method achieves 10- to 100-fold reductions in file size while allowing users to control what information is preserved2

2

. The team tested this approach on diverse datasets, including measurements of molecules and materials from several experimental techniques, solar magnetic field measurements, and photographs. The neural network demonstrated remarkable adaptability, learning which features matter for different measurements1

1

.

How Wavelet Analysis and Neural Networks Work Together

Unlike traditional compression that treats entire datasets uniformly, this AI tool employs a two-stage process. First, it uses wavelet analysis to separate features by scale. Then, neural networks compress different-scaled features separately, ensuring finer details aren't lost through generalized compression2

2

. The neural network learns a compact representation of these features to preserve them.

This method also enables selective decompression of data. Researchers can decompress only the specific region they need rather than the entire file. "If you are using a conventional compressor, you would need to decompress the entire file, which could take you minutes, hours, or days," said Zhantao Chen, assistant professor at the University of Texas at Austin, who worked on this method while a SLAC research associate. "This method can decompress only the region of interest rather than the entire dataset, so it's much more efficient"1

1

.

Implications for Scientific Research and Data Management

The tool addresses immediate needs for instruments like the X-ray photon fluctuation spectroscopy (XPFS) instrument, which will study particle movements in exotic topological and quantum materials2

2

. Beyond lowering storage costs, this approach fundamentally changes how researchers interact with massive datasets. The ability to compress large datasets without losing critical details while maintaining fast retrieval could accelerate discovery across multiple scientific domains.

The researchers emphasize this isn't a replacement but an addition to existing workflows. "Rather than replacing existing compression methods," Ni noted, "our work provides an additional AI-based approach" that can operate alongside broader data-reduction techniques1

1

. To train the neural networks, the team used Perlmutter, a computational resource at the National Energy Research Scientific Computing Center (NERSC), a Department of Energy Office of Science User Facility2

2

. As scientific instruments grow more powerful, this AI-based method offers a path forward for managing data storage and analysis at scales previously considered unmanageable.