Meta Identifies Hardware Faults as Key Cause of AI Data Corruption and Proposes New Metric

Meta Identifies Hardware Faults as Key Cause of AI Data Corruption and Proposes New Metric

Meta has pinpointed another reason for poor AI performance: hardware faults that corrupt data.

According to a recent paper and a June 19 summary, hardware faults can lead to data corruption. This isn't new information—issues like "bit flips," where data values change from zero to one, are well-known and have been linked to cosmic rays affecting memory or hard disks.

Meta refers to these "undetected" hardware faults, or errors not caught and corrected immediately, as "silent data corruptions" (SDCs). Researchers indicate that these faults in AI systems cause "parameter corruption," altering the original values of AI model parameters.This can result in inaccurate or substandard output.

"If this occurs during AI inference/servicing, it can potentially lead to incorrect or degraded model output for users, ultimately affecting the quality and reliability of AI services," according to Meta's researchers.

While bit flips are not a new phenomenon and Meta has documented their occurrence within its infrastructure, addressing these undetected faults remains challenging. Meta's latest paper highlights that the complexity and diversity of AI hardware systems increase their vulnerability to hardware faults.

To address this, Meta suggests assessing hardware faults to help AI system builders understand the risks.

The researchers propose the "parameter vulnerability factor" (PVF), a new metric designed to standardize the assessment of AI model vulnerability to parameter corruptions.

PVF can be adapted to various hardware fault models and adjusted for different models and tasks.

"Additionally, PVF can be applied during the training phase to assess the impact of parameter corruptions on the model's ability to converge," the team stated.

The paper details Meta's simulation of silent corruption incidents using "DLRM," a tool for generating personalized content recommendations. They found that in certain situations, four out of every thousand inferences were incorrect due to bit flips alone.

This is presumably in addition to the usual accuracy issues with large language models (LLMs).

The paper concludes by suggesting that AI hardware designers consider PVF to balance fault protection with performance and efficiency.