 |
| Google DeepMind's JEST Method Revolutionizes AI Training Efficiency and Speed Image credit: 9to5Google |
Google DeepMind, Google's AI research lab, has introduced a groundbreaking training method for AI models called JEST (Joint Example Selection), which claims to enhance training speed and energy efficiency significantly. This advancement arrives at a crucial time as the environmental impact of AI data centers is under scrutiny.
Traditional AI training techniques focus on individual data points, but the JEST method trains on entire batches. Initially, a smaller AI model grades data quality from high-quality sources and ranks the batches. This grading is then compared to a larger, lower-quality set. The small JEST model identifies the most suitable batches for training, and a larger model is subsequently trained based on these findings.
DeepMind researchers emphasize the importance of steering the data selection process towards smaller, well-curated datasets for the success of the JEST method. According to their paper, the JEST method achieves up to 13 times more performance and 10 times higher power efficiency than current state-of-the-art models, reducing iterations and computation significantly.
The success of the JEST method relies heavily on the quality of the training data. Without a meticulously curated dataset, the bootstrapping technique of JEST could falter. This requirement makes it challenging for hobbyists or amateur AI developers to replicate the method, as expert-level research skills are needed for the initial data curation.
The timing of this research is pivotal, as discussions about the power demands of AI are intensifying. AI workloads consumed approximately 4.3 GW in 2023, nearly equivalent to Cyprus's annual power consumption. Notably, a single ChatGPT request uses ten times more power than a Google search, and it's predicted that AI will consume a quarter of the United States' power grid by 2030.
Whether major AI players will adopt JEST methods remains to be seen. Training models like GPT-4 reportedly cost $100 million, with future models potentially reaching the billion-dollar mark. There is hope that JEST methods will maintain current training productivity rates at lower power draws, reducing costs and benefiting the environment. However, it's also possible that companies will continue to maximize power usage for rapid training output, focusing on cost savings versus output scale.
0 Comments