OpenAI Introduces Tool to Gauge AI Agents’ Machine Learning Abilities

OpenAI releases MLE-bench, an AI testing tool that checks how well machine learning agents do in engineering and how creative they can be.

Open AI researchers have made a tool that AI makers can use to test how well AI can do machine-learning engineering. The group wrote a paper about their benchmark tool and put it on the arXiv preprint site. They called it MLE-bench. The team has also put up a page on the business website to talk about the open-source new tool.

Over the past few years, computer-based machine learning and the artificial apps that go with it have grown. As a result, new types of applications have been tried out. One of these uses is machine-learning engineering, which uses AI to solve engineering puzzles, run tests, and develop new code.

The goal is to lower engineering costs while speeding up the creation of discoveries or finding new ways to solve old problems. This will allow more new products to be made more quickly.

Some people in the field have even said that some types of AI engineering could lead to the creation of AI systems that are better at engineering work than humans, which would no longer need humans to do the work. Others in the field are worried about the safety of future versions of AI tools and the idea that AI tech systems might figure out that humans are no longer needed at all.

OpenAI’s new measuring tool doesn’t directly address these worries, but it does leave the door open for making tools that stop either or both of these things from happening.

The new tool is just a bunch of tests—75 of them from the Kaggle site in total. As part of testing, you give a new AI a lot of problems to answer. All of them have to do with the real world, like asking a computer to read an old scroll or create a new kind of mRNA vaccine.

The system then checks the answers to see how well they were answered and whether they can be used in real life. If they can, they are given a score. Likely, the OpenAI team will also use the results of these kinds of tests to judge the growth of AI research.

Notably, MLE-bench tests AI systems on their ability to do engineering work on their own, which includes coming up with new ideas. I think that the AI systems being tested would need to learn from their work, maybe even from their MLE-bench scores, to do better on these kinds of tests.