At NeurIPS 2022, generative AI and LLMs are hot topics

Generative AI and LLMs were two of the hottest topics at NeurIPS 2022, which brought the AI ​​and ML community back in person for the first time since 2019 and provided “a lot of excitement,” said Alice Oh, a professor at the Korea Advanced Institute of Science and Technology. Technology and main program chair of the conference.

Part of that excitement may have been the sound of thousands of keyboards trying out OpenAI’s ChatGPT demo, which was released Wednesday and has since been the talk of Twitter, if not NeurIPS.

But the Conference and Workshop on Neural Information Processing Systems, a conference on machine learning and computational neuroscience being held in New Orleans this December, certainly had a lot of buzz of its own. According to conference leaders, more than 10,000 attended in person, with another 3,000 tuning in online. Just ten years ago, the event attracted less than 2,000.

In addition, more than 2,900 papers were accepted at NeurIPS from a whopping 9,634 submissions, covering topics ranging from neural networks and vision transformation to federated learning and offline reinforcement.

NeurIPS, as usual, was heavily focused on theoretical aspects of machine learning, Oh said, including building more efficient, accurate machine learning algorithms. But large language models (LLMs), diffusion models, and generative AI were also hot, trendy topics, along with reinforcement learning — which has been a top priority for years, she explained.

“When you go to the poster sessions, I think the generative AI models attract people, not just because the research is good, but also because they are fun to look at and to talk about,” she said.

Winning NeurIPS papers included MineDojo and Google Imagen

There were thirteen open paper receivers at NeurIPS, including two from Nvidia – one of which is described MyDojoa generalist AI agent that can perform actions based on written prompts in Minecraft.

Google AI won for a paper on its large text-to-image and super-resolution diffusion model, Imagen, which generates photorealistic images.

The Allen Institute for AI gained attention for its paper on ProcTHOR, a framework that generates interactive 3D environments used to train embodied AI. It takes the time- and resource-intensive process of building new virtual environments in which intelligent machines can be trained, and procedurally creates thousands of them, which has implications for the future of home robots, for example.

One of the winning papers, Gradient Descent: The Ultimate Optimizationwas written by MIT CSAIL and Meta researchers, including Erik Meijer, who headed Meta’s 50-person probability team who was laid off in early November.

Geoffrey Hinton says the future of computing is analog

In a closing virtual keynote at NeurIPS on Thursday, Geoffrey Hinton told the audience — as he told VentureBeat in September — that the future of computers is analog.

“What I think is that we’re going to see a completely different type of computer, not for a few years, but there’s every reason to look into this completely different type of computer,” he said.

These new ‘mortal’ computers will not replace traditional digital computers. “It won’t be the computer that manages your bank account and knows exactly how much money you have,” he said. “It will be used to put something like GPT-3 in your toaster for a dollar, so with a few watts you can have a conversation with your toaster.”

Hinton was asked to speak as part of the conference’s “Test of Time” award, which celebrates the 10th anniversary and “enormous impact” of “ImageNet classification with deep convolutional neural networks”, written with his graduate students Alex Krizhevsky and Ilya Sutskever and published in 2012. The paper, which is considered the beginning of the deep learning “revolution”, marked the first time a human-level convolutional neural network competed on the ImageNet contest for image recognition.

Hinton also released a new article, The forward-forward algorithmwhich he explained offers a new approach to neural networks called a forward-forward network, which differs from the backpropagation used in almost all neural networks.

“I think it could be something very new and important,” Oh said of Hinton’s new paper. “We’ll have to see, we’ll have to give it some time for people to process and try to replicate and make improvements.”

Ethical AI was “really big,” plus industry takeaways

Ethical AI topics were also at the center of the first week of NeurIPS, Oh said, with most of the invited speakers addressing the ethical and social implications of AI. This one included Alondra Nelsonwho leads the White House Office of Science and Technology Policy and speaks on the blueprint for an AI Bill of Rights.

Overall, there were several topics that were particularly relevant to the industry, Oh said. “One is reinforcement learning, which has been quite slow to pick up by the industry due to data or efficiency issues, but I think it will slowly but surely be adopted in the industry – to self-driving cars, robots or even natural language. processing – where reinforcement learning can be used to better predict states. ”

She also said that smaller, efficient models are an important NeurIPS topic, which will be important for companies that don’t have the computing resources of Google, Meta or Amazon.

“I think a really interesting aspect of research that can be applied is model distillation, like compressing the models to make them small, or transfer learning — to take a domain with a lot of data and then applying it to a domain where you don’t “I don’t have that much data,” she said. “That’s something that’s been going on for a while and will be very relevant.”