I have been confused for a long time why FB is not motivated enough to invest in world models, it IS the key to unblock their "metaverse" vision. And instead they let go Yann LeCun.
This would be really cool if polished and integrated with VR.
Reminds me of this [1] HN post from 9 months ago, where the author trained a neural network to do world emulation from video recordings of their local park — you can walk around in their interactive demo [2].
I don't have access to the DeepMind demo, but from the video it looks like it takes the idea up a notch.
Edit: ah original Genie paper is 2024 [3]
[1] https://news.ycombinator.com/item?id=43798757
[2] https://madebyoll.in/posts/world_emulation_via_dnn/demo/
I keep on repeating myself, but it feels like I'm living in the future. Can't wait to hook this up to my old Oculus glasses and let Genie create a fully realistic sailing simulator for me, where I can train sailing with realistic conditions. On boats I'd love to sail.
If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game studios.
Isn't this still essentially "vibe simulation"? These's no grounding in exact physical models, which is what you'd want for a training sim.
...and then, the pneumatics in your living room.
This could be the future of film. Instead of prompting where you don't know what the model will produce, you could use fine-grained motion controls to get the shot you are looking for. If you want to adjust the shot after, you could just checkpoint the model there, by taking a screenshot, and rerun. Crazy.
I feel like people are already currently doing this. Essentially storyboarding first.
This guy a month ago for example: https://youtu.be/SGJC4Hnz3m0
What’s the endgame here? For a small gaming studio, what are the actual implications?
I understand the ultimate end goal to be simulation of life. A near perfect replica of the real world we can use to simulate and test medicine, economy, and social impact.
Google Deepmind Page: https://deepmind.google/models/genie/
Try it in Google Labs: https://labs.google/projectgenie
(Project Genie is available to Google AI Ultra subscribers in the US 18+.)
Demis stays cooking
Every character goes forward only, permanence is still out of reach apparently.