This is pretty cool, but I feel as a pokehunter (Pokemon Go player), I have been tricked into working to contribute training data so that they can profit off my labor. How? They consistently incentivize you to scan pokestops (physical locations) through "research tasks" and give you some useful items as rewards. The effort is usually much more significant than what you get in return, so I have stopped doing it. It's not very convenient to take a video around the object or location in question. If they release the model and weights, though, I will feel I contributed to the greater good.
This title is editorialized. The real title is: "Building a Large Geospatial Model to Achieve Spatial Intelligence"
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
My personal layman's opinion:
I'm mostly surprised that they were able to do this. When I played Pokémon GO a few years back, the AR was so slow that I rarely used it. Apparently it's so popular and common, it can be used to train an LGM?
I also feel like this is a win-win-win situation here, economically. Players get a free(mium) game, Niantic gets a profit, the rest of the world gets a cool new technology that is able to turn "AR glasses location markers" into reality. That's awesome.
I'm pretty sure most of the data is not coming from the AR features. There are tasks in the game to actually "scan" locations. Most people I know who play also play the game without the AR features turned on unless there's an incentive.
That's good information, thank you!
It's OK to adjust the title to have more relevant facts or to fix a poorly worded one. Editorializing is more like 'Amazing: Niantic makes world-changing AI breakthrough'.
The original title was not poorly worded though. The new one was editorialized to get a certain reaction out of readers — I promise you the responses on this thread would look different with the original title.
The Harry Potter game had much better AR integration
I feel like I'm going mad, if you actually read the article it's a theoretical thing they'd like to lead in, yet literally every comment assumes it launched. The title being "announces model" rather than the actual title certainly doesn't help.
All they needed was a shit ton of pictures. The AR responsiveness (and Pokemon Go) have nothing to do with it. It was just a vehicle for gathering training data.
Not wanting to over-do it, but is there possibly an argument the data about geospatial should be in the commons and google have some obligation to put the data back into the commons?
I'm not arguing to a legal basis but if it's crowdsourced, then the inputs came from ordinary people. Sure, they signed to T&Cs.
Philosophically, I think knowledge, facts of the world as it is, even the constructed world, should be public knowledge not an asset class in itself.
Four Square just open sourced their places dataset. https://location.foursquare.com/resources/blog/products/four...
Given how expensive it is to query Google places, would love a crowdsourced open-source places API.
I’ve been saying this about Google Maps for years, especially their vast collection of public transport loading data and real time road speeds.
People are duped into thinking they’re doing some “greater good” by completing the in-app surveys and yet the data they give back is for Google’s exclusive use and, in fact, deepens their moat.
[delayed]
It's not solely for Google's benefit. They're ("we're" tbh) contributing data that improves services that we use. It has additional selfish and altruistic benefits beyond feeding the Googly beast.
As a Google maps user, I benefit from that data being in there.
Do you expect every company to release all their data to the public as well or it's just because you're not invested in this one?
I expect any company which collates information about geospatial datasets to release the substance of them, yes. Maybe there's an IPR lockup window, but at some point the cadastral facts of the world are part of the commons to me.
I would think there's actually a lot of epidemiology data which also should be winding up in the public domain getting locked up in medical IPR. I could make the same case. Cochrane reports rely on being able to do meta analysis over existing datasets. Thats value.
They found a creative way to incentivize the collection of it and paid for the processing. Anybody can collect the same data, I don't see why they would have to release it...
It would be nice of them though.
There's an "illegal child labor" angle to it, I suspect, T&Cs be damned.
I can really imagine a meeting with the big brasses of Google/Niantic a few years ago that went along
- We need to be the first to have a better, new generation 3D model of the world to build the future of maps on it. How can we get that data?"
+ What about gamifying it and crowd-sourcing it to the masses?
- Sure! Let's buy some Pokemon rights!
It's scary but some people do really have some long-term vision
Pokemon Go is built on the same engine as Inverness I think its called. When it launched they even used the same POIs. I think this was ~5-7 years before PGO launched.
Edit: I said inverness and meant ingress. Apologies.
I think you are thinking of Ingress. No idea what Inverness is.
Ingress and PGO share the same portals and stuffs and its what PGO got its data from.
Inverness is a city in Scotland
Also a tiny town in Marin County, CA and one of my favorite words. It’s just so nice to say. Inverness.
They definitely had this as a long-term vision
I still don't get what LGM is. From what I understood, it isn't actually about any "geospatial" data at all, is it? It is rather about improving some vision models to predict how the backside of a building looks, right? And training data isn't of people walking, but from images they've produced while catching pokemons or something?
P.S.: Also, if that's indeed what they mean, I wonder why having google street view data isn't enough for that.
> It is rather about improving some vision models to predict how the backside of a building looks, right?
This, yes, based on how the backsides of similar buildings have looked in other learned areas.
But the other missing piece of what it is seems to be relativity and scale: I do 3D model generation at our game studio right now and the biggest want/need current models can't do is scale (and, specifically, relative scale) -- we can generate 3d models for entities in our game but we still need a person in the loop to scale them to a correct size relative to other models: trees are bigger than humans, and buildings are bigger still. Current generative 3d models just create a scale-less model for output; it looks like a "geospatial" model incorporates some form of relative scale, and would (could?) incorporate that into generated models (or, more likely, maps of models rather than individual models themselves).
Brian Maclendon (Niantic) presented some interesting details about this in his recent Bellingfest presentation:
https://www.youtube.com/live/0ZKl70Ka5sg?feature=shared&t=12...
I'm sure the CIA already has access. [1] People were raising privacy concerns years ago. [2]
[1] https://www.networkworld.com/article/953621/the-cia-nsa-and-...
[2] https://kotaku.com/the-creators-of-pokemon-go-mapped-the-wor...
Hanke’s actually got awards from CIA for his work at In-Q-Tel investing in Keyhole/Niantic, so yeah, safe to assume that the agency invested specifically to have players collect data. Considering many Pokémon were on or near military bases around the world… not hard to assume what CIA’s real goal was.
https://futurism.com/the-byte/pokemon-go-trespassers-militar...
Google maps has more data than PGO could ever hope to have.
[flagged]
More like Celesteela, after all, you need jet fuel to melt steel beams.
People have a lot of strange beliefs about the CIA. Why would they even care about this?
Upload a picture of a bad guy in an office lobby to pokegpt and ask it where he is.
You can do that for free by sending the picture to a geoguessr streamer on twitch.
Or Google Lens. Regardless this isn't the CIA, it's the NGIA.
It may not be Geospatial data at all and I'm not sure how much the users consented but the data collection strategy was well crafted. I remember recommending building a game to collect handwriting data from testers (about a thousand), to the research lab I worked for long time back.
> For example, it takes us relatively little effort to back-track our way through the winding streets of a European old town. We identify all the right junctions although we had only seen them once and from the opposing direction.
That is true for some people, but I'm fairly sure that the majority of people would not agree that it comes naturally to them.
Somehow I always thought something like that would have been the ultimate use case for Microsoft Photosynth (developed from Photo Tourism research project), ideally with a time dimension, like browsing photos in a geo spatio-temporal context.
I expect that was also some reason behind their flickr bid back then.
https://medium.com/@dddexperiments/why-i-preserved-photosynt...
https://phototour.cs.washington.edu
https://en.wikipedia.org/wiki/Photosynth
at least any patents regarding this will also expire about 2026.
I worked on this and yes it was 100% related to the interest in Flickr. At the time Google Street had just become a thing and there was interest in effectively crowdsourcing the photography via Flickr and some of the technology behind Photosynth.
Impressive, but this is one of those "if this is public knowledge, how far ahead is the _not_ public knowledge" things
I really want to know what the NSA and NRO and Pentagon are doing training deep neural networks on hyperspectral imaging and synthetic aperture radar data. Imagine having something like Google Earth but with semantic segmentation of features combined with what material they are made from. All stored on petabytes of NVMe flash.
Conversation about ‘players are the product’ of Pokémon go aside… What are some practical applications of an LGM?
Seems like navigation is ‘solved’? There’s already a lot of technology supporting permanence of virtual objects based on spatial mapping? Better AI generated animations?
I am sure there are a ton of innovations it could unlock…
"It could help with search and rescue" jokes aside [1] this seems really useful for robotics. Their demo video is estimating a camera position from a single image, after learning the scene from a couple images. Stick the camera on a robot, and you are now estimating where the robot is based on what the robot has seen before.
They are a bit vague on what else the model does, but it sounds like they extrapolate what the rest of the environment could look like, the same way you can make a good guess what the back side of that rock would look like. That gives autonomous robots a baseline they can use to plan actions (like how to drive/fly/crawl to the other side) that can be updated as new view points become available.
I hope this tech could help make AR glasses more useful in public, day-to-day life, like a video game HUD.
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service. We receive about 1 million fresh scans each week
Wait, they get a million a week but they only have a total of 10 million, ie 10 days worth? Is this a typo or am I missing something?
A location probably requires like a million scans to be visualized properly. Think of a park near your house - there are probably thousands of ways to view each feature within.
Scans are not always of new locations. They have ~10m established nodes and they get ~1m node scans per week that might be new and might be old.
It’s possible they meant 1 million frames from scans.
Pretty sure there can be multiple "scans" per location is what they are saying
Interestingly, Pokemon GO only prompts players to scan a subset of the Points of Interest on the game map. Players can manually choose to scan any POI, but with no incentive for those scans I'm sure it almost never happens.
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service.
This 1 in 10 figure is about accurate, both from experience as a player and from perusing the mentioned Visual Positioning System service. Most POI never get enough scan data to 'activate'. The data from POI that are able to activate can be accessed with a free account on Niantic Lightship [1], and has been available for a while.
I'll be curious to see how Niantic plans to fill in the gaps, and gather scan data for the 9 out of 10 POI that aren't designated for scan rewards.
Genuinely impressed Google had the vision and resources to commit to a 10 year data collection project
Is this related to NeRF (neural radiance fields)?
I’ve published research in this general arena and the sheer amount of data they need to get good is massive. They have a moat the size of an ocean until most people have cameras and depth sensors on their face
It’s funny, we actually started by having people play games as well but we expressly told them it was to collect data. Brilliant to use an AR game that people actually play for fun
Yes it must be almost an exabyte of data.
I'm guessing this can be the new bot that could play competitively at GeoGuesser. It would be interesting if Google trained a similar model and released it using all the Street Map data, I sure hope they do.
Has anyone done something similar with the geolocated WIFI MAC addresses, to have small model for predicting location from those.
I believe I read somewhere that geoguesser AI based on street view data was mostly classifying based on the camera/vehicle set up. As in, a smudge on the lens in this corner means its from Paris.
This crowdsourced approach probably eliminates that issue.
I wonder how this can be combined with satellite data, if at all?
I don’t see why not. Photos are often combined with satellite data for photogrammetry purposes, even on large scale - see the recent Microsoft Flight Simulator (in a couple days, when it actually works)
It's usually aerial data, especially oblique aerial. Bing Maps is still pretty unique in offering them undistorted and not draped over some always degraded mesh.
Applications that I thought of as I read this:
Real-Time mapping of the environment for VR experiences with built-in semantic understanding.
Winning at geoguesser, automated doxing of anybody posting a picture of themselves.
Robotic positioning and navigation
Asset generation for video games. Think about generating an alternate New York City that's more influenced by Nepal.
I'm getting echoes of neural radiance fields as well.
Procedural generation of an alternative planet is the kind of stuff that the No Man's sky devs could only dream of.
Waymo is supposedly geofenced because they need detailed maps of an area. And this is supposedly a blocker for them deploying everywhere. But then Google goes and does something like this, and I'm not sure, if it's even really true that Waymo needs really detailed maps, that it's an insurmountable problem.
I wonder if there's a sweet spot for geospatial model size.
A model trained on all data for 1m in every direction would probably be too sparse to be useful, but perhaps involving data from a different continent is costly overkill? I expect most users are only going to care about their immediate surroundings. Seems like an opportunity for optimization.
So that's why Pokemon was notoriously impactful on battery life. They were recording and uploading our videos the whole time?
I don't think so. I wanted to voice this quickly without a detailed rebuttal as yours is the top comment and I don't think it's correct. Hopefully someone will do my homework for me (or alternatively tell me I'm wrong!).
No, that is unlikely to be the case.
Don't quite understand the application of this?
Google Maps uses this tech for AR navigation: https://www.pocket-lint.com/what-is-google-maps-ar-navigatio...
This seems like it’d be quite handy to have in an autonomous vehicle of any kind
The data marginally better than what google already have
I’m intrigued by the generative possibilities of such a model even more than how it could be used with irl locations. Imagine a game or simulation that creates a realistic looking American suburbia on the fly. It honestly can’t be that difficult, it practically predicts itself.
People complaining here that you are somehow owed something for contributing to the data set, or that because you use google maps or reCAPTCHA you are owed access to their training data. I mean, I'd like that data too. But you did get something in return already. A game that you enjoy (or your wouldn't play it), free and efficient navigation (better than your TomTom ever worked), sites not overwhelmed by bots or spammers. Yeah google gets more out of it than you probably do, but it's incorrect to say that you are getting 'nothing' in return.
I'm not sure quite what the ownership is, but Niantic isn't a subsidiary of Alphabet or Google.
The company was formed as Niantic Labs in 2010 as an internal startup within Google, founded by the head of Google's Geo Division (Google Maps, Google Earth, and Google Street View) at the time.
It became an independent entity in October 2015 when Google restructured under Alphabet Inc. During the spinout, Niantic announced that Google, Nintendo, and The Pokémon Company would invest up to $30 million in Series-A funding. Not sure what the current ownership is (they've raised a few more times since then), but they're seemingly still very closely tied with Google.
This is literally what I built my first company around starting in 2012, when Niantic was still working on Ingress
I describe it here during 500 Startups demo day: https://youtu.be/3oYHxdL93zE?si=cvLob-NHNEIJqYrI&t=6411
I further described it on the Planet of the Apps episode 1
Here's my patent from 2018: https://patents.google.com/patent/US10977818B2/en
So. I'm not really sure what to do here given that this was exactly and specifically what we were building and frankly had a lot of success in actually building.
Quite frustrating
Call an intellectual property attorney?
Fucking cool. Hi old Niantic teammates, it's me Mark Johns ;).
[dead]
The cia has to be all over this.
> I have been tricked into working to contribute training data so that they can profit off my labor
You were playing a game without paying for it. How did you imagine they were making money without pimping your data?
Niantic made 700 million dollars last year, mostly selling virtual game items.
Why would anyone think niantic would protect user-data from profit?
> Unsarcastically, a lot of people believe user data belongs to users, and that they should have a say in how it's used
I can understand that people believe this, but why do they do? Nothing in our society operates in a way that might imply this.
Is this model not a safe use of the data?
I don't think you need to be an "IT professional" to understand that not paying money doesn't imply that you aren't giving away value.
Lots of people are spending a lot of money on in app purchases in these games already.
They won't. It's the same data collection play as every other Google project
Just for clarity on this comment and a separate one, Niantic is a Google spin out company and appears to still be majority shareholder: https://en.wikipedia.org/wiki/Niantic,_Inc.#As_an_independen...
Google actually has released weights for some of their models, but judging by the fact that this model is potentially valuable, they likely will not allow Niantic for this
> Google actually has released weights for some of their models, but judging by the fact that this model is potentially valuable, they likely will not allow Niantic for this
which is totally unfair, every niantic player should have access to all the stuff because they collectively made it
“Is a farmer entitled to the entirety of your work output because you ate a vegetable grown on their farm?”
Bad analogy. I pay a farmer (directly or indirectly) for the vegetable. It’s a simple, understood, transaction. These players were generally unaware that they were gathering data for Niantic in this way.
If data is crowdsourced it should belong to the crowd.
> I don't understand this perspective. While all players may have collectively made this model possible, no individual player could make a model like it based on their contributions alone.
I don't think this is very difficult to sort out: people feel entitled to the products of their labor.
> Is a farmer entitled to the entirety of your work output because you ate a vegetable grown on their farm?
This is comparing apples and oranges: presumably the consumer didn't do anything to produce the vegetable. Hell if anything, under this analogy niantic would owe users a portion of their profits.
Most of your analysis is flawed because the model is non-rivalrous so it could easily be given to every player.
Additionally, many people can contribute to make something greater that benefits everyone (see open source). So the argument of “you couldn’t have done this on your own” also doesn’t hold any water.
The only thing that protects niantic is just a shitty ToS like the rest of the games that nobody pays attention to. There is nothing fundamentally “right” about what they did.
What you say is fair but if an individual's data doesn't matter, what happens when they ask to have their data deleted under GDPR. is there a way to demux their data from existing models?
> All of which I've directly contributed to and never (directly) recieved anything in return
To be fair, you received a service for free that you may have otherwise had to pay for. I'm not saying it's just, but to say you didn't get anything in return is disingenuous.
> any iPhone navigating with gmaps
Not saying you are saying this but it amused me how many people believe(d) that Apple wasn’t mining and hoarding location data either because well, they’re Apple and they love you. All those traffic statuses in Apple Maps on minor side streets with no monitoring came from the … traffic fairy, perhaps.
I kept wondering why a Google spinoff was named after a river and community in Connecticut, one of the least Googley locales in the country.
The connection is a ship, built in Connecticut, which brought gold rushers to San Francisco and was run aground and converted to a hotel there: https://en.m.wikipedia.org/wiki/Niantic_(whaling_vessel)
The company was named after the ship.
All companies should be truthful, forthcoming, and specific about how they will use your data, but…
If you enjoy the game, play the game. Don’t boycott/withhold because they figured out an additional use for data that didn’t previously exist.
Another way of viewing this: GoogleMaps is incredibly high quality mapping software with lots of extra features. It is mostly free (for the end user). If no one uses it, Google doesn’t collect the data and nobody can benefit from the analysis of the data (eg. Traffic and ETA on Google Maps)
There’s no reason to hold out for a company to pay you for your geolocation data because none of them offer that service.
Were you really tricked? Hard to believe that someone on Hacker News saw Pokemon Go and didn't immediately think of the data collection possibilities.
It may surprise you to learn pokemon go is nearly a 10 year old game based on 40 year old beloved IP that when it was released did not exist in the same data hellscape we do today, and even if it did, the attraction of the IP would overrule people thinking about this kind of thing. These kinds of comments are extraordinarily disingenuous sounding, particularly when anyone can spend 3 seconds and figure out their primary market is literal children.
Imagine how those of us who played Ingress (Niantic's first game) feel... We were tricked into contributing location data for the game we loved, only to see it reused for the far more popular (and profitable) Pokemon Go.
Why would anyone take issue with this? Asking as someone who tried both games at different points.
Niantic was always open with the fact that they gather location data, particularly in places cars can't go - I remember an early blog post saying as much before they were unbundled from Google. No one was tricked, they were just not paying attention.
As long as they make enough money from Pokemon Go to sustain Ingress, I OK with that.
The Google - Niantic - Ingress - borg - kubernetes conspiracy must be unraveled
> and give you some useful items as rewards
Were you tricked, or were you just poorly compensated for your time?
Frankly, with the amount of real-world walking required to progress in Ingress and Pokémon Go, most players were compensated by the motivation to get a decent amount of exercise, which had a net positive impact on their health. Most exercise apps require users to pay subscriptions for the pleasure of using them.
Frankly given the numbers of hours of entertainment most people got out of Pokémon Go, I suspect this might be one of the cases where people have been best compensated for their data collection.
Friendships too!
Do you honestly feel tricked that a gameplay mechanic which transparently asks you to record 50-100MB videos of a point-of-interest and upload it to their servers in exchange for an (often paid/premium) in-game reward was a form of data collection?
I don't think I've done any in PoGo (so I know it's very optional), but I've done plenty in Ingress, and I honestly don't see how it's possible to be surprised that it was contributing to something like this? It is hardly an intuitively native standalone gameplay mechanic in either game.
Oh yes, children, their primary market, definitely consider this. Definitely.
One of the reasons i never played pokemon go is because there was no guarantee I didnt have my data sold.
I can't tell you why other people wouldn't think of this concern
They did at least published their research, and also dataset for 655 places:
https://research.nianticlabs.com/mapfree-reloc-benchmark
This was linked the news post (search for "data that we released").
Please don't tell me you were just now realizing this
Yeah, they did the same in Ingress: film a portal (pokéstop/gym) while walking around it to gain a small reward. I've always wondered what kind of dataset they were building with that -- now we know!
Did anyone here on hackernews not seriously assume this was the real reason for the existence of that game since day 1?
You've also been tricked into making your comment, which will undoubtedly be fed into an LLM's training corpus, and someone will be profiting off that, along with my comment as well. What a future we live in!
Baba booy bbaba booy Batman bats badly barring the baristers bearing.
Magic schoolbus!
Yea, take that llm model maker
NooooooooooOooOooOo!
Lol, do you really think that? I did it from having a desire to contribute to the conversation and I was aware that that would be a future possibility :) I'm not really getting much in return or being incentivized by Y combinator
I think the joke was that it's kind of the same with Pokemon GO. You play the game mainly because it's fun or lets you get some exercise in, so it's not really a bad thing that the company used the data to train a useful model. You're still having fun or doing exercise regardless of what they do with the data. Essentially, it's a positive externality: https://www.economicshelp.org/micro-economic-essays/marketfa...
But I think your point, if I understand it correctly, is that the in-game rewards kind of "hacked your brain" to do it, which is the part you're objecting to?
I think that's part of it- but another part is a lot of people do not like what Gen AI is doing and are offended that what was a fun game is now part of that project.
Like when captchas were for making old books readable it felt a lot more friendly than now where its all driverless car nonsense
[flagged]
I don't understand this perspective. Why should I resent the creation of value from behaviours that I would be doing anyway.
Didn’t beloved New York Times photographer Bill Cunningham make a storied career out of doing exactly that?
https://www.nytimes.com/2019/09/01/style/bill-cunningham-boo...
Women on the street didn’t agree to a terms of service and didn’t choose to put content online.
The better metaphor is a woman posting her photos online and then those photos were used by a painter who then sold an abstract painting of her.
https://www.earthcam.com/cams/newyork/timessquare/?cam=tsrob...
is a webcam of Times Square, and they've got ads on the page, and they're making money off pictures of pretty men and women on that street. I don't know how okay or not I am with it, but it's the world we live in.
I don't like bad arguments like this.
Because AI is going to create a world where only a few hundred trillionaires and a few thousand billionaires exist while everyone else is in desperate poverty.
Tbh, I'm not sure its rent seeking, but whether the 'value' is for the company or for society is extremely questionable
Not really, you're assuming that your independent actions or forum comments have intrinsic value. They do not.
You think that's bad, wait till you find out about what happens at work!
[flagged]
My reaction, also.
"You used me... for LAND DEVELOPMENT! ...That wasn't very nice."
Honestly you should have assumed they were using the collected data for such a purpose. It would be shocking if they weren't doing this directly or selling the data to other companies to do this.
Assumed … or just read the Terms & Conditions / AUP like we did 10 years ago when they were using "Ingress" for location collection & tracking.
The game is free, there has to be some way for them to profit, interesting to see this was it.
This wasn’t it. It was from gems
When ever it's free, it's all about the data.
I recall having a conversation circa 2004/5 with a colleague that Google was an AI company, not a search company.
Well now by posting your thoughts to hn, you have been tricked yet again to give up free labor to train ai models.