I can't get the vector basemap to render in QGIS properly with any other projection. I love using EPSG:3301 for Europe, etc.. but loads of strange things happen. Even 4326 is an issue.
There should be enough SQL in the blog to re-purpose extracting out the Wildberries locations and seeing where they land on top of. I've never heard of this firm before you mentioned it.
From Google:
> Citibank operates over 2,300 ATMs within more than 600 U.S. branches, with a total network of over 65,000 fee-free ATMs
So the 57,163 Citibank locations are probably a combination of their branches and ATMs.
Update: I reviewed Alltheplaces a while back, they scrape company websites for store locations. They reported 68,227 locations for Wildberries. ATP is one of the sources Overture use but they seem to use 1.55M of the records from their 19M-record dataset. https://tech.marksblogg.com/alltheplaces.html
I contribute to ATP and can confirm that the author of the wildberries spider was deliberately trying to collect https://wiki.openstreetmap.org/wiki/Tag:shop%3Doutpost (online order pickup locations). It's not a common occurrence within the current set of ATP spiders to capture such features. A quick search indicates that OSM doesn't appear to have tags designed to capture pickup/dropoff partnerships between retail brands, for example, an agreement from a pet supply shop to allow collection of parcels from select fuel stations of a partner brand. Thus I think the author of the wildberries spider has used shop=outpost as the closest tag available in OSM, and Overture Map's filters wouldn't be able to omit these features from their dataset unless Overture Maps adds wildberries to their exclusion list.
Ideally ATP's "located_in" and "located_in:wikidata" fields would be populated for these wildberries pickup locations, making it clear the pickup location is part of a parent feature (e.g. fuel station, supermarket). These fields are specific to ATP and are not OSM fields. OSM would expect features to be merged and a hypothetical field such as "pickup_brands:wikidata=Q1;Q2;Q3" be used instead on the parent feature.
ATP has a much more inclusive set of features it can extract than what Overture Maps, TomTom et al care about. As Overture Maps is more opinionated on what they aggregate they will filter out ATP extracted features such as individual power poles, park bench seats, local government managed street and park trees, stormwater drain manholes, cemetery plots, weather stations, tsunami buoys, etc. I think there might be some exceptions if it helps TomTom et al with their products such as speed camera locations, national postal provider drop-off/pick-up locations within other branded retail shops, etc.
Defcon had a great talk on all the different navigational systems pilots can use and a note at the end that these shouldn't be decommissioned at the rate they're experiencing atm https://www.youtube.com/watch?v=wSVdfOn737o
From what I've seen in news reports, China has built a lot of tower blocks that are 10s of floors, rather than the 4-5-floor buildings I saw when I worked in and travelled around India.
India has ~4x the population of the US so the ratio of buildings isn't much of a surprise.
The US has only used 25% of its land. Overture published a land use dataset a while back that could go some way to verify how much land on earth is urban, covered in forest, etc.. That might only need a single SQL statement given how they structured their data.
That 5 TB of data will probably be 3-400 GB in Parquet. Try and denormalise the data into a few datasets or just one dataset if you can.
DuckDB querying the data should be able to return results in milliseconds if the smaller columns are being used a better if the row-group stats can be used to answer queries.
You can host those Parquet files on a local disk or S3. A local disk might be cheaper if this is exposed to the outside world as well as giving you a price ceiling on hosting.
If you have a Parquet file with billions of records and row-groups measuring into the thousands then hosting on something like Cloudflare where there is a per-request charge could get a bit expensive if this is a popular dataset. At a minimum, DuckDB will look at the stats for each row-group for any column involved with a query. It might be cheaper just to pay for 400 GB of storage with your hosting provider.
reply