I use OpenFreeMap for a personal site (basically only for me) and I noticed some tiles dropping yesterday-ish. I figured it would be fixed soon. But I had no idea it was such an interesting cause!
Extremely interesting insight, and surely no one expected the word of mouth for Wplace to explode into 2 million users, well even if we consider the art bots.
Anyways I've never been interested in "what the hell do we use to display tiled maps" until now, so that's really great. And holy cow, the people Cloudflare are really nice.
it's embarrassing how you keep accusing wplace of being rife with "script kiddies" who supposedly keep clearing their cookies and/or rotating their IP address to draw more, when you haven't even looked at it long enough to realise that you need to log in with a google account to draw. no curiosity for the thing you wrote nearly a thousand words about. no investigation. they're just incompetent in the simplest way you can imagine
well the thing is even if 99% of people using it are legitimate good-faith creative users, the bad-faith "script kiddies" are very much the ones generating so much traffic since they generate so many more requests-per-timeframe than a regular user
Yes, this would be absolutely wonderful! Both to argue over, as well as just to see. It's not often you see a site take off quite like this, and I would wager it's only gotten more traffic since even this article we're commenting on was posted.
For context: You get more than just 1 pixel to place, and can store up to like 50. The more you draw the more you can store - and you recharge 1 pixel every 30 seconds. It's not like the old r/place on reddit.
Hey! We've seen so many epic creations on Wplace.live and know how much coordination it takes. To support the community, we developed https://wplace.tools. It’s a simple, free tool to help teams generate pixel art blueprints from any image, making collaboration easier and faster. We hope this empowers even more incredible art on your platform!
Non-logged-in guest users (myself included) mostly visit wplace not to draw, but just to see all the artwork, which literally spans the entire globe, both zoomed in and zoomed out. That explains the high tile requests per user. I wonder if that 2 million figure refers only to registered accounts, excluding guest viewers without an account.
Would shipping a shared service worker that had aggressive client-side caching (even preventing any "last-modified" checks from hitting the servers) have helped, or would you have expected that it was fresh browser instances each time? If lots of people were scripting it, maybe the code to do it was public somewhere and could have had a PR to save the bandwidth on the client-side 😄
How did you end up solving the "too many files" issue? Just increase the OS file limit, or something else in code?
Rate-limiting your api will protect you against the bots you spotted, and instead of using a managed challenge you can directly put a Block policy -- would avoid this noisy neightboor problem you were experiencing.
Users have a bunch of pixels to paint with. Refill time is limited to 2 pixel per minute, but you have a storage of available pixels, a pixel buffer that refills when you offline. With enough time dedicated to creating art it’s possible to level up to have 200, 300, or 500 pixels available. The main case for high load probably is that users exploring the world map. I by myself spend hours peeking at art in different places of the world. I remember noticing Reddit post about Bara Asgore art with huge male chest, drawn in North Korea, LMAO. It was a really beautiful thing to see, so it was probably a Reddit effect. Personally as a user, I’m using Tapemonkey extensions that is creating overlay in browser locally, to give me a guide for drawing, because I don’t want my account to be banned, and using scripts is just cheating
In case you didn't try it, Cloudflare's smart tiered cache can help improve the cache hit ratio even more, and also coalesce the requests that go to your origin server, reduce the work load.
hi Zsolt Ero, thank you for putting this article up, so we all know what NOT to do!
I was planning to use Open Free Map's tiles for an App I am creating. I cannot predict the traffic of the App. Is it a bad idea to continue using Open Free Map?
Can you publish those data in a monthly blog? like "This Month on OpenFreeMap"—something like that. where that month's total requests, bandwidth, day of most and least requests, most mapped place/country, etc. Additionally, a comparison is made with the previous month.
If not possible monthly, then please do it quarterly.
And yes, a "Year in Review" / "OpenFreeMap wrapped" is a must.
Checking Referer header doesn't always work. If the origin website sets Referrer-Policy to "no-referrer" then their user's browsers won't send a Referer header at all. Someone that wants to abuse your free service (as opposed to a seemingly accidental situation here) would bypass your check completely.
The owner of the website sets the Referrer-Policy header. Any requests made by the browser for that page, including for embedded things like MapLibre, are affected. It's a privacy thing. Websites containing sensitive (e.g. health related) data may not want to expose possible clues about their visitors in the URLs sent as Referer.
So in a situation like this, Wplace could just disable the Referer header. You just can't rely on it, not when trying to defend against someone who wants to abuse your service.
Interesting!
I use OpenFreeMap for a personal site (basically only for me) and I noticed some tiles dropping yesterday-ish. I figured it would be fixed soon. But I had no idea it was such an interesting cause!
Extremely interesting insight, and surely no one expected the word of mouth for Wplace to explode into 2 million users, well even if we consider the art bots.
Anyways I've never been interested in "what the hell do we use to display tiled maps" until now, so that's really great. And holy cow, the people Cloudflare are really nice.
it's embarrassing how you keep accusing wplace of being rife with "script kiddies" who supposedly keep clearing their cookies and/or rotating their IP address to draw more, when you haven't even looked at it long enough to realise that you need to log in with a google account to draw. no curiosity for the thing you wrote nearly a thousand words about. no investigation. they're just incompetent in the simplest way you can imagine
well the thing is even if 99% of people using it are legitimate good-faith creative users, the bad-faith "script kiddies" are very much the ones generating so much traffic since they generate so many more requests-per-timeframe than a regular user
Would be nifty if the site had a stats.wplace thats juset a graph of unique pixels per IP or IPs per pixel placement_count/etc...
(I love all of this.
Also see this HN: https://news.ycombinator.com/item?id=44854878
and pipe a s.crypt to place QR_code_images_in_pixel_wplace_free_map. :-)
Yes, this would be absolutely wonderful! Both to argue over, as well as just to see. It's not often you see a site take off quite like this, and I would wager it's only gotten more traffic since even this article we're commenting on was posted.
My guess is, you probably have bigger worries when 2 billion requests flood in...
But yeah quite sad to see it being labeled solely as that. It's a much more creative canvas, rather than simple "script kiddies".
For context: You get more than just 1 pixel to place, and can store up to like 50. The more you draw the more you can store - and you recharge 1 pixel every 30 seconds. It's not like the old r/place on reddit.
Hey! We've seen so many epic creations on Wplace.live and know how much coordination it takes. To support the community, we developed https://wplace.tools. It’s a simple, free tool to help teams generate pixel art blueprints from any image, making collaboration easier and faster. We hope this empowers even more incredible art on your platform!
Non-logged-in guest users (myself included) mostly visit wplace not to draw, but just to see all the artwork, which literally spans the entire globe, both zoomed in and zoomed out. That explains the high tile requests per user. I wonder if that 2 million figure refers only to registered accounts, excluding guest viewers without an account.
Hi. Thanks for being chill and transparent about this, it's appreciated.
Congrats on surviving that load!
Note that the 206 status is normally.still successful. It's the code for a successful response to a Range request.
What app is that to do the natural language calculation (for the 100 million requests thing)
https://soulver.app/
thanks!
Would shipping a shared service worker that had aggressive client-side caching (even preventing any "last-modified" checks from hitting the servers) have helped, or would you have expected that it was fresh browser instances each time? If lots of people were scripting it, maybe the code to do it was public somewhere and could have had a PR to save the bandwidth on the client-side 😄
How did you end up solving the "too many files" issue? Just increase the OS file limit, or something else in code?
I don't know what Wplace's users were doing, but I helped them set up the self-hosted server and now they are running on their own infrastructure.
If they were launching new Chromium instances then no amount of caching would have helped.
I haven't solved that issue yet, I've written my questions here:
https://community.nginx.org/t/too-many-open-files-at-1000-req-sec/5796
I believe it's the combination of multi_accept + wrong limits, but I'm waiting for the experts' opinions.
Guess you could write a Worker that runs on a schedule and uses CF API to check top referers and add/remove any above X rps to the page rules
Yes, exactly, I was thinking something like that.
Great write-up!
> I need to implement bandwidth limiting by referer
In here from the cloudflare point of view the only thing you can do is applying rate-limiting rule. See references: https://developers.cloudflare.com/waf/rate-limiting-rules/best-practices/ https://developers.cloudflare.com/waf/rate-limiting-rules/best-practices/#limit-by-referrer
Also the features you'll have access to really depends on the plan you have subscribed to: https://developers.cloudflare.com/waf/rate-limiting-rules/
Rate-limiting your api will protect you against the bots you spotted, and instead of using a managed challenge you can directly put a Block policy -- would avoid this noisy neightboor problem you were experiencing.
Users have a bunch of pixels to paint with. Refill time is limited to 2 pixel per minute, but you have a storage of available pixels, a pixel buffer that refills when you offline. With enough time dedicated to creating art it’s possible to level up to have 200, 300, or 500 pixels available. The main case for high load probably is that users exploring the world map. I by myself spend hours peeking at art in different places of the world. I remember noticing Reddit post about Bara Asgore art with huge male chest, drawn in North Korea, LMAO. It was a really beautiful thing to see, so it was probably a Reddit effect. Personally as a user, I’m using Tapemonkey extensions that is creating overlay in browser locally, to give me a guide for drawing, because I don’t want my account to be banned, and using scripts is just cheating
In case you didn't try it, Cloudflare's smart tiered cache can help improve the cache hit ratio even more, and also coalesce the requests that go to your origin server, reduce the work load.
https://developers.cloudflare.com/cache/how-to/tiered-cache/#smart-tiered-cache
hi Zsolt Ero, thank you for putting this article up, so we all know what NOT to do!
I was planning to use Open Free Map's tiles for an App I am creating. I cannot predict the traffic of the App. Is it a bad idea to continue using Open Free Map?
Just curious. I love data, charts, etc.
Can you publish those data in a monthly blog? like "This Month on OpenFreeMap"—something like that. where that month's total requests, bandwidth, day of most and least requests, most mapped place/country, etc. Additionally, a comparison is made with the previous month.
If not possible monthly, then please do it quarterly.
And yes, a "Year in Review" / "OpenFreeMap wrapped" is a must.
Checking Referer header doesn't always work. If the origin website sets Referrer-Policy to "no-referrer" then their user's browsers won't send a Referer header at all. Someone that wants to abuse your free service (as opposed to a seemingly accidental situation here) would bypass your check completely.
But you'd need to actively disable it, don't you? I mean when using a normal web map library, like MapLibre, it'd always be sent, wouldn't it?
The owner of the website sets the Referrer-Policy header. Any requests made by the browser for that page, including for embedded things like MapLibre, are affected. It's a privacy thing. Websites containing sensitive (e.g. health related) data may not want to expose possible clues about their visitors in the URLs sent as Referer.
So in a situation like this, Wplace could just disable the Referer header. You just can't rely on it, not when trying to defend against someone who wants to abuse your service.