Botnets are always on the hunt for an insecure website, or the operators of it just like to take down websites for the “lulz”. In my case it was the latter.

The Soft Underbelly

Traditional PHP framework websites have always suffered from several facts:

1. Rendering “dynamic” pages on the fly with PHP is slow.
2. PHP just can’t keep up with demand so you have to introduce lots and lots of caching (Like varnish for example)
3. No matter how much you cache, some script kiddie will find a vulnerable area of the site.

Enter Magento

Magento is an ecommerce website with a large installation base. It aims to be a fully featured CMS and ecommerce solution with integration into 3rd party APIs and more.

Since it’s entirely rendered with PHP every time someone visits it, it doesn’t scale well… at all. In fact, if you turned off its own internal caching the site would slow to a crawl and most likely stop. There’s many layers of caching it needs, from Redis to Varnish and PHP Opcode cache. It’s a beast.

The site that was attacked was a Magento 2 site running PHP 7.4, and it had quite a substantial caching system using Varnish. All the recommended “best practices” were applied.

Barbarians at the Gate

There’s one drawback with Magento 2: its search system. Whenever you search for a product on a magento site, you will go to a URL like /catalogsearch in the URL. This endpoint talks directly to PHP and it cannot be cached. (There is actually a way to cache search results but I think most developers are too afraid to use it for fear of it returning stale results, and that’s a good reason.)

The first attack lasted for just over an hour in which 943 different IP addresses would GET a /catalogsearch/results/?q=0 URL. Often 2 or 3 times within 5 minutes, but if you multiply this by 943 you get a large amount of hits. Perhaps not enough to make the site fully fall over, but performance will be impacted. Obviously searching for ‘0’ will just get you “No results found” but it still has to take an expensive trip into PHP which then has to bootstrap Magento, query the database, and come back with the answer.

Did I mention you can’t cache this?

The Second Wave

The next day, roughly 12 hours later a sustained campaign started. Luckily we were able to stop /catalogsearch/results/?q=0 at the load balancer, but then the bots started querying other numbers. With a bit of magic in the web server, we were able to completely stop the bots from overwhelming the server. They eventually stopped attacking.

The second wave consisted of 2,470 unique IP addresses from all over, but I noticed the vast majority were from a hosting company in India and the other was in the United Kingdom.

Mitigation

We employ fail2ban on the web servers themselves, but this doesn’t help if you are using multiple web servers in an autoscaling setup behind the load balancer as fail2ban will only see maybe 1-4 hits from an IP before not seeing that IP again (there’s 2,469 more that can hit it!) so fail2ban is a complete wash.

The only way to stop it was to block the most egregious searches, but this can result in limited search functionality.. what if you know a product number and want to search for that?

There’s probably other ways to block or mitigate this stuff, WAFs (Web application firewalls), Cloudflare, or others.. but you end up having to pay a lot of money to protect your vulnerable website.

A Better Way

There’s been a movement to fix this web scalability and it’s called Jamstack. Imagine: an ecommerce server where all the product pages are rendered once into static html, images and javascript. The javascript can talk to 3rd party APIs for payment or perhaps updating stock, but the rest of the site is static. No PHP, no database to talk to, nothing to slow it down!