Disruptions, bugs etc

[Danzar]

Apologies for that downtime, I overlooked setting up an auto purge of the database archives so ran out of disk space. That sent the website down. All fixed and properly set up now.

Setting this up as a sticky for future updates.

[Danzar]

RWO update – hopefully the end of the outages

As everyone knows, RWO has had far too many outages and periods of painfully slow performance. The good news is that after a lot of digging over the past few days, I think we've finally found the underlying causes and fixed them.

There's been quite a bit done:

• RWO was being hammered by bots and automated crawlers. The worst of these have now been blocked and much stronger controls put in place.
• A couple of problems within the forum software itself were causing RWO to do vastly more work than it needed to. These have been identified and fixed.
• Years of accumulated junk behind the scenes has been cleaned out.
• The server and database have been thoroughly checked, cleaned up and tuned.
• The server software has been updated.
• RWO itself has been upgraded from vBulletin 6.0.6 to 6.1.3.

So what was actually going wrong?

There wasn't one big thing wrong with RWO. It was really a combination of a few things that had gradually got out of hand and, together, were periodically overwhelming the site.

The most interesting one was these damn bots. The site was being absolutely hammered by automated crawlers from all over the world. Some are legitimate search engines, but there were also huge numbers of less useful bots, AI crawlers and automated services constantly trawling through old threads and pages.

At one point RWO was showing more than 15,000 guest sessions from around 15,000 different internet addresses. I think we hit a record active number of users of nearly 38k a week ago. For a relatively small footy forum in Australia, that's obviously ridiculous, and I wonder whether there's something suss about this...

The other problems were inside RWO itself. Over the years the software had accumulated an enormous amount of unnecessary temporary information, while one particular part of vBulletin was doing an extraordinary amount of unnecessary work every time it ran. Neither was particularly disastrous on its own, but combined with all that automated traffic it basically kept filling up all disk space and crashing the server.

Those problems have now been cleaned up or fixed. RWO is currently doing a fraction of the unnecessary work it was doing before, and the server has heaps of spare capacity.

What's new?

The vBulletin upgrade also brings a few new things:

• Bookmarks – you can now bookmark individual posts and find them again later.
• Collapsible forum categories – you can hide sections of the forum you don't want to see and RWO will remember your preference.
• Push notifications – the new version supports browser notifications for things like replies and mentions. I'll have a look at how best to configure this.
• There are also various smaller fixes and improvements.

I'm pretty confident we've finally dealt with the reasons it has been so unreliable, rather than just getting it running again each time it breaks.

So, hopefully, this is the end of the regular outages.

Thanks for everyone's patience while I've been working through it.

[Velour&Ruffles]

And a huge thanks to you for everything you do. I am extremely grateful for your efforts.