News from behind the scene

Feel free to discuss here about news on our blog
Post Reply
User avatar
jvoisin
Posts: 295
Joined: 11 Aug 2011, 18:47
Contact:

News from behind the scene

Post by jvoisin »

We took some time this year to overhaul, simplify and awesomify our web presence and infrastructure.

Our blog

Ten years using WordPress and still kicking! But it was time to clean up the dust a bit.

First of all, we removed unnecessary plugins like akismet and its friends to only keep:

- our Cloudflare integration, since our server is not super powerful and we've been slashdot'ed before. So better safe than sorry. Also, no need to fetch your pitchforks if you're using Tor or a VPN: we configured Cloudflare to not annoy you, so it's only used as a CDN/cache.
- Polylang, since WordPress doesn't support internationalization by default and some of our blogposts are translated into a couple of languages.
- Jetpack because our posts are written in Markdown.

Moreover, since more and more people are browsing the web via their phones, we are more than happy that vtastek took the time to make the blog responsive, so that it looks great on mobile as well!

If you have a look at the sidebar, you'll notice a handful of badges: GitHub, GitLab, Discord, matrix, Twitter, Reddit - all without a single line of JavaScript, as in, no tracking. Speaking of tracking: we also removed Google Analytics, which we weren't using anyway.

On the backend side, we cleaned the database up by removing comments (which had been turned off years ago), post revisions, unnecessary users, and much more, reducing the size of our database by 60%.

All those small changes lead to a significant improvement of loading times: from around 9 seconds to 1 second!

Our plan for the future is to move away from WordPress to take advantage of GitLab Pages to run a static website instead. Any help to make this happen is more than welcome!

Forum

We've been using phpBB for the forums even before we had our blog, and it's been working great. So not a lot has changed on this side except that we upgraded it to the latest available major version, granting us a minor performance boost and improved security. We also switched to MySQL for the search engine which reduces the database table and makes the search not only better but also faster. Moreover, we stepped up our anti-spam game: the forum should be kept tidy now with less human intervention necessary, and we're now reporting spammers to Stop Forum Spam.

Wiki

We've long been using MediaWiki (the same wiki software used by Wikipedia itself!) to document everything, along with a plugin to bridge accounts between the forum and the wiki. Now that OpenMW is slowly reaching feature parity with Morrowind's original engine, most of the information on the wiki is no longer being updated. So we made our wiki read-only and available at old-wiki.openmw.org and moved the content to GitLab's wiki, reducing the maintenance cost of our MediaWiki. Switching our old wiki to read-only allowed us to disable a ton of features. It also allowed us to make use of aggressive caching which significantly improves its performanc and makes it way less taxing resource-wise.

Alerting

We've set up a status page at openmw.org/status, thanks to UptimeRobot, to monitor all of our services and to get notified in case one of them goes down. Also, since this information is available to everyone, you can check for yourself if "openmw.org is down for everyone, or is it just me"? This uncovered an issue with our backup system: in the middle of the night (European time) our backups are kicking in, and were doing some heavy compression operations which result in a massive slowdown of the website, generating timeouts. We lowered the process priority of the backups, which fixed the issue and appeased our monitoring.

Backend

Our nginx config was overhauled and factorized to make our services' configurations more uniform: same modern TLS parameters, usage of http2, same Cloudflare configuration, more efficient serving of big files for downloads.openmw.org, etc.

Since we're automatically building OpenMW on every commit using GitLab's continuous integration to avoid hammering NuGet, Chocolatey, Debian and co. too much, we're using Nexus Repository OSS to act as a caching proxy. It has been updated to the latest available version, fixing a lot of nasty security issues. We also spent some time improving its loading time.

We ran OptiPNG and jpegoptim on icons/media to drastically reduce the size of images and, thus, our bandwidth consumption.

Our database engine, MariaDB, was tuned for performance as well as cleaned up: we purged old data, and optimized the rest.

Since we're in 2021, all of our domains are now fully accessible over IPv6.


Automatic updates have been enabled via UnattendedUpgrades, so that our beloved sysadmin crew don't have to waste time on mundane tasks and can sleep better at night knowing that security upgrades are installed as soon as they become available.

Finally, speaking of the sysadmin crew: we took steps to increase the bus factor, bringing nice side effects like increased motivation (it's always funnier to do things with friends) and reduced burnout risks.
User avatar
psi29a
Posts: 5168
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: News from behind the scene

Post by psi29a »

Thank you #infra team for making things better under the hood. :)
User avatar
kyleshrader
Posts: 1
Joined: 26 Sep 2020, 08:39
Gitlab profile: https://gitlab.com/kyleshrader

Re: News from behind the scene

Post by kyleshrader »

I appreciate that you broke down this information and shared it. I learned about some new solutions and even some new problems to consider.

Great work, everyone.
User avatar
piotrekdg
Posts: 4
Joined: 11 Oct 2015, 21:08

Re: News from behind the scene

Post by piotrekdg »

Great job. This is basically a very good cleanup that a lot of the big players fail at miserably.
Post Reply