Posted in: Coding
Posted 2019-02-25 at 22:55 by Senshi
You all know (and rightly complained 
) about the recent downtime of the webserver.
And it allows me to finally show the reason why I have that shiny blue [R-DEV] next to my name. It's not just because I asked nicely. Honest!
Over the past ten or so years, the server only received rudimentary attention, mostly on an "as-needed" base. We managed to drag it along like that for surprisingly long, but sooner or later progress is unavoidable. That moment is now.
The incident
We had an incident where ALL user passwords were overwritten to be the exact same. Obviously we noticed that fairly quickly, so as a precaution we shut down the websever to investigate and get ready to bolster our defenses in the future. As a secondary immediate - probably unnecessary in hindsight - precaution, we regenerated all passwords with cryptographically strong, random passwords.
Investigation revealed that nothing else happened to our database. Luckily, despite my initial fears that an attacker might have gained SSH access to our server, Fritz had already configured SSH access to be very safe. No way anyone gets into those without physically breaking into our home and hacking our computers.
The password change itself was done in a way that would not allow anyone actual access to your accounts. Passwords always have been stored in an encrypted fashion using multiple tokens, and only one such token was overwritten (with a faulty encryption, no less).
This led us to believe that this was either not done by a professional, or it was simply a random bot roaming the internet looking for easy prey.
What does that mean for you?
All of you will have to reset your password this one time. Just enter your email here:
https://www.realitymod.com/forum/login.php?do=lostpw
and follow the steps.
As always, we recommend using a strong password, and to not use the same password for multiple pages. Use a password manager if you're lazy like me! There's plenty of great ones out there (Keepass, Bitwarden, ...), some of which integrate very comfortably in all kinds of devices and browsers so you can be even more lazy and keep auto-filling your forms. Which obviously is a security risk in itself again, but it's a lot better than using simple dictionary words, especially if you use the same password here and for your banking services...
With this scary bit out of the way, let's see what else we did!
Why was the server down so long?
A bunch of core technologies we use are close to reaching their end of life. This is the part we knew about for months or even years, and thus I had already spent many hours evaluating and preparing the necessary upgrades, so I could apply them fairly quickly.
However, there's a bunch of things that you simply cannot do on a live system easily, such as kernel upgrades and other low level security updates. Also, doing that live could have opened new security and compatibility issues for the not-yet updated services. So it was convenient to do right now as well. That's why we locked most of the homepage away so we could developed and test the updates without risk of interfering with live usage.
So what did we do?
This is a pretty impressive list and almost no part of the webserver was left untouched. Considering that a lot of the plugins and code we use is either custom-made and/or hasn't been updated in years, it took me a couple of days of wading through it all and making sure everything is airtight for hopefully at least another ten years
.
If we did everything right, there should be very little that you will notice when browsing the homepage or the forum. I can't guarantee that, so I'm counting on you informing us about any remaining bugs you spot!
Summary
We had someone/something possibly use a PHP exploit to inject a SQL command. No account was compromised.
We upgraded the webserver and everything on it, making everything state-of-the-art secure.
A sweet bonus: The homepage and forum are now served about 60% faster than before, leading to an impressive ~300-400ms response time (TTFB) in the forum. Given the size of our database and the complexity of our code, this is amazing.
Just so you get an idea of the scale of things:
Our database contains just shy of one hundred thousand forum users and over 2 million(!) posts.
It not only includes a wealth of user data (emails, passwords etc.), but also a huge amount of knowledge regarding PR and BF2 in general, and not least of all sweet memes and raw history.
I am an advocate for transparency, as I believe only by being open about what happened and our response to it can you trust us in the future.
If you have feedback, questions or concerns, feel free to ask, I'll try to answer to the best of my ability.
) about the recent downtime of the webserver.And it allows me to finally show the reason why I have that shiny blue [R-DEV] next to my name. It's not just because I asked nicely. Honest!
Over the past ten or so years, the server only received rudimentary attention, mostly on an "as-needed" base. We managed to drag it along like that for surprisingly long, but sooner or later progress is unavoidable. That moment is now.
The incident
We had an incident where ALL user passwords were overwritten to be the exact same. Obviously we noticed that fairly quickly, so as a precaution we shut down the websever to investigate and get ready to bolster our defenses in the future. As a secondary immediate - probably unnecessary in hindsight - precaution, we regenerated all passwords with cryptographically strong, random passwords.
Investigation revealed that nothing else happened to our database. Luckily, despite my initial fears that an attacker might have gained SSH access to our server, Fritz had already configured SSH access to be very safe. No way anyone gets into those without physically breaking into our home and hacking our computers.
The password change itself was done in a way that would not allow anyone actual access to your accounts. Passwords always have been stored in an encrypted fashion using multiple tokens, and only one such token was overwritten (with a faulty encryption, no less).
This led us to believe that this was either not done by a professional, or it was simply a random bot roaming the internet looking for easy prey.
What does that mean for you?
All of you will have to reset your password this one time. Just enter your email here:
https://www.realitymod.com/forum/login.php?do=lostpw
and follow the steps.
As always, we recommend using a strong password, and to not use the same password for multiple pages. Use a password manager if you're lazy like me! There's plenty of great ones out there (Keepass, Bitwarden, ...), some of which integrate very comfortably in all kinds of devices and browsers so you can be even more lazy and keep auto-filling your forms. Which obviously is a security risk in itself again, but it's a lot better than using simple dictionary words, especially if you use the same password here and for your banking services...
With this scary bit out of the way, let's see what else we did!
Why was the server down so long?
A bunch of core technologies we use are close to reaching their end of life. This is the part we knew about for months or even years, and thus I had already spent many hours evaluating and preparing the necessary upgrades, so I could apply them fairly quickly.
However, there's a bunch of things that you simply cannot do on a live system easily, such as kernel upgrades and other low level security updates. Also, doing that live could have opened new security and compatibility issues for the not-yet updated services. So it was convenient to do right now as well. That's why we locked most of the homepage away so we could developed and test the updates without risk of interfering with live usage.
So what did we do?
- Kernel and all base packages upgraded to latest
- Upgrade from PHP 5.3 & 5.6 to PHP 7+
- Upgraded forum software
- Upgraded all plugins and homepage to use safer (and more performant) code and DB connections
- Upgraded account security
- Updated email authentification so Google doesn't throttle emails by the forum anymore (no more hour-long delays in email delivery)
- Added TLS encryption to outgoing emails. Now we truly serve everything via TLS! Thanks, Fritz!
This is a pretty impressive list and almost no part of the webserver was left untouched. Considering that a lot of the plugins and code we use is either custom-made and/or hasn't been updated in years, it took me a couple of days of wading through it all and making sure everything is airtight for hopefully at least another ten years
.If we did everything right, there should be very little that you will notice when browsing the homepage or the forum. I can't guarantee that, so I'm counting on you informing us about any remaining bugs you spot!
Summary
We had someone/something possibly use a PHP exploit to inject a SQL command. No account was compromised.
We upgraded the webserver and everything on it, making everything state-of-the-art secure.
A sweet bonus: The homepage and forum are now served about 60% faster than before, leading to an impressive ~300-400ms response time (TTFB) in the forum. Given the size of our database and the complexity of our code, this is amazing.
Just so you get an idea of the scale of things:
Our database contains just shy of one hundred thousand forum users and over 2 million(!) posts.
It not only includes a wealth of user data (emails, passwords etc.), but also a huge amount of knowledge regarding PR and BF2 in general, and not least of all sweet memes and raw history.
I am an advocate for transparency, as I believe only by being open about what happened and our response to it can you trust us in the future.
If you have feedback, questions or concerns, feel free to ask, I'll try to answer to the best of my ability.
-
Awesome stuff Senshi and Fritz!Posted 2019-02-26 at 09:00 by Rhino 
-
As always: A big THANK YOU to those who invest their time and brains working on PR to keep it as what it is: Best game ever!Posted 2019-02-26 at 16:57 by captainfuture
-
Good work guys! I do know how it is to have a forum needing version upgrade for years, but it just has so many issues, one being the time and courage to start
Btw, any chance to have forum editor WYSIWYG mode back? It disappeared (at least for me) at some point last year and i've asked it a couple of times. Would be awesome, editing long event announcements in BBCODE is nice, but i would still enjoy an easier wayPosted 2019-02-27 at 00:08 by temexter
-
If you don't sure the way how attackers did it, so it's not possible to be sure that our data is not compromised.
Posted 2019-02-27 at 03:30 by fenriz9000
Updated 2019-02-27 at 03:35 by fenriz9000 -
PR website is ready for another 10 years now))) Good work!Posted 2019-02-28 at 13:44 by Hokunin
-
Great work guys, many thanks!
As Hokunin says, 10 more years of realitymod.com is Wunderbar!
And importantly, as Senshi pointed out, sooo much great content on this site! A lot of it is still useful, interesting or fun, in some of us's estimation.
I'll make another overdue donation.Posted 2019-03-02 at 18:20 by TheMerchantOfMenace
-
Quote:Originally Posted by fenriz9000
If you don't sure the way how attackers did it, so it's not possible to be sure that our data is not compromised.
The data from the SQL database might have been pulled, that is perfectly possible. But there's not a lot of data in there, the most valuable being your email address and the securely encrypted passwords.
Hence the recommendation to ABSOLUTELY change your password, and not to just use the same one again. An attacker now has all the time in the world to bruteforce the PWs.Posted 2019-03-03 at 13:22 by Senshi
