The Recent Downtime

[ENTERPRISE]

Hello all, 

 

So we have been down for a few days now much to my regret and disappointment.  There were a couple of factors that led up to this issue. 

 

1. We were experiencing some template conflicts which lead to posts and the editor disappearing. Many of you will have seen this. 

2. Breach & Data deletion.

 

These two combined meant that having the site online was untenable. From my current investigations I have tracked a couple of logins within our infrastructure, seemingly from an old administrator who's access was revoked, or at least that was what was thought. Unfortunately this person decided to delete some of the data that related to our uploads folder. This would include such things as anything image based as well as attachments.  The individual also disabled our data versioning, which is essentially the backup solution to the main storage, they also deleted a large portion of our backups. Fortunately I had a backup of the sites data, but unfortunately it is not a backup that covers the entire lifetime of the site. 

 

This was a malicious attack only. I and my two admins have checked and there was not any data removed nor compromised. Plus any data obtained would have been encrypted in any case. So rest assured everyone's user data is safe.

 

We do not believe the template issue was related to the data loss, this looks to be an inherent issue with the caching technology being used at the time. We are no longer using it, as such we expect the template issue to be resolved.

 

How on earth did this happen ? 

 

Ultimately the buck stops with me. While I employ other members of staff present and in the past to help with administration and development, it is my responsibility to ensure continuity. While we indeed had backups in place, these do not help if another has access to be destructive. Unfortunately it seems that a past admin, all of a sudden decided it was time to cause issues. I had asked one of our more recent admins to ensure that all security protocols were in good standing, which included the removal of old logins/access. However evidently this was either not completed, or simply not undertaken properly. However that being said, I should have double checked. As I say the buck stops with me, I can only apologise for this oversight.

 

So what does this mean ? 

 

Moving forward there will inevitably be missing images and attachments from across the site. This includes user avatars, so if yours is not showing, that will be why. At this point for anything that was uploaded to the site that is no longer appearing, it is gone, as we do not have additional backups in order to restore it. I can only profusely apologise for the inconvenience this has caused. 

 

What has changed now we are live?

 

During the downtime I decided that we would do a full migration of the site from the infrastructure we were on from AWS to a dedicated server. The reason for this is mainly due to management and efficiency. While AWS is impressive, it is by no means efficient, especially when it comes to the deployment of software updates and custom development work. After months of fighting with it, I saw an opportunity to move while the site was down. It was the case of making the best of a bad time. Furthermore, due to this incident we now have multiple offsite backups in place as well as the local backups.

 

What now ? 

 

We are working on restoring some of the missing visual elements to the site. We have reverted back to a default template and we will be re-introducing some of the missing visuals that we had all become accustomed to. This includes the Rep Meter, User group titles (on mobile), Increased avatar sizes both on desktop and mobile & more. So bare with my while I work to get everything back to normal.

 

Ok, so I am back, is there anything I could do ? 

 

There is nothing you as a user you need to do, however if you would like to help out, you could try doing the following. 

 

1. Re-upload your avatar (if it is one the missing ones) 

2.Re-add images or attachments to your content if it has been affected.

3. If you come across some strange behaviour or something you think could be a bug, please report it in our Submit A Bug section

 

Anything new ? 

 

Well we have installed a couple of new features and we are just finishing up the first phase plugin which will include Dark Theme, Custom User Status's & Social Badges. However we will reveal all this in full in the coming days.

 

.............................................................................................................................................................................

 

In summary I can only carry on apologising for the current state of affairs. We aim high but there have been some rough times and stumbles, this being one of them. I hope to carry on seeing you all round these parts despite the frustrations and inconvenience caused to you all. 

 

Kindest Regards,

E

[UltraMega]

Glad the site is back up. Bummer about the former admin but I'm sure this will just be a road bump in time. 

 

Spill the tea, who was it?

[ENTERPRISE]

Never spill the tea! It is a crime. 

 

[BWG]

Can we get a tutorial on Dark Theme implementation? I can't seem to find it. 

[ENTERPRISE]

2 minutes ago, BWG said:

Can we get a tutorial on Dark Theme implementation? I can't seem to find it. 

 

Not sure if serious, however it is not yet available. Should be in the next 24 hours or so, literally got one or two kinks to work out.

[iamjanco]

Thanks, E. Glad to see the site back up.

 

As for person who used to work for you, there are blacklists that can be employed to help ensure they never do that to another site. 

[BWG]

Just now, ENTERPRISE said:

 

Not sure if serious, however it is not yet available. Should be in the next 24 hours or so, literally got one or two kinks to work out.

 

Can we just agree to disagree  I was actually serious. You're acting like that shiny 3090 FTW3 Ultra Hydrocopper in my EVGA queue 1 second away from the last reported notify time fulfilled.

 

But, I'm very glad we're back. I'm sorry that all happened. You didn't deserve that at all. 

[ENTERPRISE]

14 hours ago, iamjanco said:

Thanks, E. Glad to see the site back up.

 

As for person who used to work for you, there are blacklists that can be employed to help ensure they never do that to another site. 

 

Not to worry, the blacklisting will be getting used !

14 hours ago, BWG said:

 

Can we just agree to disagree  I was actually serious. You're acting like that shiny 3090 FTW3 Ultra Hydrocopper in my EVGA queue 1 second away from the last reported notify time fulfilled.

 

But, I'm very glad we're back. I'm sorry that all happened. You didn't deserve that at all. 

 

Well not long to wait bud, the Dark Theme I am hoping to launch later today, but I will only do it if you are being a good boy.  Oh nice ! I have one of those GPU'S. Overkill, but nice haha.

[Avacado]

36 minutes ago, ENTERPRISE said:

 

Not to worry, the blacklisting will be getting used !

 

Well not long to wait bud, the Dark Theme I am hoping to launch later today, but I will only do it if you are being a good boy.  Oh nice ! I have one of those GPU'S. Overkill, but nice haha.

But he's NOT a good boy. No dark theme for you. 

[Bastiaan_NL]

Glad we are back online! 

[Supercrumpet]

Glad we're back, and thanks for the info and explanation E!

[GanjaSMK]

Thanks for the update.  Bummer you and admins had to deal with that kind of thing.  Lame. 

 

Glad the site is back up !

[ENTERPRISE]

Yeah we are super glad to be back and with a few new things haha.

 

Also just incase anyone has not seen It yet, take a look :

 

https://forums.extremehw.net/topic/1360-recover-re-insert-images/

 

It is possible for you to rescue broken images in some instances. 

[iamjanco]

26 minutes ago, ENTERPRISE said:

Yeah we are super glad to be back and with a few new things haha.

 

Also just incase anyone has not seen It yet, take a look :

 

https://forums.extremehw.net/topic/1360-recover-re-insert-images/

 

It is possible for you to rescue broken images in some instances. 

It works, though it's a bit of a tedious process. I'm about halfway through my previous posts and I'll finish up the other half later today.

 

emojis you have to rebuild though; and some embedded Youtube videos.  And anything embedded in something quoted can only be rebuilt by waiting on the person who originally made the quoted text/image combos to rebuild them, then requote what was originally quoted. 

 

Mind you, not everything image-wise seems to be broken; but you won't know what is broken without reviewing all your previous content. 

[ENTERPRISE]

24 minutes ago, iamjanco said:

It works, though it's a bit of a tedious process. I'm about halfway through my previous posts and I'll finish up the other half later today.

 

emojis you have to rebuild though; and some embedded Youtube videos.  And anything embedded in something quoted can only be rebuilt by waiting on the person who originally made the quoted text/image combos to rebuild them, then requote what was originally quoted. 

 

Mind you, not everything image-wise seems to be broken; but you won't know what is broken without reviewing all your previous content. 

Yeah its a little tedious for sure. Im going to look to see if there is a safe way to automate these things in the coming days, I know that it can be done but may have to speak to my admin about it as likely will involve a MySQL query as I don't think re-build tools are native to this platform BUT that is what I will be looking into.