Occurs on 3 different devices on 3 different networks... Jaysen working overtime?! Keep up the good work. I know it may be painful now but it will help BBC out which means it is helping us too!
http://www.boatbuildercentral.com/
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@boatbuildercentral.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
500 internal Server Error
Re: 500 internal Server Error
Thank you BB!! Yes, Jaysen is working the Forum as we speak!!! Jeff
- Jaysen
- * Bateau Builder *
- Posts: 6500
- Joined: Wed Aug 05, 2015 7:59 am
- Location: St Helena Island, SC
- Contact:
Re: 500 internal Server Error
A bit of clarity on both the work, the problem and why it impacted the forum. then I will explain why we are doing all the work we are doing this quarter.
work
Today we are working to ensure "backward compatibility" to allow BBC to reduce the number of live sites that need to be maintained. This work equires some DB and OS level changes as well as software config. All of that work is on the boatbuildercentral.com website.
problem
The problem today was caused by a couple of "rouge" processes that conflicted with our new backup process. This conflict create a backup of request for I/O driving system load very high. The rouge process are of unknown origin, but the backup process was Jaysen's.
why it all broke
Currently BBC uses a single dedicated host for ALL the sites. This means that if one site or the server has heavy load, all sites get killed. While the high I/O was not site traffic related, it caused the DB server (software runs on the same dedicated host) to be slow which broke ALL the sites.
what's the point of this work
One of the things we are doing it providing more isolation between sites to prevent this exact problem. BBC will migrate to new hardware with out of band backup. Each software package will be hosted to optimize cost, availability and reduction of management on Jeff's side. at a high level this means:
* BoatBuilderCentral.com will become the core of all BBC services
* The store site will move to a managed service provider on clustered systems.
* The forum and gallery will move to a unified cloud instance (techno-nerds: containers in AWS)
* URLs will be changed but all OLD urls will automatically send you to the correct place
* All unused sites will be purged
* All remaining legacy sites will be hosted appropriately based on usage.
The net effect to end users will be ... updated software with more features that runs faster and has less downtime.
The net effect to BBC is ... they can stop worrying about IT and focus on boat building products and services. Oh.. and they save a couple $$ too.
Lastly, we are working to improve the security of the sites all around. Part of this is via simple software upgrades. Some of it is removing accounts and backdoors that we did not know existed. While we do not believe there has been any breach of data (no evidence, and I did look) we are certain that we have closed a LOT of doors that could have been exploited (we still have a few more to close).
I'm happy to answer any questions anyone has about this event or the plans. I may have to hide some details but I will be as open as possible. Fire away.
work
Today we are working to ensure "backward compatibility" to allow BBC to reduce the number of live sites that need to be maintained. This work equires some DB and OS level changes as well as software config. All of that work is on the boatbuildercentral.com website.
problem
The problem today was caused by a couple of "rouge" processes that conflicted with our new backup process. This conflict create a backup of request for I/O driving system load very high. The rouge process are of unknown origin, but the backup process was Jaysen's.
why it all broke
Currently BBC uses a single dedicated host for ALL the sites. This means that if one site or the server has heavy load, all sites get killed. While the high I/O was not site traffic related, it caused the DB server (software runs on the same dedicated host) to be slow which broke ALL the sites.
what's the point of this work
One of the things we are doing it providing more isolation between sites to prevent this exact problem. BBC will migrate to new hardware with out of band backup. Each software package will be hosted to optimize cost, availability and reduction of management on Jeff's side. at a high level this means:
* BoatBuilderCentral.com will become the core of all BBC services
* The store site will move to a managed service provider on clustered systems.
* The forum and gallery will move to a unified cloud instance (techno-nerds: containers in AWS)
* URLs will be changed but all OLD urls will automatically send you to the correct place
* All unused sites will be purged
* All remaining legacy sites will be hosted appropriately based on usage.
The net effect to end users will be ... updated software with more features that runs faster and has less downtime.
The net effect to BBC is ... they can stop worrying about IT and focus on boat building products and services. Oh.. and they save a couple $$ too.
Lastly, we are working to improve the security of the sites all around. Part of this is via simple software upgrades. Some of it is removing accounts and backdoors that we did not know existed. While we do not believe there has been any breach of data (no evidence, and I did look) we are certain that we have closed a LOT of doors that could have been exploited (we still have a few more to close).
I'm happy to answer any questions anyone has about this event or the plans. I may have to hide some details but I will be as open as possible. Fire away.
- BB Sig
- Very Active Poster
- Posts: 1750
- Joined: Thu Sep 29, 2011 9:20 am
- Location: Tallahassee, FL
- Contact:
Re: 500 internal Server Error
Thanks for the back ground. Just know that we appreciate you doing this! It is a ton of work....
Who is online
Users browsing this forum: No registered users and 15 guests