OT(ish): Move MgoBlog to the Public Cloud?

Submitted by Sinsoftheschafer on February 3rd, 2016 at 4:23 PM

I want to thank our fearless leader for his effort in this site; as we sit here in the after-Harbaugh /pre-Gary glow I think its important to realize that none of this would have likely come to pass if it wasn't for him and his efforts.  I don't think this is hyperbole; his email outing was the final nail in the Brandon coffin.

That said, I feel like the site infrastructure lets us down on days like today.  We basically punted the server load to twitter to avoid the Gary rush.  In this day and age, it feels like elastic computing has evolved to the point where we shouldn't have to do this.  A comination of pre-packaged dupal (Cloud Formation/Cloud Laucher on AWS/GCP) and elastic load balancing/autoscaling should enable our leader to a)lower is bill and b)support #chaos.  Sooooo: are there any mgocloud arcitechts out there who would love to chip in a reccomend a solution to keep the site performant at all times?



February 3rd, 2016 at 4:28 PM ^

Yeah. I don't want to sound like an ungrateful jerk for all the free content but the site is impossible to get into on days like this and even on other random days I get booted out. I'd contribute money to a new server.


February 3rd, 2016 at 4:28 PM ^

I know (or highly suspect) that you were writing in English, but I have no idea what you just wrote...is a "Cloud Formation" a variation on the single wing offense? Maybe it's derivative of the 46 defense of the '85 Bears?




February 3rd, 2016 at 4:29 PM ^

On that same note, any good developers interested in working on a social media app protect with me? Would be iOS or cross platform ideally. I started building it in swift but since Parse went down I'm looking at AWS with DynanoDB for a backend. I don't have much free time to devote to this so would love to partner up on it with someone!

Sent from MGoBlog HD for iPhone & iPad


February 3rd, 2016 at 7:03 PM ^

Honestly, it wouldn't need to be AWS, but I have a strong suspicion that the infrastructure on this site isn't very elastic, so any design that could semi-easily ramp up for heavier loads would be good. AWS just is a nice option.

And when we've used AWS, the bill wasn't much different than we saw with our own boxes being run at the datacenter.

Rufus X

February 3rd, 2016 at 4:37 PM ^

But I am in agreement, with the requisite qualifier "I love the site, but..." 

It seems to me that this fine website being on such an unreliable infrastructure is like Al Borges putting Denard Robinson under center despite ovewhelming evidence to it's ineffectiveness. 

And no, I don't even know if "unreliable infrastructure" is the correct syntax, but you get my point.


February 3rd, 2016 at 4:39 PM ^

It looks like the site already uses Cloudflare as a CDN for statically served content. As for what has been configured and deemed to be static vs dynamic only the admins can say, but it's possible that it could be optimized.



February 3rd, 2016 at 4:42 PM ^

I've been a web developer for 16 years and this site has taught me that Drupal is to be avoided at all costs.  There is no way that any website the size of this blog should be getting 503 errors like they do.  I'd be more than willing to look at what they are using and make cost effective recommendations.  I've seen one server crash because of traffic and if this site is getting that kind of traffic.  They don't need to worry about money.


February 3rd, 2016 at 8:48 PM ^

I'm sure many of us would be eager to chip in a few bucks to alleviate some of the frustrations inherent in using these boards. 

If there's any concern that it would take away from beveled guilt, fuck it, make it that you have to make a matching donation to beveled guilt. That would probably end up getting people to donate who haven't donated in the past. 


February 3rd, 2016 at 4:44 PM ^

Server upgrades, drupal upgrades, CSS upgrades to allow better ad placement, fixing voting, it has all been talked about here and either not happened or taken much longer to implement than it should.  I see Brian tried Cloudflare today, but it didn't get the job done.  Perhaps MGoBlog Drupal needs to go the way of the Haloscan variant.  Drupal itself is never that pleasant and perhaps this site just has so many customizations it is time to dump it.

We could use the upgrade though, MGo used to be one of the better sites in terms of layout, stability, etc.  Now compared to Shaggy, /r/cfb in reddit, etc it is lagging behind. 


February 3rd, 2016 at 8:31 PM ^

Its main benefit is you are outsorcing a lot of the management of the system, companies guarentee uptime (99.9999%), and you arent tied to the physical hardware as cloud companies generally allow you to delete/add new instances at will. Basically instead of physically running your own server (either at home or hosted in a datacenter) or renting out a single actual box in a server farm, you are renting out a virtual machine which can be running on whatever hardware. 

Cloud companines are basically renting out space from their own server farms and slapping a self-service interface on top for adding/deleting vms and some other customer service benefits


February 3rd, 2016 at 4:53 PM ^

From having worked with heavy and spikey web traffic I speculate that the MGoBlog site is suffering from Drupal(PHP)/MySQL-itis in that one server can only handle roughly 600-1000 simultaneous requests. I don't know what the configuration of this site is like but to make it work for, let's say 10,000 simultaneous requests, they'd have to do some type of load balancing, buy more servers/nodes, which equals more $$$. However, this site deserves to be accessible when there's news, and it's let us down on those occasions. 

It can get very expensive to try to work around the limitations of PHP/MySQL. A better idea is to switch to a lower overhead and less blocking tech.

However, I'd like to help! Please let me know if I can chip in. I've successfully transitioned PHP/MySQL applications to Node/Mongo and seen the vast improvements in load and response times.  


February 3rd, 2016 at 6:57 PM ^

This is probably the case, especially the timeouts. My guess is the stack traces are all about an inability to access MySQL or long queries.

There are other PHP frameworks out there (silex and lumen are quicker, Symphony and Laravel are more robust and Drupal-like), and others have suggested Ruby and Python alternatives. And my guess is that some sort of caching (say with redis or mongo) of parts of pages (like comments and main text) would help immensely. But it sounds like Drupal is here to stay, so my guess is that the likely solution is just more boxes.


February 3rd, 2016 at 4:51 PM ^

I did not experience a lot of problems today relative to some of the other things I've heard from folks, but about an hour ago I was kicked out and then unable to log back in for about 20-30 minutes as the site was intermittently up and down for a while. One thing that I did notice today is that the 524 errors (connection timed out) started again during peak traffic after a long absence - they were prevalent during the height of football season. If you open multiple tabs for threads, as I do when modding, some were opening and some were not so bandwidth was definitely an issue.


February 3rd, 2016 at 6:54 PM ^

Bandwidth upgrades are in desperate need around here. Start a funding campaign, I guarantee we can meet it.

This site is basically unavailable on gamedays, announcements, other events etc.

The exact time it should be readily available. Not saying anything new here, just adding that "it's getting better" is not correct.

I hate sending my traffic and dollars elsewhere. Please make it easier to give you my traffic and revenue.


February 3rd, 2016 at 5:00 PM ^

"Put it in the cloud" is easy to say but much harder than you would expect.  Just hosting the site on cloud products would be easy but the elastic part requires a lot of engineering and staff.  It's not nearly as automated as you would think.  In particular, scaling backend services is much more difficult than front end.


At any rate, I recall a few months back Brian had a front page post looking for a developer with Drupal experience, so I expect he's got some plans around this.  It's going to be a long-term process.


February 3rd, 2016 at 5:00 PM ^

Wouldn't a caching solution would be cheaper for the (relatively short) surges in traffic that the site sees?  Switching to a full AWS cloud architecture with properly sized instances and auto scaling groups would be fairly expensive*, require more attention to maintain, and a little overkill for the normal traffic patterns and days. A caching layer or CDN may not give you up to millisecond accurate data or posts but it would allow the site to continue to server traffic without breaking the bank

* I say this with zero knowledge of the existing infrastructure, cost, or budget.