One of the big buzzwords in technology these days is the idea of “the Cloud”. While this is not really a new idea, some services that are associated with cloud computing have recently become more viable. At CRT we recently had a chance to use a cloud service for our RETS compliance tester, specifically the virtual server service, the Amazon Elastic Compute Cloud (E2) from Amazon Web Services. Our experience was captured in more technical detail in a white paper, but for those that don’t want to get mired into the technical details I will give a high level overview and discuss when using E2 might be advantageous, as well as the potential drawbacks of using E2 and other similar cloud services.
First off, what is E2. The short answer is that it is a virtual private server service, where customers can rent the use of a virtual machine to run online applications on. These virtual server instances can run any software that you want, and you are given full access to them, configuring and running them very much as if they were physical machines in a data center. As an Operating System, E2 supports Linux, Solaris and Windows 2003.
Since E2 makes heavy use of virtualization, it allows you to easily scale up and down as demand requires. E2 lets you save your configured instances so you can create other instances with the same configuration, in addition to offering a large collection of preconfigured instances designed to give you a start for different types of servers (web server, database, etc.). A host of other services also support E2 to allow users to create robust and scalable applications, including Elastic Block Storage, Elastic IP addresses, Automated Scaling, Elastic Load Balancing and monitoring.
So what does all this cost? That depends, as E2 is mainly based on an hourly cost for each instance running. The hourly cost depends on the type of instance, the more power you want, (CPU power and/or memory) the greater the hourly cost. There are additional costs for the other supporting services as well as for bandwidth used beyond a certain base amount. All this means that it’s not necessarily cheap. Amazon’s page for E2 has the service and pricing details.
What are the disadvantages of E2? The biggest one is that you need to trust that Amazon will be able to keep the service up and running without an extended outage or a failure that destroys your data. In light of this, you would likely want to have backups held elsewhere. Also, there is a learning curve to getting the most out of E2, and you will still need someone to maintain your servers, so E2 does not obviate the need for a competent System Administrator.
So the question then becomes, when would using E2 be advantageous? E2 is flexible in a way that would benefit people who have an unpredictable demand on their online services, demand that requires them to have hardware to make sure the service is responsive, but that is not always being fully utilized. With automatic scaling and the preconfigured instances, E2 can allow you to scale up and down with demand needs, so you are only paying for the resources you need and are using. Also, the preconfigured instances can make service maintenance and set up faster and less prone to error. But if you have modest server requirements and little fluctuation in demand or light demand, E2 is likely to be more complex and expensive then a well-maintained physical box.
But if you are someone who finds they need a flexible server environment, or who just wants to make your server setup and maintenance easier, E2 might be a valuable solution for you
Amazon Elastic Compute Cloud (Amazon EC2) is neat stuff, but it and other cloud services are not panaceas for scaling a web application … yet.
One important issue to keep in mind (using Amazon as an example) is that you must re-write your application to work in the Amazon cloud using storage mechanisms such as Amazon S3. This isn’t an insignificant effort, and once you go through it it’s not like you can then easily move your app to another cloud if you are unhappy with the service – you would have to re-write your application for another cloud or to go back to your own infrastructure.
Another issue to keep in mind is that you are responsible for bandwidth and other resources you use BUT Amazon has no Intrusion Detection Systems (IDS) or Intrusion Prevention Systems (IPS) built in … and you can’t use any hardware-based IDS/IPS or integrate a software one with Amazon’s firewalls. This leaves you with all of the responsibility but none of the capability to manage the risk – a tricky spot.
I’m by no means ‘anti-cloud’ … I’ve seen some great things done in the cloud and spent a lot of time talking with cloud providers … but these are just a few of the things to keep in mind when considering cloud deployment. I’m hopeful that in the months and years ahead that these issues are solved, that standards for cloud infrastructures emerge to encourage application portability and competition, and that cloud providers solve the uptime issues and other questions that have impeded adoption.
Matt,
As to your first point, that was not the case when we used E2. We were able to put the Compliance Tester on E2 with no code changes at all. I agree if we decided to use S3 in the app itself it would have required a change, but that is not a requirement of deploying an app to E2, and in fact if you hosted an app yourself and used S3 for storage you would have that issue. An E2 instance works just like a physical box and with Elastic Block Storage you can add any number of persistent volume that act just like any other mounted volume. We use EBS and it’s really powerful and simple to use. It even has easy snapshots that don’t require the application to be halted.
As to the second point, E2 provides a service called Amazon Virtual Private Cloud, which would allow to integrate your Amazon instances with your existing security structure. In this case you would be able to use hardware IDS and IPS just as if you were managing a physical server in an actual data center. Although it’s an additional cost, they do give you the ability to manage your security fully, so the tricky spot is not as bad as it seems.
So I would say, those issues are pretty much solved already.
The white paper mentions the recent T-mobile Sidekick data loss, which true tech people realize was likely not specifically a “Cloud” issue, although the media has run with that. It actually had NOTHING to do with Amazon or their service, so I’m not sure it’s relevant to connect Amazon to the Sidekick issue. Of course, backup is an issue in ANY setup..
As an IDX vendor looking at possibly using Amazon’s service in the future, my first question was about the security requirements of the IDX agreements. Among what I’ve had to pass was stringent “Who owns and has access to the server hardware” questions, as well as having to specify a single IP address that will connect to the IDX servers to access the data.
I hope this article will help the MLS people understand the “Cloud” setup when I get to that point.
Thanks for sharing!
Peter Janett
http://www.NewMediaMLS.com
Andrew,
You’re 100% right that EBS solves the storage problem – but at a cost. I’ve looked into this for hosting web applications and didn’t find it cost effective … leaving us back at re-writing the app to use S3 – not pretty!
Same on the second point … VPC costs $$ and is not the normal cloud deployment.
Peter,
E2 has Elastic IP Addresses, which allows you to create static IPs that you can assign to specific instances, so that could address you concerns about the single IP issue. The hardware issue is a little trickier, and could scuttle someones plans of using E2, but that would be the case for any virtual hosting service.
As to the mention of Sidekick, it was more to stress the point that you have to trust the people that run a cloud service, in this case Amazon. I think you can, but it’s something that each person is going to have to decide for themselves.
Andrew
Matt,
It’s true that those services cost money, but then again S3 costs money as well and I cannot see S3 being so much cheaper that you don’t lose the savings by going to S3 in developer costs.
My main point was that you can address the issues you mentioned. Each person will need to do a cost benefit analysis of course, but the issues CAN be addressed.
The question of E2 being cost effective, as I say in the post, it’s likely to be so if you have unpredictable demand. If you have periods of time, lets say a week out of a month that you get a four fold or more spike in demand E2 can make that pain easier. If you use your own hardware and you want your app to not slow to a crawl, you have to buy enough hardware to handle the high load periods. But then that hardware sits mostly idle the during the long slow times. Even worse is if you get an unexpected spike in demand that exceeds your capacity. With E2 you scale up your instances, paying more of course, but you can scale DOWN when the demand goes back to normal levels. If you use AMIs properly and have pre-configured instances, you can scale up in almost no time, and scale down even faster. You stop paying for those instances when they are terminated, where as the hardware you bought still cost you money.
Amazon even has services that will automatically scale up and down your instances based on demand using your preconfigured AMI. It does cost cost extra, but do give you more flexibility. I also feel it could lower the administration requirements so you can have fewer sys admins, again saving money in salary.
If you have steady demand E2 is not cost effective IMO, but for those that it can benefit the savings in hardware, developer and admin costs could be significant.
Andrew
Andrew,
I definitely agree that there are great potential cost savings in the cloud – the ability to scale up and down as needed is awesome. Yes, you can solve the security problem if you don’t use the standard cloud offering. And yes, if one wants to pay the freight on elastic block storage, it solves the rewrite problem. But, if one is thinking of using the cloud ‘normally’ – using the S3 for storage (block storage is usually just suggested for databases, not the quantity of media files we see in many RE apps) and the VPC is quite the extra – these are concerns to consider. Apps with extremely variable processing requirements – that type of thing – the cloud is a no-brainer.
Matt
Lynx Geosystems has a RETS to Cloud solution available. It stores the images in S3 and is quite efficient.