Updated 1:38 p.m ET: After prominent source-code hosting provider Code Spaces was forced to shutter its operations following an attack against its Amazon Web Services (AWS) control panel that deleted troves of irreplaceable customer data, a second provider has reported similar AWS infrastructure compromises in what may be a series of attacks against AWS-based cloud providers.
Thursday morning Websolr and Bonsai, two search application infrastructure services offered by One More Cloud LLC, reported that all its EC2 instances were unexpectedly terminated within a two-minute span.
"We're treating our AWS account as compromised, and responding accordingly," the Websolr support team said on Twitter.
As of Thursday afternoon, both Websolr and Bonsai were undergoing the process of reprovisioning instances and restoring from backups where necessary. It's unclear whether the incidents resulted in the irretrievable loss of any customer data.
Earlier this week, Code Spaces' incident began on June 17, when its servers were subjected to a distributed denial-of-service attack, according to a statement on the front page of codespaces.com. Code Spaces deals with DDoS attacks "quite often," the statement noted, but in this instance, attackers also gained access to the company's login credentials for its Amazon EC2 control panel.
The attacker left messages on the panel demanding a ransom in exchange for ceasing the DDoS attack. After confirming that the attackers lacked the private encryption keys necessary to access its machines, Code Spaces moved to take back its control panel by changing the stolen Amazon credentials.
"However, the intruder had prepared for this and had already created a number of backup logins to the panel, and upon seeing us make the attempted recovery of the account, he proceeded to randomly delete artifacts from the panel," Code Spaces' statement said. "We finally managed to get our panel access back, but not before he had removed all EBS snapshots, S3 buckets, all AMIs, some EBS instances and several machine instances.
"In summary, most of our data, backups, machine configurations and off-site backups were either partially or completely deleted."
The statement went on to explain that certain repositories of Subversion code using the URL format "https://svn.codespaces.com/[ACCOUNT]/[REPONAME]" are available for export, but otherwise, all such backups and snapshots were deleted. Git repositories suffered a similar fate, while practically all Code Spaces' machines and Elastic Block Store (EBS) volumes were also wiped out.
Martin Howes, a director with Luton, U.K.-based software design and consultancy firm Springwater Software Ltd., said his organization used Code Spaces' SVN repositories for more than five years before the abrupt shutdown. He said he learned about the incident through Twitter and the statement on the company's website, but had yet to be contacted directly by Code Spaces.
Howes noted that one of the deciding factors in choosing Code Spaces was that the company utilized Amazon Web Services, one of the largest providers of cloud hosting providers in the world, rather than some of its competitors who tried to run their own hosting services.
The impact of the Code Spaces shutdown, Howes said, was minimal for his organization. Springwater Software had both local copies of its data and made weekly backups that can still be accessed, so while the company may have lost access to specific backups that were stored on Code Spaces, Howes said the organization had most of what was needed to move forward with business until a new cloud hosting provider can be chosen.
The company paid for Code Spaces' services in six-month blocks, according to Howe, and there was only about one month left on the current order, so he wasn't overly concerned with the need to seek financial recourse.
"It's just normal caution," said Howes. "Everyone is aware of the risks of putting things in the cloud and wouldn't rely on it completely."
Code Spaces failed to elaborate on how attackers gained access to login credentials, only noting that they had no reason to think any current or former employees were involved, and further details would come after customers' needs were handled.
Though the statement is all that remains of the Code Spaces' website, a version cached by Bing showed the company touting both its security and data backup capabilities.
The cached site said both encryption and SSL-enabled connections were available, while also pointing to an Amazon document detailing the level of security in place at the cloud giant.
Code Spaces also specifically mentioned the ability to offer full redundancy for customers' data, including providing storage through data centers on three continents and a 99% uptime guarantee for its servers. The site too mentioned real-time backups for data, the latest of which was available to "download anytime," as well as a full recovery plan that was "proven to work time and time again."
Despite those assurances, customers are now left scrambling to collect whatever data might remain from Code Spaces.
"Code Spaces will not be able to operate beyond this point; the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in an irreversible position -- both financially and in terms of ongoing credibility," Code Spaces' statement said. "As such, at this point in time, we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us."
Dave Shackleford, founder and principal consultant for Atlanta-based Voodoo Security, said that Code Spaces likely made the mistake of failing to implement multi-factor authentication for its AWS control panel, meaning attackers could have discerned the company's login credentials in any number of ways.
"It is pretty common, unfortunately, even though Amazon makes it fairly simple to [implement multi-factor authentication]," said Shackleford. "It's still pretty standard to just find username and password."
Rich Mogull, CEO of Phoenix-based security consultancy Securosis, said that Amazon does a good job of emphasizing the importance of applying multi-factor authentication and that the company provides a range of options.
Mogull said that multi-factor authentication should be used for all administrator accounts with Web access, and in situations involving API access where MFA isn't available, organizations should look to lock down rights with Amazon's identity and access management, particularly the IAM privileges themselves.
Mogull also highlighted the mistake Code Spaces may have made in responding to the incident, specifically not being prepared to quickly lock down access.
"You don't want to try and play Whac-A-Mole manually in the Web interface," said Mogull. "Write a script to lock it all down at a high rate of speed using the API so the attacker can't respond."
An issue that is just as pressing as how attackers gained access to Code Spaces' Amazon credentials, Shackleford said, is why the company depended on a single-point-of-failure strategy. While Code Spaces' redundancy claims were likely true, Shackleford noted that the company's backups and seemingly its entire business were based in the Amazon cloud – a strategy that left the organization open to exactly the sort of attack that occurred against its management console.
For nervous enterprise customers of other cloud services providers, Shackelford said this incident shows the importance of pressing for answers to key questions, such as whether a provider has a backup strategy outside its cloud environment, and whether it has some sort of secondary facility or outside cloud service where it stores virtual machine images of copies of its infrastructure.
"Really this is a huge failure of not only authentication, but also really a redundancy strategy," said Shackleford. "These guys put all their eggs in one basket, and that basket got compromised. It's a perfect example of the wrong way to securely manage your cloud infrastructure.
"This thing," Shackleford added, "is going to be a case study for everybody's cloud security forever."
Executive Editor Eric B. Parizo contributed to this report.