A lot of testers we've spoken to are being exposed to mobile testing for the first time and most of them don't think about some of the things that are important.
vice president of global quality and testing services, Capgemini
The fourth annual World Quality Report (.pdf), conducted by Paris-based technology consulting firm Capgemini, analyzed the state of software quality in the face of growing use of smartphones and the bring your own device (BYOD) movement. This year's focus was on mobile application security. The global survey was designed by the firm's quality assessor (QA) and testing experts, and reached more than 1,500 CIOs, IT directors and application and testing managers in 25 countries.
The study found that 31% of respondents across the world currently test mobile applications. Of those that do test, 64% of firms focus on efficiency of performance, rather than functionality, usability or security.
"Mobile is a completely new paradigm that presents a lot of new challenges and people haven't really caught up yet," said Charlie Li, vice president of global quality and testing services at Capgemini.
Different mobile platforms coupled with devices supporting various firmware versions, combined with a complex entanglement of mobile carriers, creates serious testing challenges for many firms, Li said. Those surveyed readily admit to being ill-equipped for mobile testing with 65% citing a lack of tools to conduct testing, he said. Emulators can be used to test how software runs on different platforms, but it doesn't test how the application runs on a specific device. Another 52% cited the lack of access to the required devices to conduct testing. Other tools can allow firms to emulate network connections, but Li said most firms want to see the mobile application run on a real network.
In addition, some firms said their QA teams lacked the expertise to test mobile applications against security and functionality requirements. One-third of organizations lack the testing methodologies and processes, and 29% fail to have the specialists necessary to effectively certify mobile applications, according to the Capgemini study.
"The way developers write code is not fundamentally so different," Li said. "A lot of testers we've spoken to are being exposed to mobile testing for the first time, and most of them don't think about some of the things that are important, such as Edge or 3G versus a 4G network. ... There's no formalized training of any kind. Most testers learn on the job."
Li said he engaged a client a few years ago on mobility testing. The firm offshored all of its testing facilities to India and wanted to conduct mobility testing from the country. But the tests needed to be run on mobile devices in different parts of the world. "The roaming charges ended up being more than the actual cost of labor," Li said.
While enterprise IT security teams are addressing security policies and controls to reduce the risks posed by smartphones and other devices in the enterprise, software development and testing teams creating enterprise mobile apps are also addressing security concerns, said Chris Wysopal, CTO of Burlington, Mass.-based application testing services firm Veracode Inc. In a recent interview with SearchSecurity.com, Wysopal said companies often fail to see the weaknesses that can be exploited by attackers.
"It used to be the client system was a Web browser and the back-end system was a Web application and people would test the Web application," Wysopal said. "When you make the front-end a mobile device, a lot of people aren't testing that back-end Web service as they would if it was a Web app. Every time something changes, they think all the risks that were there are gone, but they're not. The risks are still there, but the attackers have to just change their toolset."
If an attacker can create a malicious application and make money through fraudulent SMS charges, it won't be long before a mobile app is built that directly steals intellectual property, he said. A number of firms, including Sunnyvale, Calif.-based Good Technology Inc., enable mobile application developers to create enterprise apps that are contained in a sandbox protecting data stores and data transmission. Even with a sandbox, it's important to test the application, he said. Implementing a sandbox incorrectly enables an attacker to find holes or bypass the sandbox altogether, Wysopal said.
Li advocated a test-driven development model to address mobile, a concept that has been around for years, experts said, but is rarely ever fully implemented. The model emphasizes repeated testing on source code. Companies often aren't ready to make the significant investment needed to change the processes that have been ingrained for years, and some software development and testing teams often push back fearful of the amount of work required. But the process allows clients to start thinking about the quality of the application much earlier than after it is built, Li said.
"We're trying to help clients think about quality, performance and security before a single line of code is written, Li said. "It definitely takes a political change in most places. If done properly, most companies will report that it reduces a significant amount of work down the line."
Mobile fuels interest in cloud-based testing
Cloud-based models for testing have seen slow adoption, but Capgemini and other security firms believe adoption will begin to grow. Testing as a Service (TaaS) is of interest to survey respondents with 78% of them indicating their firm plans to move to TaaS in the next two years, rising to 89% by 2015.
TaaS prevents clients from the costs associated with building their own mobile testing labs. It encompasses everything from the environment and the tools to the techniques and processes necessary to meet the requirements of the application, Li said.
"As a consumer, you're buying a piece of mind that when the app is released, less than 2% of defects will go into production," he said.