Solutions Methodology of Mobile App Testing on Different Platforms/Devices into the Cloud, Automation and Manual environment.

Hi,

As we know the "Testing" phase play very important role into the SDLC process especially to improve the Quality and standard of the products from different-different corners. Since, the current generation of IT (i.e., enterprise and consumer organization) is moving towards into the mobile computing, day-by-day launching new kind of devices for different-different platforms with various kind of features makes little challenge for the tester to test the app (i.e., mobile app) into the several devices. In this post, I am going to highlight the standard approach and methodology of testing the application for different device platforms on top of their internal features and functionality. Nowadays, the mobile market has 4 main OS (Android, iOS, Blackberry and Windows Phone 7) that are applied to a wide range of handsets. This implied uniformity in the way an application looks across all devices that run the same OS. So, for example, an Android application would look practically identical on any Android mobile phone. Therefore, an application vendor can create a test on one android device and run – the very same test – on all Android devices (phone, tablet etc).

Here, I am introducing and highlight the existing solutions which is mainly used as the powerful mobile automation tools. They are:

1. PerfectoMobile,
2. Keynote DeviceAnywhere, 
3. SeeTest by ExperiTest, 
4. FoneMonkey (MonkeyTalk), and
5. JAMO, etc.

 *Note: Except of the FoneMonkey (MonkeyTalk) - all others are commercial tools and not free.

1. PerfectoMobile: Perfecto Mobile is a global leading provider of cloud-based testing, automation and monitoring solutions for mobile applications and websites, utilizing a wide selection of REAL mobile devices accessible via the web.
The MobileCloud enables developers and testers, located anywhere in the world, to access a comprehensive range of the latest mobile handsets and tablets via the Internet. Users can develop, test, deploy and monitor their mobile applications and services without having to physically obtain the handsets. The MobileCloud handsets and tablets are connected to LIVE cellular networks.

Perfecto Mobile solutions are specifically tailored to address mobility in the various stages of the application life cycle and to be used by the various departments. 

Enterprise Class Mobile Testing Solution:

• Automate - test cases and re-use them across multiple devices
• Test new smartphones & tablets as soon as they reach the market
• Share & collaborate with globally distributed teams and partners
• Integrate with existing ALM systems
• Constantly monitor performance & availability on real handsets

1.1 MobileCloud for QTP

A fully integrated mobile testing solution for Quicktest Professional 

MobileCloud for QTP is a complete solution for automated functional and regression testing of mobile applications within HP's QuickTest Professional (QTP) product. MobileCloud for QTP seamlessly integrates native QuickTest Professional scripting and flow control with Perfecto Mobile's comprehensive cloud-based automated mobile testing solution. This breakthrough solution enables you to naturally extend your existing HP Functional Testing and Application Lifecycle Management (ALM) environment to mobile applications.

Why Use MobileCloud for QTP?

Innovative design and natural integration with QTP enable you to cost-effectively extend your functional and regression testing environment to mobile:

• Native QTP scripting for mobile - No need to learn new scripting language facilitates script creation and maintenance
• Fully integrated user interface - Use the same UI for normal QTP scripts, mobile scripting and hybrid scripting - Screenshots and video improve collaboration among globally distributed teams and shortens correction cycles
• Rich image-enhanced reports - Single centralized test management platform maximizes cross-organizational visibility and reduces TCO
• Seamless integration with HP Quality Center ALM - Single centralized test management platform maximizes cross-organizational visibility and reduces TCO
• Cloud-based access to REAL handsets in live networks - Improved quality without the hassle of procurement and network allocation
• Device-agnostic mobile scripting - Build test cases that can be easily maintained, reused and ported to multiple mobile platforms

The Best in Cloud-Based Mobile Testing - Inside QTP!

MobileCloud for QTP lets you enjoy the best of both worlds – you get the world's most advanced mobile test automation solution, while preserving your existing ALM frameworks and methodologies. Key features and capabilities include: 

• Secure cloud-based access to real devices - testers can select and immediately access hundreds of mobile devices from inside QTP. 
• Fast, easy and maintainable mobile scripting – testers can use high-level intelligent mobile-specific functions that emulate real user operations (e.g., press keys, open browser, select image, type text, etc.)
• Device-agnostic keyword-based script creation - ScriptOnce technology enables keyword-based scripting, dynamic search for text and images, scrolling and visual relationship association. 
• Tight UI integration - users work on the device directly from the familiar QTP UI, using the same QTP feature set and scripting environment for normal QTP scripts, mobile scripting and hybrid scripting.
• Rich image-enhanced reports - complete testing information, include active screenshots and video recording, is embedded into the QTP report. 
• Full back-end integration with HP Quality Center – Native QuickTest Professional scripting, data and report structures enable centralized test management and cross-organizational visibility of your mobile apps via HP Quality Center.

1.2 MobileCloud-Interactive

The MobileCloud-Interactive service enables users to test mobile applications, websites or services on a multitude of real handsets  and tablets via the web and control them as if holding them in their own hands. Using  MobileCloud-Interactive, you'll be able to streamline development, testing and support processes, perform more tests and ensure optimal compatibility of your product or service, on more handsets, networks and locations.

Main Features

• Directly install applications from your PC to the device
• Send text and numbers to the device using your keyboard
• Simulate handset events in a single click (Call Me and SMS Me)
• Capture screenshots and record videos
• Share test reports with other colleagues and partners
• Share devices with others to demo new apps/features, reproduce a bug
• Reserve handsets for use whenever you need

1.3 MobileCloud-Automation

MobileCloud-Automation allows users to execute automated testing on real handsets and tablets located anywhere in the world. Users can automate their testing using the intuitive web-based user interface, which allows the creation of keyword-based ScriptOnce scripts. Once the scripts are done, users anywhere in the world can access the devices and run the scripts.

Benefits

• Easy and efficient script creation – Reduced manual testing efforts on all platforms 
• Full flow control – including conditions, loops, flow commands, expressions and data-tables
• Detailed reports  - Reporting of each execution includes vast variety of information such as actual results, screen captures in digital format, continuous video recording of the entire script and graphs showing statistics about the execution
• Enhanced collaboration - saves time and money on problem reproduction by using real time online handset sharing with multiple participates  
• Open REST API - allowing the integration into 3rd party tools such as test management, monitoring, defect tracking and ALM in general.

Benefits at a Glance

• Fastest Time to Publication – Pre-certification and automated testing drastically reduce queues and rejects, resulting in huge time savings (up to 75%). Thus, premium applications are available in the store as soon as a new handset is launched
• Increased quality and reduced risk – Automated testing and monitoring means you can test on many handsets, ensuring the highest levels of quality and user experience
• Gain a competitive advantage – Simplifying the certification process will attract more developers to your store, enhance your reputation and reduce operational costs
• Cost-effective - Automation lets you perform more tests on more handsets, locations and networks in less time and with less effort from your testing team, and less cost 
• Flexibility - Handsets can be added or replaced within a matter of hours with only a simple software update, enabling app stores to quickly respond to market trends

2. Keynote DeviceAnywhere: Provides a service for planning, testing and monitoring the functionality, usability, performance and availability of mobile apps and websites.

2.1 Enterprise-Class Mobile App Lifecycle Management (mALM):

Keynote DeviceAnywhere’s Test Center Enterprise (TCE) cloud includes interactive manual testing, test automation, and monitoring solutions allowing companies to bring higher quality mobile apps and websites to market faster, at a lower cost. TCE also provides intuitive planning, collaboration, management and reporting tools enabling customers to manage their mobile app lifecycles completely in the cloud.

Cloud-Based, SaaS Architecture

• 24x7, remote online access to all popular mobile devices including Android, iOS, BlackBerry, Windows Phone 7, and WebOS Smartphones and Tablets.
• An always-on, globally accessible device bank saves significant cost and effort vs. establishing and managing mobile QA labs for distributed teams.
• Test and monitor across multiple devices or test device-to-device interactions from any location in the world.
• Online reporting (including video and screen capture) and collaboration enable efficient review and sharing with colleagues.
• Built for on-demand use in our enterprise cloud environment as well as for private cloud deployments.

100% Mobile Ready – No Restrictions

• Test or monitor any mobile app or website – native, HTML5, network-aware, streaming, multi-media, etc.
• No infrastructure investment required.

         Overview:

 

• 2.1.1 Planning: Keynote DeviceAnywhere helps enterprises target their development and test efforts on the most relevant devices and platforms with two free, valuable tools:
  
  • Device Planner is a tool aimed at helping developers and businesses decide which devices and platforms to develop on, to reach the largest potential audience for their app. 
  • Test Planner is a tool which provides companies with a recommended list of devices for testing their apps on, to generate optimal test coverage using the least resources.
  
  

• 2.1.2 Execution:
  
  • Test Center Enterprise Automation is a SaaS enterprise software service that automates testing of mobile apps and websites.The product aims to ensure reliable, repeatable and reportable results for any mobile app, on any device, operating system and network. Using the service, engineers can perform unit testing on real devices. Similarly, testers are able to troubleshoot field issues and perform the following kinds of tests: UAT; BAT; Functional Testing; Regression Testing; and Integration Testing.
  • Test Center Enterprise Interactive is a cloud-based service designed for manual testing of mobile apps, designed for enterprise users. The service allows enterprises to remotely interact with and control a device’s functions including pressing handset buttons, viewing LCD displays, listening to ringers and tones, playing videos, tapping and swiping touch screens, muting, powering on/off, increasing volume and other functions as if they were holding the physical device in their own hands.
  • Test Center is a cloud-based service designed for manual testing of consumer-facing mobile apps.
• 2.1.3 Analysis:  
• Test Center Enterprise Monitoring is a SaaS platform targeted at enterprises, which provides monitoring of mobile services to detect potential problems early and ensure quality of service.

3. SeeTest by ExperiTest: Experitest’s SeeTest product suite is based on a patented technology that uses self-learning diagnostic and matching algorithms and a modular self- enhancing image recognition technology.SeeTest uses a client-agent architecture and plugs into all existing test automation environments, such as QTP, C#, JUnit, Python, and Perl.

ExperiTest does have 3 major services:

1. Mobile Automation:Automation tool. Records on real devices. And on Emulators. Plugs into QTP, TestComplete, C#, RFT, Java, Perl, Python.   
2. Mobile Manual:Manual testing tool. View the mobile device on your computer screen & test using your computer mouse & keyboard. 
3. Mobile Manual:The mobile device remote access & scheduling platform.

Features   

• Connect all mobile devices to one central hub/server that resides in your organization  
• Enable remote access from anywhere in the organization to any device
• Book & Schedule devices
• Advanced booking options including: device filter by OS, SW version, tablet/device and more
• Enforce device usage and remote access based on user class 

Benefits 

• Top security – because it is your own organization’s security
• Easy remote access to a large inventory of mobile devices
• Efficient usage of mobile devices for testing by sharing resources (by accessing remotely) across the organization
• Support for any device, including new devices and proprietary hardware devices that have not yet been commercially launched

4. FoneMonkey (MonkeyTalk):
It's free & open source functional testing tool for iOS and Android Mobile Applications

• MonkeyTalk is the leading tool for automated testing of iOS and Android applications 
• Robust, cross-platform recording and playback 
• Test native, web, or hybrid apps, on iOS simulators, Android emulators or real devices -- no jailbreaking required 
• Everything from simple "smoke tests" to sophisticated data-driven test suites 
• Another great free and open source functional testing tool from Gorilla Logic

 For more details: Please refer below link to understand the services into video format.

 http://vimeo.com/monkeytalk/part1

Other features include:

• Robust script recording and playback – more comprehensive recording and playback for user interactions, including taps, keyboard input, dragging and scrolling for all Cocoa Touch components.
• Code generation options – FoneMonkey can now generate ready-to-run test scripts in Objective-C that can be run under OCUnit, or JavaScript that can be run with Apple’s UI Automation tools.
• Optimized for both iPhone and iPad platforms and user interfaces.
• Works on both simulators and real devices.
• Test playbacks can be completely automated.

Please refer "QuickStart demonstrates using FoneMonkey" and "Setup process with different versions of XCode" via hlinks and their respective screenshots and videos.

5. JAMO: Jamo Solutions automates testing of mobile applications by using the same techniques on mobile devices that made automated testing on the PC so successful. Using the Jamo solutions M-eux Test software  will be able to replay the actions of the end user directly on the device.

M-eux Test 

The M-eux Test software allows you to automate, monitor and maintain your mobile test cases. M-eux Test recognizes the GUI elements of your mobile application resulting in test cases that are :

• Device model independent.
• Completely re-useable across different versions of the service/application under test.

M-eux test integrates flawlessly with existing well known environments such as Eclipse, QuickTest Professional from HP and Visual Studio from Microsoft. This approach results in scripts that are :

• Easy to create and maintain

Architecture:An agent is installed on the mobile device. The agent is responsible for the recording and the replay of the test commands The Device is connected to the PC using either WIFI or a USB connection. The Device Manager is the gateway between the connected devices and the extension inside Visual Studio. The architecture allows the connection of multiple devices. One test script can execute against multiple devices.

 Supported Mobile OS:

Device Operating System	System Supported versions
Android	Android 1.5 Android 1.6 Android 2.0 Android 2.0.1 Android 2.1 Android 2.2 Android 2.3 Android 2.3.3 Android 3.0 Android 3.1 Android 3.2 Android 4.0.1 Android 4.0.3 Android 4.0.4
BlackBerry	BlackBerry OS 4.6 BlackBerry OS 4.7 BlackBerry OS 5.0 BlackBerry OS 6.0
iOS (iPhone, iPad and iTouch)	iOS 4.2.x iOS 4.3.x iOS 5.0.1 iOS 5.1 iOS 5.1.1
Windows Mobile	Windows Mobile 2003 Pocket PC second edition Windows Mobile 2005 Pocket PC Windows Mobile 6.0, 6.1 and 6.5 Classic, Standard & Professional
Windows CE	Windows CE 5.0 Windows CE 6.0
Windows Phone	7.5

 Here users can get the lists of sample videos for the demo of the tools and it's methodology:

" Video Sample "