Testing with Selenium

Introduction:

• Global enterprises are challenged to surface performance and load issues, regression and functional issues, and to accomplish business service production monitoring in Rich Internet Application (RIA) environments that use Ajax technology.
•  Selenium Training is a popular and widely used open-source test development tool. Selenium enables testers to record functional tests of applications in Selenium IDE, a Firefox-based record/playback test development environment. 
• Playback these tests in Firefox, Internet Explorer, Chrome, Opera, Safari and other browsers.

Features and Benefits:

TestMaker integrates Selenium to provide the following features and benefits:

• TestMaker repurposes a single Selenium testing as a functional test, load and performance test, and business service monitor. 
• Scale-up your Selenium testing is to operate in your QA lab, in the PushToTest OnDemand Cloud Test Environment, or both! 
• TestMaker data-enables Selenium testing for test-driven development. 
• Dramatically reduce test operating costs. TestMaker efficiently operates multiple Seleniums test in a grid concurrently. 
• TestMaker runs Selenium tests for browser compatibility testing of your Web application. 
• Produce actionable knowledge by analyizing the results into 350 or more performance and functional charts. 
• Transform Selenium tests into Java unit tests for additional features: looping, variables, conditionals
• Run At User Levels Never Before Possible - Building a test lab used to be prohibitively costly for medium and large scale load tests. Cloud computing delivers bandwidth and computing power to achieve tests in the millions-of-users levels at commodity prices.

Repurposing Methodology and Architecture:

1. PushToTest Repurposing Methodology applied to Selenium testing  enables many types.
2. Functional Testing in the Firefox Web Browser - Record test scripts using Selenium IDE, a plug-in to the Firefox browser. Play the test scripts in the Firefox browser. 
3. Browser Compatibility Tests - Play recorded Selenium testing in Internet Explorer, Safari, Opera, Chrome, and other browsers. Play the tests on a grid of machines with multiple operating environments and multiple browser versions.
4. Functional Testing - Play recorded Selenium tests as smoke tests, regression tests, and integration tests. 
5. Load and Performance Testing - Determine the Scalability Index of your application by observing the throughput of your application at increasing levels of concurrently running virtual users. Each virtual user operates a test use case with one or more Selenium tests.
6. Production Monitoring - Prove compliance with a Service Level Agreement (SLA) by periodically running a Selenium test script. TestMaker sends notification emails when the Selenium script fails or encounters an error condition.
7. Stress Testing - Understand the behavior as your application comes under unexpected high load conditions.
8. In each of the above types of tests TestMaker repurposes a single Selenium test script. The TestMaker architecture make repurposing possible.
9. PushToTest TestMaker operates Selenium tests as load and performance tests by running the tests in one or moreTestNodes. 
10. The TestNode operates the test in a browser (such as Internet Explorer, Safari, Opera, Chrome) or using HTMLUnit. HTMLUnit is an open-source headless browser framework that provides a programmatic simulation of a Web browser.
11. HTML Unit uses Rhino, the Mozilla JavaScript engine, to run JavaScript and Ajax programs. This approach makes it easy to run hundreds to thousands of simulated users in a load and performance test. 
12. The alternative would be costly test runs that start a browser for each simulated user. TestMaker makes it easy to pivot from one type of testing to another. For example, functional testing is often the time to ensure the test scripts work as expected. 
13. With a click of a button in the TestMaker Editor a functional test becomes a load and performance test. While loading the application under test with several virtual users your test may uncover another functional issue. 
14. Pivoting back to functional testing is easy with TestMaker. Once you record a test, TestMaker repurposes the test to be a functional test, and load and performance test, and a production monitor.

Often TestMaker users will following this methodology:

a) Record a Selenium test in Selenium IDE using Firefox browser. Save the test in the Selenese file format. Confirm the test runs by playing the test against the application in the Firefox browser.

b) Run the Selenium test in a TestNode in the Firefox browser. Create a TestMaker TestScenario using the TestMaker Editor. Identify one or more Selenium tests to run in a test use case in the Editor's Use Case tab. 

Use the Selenium RC test type and identify *firefox as the browser. TestMaker sends the Selenium test to the TestNode. The TestNode launches Firefox and plays the test. 

The TestNode records the time it takes for each step of the Selenium test to operate. TestMaker plots a set of results charts to identify functional issues in the test and application.

c) Run the Selenium test in a TestNode in the HtmlUnit browser. Same operation but this time using the headless HtmlUnit browser.

d) Run the test as a Load and Performance test. Operates the Selenium test at several levels of load. Identifies the Scalability Index of the application. Surfaces performance bottlenecks and functional issues.

e) Run the test as a Production Monitor. Operates the Selenium test periodically to prove compliance with an SLA and to surface functional issues.

Tutorials:

Please read the tutorial on Selenium testing in TestMaker. The tutorial shows step-by-step instructions to accomplish functional testing, load and performance testing, and business service monitoring of Selenium tests in TestMaker.

Selenium Tutorial: Building Tests of Rich Internet Applications Using Ajax and Selenium

These additional resources are available:

Screencast: Watch the 8 Minute Screencast on TestMaker

DZone RefCard: Introduction To Selenium

Selenium Documentation Project

Open Source Test Workshop

Open Source Test Workshop:

This Open Source Test Workshop is for QA testers, software developers, and network and server managers needing visibility, tools, and methodology into all the demand for IT. This is a deep-technical Workshop to answer your issues:

• TestMaker repurposes a single Selenium test as a functional test, load and performance test, and business service monitor. 
• How To Use Selenium in Rich Internet Application (RIA, using Ajax) Environments
• How To Build Test Suites of SOAP and REST-based Applications using soapUI
• How To Data Enable Selenium Tests Using TestMaker
• Testing in Flash/Flex Environments Using AMF Protocols
• Integrating TestMaker Tests Into Your Continuous Integration Environment
• How To Analyze Test Results Into Actionable Knowledge
• Using Glassbox and DynaTrace for Root Cause Analysis
• Sign-up for the Workshop.

Questions And Answers :

PushToTest, the open source application performance management and load test solution company, participates in the Selenium project. Many times PushToTest's response to a customer issue is in the form of an improvement to Selenium iteself. The following questions and answers originally appeared in DZone.

1. What exactly is Selenium?

• Selenium is a combination of tools and a domain specific language (DSL) to build and operate functional, smoke tests, and integration tests of Web applications, including Rich Internet Applications (RIA, using Ajax.) 

• Selenium provides a record/playback tool that plugs-into the Firefox browser. The recordings use the Selenese DSL to express the steps of a test. Selenium plays-back the test steps in the browser using a Javascript-based Browserbot daemon. 

• Selenium Remote Control (RC) runs the test in Internet Explorer, Firefox, Opera, Safari, Chrome and other browsers. Selenium Grid makes it easy to run these tests in a grid of test machines.

2. How long has the project been around? And how long have you been using it?

• Selenium had its start at ThoughtWorks, a system integrator and global consulting firm. The core developers released Selenium into an open source distribution in 2006. PushToTest began using Selenium since 2007. 

• PushToTest was looking for a way to repurpose Selenium tests as load and performance tests, and business service monitors when Olivier Dony and Dominique de Waleffe contributed a TestMaker package to run Selenium tests using the HtmlUnit headless high performance browser. 

• PushToTest enhanced this work to make the Selenium tests data-driven, to provide root cause analysis and results analysis functions, to provide compatibility to test Ajax applications, and to run these tests in a Cloud testing Environment. 

• This has been hugely beneficial to organizations needing to reduce costs and take advantage of Selenium's easy-of-test-authoring features.

3. What are the basic steps involved in getting setup with Selenium? (jars/server setup etc)

• Setup to record and playback tests of Web browser-based applications using Selenium takes about 5 minutes. The Selenium IDE plugs into Firefox as a standard add-on using XPI installation techniques. 

• To playback the tests in Internet Explorer, Opera, Safari, Chrome and other browsers requires installation and configuration of Selenium RC. Selenium RC uses a daemon architecture. 

• Start the daemon using a simple command-line call to start Java and run the Selenium RC JAR package. Selenium RC installations take an hour or less. Selenium Grid installs Selenium RC services on a farm of servers.

4. What language is used to write the tests?

• Selenium implements a domain specific language (DSL) for testing Web applications. For example, the DSL implements commands to click a button, type characters into a test field, and wait until text appears in the page. 
• Selenium implements the DSL as class library in Java, package in Python, and Ruby, Groovy, C#, Perl, and PHP. The DSL has approximately 200 commands. 
• PushToTest teaches a 3-day introductory class in Selenium. It takes 2-3 days for most students to become proficient in the Selenium DSL.

5. How well does the record/playback function work?

• Selenium IDE is designed for Web 1.0 applications. In Web 1.0 environments Selenium's record/playback function works very well. For Rich Internet Applications that use Ajax techniques and widgets, not so much! 
• Ajax is the Wild West of software development. Everything goes, there is no sheriff, and no standards body exists. Consequently Selenium does not know how to handle asynchronous applications where objects load at their own pace and there is no "wait until idle" function. 
• Selenium misses Javascript driven clicks, Javascript created frames and pop-up windows. Each miss means the test author needs to manually add Selenium DSL commands.
• Selenium IDE record/playback does not work well for tests that require looping and conditional branching. These are possible by manually coding Selenium DSL commands in a Java or other scripting language environment. 
• The solution is to use Selenium IDE's record feature to create simple test use cases, transform them into a Java class, and treat each use case as a component incorporated into a test suite. 
• There are emerging frameworks to help implement a component approach to Selenium test authoring.

6. Is there a coverage tool that plugs into Selenium?

• ThoughtWorks Twist is a tool to facilitate a continuous test and integration methodology built on Selenium tests in a team-oriented development environment. There may be others. 
• There is a lot of excitement and energy in the Selenium space so PushToTest wouldn't be surprised to find coverage tools.

7. Are there any missing features in Selenium? Is there a new release due soon? 

• A key drawback is Selenium's requirement to playback tests in a real browser. When the browser needs to interact with a user the Selenium test stops running. 
• For example, when a browser moves from an HTTPS secured page to an HTTP page many browser's security policy kicks in and the browser asks the user to confirm the action. 
• The Selenium project is working on a next generation technology called WebDriver. WebDriver uses each browser's native API to operate the test commands.
• Selenium lacks a data production library system to provide operational test data, lacks a way to repurpose tests into load tests, lacks a platform to manage Selenium tests as test assets, and lacks a way to derive the root cause to functional and performance issues. 
• Thankfully these are provided by many side projects in the Selenium biosphere, including PushToTest, BrowserMob, SauceLabs, and others.
• The words "coming soon" and Selenium are mutually exclusive. It took the Selenium project 18 months to move from "beta" to "release candidate." 
• PushToTest is optimistic but not hopeful that Selenium 2.0 with WebDriver technology will appear in 2010.