BI4 Web Intelligence Performance Test with JMeter

Overview of the Business Case

Designing and deploying a BI environment can be challenging in and of itself, but it is most likely to fail when released to business users without validation of performance and reliability.  Chances are business users will be displeased with a BI platform if it does not meet their expectations in terms of response and the overall user experience.  While use of sizing guides and sizing calculators is recommended, they can never take into account the specific nuances of your data, security model, and infrastructure including network latency, corporate proxy servers, etc.

In order to deliver a solution that maximizes your Return on Investment (ROI) it is critical that all deployments be performance tested.  In some organizations you may have access to a dedicated team that is responsible for designing and executing performance tests with a commercial tool such as HP LoadRunner.  Some organizations, however, have no dedicated team or do not have access to a tool such as LoadRunner.  Performance testing is still every bit as critical and a workable solution must be identified.  Apache JMeter offers such a solution, one that is open source and free to use.  These exercises will introduce you to JMeter as a tool and demonstrate how it can be used with SAP BusinessObjects Business Intelligence Platform 4.0.

Exercise 1:  Record a Test Plan

JMeter HTTP Proxy Server Component
JMeter allows you to create a set of actions in a number of different ways.  HTTP calls can be constructed manually, for example, by defining a particular URL and the headers and parameters needed to execute it.  This is great for developers that know their own application very well, but with enterprise software such as BI 4.0 it is unlikely you will have this level of knowledge.  JMeter therefore also allows you to record a specific workflow by performing it manually in your browser.  By acting as an HTTP Proxy Server, it can record all of the requests that you perform so that you can play them back at a later time.

In this exercise, you will learn how to launch JMeter, create an HTTP Proxy Server component, and record a simple workflow involving a Web Intelligence report.  We will then play the test plan back for 5 users and measure the performance of the BI 4.0 system.

You must have downloaded Apache JMeter and the JMeter Plugins library to complete this exercise.  Refer to the Performance Testing Wiki for instructions on configuring JMeter for the first time and this SCN Blog for JMeter Plugins.

1. Double-click jmeter.bat to launch the application.
   
   
   
   
2. Add the HTTP Proxy Server component by right-clicking on the Workbench > Add > Non-Test Elements HTTP Proxy Server.
   
   
   
   
   
3. Set the listening port of the HTTP Proxy Server component to 48080.  The default selection of 8080 is already in use by the local instance of Tomcat.
   
4. Under Test plan content change the Target Controller property to WorkBench > HTTP Proxy Server.  This instructs JMeter to store all recorded samples under the WorkBench so that we can move them to the Test Plan in our preferred format.
   
5. Change the Grouping option to Add separators between groups. 
   
6. Check the box labeled Regex matching.  The HTTP Proxy Server component screen should now look something like this:
   
   
7. Click Start at the bottom of the HTTP Proxy Component to launch the JMeter proxy server.
   
8. Next, launch Internet Explorer by executing Start > Internet Explorer
   
   Tip: It is recommended that you practice the desired workflow before recording it.  This way you are comfortable with all of the steps and can execute them quickly and effectively.  This has the added benefit of caching static content locally on your machine, so that the recorded test plan is cleaner and less cluttered.
   
9. Access the BI Launch Pad and login using the desired credentials.  
10. Click Documents > Expand the Folders option from the accordion menu > Access Public Folders > Web Intelligence Samples > Double-click the Formatting Sample report.
    
    
    
11. Refresh the report and note that the last refresh date updates as below.
    
    
    
12. Close the report tab and accept the warning that appears.  Click Log Off to complete the process. 
    
    
13. Click Tools > Internet Options > Connections > LAN settings
    
    
14. Uncheck the box Use automatic configuration script.  You can re-enable this setting once you’ve completed the exercise.
    
15. Enable Proxy Server and click the Advanced button under Proxy Server.
    
16. Change the HTTP proxy address to localhost and set the port to 48080. Delete all exceptions from the box at the bottom.  Click OK all the way out.
    
    
17. Repeat steps 8-12 to record the methods needed for the test plan.
    
    Tip: Ensure you access the machine by the server name as using the addresslocalhost will not reach the proxy server.
    
18. Return to JMeter and expand the HTTP Proxy Component.  There should now be a number of requests present there that look something like this:
    
    
    Note: These requests represent all of the HTTP calls performed during the workflow above.  We can now customize them to extract session related data and play the actions back to emulate high concurrency.
    
19. Next, create a Thread Group within the test plan to store the requests captured by the HTTP Proxy Server component.  Thread Groups control the number of users emulated during test plan execution and are the component that our HTTP samples will reside.
    
20. Right-click the Test Plan icon, select Add > Threads (Users) > Thread Group
    
    
21. Rename the Thread Group to Web Intelligence Refresh Test, set the Number of Threads (users) to 5, and set the Ramp-Up Period (in seconds) to 10 as below.  This configures the test for 5 users and launches one user every 2 seconds, so that after 10 seconds all 5 users will be actively running the test. Set the Loop Count to 2 so that the test is repeated for a total of 10 executions.
    
    
    
22. Next, create a Once Only Controller underneath the Thread Group by right-clicking Web Intelligence Refresh Test > Add > Logic Controller > Once Only Controller.  Once Only Controllers execute the samplers contained in them a single time per thread, no matter how many times other controllers are configured to process.  They allow us to configure a test to login and logout only once, while potentially repeating other aspects of the test numerous times.
    
    
23. Rename the Once Only Controller to Login.
    
24. Shift+click to select all of the requests from the first 2 groups under the HTTP Proxy Server component.  Right-click and select cut.  Next, right click on the Login controller and selectpaste. Expand the Login controller and note that the requests have been entered there.
    
25. Right-click the Web Intelligence Refresh Test again, select Add > Logic Controller > Transaction Controller.  Transaction Controllers allow us to measure the aggregate time a group of samplers take to execute when we’re more interested in an action, such as navigating or refreshing, than we are each individual HTTP method.  Rename the Transaction Controller toNavigate BI Launch Pad.  Check the box Generate Parent Sample which instructs JMeter to collect only the performance statistics associated with the collection of samples.
    
    
    
26. Shift+click to select all of the requests from the next 4 groups under the HTTP Proxy Server component.  Right-click and select cut. Next, right click on the Navigate BI Launch Padcontroller and select paste.  Expand the Navigate BI Launch Pad controller and note that the requests have been entered there.
    
    
    
27. Right-click the Web Intelligence Refresh Test again, select Add > Logic Controller > Loop Controller.  Loop Controllers allow us to repeat a group of samplers a specified number of times.  Set the Loop Count to 2.
    
    
    Tip: It is possible to nest controllers within a Loop Controller to add complexity.  You could nest a Once Only Controller, such as the Login controller used above, along with several transaction controllers.  In this fashion the Once Only Controller would process the first time only, but the remaining transaction controllers would be processed according to the Loop Count setting above.
    
28. Right-click the Loop Controller and select Add > Logic Controller > Transaction Controller.  Rename the Transaction Controller to View and Refresh Web Intelligence Report and check the box Generate Parent Sample.
    
29. Shift+click to select all of the requests from the next 3 groups under the HTTP Proxy Server component.  Right-click and select cut. Next, right click on the View and Refresh Web Intelligence Report controller and select paste.  Expand the View and Refresh Web Intelligence Report controller and note that the requests have been entered there.
    
    
30. Next, create another Once Only Controller underneath the Thread Group by right-clickingWeb Intelligence Refresh Test > Add > Logic Controller > Once Only Controller.  Rename the controller to Logout.
    
31. Shift+click to select all of the remaining requests HTTP Proxy Server component.  Right-click and select cut.  Next, right click on the Logout controller and select paste. Expand theLogout controller and note that the requests have been entered there.
    
    
    
32. The Test Plan should now look something like this:
    
    
    
    Click File > Save Test Plan As and save the file as RefreshWebi.jmx 
33.  

1.  Exercise 2:  Customize Test Plan for Dynamic Data

1. Dynamic content is what makes enterprise software such as BI 4.0 truly function.  Each time a user logs into the application a session is created which is then referenced by future calls into the platform.  Similarly, viewing a Web Intelligence report creates a session on the Web Intelligence Processing server to hold temporary files associated with the request.
   
   In order to have JMeter emulate distinct user sessions, it is necessary to extract this dynamic data from the BI platform and use it in future HTTP samplers. Some of this data is stored in cookies, some in HTTP Headers, and still other data is passed within the response body of the HTTP request itself.  JMeter is able to store cookies and header values automatically, but data passed in the response body needs to be extracted manually.  To do so, JMeter provides a number of post-processors, components that perform a function after an HTTP request has been executed.  The Regular Expression Extractor is the most important post-processor used with BI 4.0.
   In this exercise, you will attach a series of post-processors to the test plan created in the previous exercise and use regular expressions to capture dynamic data from the BI 4.0 environment.  You will also add an HTTP Cookie Manager so that JMeter captures cookies for each user/thread.
   
   
   Note: For BI 4.1 users, you must also retrieve the com.sun.faces.VIEW string out of the first logon.faces as explained here:
   Performance Testing in BI 4.1
   
   
   1. The first step necessary is to extract the session token returned by the logon method.  This object is called bttoken in BI 4.0 and is returned in the response body of the logon.objectsampler.  Right-click the logon.object sampler and select Add > Post Processors > Regular Expression Extractor
      
      
      
   2. Rename the extractor to ***bttoken extractor*** and configure it as follows:
      
      Applies to: Main Sample Only
      Response Field to check: Body (unescaped)
      Reference Name: bttoken
      Regular Expression: bttoken=(.+?)&
      Template: $1$
      Match No. (0 for Random): 1
      Default Value: null
      
      Note: In BI 4.1 SP3 and earlier, the regular expression has been modified slightly.  You will need to use a regex of:
      
      Regular Expression: bttoken=(.+?)"
      Note: In BI 4.1 SP4/SP5 the bttoken is now returned as part of a hidden input so the regex looks like this:
      
   
         Regular Expression: name="bttoken" value="(.+?)"
   
   
   
1. Select the next request in the sampler, which should be main.do.  This represents the entry point into the BI Launch Pad application and is the first place the bttoken property is used.  Double-click the value field, copy the string to the clipboard using CTRL+C, and replace the text with ${bttoken}.  
   
   The existing bttoken string should look something like this but will be different in every environment:
   
   MDAwRGNfQzxgaGFcVzI9ZDNobTJoXGltaTgPLUDNVOzAEQ
   
   This represents a JMeter variable, called bttoken, which we extracted in the previous step.  Click File > Save to update the test plan.
   
   
   
   Tip: The bttoken value will need to be updated in numerous locations throughout the test plan.  While it is possible to update the value manually, it is a time consuming and tedious process.  Instead, we can use a text editor such as Notepad to update every occurrence of the value.  The JMX format of a JMeter test plan is clear text and be modified easily using any text editor.
   
2. Navigate to the location where you saved the test plan (default will be in the JMeter bin directory) and locate the file RefreshWebi.jmx.  Right-click the file, click open, select a program from a list of installed programs, and select Notepad.  Select Edit > Replace and type the string value for bttoken in the Find what: box.  Enter ${bttoken} in the Replace with: box and select Replace All.  Click Cancel and File > Save.  Close the JMX file.
   
   
3. Return to JMeter, click File > RefreshWebi.jmx.  If you are prompted to save the file say Noas it would overwrite the changes we have just made.  When the test plan is reloaded, expand the thread group and Login controllers, and select the sampler home.do.  Note that the bttoken value has been updated to reflect the JMeter variable.
   
   
   
4. Next, it is necessary to extract the Web Intelligence sEntry, a value that represents a Web Intelligence document state.  An sEntry is generated when you view a report, and another sEntry is created when you refresh a report.  Each value represents a distinct document state within Web Intelligence (that is, the state of the document when it is opened with saved data, and the state after it is refreshed with live data).  
   
   Expand the Loop Controller > View and Refresh Web Intelligence Report and select theviewDocument.jsp sampler. Right-click viewDocument.jsp and select Add > Post Processors > Regular Expression Extractor.
   
   
   
5. Rename the extractor to ***sEntry Extractor*** and configure it as follows:
   
   Applies to: Main Sample Only
   Response Field to check: Body (unescaped)
   Reference Name: sEntry1
   Regular Expression: "strEntry":"(.+?)"
   Template: $1$
   Match No. (0 for Random): 1
   Default Value: NONE
   
   
   
   Tip: Regular Expressions provide a concise and flexible means to match strings of text.  In the example above, the post processor examines the response body of viewDocument.jsp for the string “strEntry”:” and returns every character until it encounters a closing set of double quotes.  The parentheses open and close the function, while the .+? combination tells the processor to capture every character.  A useful tutorial on regular expressions can be found on the Apache HTTPD website
   
6. Select the next request in the sampler, which should be report.jsp.  Double-click the sEntryvalue field, copy the string to the clipboard using CTRL+C, and replace the text with ${sEntry1}.
   The existing sEntry string should look something like this but will be different in every environment:
   
   we0001000078432c722f78
   Click File > Save to update the test plan.
   
   
7. Navigate to the location where you saved the test plan and locate the file RefreshWebi.jmx.  Right-click the file, click open, select a program from a list of installed programs, and select Notepad.  Select Edit > Replace and type the string value for sEntry in the Find what: box.  Enter ${sEntry1} in the Replace with: box and select Replace All.  Click Cancel and File > Save.  Close the JMX file.
   
   
8. Return to JMeter, click File > RefreshWebi.jmx.  If you are prompted to save the file say Noas it would overwrite the changes we have just made.  When the test plan is reloaded, expand the thread group > Loop Controller > View and Refresh Web Intelligence Reportcontrollers, and select the 1st instance of getImage.jsp.  Note that the sEntry value has been updated to reflect the JMeter variable ${sEntry1}.
   
   
   
9. The final piece of dynamic data to be extracted is the sEntry that corresponds to the document state after refreshing the report.  Right-click the refreshDocument.jsp sampler and select Add > Post Processors > Regular Expression Extractor.
   
10. Rename the extractor to ***sEntry2 Extractor*** and configure it as follows:
    
    Applies to: Main Sample Only
    Response Field to check: Body (unescaped)
    Reference Name: sEntry2
    Regular Expression: sEntry=(.+?)&
    Template: $1$
    Match No. (0 for Random): 1
    Default Value: NONE
    
    
    
11. Select the next request in the sampler, which should be report.jsp.  Double-click the sEntry value field, copy the string to the clipboard using CTRL+C, and replace the text with ${sEntry2}.
    
    The existing sEntry string should look something like this but will be different in every environment:
    
    we00020000105bca428762
    
    Click File > Save to update the test plan.
    
12. Navigate to the location where you saved the test plan and locate the file RefreshWebi.jmx.  Right-click the file, click open, select a program from a list of installed programs, and select Notepad.  Select Edit > Replace and type the string value for sEntry2 in the Find what: box.  Enter ${sEntry2} in the Replace with: box and select Replace All.  Click Cancel and File > Save.  Close the JMX file.
    
13. Return to JMeter, click File > RefreshWebi.jmx.  If you are prompted to save the file say Noas it would overwrite the changes we have just made.  When the test plan is reloaded, expand the thread group > Loop Controller > View and Refresh Web Intelligence Reportcontrollers, and select the 2nd instance of getImage.jsp.  Note that the sEntry value has been updated to reflect the JMeter variable ${sEntry2}.
    
14. Finally, in order to capture any cookies generated by BI 4.0, right-click the thread group Web Intelligence Refresh Test and select Add > Config Element > HTTP Cookie Manager.  Check the box Clear cookies each iteration? to ensure new cookies are stored for each loop. Click File > Save to complete the test plan customization.
    
    

Exercise 3:  Single User Test
It is beneficial to execute a single instance of the test plan to ensure the test functions as expected.  A single typo can cause the test to fail and it is much better to identify any problems with 1 user as opposed to 1000!  In this exercise you will confirm core functionality of the test before moving on to the final execution.

1. Configure the test plan to run for a single user as in the screenshot below.
   
   
   
2. Right click the Web Intelligence Refresh Test thread group and select Add > Listener > View Results Tree.  This component captures every request and response in the test plan for review and is beneficial for debugging a test.  The component consumes large amounts of memory and is not recommended for use in large scale testing.
   
   
   
3. Right click the Web Intelligence Refresh Test thread group and select Add > Sampler > Debug Sampler.  This component adds a record of all JMeter variables collected to the bottom of the View Results Tree component.  This allows us to confirm that the dynamic data configured in the previous exercise has been extracted correctly.
   
   
   
4. At the top of the application, click Run > Start to kick off the test plan.  Note that the thread count increments in the top right corner to reflect the number of active threads.
   
5. When the counter returns to 0/1, select the View Results Tree component and locate the View and Refresh Web Intelligence Report controller.  Expand it and select the first report.jspsampler.  Set the drop down menu below to HTML and click the Response data tab in the right hand pane.  Confirm that report data appears.
   
   
   
   Note: The Loop Controller may cause navigation to fail the 2nd time this controller is executed.  You may disregard a 500 error associated with the ajaxRequest sampler located there.  The report refresh is still correctly executed the 2nd time.
   
6. Scroll to the very bottom of the View Results Tree component and select the Debug Sampler.  Confirm that bttoken, sEntry, and sEntry2 are correctly populated.
   
   
   
7. Next, remove both the View Results Tree and Debug Sampler components by right-clicking them individually and selecting Remove.

Exercise 4:  Configure Results Analysis Collectors and Execute Test Plan

You have now created a simple test plan that is ready for execution. No test plan would be complete, however, without a means to measure the results.  In this exercise, you will add JMeter components to your test plan that allow you to measure throughput and response time from within the JMeter console. You will also configure an extension called JMeter Plugins that allows you to monitor system resources such as CPU and Memory from within the JMeter console.

You will use the plugins you configured by following the blog post listed earlier in this document.

1. Right click the Web Intelligence Refresh Test thread group and select Add > Listener > Graph Results.  Deselect Data, Median, and Deviation from the Graphs to Display section, leaving only Average and Throughput selected.
   
   
   Tip: The Average measurement displays the average response time of a given transaction. Throughput measures how many transactions occur in a given interval (second, minute, etc).
   
2. Right click the Web Intelligence Refresh Test thread group and select Add > Listener > jp@gc - PerfMon Metrics Collector.  This component allows you to retrieve resource consumption statistics for the machine such as CPU and Memory.
   
3. Select the jp@gc - PerfMon Metrics Collector component and click Add Row to add a CPU counter.  Click Add Row a second time and change the Metric to collect drop down toMemory.  Leave all other options as the default.
   
   
   
4. Launch the JMeter Plugins server agent which monitors the server and delivers statistics to JMeter.  Navigate to the location where you extracted JMeter Plugins and double clickstartAgent.bat
   
   
   
5. Ensure the Thread Group is configured for 5 users, 10 second ramp-up time, and 2 loops.
   
6. Click Run > Start to execute the test plan and note the counter increment to 5/5.
   
7. Once the counter returns to 0/5 select the Graph Results component and note the metrics returned. In this example, an average request took ~500 ms and throughput was ~376 transactions per minute.
   
   
   
8. Select the jp@gc - PerfMon Metrics Collector component and note the metrics returned. In this example, CPU fluctuated and peaked at around 98%.  Memory remained constant at ~82%.
   
   

Although this is a simple test, the components demonstrated in this exercise can be used to create complex test plans that touch every area of the SAP BusinessObjects Business Intelligence platform.  By running performance tests before releasing an environment to business users, you can be confident in the usability and response time of your deployment.