When Should I Do Performance Testing?

Your store's performance should be on your mind from the moment you have initiated an ecommerce project and for as long as your store is online.

The truth of the matter is that you cannot test performance into the system. You need to design, develop, and deploy online store implementation while maintaining continuous focus on performance. Project teams that delay performance testing to the last moment before going live are setting themselves up for nasty surprises.

The diagram below shows how performance testing activities should be aligned with standard phases of an IT project.

Sizing and Capacity Planning

You should start by developing a capacity planning model to predict your store's traffic based on the expected number of store visits, conversion rates, and typical store usage scenarios.

During this stage you need to:

• Understand and size your data: catalog size and structure, number of stores, number of registered customers, number of promotions, etc.
• Understand site usage: number of unique visitors, user sessions, page views, orders per hour
• Define performance goals
• Define reliability goals

Performance Engineering

With load requirements documented and understood, it is time to design your tests:

• Identify high-load scenarios
• Design benchmark and focus performance tests
• Review design of proposed customizations to understand their impact on performance and reliability
• Select and obtain load-generating tools
• Design and prepare a load test environment
• Develop test monitoring and test result processing scripts
• Define requirements for deployment environment and hardware

Performance Testing

Now you are ready to start testing. As I mentioned at the beginning, doing a performance test right before the launch day is not helpful. You need Calendar time!

Many people compare performance testing with pealing an onion. You achieve performance goals by laboriously identifying and removing layer after layer of performance bottlenecks in multiple test/analyze/fix and test again cycles.

Your performance team needs to have the right mix of skills to be successful. You need to have people proficient in networking, software development, and web application tuning. Otherwise, you may end up with a Java developer optimizing his Java code while your real bottleneck is caused by a misconfigured load balancer.

If you don't have the right expertise in-house, look for external help.

Monitoring and Troubleshooting

An ecommerce site is a constantly evolving complex system and initial performance testing is not enough to ensure that your customers continue to have a great shopping experience. As you enhance the site with new features and integrations or upgrade the deployment infrastructure, there is always a chance that even a small change can negatively affect the site's performance.

To make sure that this does not happen, you need to set up performance monitoring scripts to detect performance issues as they occur and repeat performance tests in the staging environment before deploying a new code.

From Crisis Resolution to Performance Enhancements

What should you do if you have not done all this work ahead of time and encounter a stability or performance problem?

• Collect data
• Identify performance problem
• Diagnose the cause
• Resolve the issue through tuning and configuration

Don't stop after an immediate crisis is resolved! Start working on a long term solution using the approach described above.

Remember, the right answer to the question "When should I do performance testing?" is - ALWAYS!