Testing

Map of Outcomes to Testing Techniques

The table below maps outcomes -- the results that you may want to achieve in your validation efforts -- to one or more techniques that can be used to accomplish that outcome.

When I am working on...	I want to get this outcome...	...so I should consider
Development	Prove backward compatibility with existing callers and clients	Shadow testing
Development	Ensure program logic is correct for a variety of expected, mainline, edge and unexpected inputs	Unit testing; Functional tests; Consumer-driven Contract Testing; Integration testing
Development	Prevent regressions in logical correctness; earlier is better	Unit testing; Functional tests; Consumer-driven Contract Testing; Integration testing; Rings (each of these are expanding scopes of coverage)
Development	Validate interactions between components in isolation, ensuring that consumer and provider components are compatible and conform to a shared understanding documented in a contract	Consumer-driven Contract Testing
Development; Integration testing	Validate that multiple components function together across multiple interfaces in a call chain, incl network hops	Integration testing; End-to-end (End-to-End testing) tests; Segmented end-to-end (End-to-End testing)
Development	Prove disaster recoverability – recover from corruption of data	DR drills
Development	Find vulnerabilities in service Authentication or Authorization	Scenario (security)
Development	Prove implementation correctness in advance of a dependency or absent a dependency	Unit testing (with mocks); Unit testing (with emulators); Consumer-driven Contract Testing
Development	Ensure that the user interface is accessible	Accessibility
Development	Ensure that users can operate the interface	UI testing (automated) (human usability observation)
Development	Prevent regression in user experience	UI automation; End-to-End testing
Development	Detect and prevent 'noisy neighbor' phenomena	Load testing
Development	Detect availability drops	Synthetic Transaction testing; Outside-in probes
Development	Prevent regression in 'composite' scenario use cases / workflows (e.g. an e-commerce system might have many APIs that used together in a sequence perform a "shop-and-buy" scenario)	End-to-End testing; Scenario
Development; Operations	Prevent regressions in runtime performance metrics e.g. latency / cost / resource consumption; earlier is better	Rings; Synthetic Transaction testing / Transaction; Rollback Watchdogs
Development; Optimization	Compare any given metric between 2 candidate implementations or variations in functionality	Flighting; A/B testing
Development; Staging	Prove production system of provisioned capacity meets goals for reliability, availability, resource consumption, performance	Load testing (stress); Spike; Soak; Performance testing
Development; Staging	Understand key user experience performance characteristics – latency, chattiness, resiliency to network errors	Load; Performance testing; Scenario (network partitioning)
Development; Staging; Operation	Discover melt points (the loads at which failure or maximum tolerable resource consumption occurs) for each individual component in the stack	Squeeze; Load testing (stress)
Development; Staging; Operation	Discover overall system melt point (the loads at which the end-to-end system fails) and which component is the weakest link in the whole stack	Squeeze; Load testing (stress)
Development; Staging; Operation	Measure capacity limits for given provisioning to predict or satisfy future provisioning needs	Squeeze; Load testing (stress)
Development; Staging; Operation	Create / exercise failover runbook	Failover drills
Development; Staging; Operation	Prove disaster recoverability – loss of data center (the meteor scenario); measure MTTR	DR drills
Development; Staging; Operation	Understand whether observability dashboards are correct, and telemetry is complete; flowing	Trace Validation; Load testing (stress); Scenario; End-to-End testing
Development; Staging; Operation	Measure impact of seasonality of traffic	Load testing
Development; Staging; Operation	Prove Transaction and alerts correctly notify / take action	Synthetic Transaction testing (negative cases); Load testing
Development; Staging; Operation; Optimizing	Understand scalability curve, i.e. how the system consumes resources with load	Load testing (stress); Performance testing
Operation; Optimizing	Discover system behavior over long-haul time	Soak
Optimizing	Find cost savings opportunities	Squeeze
Staging; Operation	Measure impact of failover / scale-out (repartitioning, increasing provisioning) / scale-down	Failover drills; Scale drills
Staging; Operation	Create/Exercise runbook for increasing/reducing provisioning	Scale drills
Staging; Operation	Measure behavior under rapid changes in traffic	Spike
Staging; Optimizing	Discover cost metrics per unit load volume (what factors influence cost at what load points, e.g. cost per million concurrent users)	Load (stress)
Development; Operation	Discover points where a system is not resilient to unpredictable yet inevitable failures (network outage, hardware failure, VM host servicing, rack/switch failures, random acts of the Malevolent Divine, solar flares, sharks that eat undersea cable relays, cosmic radiation, power outages, renegade backhoe operators, wolves chewing on junction boxes, …)	Chaos
Development	Perform unit testing on Power platform custom connectors	Custom Connector Testing

Sections within Testing

• Consumer-driven contract (CDC) testing
• End-to-End testing
• Fault Injection testing
• Integration testing
• Performance testing
• Shadow testing
• Smoke testing
• Synthetic Transaction testing
• UI testing
• Unit testing

Technology Specific Testing

• Using DevTest Pattern for building containers with AzDO
• Using Azurite to run blob storage tests in pipeline