Testing and QA
Interview Questions

We kind of have ranges like the last question—"above zero" and "zero or below." But other than that we don’t have any bounds on the input or any changes in behavior based on the input. Sure, 0 and 1 will be important to test, but those aren’t enough on their own.

In general for any method that takes numbers as input, it’s a good idea to consider testing:

• Negative numbers
• 0
• 1
• 2
• 10, and higher by orders of magnitude
• The highest Integer (like the highest 32 bit integer, or something more specific to your system or language)

This generally covers a basic set of edge cases, potential special cases, base cases, and a reasonable range of values.

Let’s start by looking at the possible values for each parameter.

The size of the cake is easy—it can only be one of two options, ‘1-10 guests’ or ‘11-20 guests’. The radio button with a default is nice because we know we don’t have to worry about invalid or blank input.

But wait—do we know we don’t have to worry about invalid or blank input? We got specifications of the field types and defaults in the UI, but what if we have an API and someone directly hits it?

This is a good question to ask your interviewer because it shows an ability to think of things that could go wrong. In this case, the spirit of the question doesn’t involve catching edge cases, so we'll assume the arguments are reliably and intentionally restricted, and you can count on valid input.

The writing on the cake can be blank or present. We don’t have to worry about it being too long because the input is already limited to 30 characters. We might want to ask our interviewer if we allow any 30 characters. What about obscure special characters or offensive words? We’ll say there’s no restriction—these problems are rare enough and messy enough that we can just follow up by phone if we’re concerned about the message.

For pickup time, looks like we only have 2 values to look for. Times today or times later than today. We know the input is a calendar, has a default, and doesn’t allow times when the bakery is closed. So we don’t have to worry about blank or invalid input. Although a reasonable question would be to ask about a limit in the future—how far in advance can someone order a cake? We’ll say there’s no limit here.

Ok, let’s lay this all out. What are all the combinations of inputs we can have?

Well, we have 2 possibilities for size, 2 possibilities for writing, and 2 possibilities for pickup time. So that’s 8 combinations (2 x 2 x 2 = 8).

We know enough to answer the question now, but let’s take it a step further and look at the price for each scenario.

This kind of table—showing the possible combination of values for a set of conditions, and identifying the actions we take—is called a decision table. This kind of thinking will get even more helpful as the total number of parameters and possible values grows. What if we had different prices for different flavors? What if the inputs could be blank or invalid? Decision tables help us methodically design comprehensive tests.

Sometimes decision tables can also help combine scenarios that have the same action (for example, we could combine the first 2 columns and say "if a cake is for 1-10 guests and writing is blank, it doesn’t matter what the pickup time is—we charge $45"). But this can be dangerous for testing—what if a bug factors in the pickup time when it shouldn’t?

So for this this method, we should test 8 scenarios—every possible combination of inputs.

One way to start deciding what to test is risk-based testing, focusing on what’s most likely to have a bad impact. For example, failure in the signup flow might completely halt new business. Or if we charge customers the wrong amount we could make users angry and even get in legal trouble.

We can also break down the severity and priority of potential problems. Sometimes there’s a tricky balance between the two—for example, are we more worried about a feature that fails occasionally for the 80% of users using Chrome or Safari, or a feature that’s totally unusable to the 5% on Internet Explorer?

A specific way to get reasonable coverage with fewer tests is combinatorial testing. Let’s say we have an app to regularly check and record our bakery’s refrigerator temperatures. We have 2 digital thermometer models (T and TX), 2 app versions (iOS and Android), and 2 connection methods (bluetooth and USB). Similar to a decision table, we can lay out every possible combination. There are 8 configurations for getting refrigerator temperatures:

But with combinatorial testing, we can reduce this to 4 tests by just testing pairs. Instead of asking "Does every model work on every version with every connection?" we can separately ask:

• Do all the models and versions work together?
• Do all the models and connections work together?
• Do all the versions and connections work together?

We've reduced our total tests by half, and this approach becomes even more powerful as the number of conditions and the possible values for each condition grows. It’s not unrealistic to reduce a quintillion (a billion billion) possible combinations down to hundreds of tests.

If we're worried that just testing pairs isn't enough, we can test the interaction of 3 values or any n values. Research shows that the interaction of 6 values is usually enough to find all the bugs in a system, even for complex and dangerous systems like air traffic collision avoidance, medical devices, and space flight.

So combinatorial testing can save us a huge amount of time with no cost or a low cost to effectiveness.