What is Statistical Significance?

Imagine you read a far-fetched news headline about a recently published study. The headline reads: “Study Shows Running in Circles Prevents Cancer”. You’d probably scratch your head in disbelief. 

Then you read on to discover the sample size was five people. And of those five people who ran in circles, one person didn’t get cancer. So was this coincidence or does the study indicate that running in circles prevents cancer? Probably the former.   

The study for such a conclusion doesn’t have statistical significance — though the study was indeed performed, its conclusions don’t exactly mean anything because the sample size was so incredibly small. 

This Kissmetrics guide will take you through everything you need to know about statistical significance and how to calculate it.

What is Statistical Significance and How Is It Calculated?

As we mentioned above, the fake study about spinning in circles isn’t statistically significant. This means that the conclusion reached isn’t valid because there’s simply not enough evidence to support that what happened was not random chance or due to luck. 

A statistically significant result occurs when you reach a certain degree of confidence in the results after rigorous testing; we call that degree of confidence the confidence level, which demonstrates how certain we are that our data was not skewed by random chance or luck. 

More specifically, the confidence level is the likelihood that an interval will hold values for the parameter we’re testing. 

Statistical significance helps quantify whether the result is likely due to chance or some factor of interest. When a new finding is significant, it means you can feel confident that it’s real — not that you just got lucky (or unlucky) in choosing that specific sample.

Calculating statistical significance accurately by hand can be a pretty complicated task that requires a solid understanding of calculus and statistics. When you calculate by hand, however, it’ll help you more fully to understand the concept. 

Follow these steps for calculating statistical significance:

  • Create a null hypothesis.

The very first step in calculating statistical significance is to determine your null hypothesis. This should state that there’s no significant difference between the sets of data you are using. Keep in mind that you don’t need to believe the null hypothesis. 

  • Create an alternative hypothesis.

Next, you’ll need to create an alternative hypothesis. Typically, this is the opposite of your null hypothesis since it will state that there is, in fact, a statistically significant relationship between your data sets. 

  • Determine the significance level.

The next step involves determining the significance level, or rather, the alpha. This refers to the likelihood of rejecting your null hypothesis even when it is true. A common alpha is five percent (i.e., .05). 

  • Decide on the type of test you will use.

Once you’ve created both hypotheses and have determined the significance level, you’ll need to determine if you’ll use a one-tailed test or a two-tailed test. Whereas the vital area of distribution is one-sided in a one-tailed test, it is two-sided in a two-tailed test. In simpler terms, one-tailed tests analyze the relationship between two variables in one direction, while two-tailed tests analyze the relationship between two variables in two directions. If the sample you happen to be using lands within the one-sided critical area, the alternative hypothesis is considered to be true.

  • Perform a power analysis to find out your sample size.

Next, you will need to do a quick power analysis to determine your sample size. This involves the sample size, effect size, statistical power, as well as significance level. For this important step, it might be best to use a calculator.   You’re going  to see how big a sample size you’ll need in order to learn the effect of a given test within a degree of confidence. In other words, it will let you know what sample size is best to determine statistical significance. If your sample size ends up being a bit too small, it won’t give you the accurate result you’re looking for. 

  • Calculate the standard deviation.

Now it’s time to calculate the standard deviation. To do this, you will need to use the following formula:

standard deviation = √((∑|x−μ|^ 2) / (N-1))


∑ = the sum of the data

x = individual data

μ = the data’s mean for each group

N = the total sample

  • Use the standard error formula.

Next, you’ll need to use the standard error formula. For our purposes, let’s just say that you have two standard deviations for your two groups. The standard error formula would look as follows:

standard error = √((s1/N1) + (s2/N2))


s1 = the standard deviation of the first group

N1 = group one’s sample size

s2 = the standard deviation of the second group

N2 = group two’s sample size

  • Determine the t-score.

For your next step, you will need to locate the t-score. Here is the equation:

t = ((µ1–µ2) / (sd))


t = the t-score

µ1 = group one’s average

µ2 = group two’s average

sd = standard error

  • Find the degrees of freedom.

Once you’ve determined the t-score, it’s time to find the degrees of freedom:

degrees of freedom = (s1 + s2) – 2


s1 = samples of group 1

s2 = samples of group 2

  • Lastly, use a t-table.

Once you’ve completed all the steps listed above, you’ll calculate the statistical significance using a t-table. Begin by looking at the left side of your degrees of freedom and find your variance. Then, go upward to see the p-values. Compare the p-value to the significance level (aka, the alpha). 

Keep in mind that a p-value less than 5 percent is considered statistically significant — this is where you’ll often see the famous (p=.05) or (p=.01) in whatever study you’re looking at. 

Don’t worry, you can easily determine the statistical significance of experiments — without any math (or headaches) — using an analytics tool like Kissmetrics

Where Does Statistical Significance Factor Into Data Analysis, and How Can It Help My Company?

Statistical significance plays a huge factor in data analysis because brands use it to understand how strongly the results of an experiment, poll, or survey they’ve conducted should influence the decisions they make. 

To give you a quick example, if a marketing manager runs a pricing study to understand how best to price a new product or service, she will calculate the statistical significance so that she knows whether the findings should affect the final price. 

Statistical significance can help you and your company immensely — and it’s pretty easy to see why. Think about it: would you rather make a critical business decision based on a study with a small sample size, or would you prefer to make that same crucial decision based on accurate statistics and testing?  

At the end of the day, data is not valid if it’s not statistically significant.

 Real Life Example: Trying to Determine the Correlation Between a New Ad Campaign and Sales

Now that you understand exactly what statistical significance is and how to calculate it, let’s take a look at a real-life example:

Let’s say you want to increase sales by attracting more customers to your growing business — so, you decide to run a new ad campaign. In doing so, you take into account how many advertisements should be made in print, and many should be made digitally. 

You rely on your past marketing campaigns using ads to forecast how many you will need of each. If you determine that your p-value is about 5%, you will end up with a result that is not statistically significant. This means that there is a greater than 5% chance that the relationship between the two ads was left up to chance. 

Therefore, this result would indicate that it’s not reasonable to use the previous ad campaign as a guide to drive in new sales. 

The Takeaway

Statistical significance is an extremely useful tool for marketers to validate their insights and provides credibility to their research. Although calculating statistical significance by hand is cumbersome, there are some incredible analytic tools like Kissmetrics that can help. 

With the click of a button, Kissmetrics can help managers and researchers alike have confidence in their business decisions by providing them with the tools they need to run accurate tests and gain valuable insight into product and marketing campaigns.  

Statistical significance is a powerful tool — are you using it? 

Check out Kissmetrics today to add the power of p-values to your arsenal to increase your product’s ROI. 





What is High Touch vs. Low Touch?

In many cases, Customer Success Managers face a common doubt in their workplace: What engagement model should they use for their customers? Or, what is the correct number of touchpoints they should maintain for the best results? 

There are quite a few factors at play here. 

For example, a low touch customer success is useful for low customer support cost and reaching out to a much wider audience. However, on the other hand, a high touch customer success helps you give a more personalized experience. 

Over the last few years, customers of the Software-as-a-Service (SaaS) world have completely changed the expectations of the approach. 

Today, consumers need more hands-on interactions from brands more than ever, with a constant line of communication — high touch service is the modern way to avoid churn. Plus, most surveyed businesses agreed that it’s more cost effective to try and retain current customers than trying to acquire new ones.

So, before going into which engagement model is best for you and your growing business, let’s first get a good understanding of the difference between high touch vs. low touch. 

The Basics of Customer Engagement

We already know quite a bit about the customer journey — how it’s made up of numerous touchpoints, from search to buying decisions to post-purchase support. And, we know that providing a memorable customer experience at each of those points is critical to not only building a solid reputation for your brand, but also maintaining it. 

However, the unfortunate truth is that the value of customer engagement is often underestimated, even though it’s vital to nudging customers toward conversion along their journey. 

You see, customer engagement is all about encouraging your customers to interact with and purchase from your brand over other brands. And if you do it right, you’ll grow your brand and build customer loyalty, ultimately driving revenue. 

In fact, there is actually a driven — and proven — correlation between the level of customer engagement and business profitability. According to research, companies who improve their customer engagement can increase cross-sell revenue by 22% and up-sell revenue by 38%.

Despite the incredible financial impact engaging with customers can have, some brands are still not putting effort into customer engagement at any point in the sales funnel, and if they do, they don’t have a concrete strategy that commits to either high touch or low touch techniques. 

What is the Main Difference Between High Touch and Low Touch?

Over the years, these two popular approaches have been pitted against each other.  This often happens as brands transition from high touch to low touch when scaling their business, or, conversely, from low touch to high touch when they realize all of their expendable funds are going toward customer acquisition just to end up with a high churn rate.

But why are these two pitted against each other in the first place? Why are both not seen as equal techniques in an all-encompassing customer success strategy?

In fact, taking a hybrid approach actually is one of the best strategies to use — but, the only way you can do this effectively is to know the advantages and disadvantages of each approach.

What Does High Touch Involve?

The high touch approach involves providing a personalized experience to your customers by catering to their specific wants and needs throughout the customer journey. This not only elevates the customer’s experience but also ensures a strong long-term relationship between the brand and the customer. With this effective approach, the sales representative may gauge the customer’s specific needs prior to the decision stage and pitch solutions to their pain points accordingly. 

Efforts to reach out and connect during any point of the sales process are generally well-received by the modern consumer, so developing a connection early on and maintaining that level of service throughout the journey can make a solid impression that later equates to customer loyalty. 

However, the truth is that this customized, one-on-one level of service is not always scalable — and it’s definitely not cheap. 

What Does Low Touch Involve?

Also known as tech-touch, this effective strategy involves digital engagement and the use of non-designated customer success associates as needed. 

Essentially, there is very little human interaction with the customer, but automatic check-ins are frequent and managed by a CMR-type software. This data-driven style of proactive management helps customers to feel valued, and lets them know help is available if they need it. 

This approach frees up customer support agents so they can tend to high-needs clients or specific high-touch situations. Pre-programmed temperature checks can also be implemented as a part of a retargeting strategy.

As amazing as the low touch approach may seem, the drawback here is pretty obvious: without at least some human interaction, customers may begin to see the brand as cold or distant. 

Even more, they may begin to view the brand as uninterested in actually solving their problem. It’s also difficult to forecast churn when you’re not able to talk to your customers directly.

Is One Method Better Than the Other For Certain Markets?

Smart Customer Success Managers (CSMs) maintain a hybrid high touch-low touch model to cater to different customers. It all depends on various factors like the number of customers going through the sales process at once, customer portfolio, the number of available and appointed CSMs, the industry of the business, and more. 

You see, the truth is that there is no “one-size fits all” high touch low touch ratio for all SaaS companies. 

One approach is not necessarily better than the other by default because every brand has a slightly different target market, and where one brand might benefit greatly from a mostly low touch approach, others may benefit from a mostly high touch approach. 

Whether or not a high touch model would fit better compared to a low touch model is entirely up to the product or service being offered and the desired outcome. 

If your product or service is generally user-friendly enough that installation, implementation, etc. can be done by the user, it’s often best to go with an engagement model that doesn’t overwhelm the customer with aggressive — and sometimes annoying — communication. 

If your product or service is designed as an expert-designed solution for novices to the subject area, a high touch approach may be better received since your consumers are working with something they don’t have much familiarity with. 

One of the best tactics you can use to decide which model is more appropriate is to step back and take a look at your sales funnel — is it weaker at the top of the funnel, the middle, or the bottom? Weaker conversion at any point of the journey may indicate the need for a high touch approach just in that stage, while you might be able to get away with a low touch everywhere else. 

So, before diving into either a high touch or low touch approach, really think about who your customers are (think customer persona). 

Real Life Example: How an Online Shop Could Take Either Approach

By now, you understand the difference between high touch engagement and low touch engagement, so let’s take a look at a quick example.

Let’s say you have an eCommerce web store that sells custom t-shirts in bulk for companies, non-profits, sports teams, etc. Your store, while it does have its own Shopify backend, also has Facebook and Instagram to help with advertising efforts. 

A high touch engagement approach may involve having each new customer work with a designated designer from the very beginning of the process to help walk them through the design process, ordering process, manufacturing process, and fulfillment. The designer would work with them every step of the way to ensure that the custom order is not only fulfilled correctly, but that any potential pain points are addressed as soon as possible. After fulfillment, the designer may follow up within a week of delivery to ensure all the shirts fit and look as planned. 

A low touch engagement approach may involve having an automatic pop-up message come up on the site as the customer is browsing shirts or even as they’re mocking up a design. The message may say something like “Hi! Do you need help with anything?” and the customer can then either reply free-form or select a pre-designated response to get further assistance from a human support agent. 

The Takeaway

High touch and low touch customer success strategies are not in opposition to each other — you should use a combination of both that is tailored to your specific customer persona, journey, and product experience. 

But one rule is common: for anything generic, a low touch approach is more than enough. For dealing with valuable or high maintenance customers, the high touch model is most useful. 

By using a dedicated analytics program like Kissmetrics, you can gain valuable insight about the impact of your engagement efforts to help you figure out which approach is best for their specific needs, even if those needs differ per-cohort. 

Kissmetrics gives you the information you need to acquire qualified prospects, convert more leads into paying customers, and reduce churn — everything you need to ensure the success of your growing brand. 

If you’re ready to get started, click here to Schedule a Demo with Kissmetrics, and see how product and marketing insight can help you drive revenue.