A/B testing the Newsletter sign-up modal

...or reducing user annoyance by reducing size

After launching the initial A/B test successfully, I turned my attention towards other quick wins. A quick discussion with project stakeholders showed that our Newsletter sign-ups on the Outriders homepage were lacking. Keeping in mind that 80% of our traffic comes from mobile users, I suspected it might be because the modal took up the entire viewport, as can be seen in the screenshot on the left. This is a big issue particularly on mobile that could cause users to abandon the page since they felt like the content is blocked unless they sign up to the Newsletter and could give them the impression that they are being upsold, eroding trust. Additionally, the bounce rate was quite high at about 40%.

The Problem

After making the suggestion of reducing the size of the Newsletter modal to a less intrusive, small pop-over to avoid the above-mentioned issues, a new A/B test was approved.

Hypothesis

By reducing the size of the Newsletter sign-up modal, we'd like to see an increase of users interacting with the modal and signing up to the Newsletter. We deem the experiment successful when we see an increase of Newsletter sign-ups through the smaller, non-intrusive modal and a reduction of the bounce rate.

The Prototype

Once more, I created an interactive prototype in Adobe Xd to illustrate changes. The Newsletter component is handled by our Digital Platform team who provide internal tools such as CMS and retailer APIs. For their reference, I designed two different versions of the modal. One followed their initial design a little more closely and the second one I kept as minimal as possible.

Image showing the two version of the pop-over component. On the left is the version that is closer to the original while the right is the new, minimalist component.

The pop-over containing the input field expands when the user clicks into it to enter their email address.

gif showing the full flow of the interactive prototype from start to finish.

The stakeholders of the Digital Platform agreed with making the modal as non-intrusive as possible so to not block users from seeing content as obscuring content was affecting the bounce rate of the page negatively. After a quick check-in with the legal team, the modal however grew slightly in size again to include important GDPR information.

Results

The test was successfully launched about a month later together with a major game update for Outriders after testing the functionality of the new component.

Full flow as it appears live on the website.

As of right now, the experiment is live. Preliminary data from the Google Tag Manager shows an uptick in users signing up to the Newsletter when shown the new component.

This case study will be updated with more data once the experiment has been turned off.

← Back to Portfolio Overview