A survey of user studies as evidence for dark patterns in consent banners

Almost any website today presents the user with the choice to be made in a consent banner: Do you want to accept or reject online tracking?

Design space for consent banners is enormous, and it’s still up to the website owners to decide how exactly their banner will look like. While the ePrivacy Directive requires a valid consent before reading or writing of cookies and other tracking technologies, GDPR has set only high-level requirements for such consent to be valid, leaving website owners a large design space for testing various consent banners on users.

Such situation, where law gave space for interpretation, and design space was unlimited, gave rise to the use of manipulative tactics in UX/UI commonly known as dark patterns (Gray et al. 2018). These design patterns have been often used by website owners in consent banners to increase “consent accept rate”, or in other words to nudge more users to click on “accept” rather than searching for a way to reject cookies and other trackers (Gray et al. 2021, Habib et al. 2022).

 

Since 2018, numerous case laws and various recommendations by DPAs (Santos et al. 2020), including CNIL’s latest guidelines and recommendations  gave more concrete requirements on the design of acceptance and rejection modalities in consent banners. However, it is still largely unclear which UX/UI design patterns could manipulate the otherwise free and active choice of end users: would users be manipulated if the “reject” button is not accessible on the first layer of the banner? Or would users accept cookies because “accept” is highlighted and much more visible than “reject”?

To answer these questions, researchers have performed studies with end users, testing and trying to quantify how much different designs of cookie banners influence users’ decision making (Utz et al. 2019, Nouwens et al. 2020, Machuletz et al. 2020, Habib et al. 2022). First, researchers present a “base line” cookie banner to participants – that is, a banner where accept and reject buttons are shown in exactly the same way. This is needed to understand users’ “usual” behavior when they are not manipulated. Then, participants interact with another banner that contains a studied dark pattern. Researchers would collect data about interaction with both banners by many such participants, and using statistical tests, prove that the difference in behavior is strong enough and is statistically significant. Below we show the acceptance rate in cookie banners that we derived from one of the first academic articles that performed a user study on cookie banners (Utz et al. 2019).

While such research studies are extremely valuable to policy makers, they are scattered across various disciplines, such as computer science, social science, design, law and sometimes not known to policy makers or regulators at the moment of decision making. 

LINC have decided to share an overview with all stakeholders, policy makers, and others: we collected and analyzed all 10 research user studies that exist up to date on how dark patterns in consent banners impact users’ decision-making. We then discuss the difficulties that arise when comparing the results of user studies and propose ways to evaluate be ecological validity of such studies. We propose our readers to reflect about the reasons why they may be useful by the policy makers and regulators in their guidelines and decisions. For example, five different studies reported different acceptance rate for cookie banners with both "accept“ and “reject” buttons, presented in identical way: while two studies reported 72% acceptance rate, other three study reported only between 53% to 60%. Finally, we provide the five basic conditions for the policy makers and regulators to perform such studies based on the overview of the research literature and our experience at the LINC.

LINC believes more policy makers and regulators should be aware of the research done in the area and the caution that is needed when reusing such studies or building new ones, because, when properly built, user studies may be used as evidence of dark patterns in cases on consent banners in the future.