Personalize grocery stores with Artificial Intelligence
By Frederikke Bloch, Michelle Palsgaard Grassi, Finn Petersen and Emil Bajlum
Artificial intelligence (AI) is concerned with developing intelligent machines that, at best, act human-like or even more efficiently and solve problems in the process. The development that machines can learn is a crucial step in this direction. Rigid control systems no longer have to be programmed, the machine can analyze examples using data and recognize specific patterns and regularities with the help of self-learning algorithms. In this way, data can be linked with one another, and conclusions and predictions can be made. These developments are very interesting for the marketing sector. The automated action of tools for evaluating data or communication supports companies in knowing their customers and target group as precisely as possible and in addressing them personally.
Data and Analytics can help grocery stores achieve a deeper understanding of their customer's behaviour. As Yang et al. (2016) explains, online personalization benefits both the user and the provider.
“Online personalization benefits both users and service providers. Successful personalization anticipates users’ intentions and reduces their navigation (number of screens) and selection efforts (number of taps and inputs).” (Yang et al., 2016, p. 566).
The potential for grocery stores would be limitless if they can gather and track customer data, such as:
- who are the customers (e.g. age, gender, occupation, etc.)
- what kind of items they purchase
- when and where they purchase the items
With the help of advanced analytics, retailers can derive meaningful insights from a massive amount of data — for example, recognizing high-value customers, their motives behind the purchase, their buying patterns behaviours, and which are the best channels to market to them and when.
User needs
For humans and AI to work together successfully, sufficient trust in AI systems is crucial. To be trustworthy, AI’s decisions must be understandable, explainable and reliable. At the same time, the increasing complexity of advanced AI systems makes it more and more difficult to understand how they work in detail. Kliman-Silver et al. (2020) arguments that explainability and trust are essential to convey the essence of AI in the users’ context.
“Since AI-driven experiences are only partially predictable, users may not initially trust the system, heightening the user’s need for explainability: if the system explains how it makes its decisions, the user is more likely to build trust in the system based on the explanations. In other words, if the user understands how the system is making its judgments, they are likely to trust it more.” (Kliman-Silver et. al., 2020, p. 2)
This is where Explainable Artificial Intelligence (XAI) comes in. With XAI, the decisions and activities of an AI should be made transparent and understandable for people. The basis is created so that users and decision-makers understand how AI systems work and achieve specific results. This transparency forms the basis for the trust and acceptance of artificial intelligence — the prerequisites for its successful use.
About our case, it is therefore essential to explain how the application responds to their actions in the supermarket. Consequently, we have focused on the user choosing predefined preferences to start with and kickstart the AI. To maintain the user's curiosity, the personalized offers and recommendations must suit the needs of the user. When the recommendations do not match the users’ preferences, it is impossible to build trust in the application.
Mapping the product
We initiated a design sprint with a classic brainstorming board (Innovation værktøjskassen 1, 2021). This was an effective way to share ideas across the group members and map out five products from that starting point. Working in a design sprint, it is essential to know the iterative process through the four phases: 1) Idea, 2) Build, 3) Launch, and 4) Learn. These four phases can loop in 1) Idea and 2) Build for several ideations before passing onto the next phase (Google Ventures, 2021).
After setting the stage in the understand and define phase, we used “How can we…?” (Innovation værktøjskassen 2, 2021) questions to frame and finalize the main product ideas and lastly, we used dot-voting to choose the final idea amongst the sticky notes (Gibbons, 2017), which we continued unfolding.
The product idea was to create a mobile application that will help customers get discounts on their most bought items through artificial intelligence. Likewise, to help them remember if they are ‘low-in-stock’ in their at-home warehouse, for example, on basic items like; toilet paper, soap, milk and so on.
Next up, in the design process, we looked at related work to gather more knowledge and inspiration about grocery stores and how they engaged with their customers. Here we found similar situations from the danish grocery stores, Netto and Bilka. Among others, we used the two cases to map pain points and gain points for our design and what the application should consist of (Frankly Studio, 2021).
Furthermore, we also looked at other online solutions, e.g. online-streaming services like; Netflix, Amazon, YouTube and Spotify. The three services’ all had in common that they gathered information about what the user liked to listen to or watch. On top of that, also what they bought from that knowledge on how suggested new or similar music or movies that the service prospected that the user would like. We also noticed that the video and movie streaming-services, mentioned had a function that intrigued the user to keep watching. We used Amazon as an interactions-critique on the recommendation system and how the purchase flow is affected by the company and the ads they push out. With this in mind, we started mapping out how an application could nudge the user to buy specific groceries that they could need in their households. These suggestions should come through an AI assistant who learned what the customers, such as eating and drinking, based on the customer’s purchased history.
Later on, we used the Crazy 8’s method. This fast idea-sketching exercise was set up to ideate eight distinct ideas in eight minutes (Design Sprint Kit with Google, 2021). Our goal was to challenge creativity and create a wider variety in our solution for our product idea. After each group member sketched individually, they presented their unique ideas for the rest of the group. This exercise was important to the idea development of the chosen product because it challenged the team members to think more creatively within the product idea.
In this process, we found out that we were very inspired by the Netto- and Bilka mobile application, and from that ‘crazy idea’, we started out designing the visuals for our own product.
Use scenario
The use scenario of the Netto+ application is the following: the customer shops groceries in the local Netto and has therefore downloaded the new Netto + application to get and find more personal offers. Before the customer goes out shopping, the application is checked to see which relevant offers the customer should specifically keep an eye on. Here the customer can add these to his or her shopping list. The customer goes to his or her’s local Netto and opens the application to see which items to buy. Here the user checks the shopping list once again and finds the items using the application. The customer buys the groceries and scans the Netto+ application at check-out. The application saves the customer’s purchase and will now gather more knowledge about the customer. The Netto+ app notifies the customer when there are offers for the customer’s favourite item.
To analyze the context the product engages, we use the framework from Kliman-Silver et al. (2020). The first dimension distinguishes between personal and social experiences. Our product has focused on the individual user experience as the user one-by-one interacts with the product and the AI.
The second dimension is the discretionary and non-discretionary nature of the experience. Our product is a discretionary experience, a temporary, task-based, non-committal experience (Kliman-Silver et al., 2020). The last dimension describes the intelligent systems level of independence within the interaction. We have focused on high independence, as the product works as a reactive/proactive AI.
Prototyping
After our design sprint and based on our sketches, we started designing our product. Our product design is based on Netto’s current app with incorporated features and categories. To design our product, we have used the program Figma, which allows us to design and collaborate simultaneously. Lastly, Figma makes it possible to connect the different screens into a working prototype that is clickable and, in this way, simulates an authentic experience when testing the app.
Low fidelity wireframes
First, we started creating low fidelity wireframes (Babich, 2017) to focus on the actual functionality instead of the visuals. The purpose was to decide where the different features should live in the app and only include the essential elements. Before developing the final prototype, we tested the low fidelity wireframes on ourselves to see if the added features and categories made sense. Here we made some adjustments concerning the wording in the different categories.
High fidelity prototype
After creating the low fidelity wireframes, we began designing the high fidelity prototype (Babich, 2017) to make it easier for the user to understand the context by applying design and interaction animations. We have developed a suitable prototype for the different test tasks we want to do when testing the app on a few respondents.
Our product
In our solution, we have focused on improving the Netto + app by incorporating AI to personalize the purchasing experience in the supermarket Netto by tracking the customer’s purchase history and using this data to create more individual and relevant offers for the customer. Practically the customer’s payment information is linked to the app. When the customer adds a product offer to his/her shopping list, this will be taken into account at checkout when paying at the supermarket. As the transaction is completed, Netto can collect this data and thereby learn more about the consumer’s buying habits.
Preferences
When downloading the app and using it for the first time, the customer has to complete the sign-up, where the customer is asked to choose his/her preferences. Here the customer is presented with several keywords to choose from, e.g., Meat, fish, vegetables, organic and gluten-free. Based on the selected keywords, we will use AI to create valuable and relevant offers from the start to the customer.
Product feed with personalized categories
After having completed the sign-up process, the customer will be presented for a product feed. In Netto’s existing app, the product feed consists of two offer categories; Netto’s offers and Personal Offers. Here we have chosen to expand the categories to use the purchase data to better target offers and meet the customer’s specific needs. So for the improved app, we have chosen the following categories; “Offers for you!” “Time to stock up again?” and” Others like me purchased!”. The category; “Offers for you!” will be based on the type of products the customer buys in the supermarket daily. This will appear in the customers ' offers if the customer mostly buys vegetables and lactose-free products. Next, we have the category; “Time to stock up again?” which will be based on the products that the customer buys at intervals of a few weeks, including toilet paper, washing-up liquid, and many others. Lastly, we have the category; “Others like me purchased!” which will be based on the purchase history of others from the same segment. This category is meant to inspire and introduce the customer to other products that could be interesting. This feed will automatically adjust to the customer’s habits and needs. Here we have also made it possible for the customer to delete an offer- in this way, the customer can influence the AI and what offers are being shown.
Product view
When the customer scrolls through the product feed and clicks on a specific product offer, the customer will see a product view, where the customer can read further details about the particular product. It is possible to add a product to the shopping list from the feed and the product view.
In this way, we use AI to create a more personalized experience that adapts to the customer. To actually see how this design concept works in real life, the next step will be user testing.
User testing
When testing the prototype, we created a think-aloud test, where we have made four specific tasks for the respondent to solve. Tomitsch et al. (2018) refer to that a think-aloud test can provide insight into the users thinking behind the interactions performed.
“The think-aloud protocol method allows designers to gain access to people’s thought processes while they are interacting with a product or service” (Tomitsch et al., 2018, p. 124).
These tasks have focused on testing the respondent’s overall experience and test if the user understands how the offers are personalized for the individual customer and lastly if the app could be interesting for them to use. For the user test, we have recruited two respondents; Markus and Morten.
When solving the tasks, we asked the respondents to think aloud during the whole test to hear their thoughts and observe how they interact with the app.
We have used Figma Mirror for the test to connect the prototype to the respondent’s phone directly.
Both respondents had an overall good experience with the app and had an easy time solving the various tasks. They also understood how the offers are being generated based on the customer’s purchase history patterns and how the delete function and preferences affect which offers are presented. On the other hand, the respondent Markus was unsure if the offers were only targeted at him or others got the exact offers. Then he was also confused about the category; “Others like me have purchased” because he found it odd that he should receive recommendations based on what others had bought. The other respondent Morten wanted it to be more clearly how one’s data is being used and how the preferences impact what offers one receives.
Reflecting on user feedback and the design process
Reflecting on what we have learned regarding user feedback, the “Others like me have purchased” category can arguably be considered worded inadequately. Given that the app’s extension aims to personalise the user’s grocery shopping experience, it can be deemed contradictory that the app would suggest what “others like” the user have purchased, disrupting the personalised user experience in the process. Wording the category differently, such as “example”, can make it less confusing for users such as Markus, who prefers the personalised experience the app’s extension offers.
The Wizard of Oz method is a flexible approach that allows concepts to be tested and modified without worrying about potentially tiresome code changes, breaks in daily testing schedule or total development costs. Allowing designers to learn how users will react to a system that does not exist yet, this method is very efficient for testing new and unique concepts or projects requiring substantial effort to create. Considering how our concept of “Netto’s Personal Offers” is an extension of Netto’s existing app, most users already have a pre-understanding of interacting with the concept. For this reason, we have decided not to carry out Wizard of Oz testing in the design process, as it would arguably add unnecessary time to the development of our concept.
The same can arguably be said when we choose not to test out the low-fidelity prototype on others than ourselves. According to UX Designer Carol Barnum, low-fidelity prototypes are good for “assessing the clarity of terminology and iconography” by learning from users whether the words and images in the user interface carry the intended meaning” (Six, 2018). However, in practice, many UX Designers will “not have the luxury of creating and testing a series of mockups”, and often choose to begin developing high-fidelity prototypes directly. Add to this that most projects arguably “won’t benefit at all from low-fidelity prototypes” and those that would are “usually innovative in some way”, according to Jordan Julien (Six, 2018). With these considerations in mind, we decided we would be better off testing the high-fidelity prototype instead of spending time on user feedback for our low-fidelity prototype.
Reflecting on practical and ethical issues with AI
Consumers are often tasked to choose between hedonic and utilitarian goods. When it comes to consumer products, hedonic foods and liquids are mainly consumed for pleasure (such as beer, fast food and chips). In contrast, utilitarian foods and beverages are consumed primarily because of their nutritional value (milk, broccoli and almonds) (Maehle et al., 2015). Experiments conducted by professors Chiara Longoni and Luca Cian (2020) suggest that if someone is focused on “utilitarian” qualities, the word of a machine is more effective than human recommenders. For someone focused on hedonic qualities, human recommenders are more effective. One of the experiments conducted tasked 144 participants to taste two different chocolate cakes, one made from an AI-generated recipe and another made from a human recipe. Participants ultimately found the AI-recommended cake less tasty but healthier than the cake recommended by the human chocolatier (Longoni & Cian, 2020).
Reflecting on the experiments’ results, one might wonder if an AI can generate satisfying special offers for the user’s hedonic needs. Is the AI able to adapt to creating tempting offers on unhealthy foods and beverages on a Friday if it has spent the entire week recommending special offers on healthy greens and organic juices? As Logoni and Cian argue, the AI very likely could adapt to hedonic needs but not understand them the same way a human would. This can lead to anti-climatic user experiences for those hoping to buy pleasurable foods and drinks for a lower price.
The question of how much AI should supply to humans’ hedonic needs also leads to a more considerable ethical discussion regarding how much AI should intervene in our private lives. If a specific user buys mostly hedonic foods and drinks, the app should by design generate special offers on even more hedonic goods. This would likely cause the user to continue their unhealthy diet unless the AI is programmed to intervene and generate offers on healthy hedonic food instead. Whether or not AI should “make the final decisions”, Admond Lee (2019) argues that it is nearly impossible for humans to “make accurate justification and selection without intensive computation within a short period”. This is why we delegate mapping decisions to AI when choosing the best route to a specific destination on services like Google Maps (Lee, 2019). Would AI not also be better than humans at making “accurate justification and selection” in a “short period” when it comes to calculating and choosing the best consumption products based on nutritional value?
Anna Trunk, Hendrik Birkel & Evi Hartmann (2020) clarify how researchers have argued that including ethical guidelines into algorithms is “only possible to a limited extent” and is “always influenced by the people designing them”. Researchers have argued further that it is necessary to retain “veto power” when “the decisions can have far-reaching consequences for human beings” (2020). In the case of our prototype, giving too much power to AI deciding what consumption products are best for us could have the far-reaching consequence of humans losing control of their own consumption. Considering how purchasing hedonic goods to “seek pleasure” and “avoid pain” is a motivational part of what makes us human (Higgins, 2006, pp. 440), if AI slowly removes this hedonic motivation, it could be unhealthy for us on a more psychological level. Nudging users to have a healthier lifestyle can also backfire and cause the user to be unhealthier in other scenarios (Hollingworth & Barker, 2020). Therefore, it can be argued that who decides what goods are best for us to purchase should be divided between AI and humans, allowing AI to “automate tasks” while allowing humans to “focus on work that will add value” (Trunk et al., 2020).
Another problem is that AI is fundamentally dependent on data. If the system does not have enough data on its users, the algorithm cannot draw any conclusions and make good recommendations until it eventually has gathered the data to do so (Franklin, 2021). This requires users to be patient with the AI learning and adapting to their buying patterns. For the Netto AI, this “cold start” can be fixed to some extent by making the app generate offers based on pre-existing data on offers the average user is most likely to capitalise on. This lets the AI make attractive offers to most users during the first few weeks while it slowly learns the specific buying pattern of the user. But if the user capitalises too much on the early offers made by the AI, the AI might think the user has no special shopping habits and would possibly continue generating the same offers it has done since the user started using the app, creating a repetitive and frustrating experience for the consumer.
If the AI ultimately fails to recognize the user’s unique shopping trends, it can negatively impact the user experience. The user might think the AI ignores the user’s shopping tendencies due to bad programming. Or they may begin to think their buying habits are more customary than they thought, which might, in a worst-case scenario, cause the user to express self-doubt.
However, if the Netto+ App’s AI succeeds at predicting the user’s favourite utilitarian and hedonic foods and drinks, it can create an incredibly satisfying user experience. Not only will the user get just the conveniently right offer for their favourite foods and beverages, but the user will also feel their buying habits are being noticed, creating a more personalized grocery shopping experience for them. When users think any shop understands their needs, they tend to buy more from that specific shop (Hyken, 2017), which will lead to revenue increase for Netto in the case of this app.
In conclusion, recommendation AI’s are most efficient for predicting utilitarian user needs but sufficiently less capable of predicting hedonic user needs. Therefore, it can be argued that as long as the user does not expect too much of the Netto app predicting what special offer on hedonic goods they would want at the end of the weekend, they will be most satisfied with the AI recommendations, especially when it comes to generating special offers on utilitarian needs such as greens and dairy products.
With these thoughts and reflections in mind, we have learned you can never fully predict how an AI will process the user’s store purchases over a more extended period and how the user will react to the information given by the AI.
Video of the prototype
References
- Babich, N. (29. 11 2017). The Difference between Low-Fidelity and High-Fidelity Prototypes and When to Use Each. Last accessed May 10th 2021: : https://blog.adobe.com/en/publish/2017/11/29/prototyping-difference-low-fidelity-high-fidelity-prototypes-use.html#gs.107qyr
- Design Sprint Kit with Google (s.a.) “Crazy 8’s. Core Method”. Last accessed May 10th 2021: https://designsprintkit.withgoogle.com/methodology/phase3-sketch/crazy-8s
- Frankly Studio (s.a.) “Reframing a pain point to a (design) opportunity”. Last accessed May 10th 2021: https://frankly.studio/reframing-a-pain-point-to-a-design-opportunity/
- Gibbons, Sara (2017) ”Dot Voting: A Simple Decision-Making and Prioritizing Technique in UX” NN Group. Last accessed May 10th 2021: https://www.nngroup.com/articles/dot-voting/
- Google Ventures, The Design Sprint (s.a.) “The Design Sprint”. Last accessed May 4th 2021: https://www.gv.com/sprint/?fbclid=IwAR1gDr13YAysuSYMiP2PnL2KlmmZO6ygZWpoC7VsS8cBhS1FZ0kbKoN18XA
- Greene, T. (2018). AI will soon decide what we eat. TNW. Last accessed May 10th 2021: https://thenextweb.com/news/ai-will-soon-decide-what-we-eat
- Higgins, T.E. (2006). Value from hedonic experience and engagement. American Psychological Association. Vol 113, №3, 439–460.
- Hollingworth & Barker (2020). The Holistic Impact of Nudging. The Marketing Society. Last accessed May 10th 2021: https://marketingsociety.com/think-piece/holistic-impacts-nudging
- Hyken, Shep (2017) Personalized Customer Experience Increase Revenue And Loyalty. Last accessed May 10th 2021: https://www.forbes.com/sites/shephyken/2017/10/29/personalized-customer-experience-increases-revenue-and-loyalty/?sh=7f27e1654bd6
- Innovation værktøjskassen 1 (s.a.) “Metode: Klassisk Brainstorm”. Last accessed May 10th 2021: https://innovation.sites.ku.dk/metode/klassisk-brainstormi/
- Innovation værktøjskassen 2 (s.a.) “Metode: Hvordan kan vi…? — spørgsmål”. Last accessed May 10th 2021: https://innovation.sites.ku.dk/metode/byg-mulighedsrum/
- Kliman-Silver, C., Siy, O., Awadalla, K., Lentz, A., Convertino, G., & Churchill, E. (2020). Adapting user experience research methods for AI-driven experiences. Conference on Human Factors in Computing Systems — Proceedings, 1–8.
- Lee, A. (2019). Should AI or Humans Make the Final Decisions? Towards Data Science. Last accessed May 10th 2021: https://towardsdatascience.com/should-ai-or-humans-make-the-final-decisions-6592519bac50
- Longoni C. & Cian L., (2020). When Do We Trust AI’s Recommendations More Than People’s? Harvard Business Review. Last accessed May 10th 2021: https://hbr.org/2020/10/when-do-we-trust-ais-recommendations-more-than-peoples
- Maehle, N., Iversen, N., Hem, L. and Otnes, C. (2015) “Exploring consumer preferences for hedonic and utilitarian food attributes”, British Food Journal, Vol. 117 №12, pp. 3039–3063. Last accessed May 4th 2021: https://www.emerald.com/insight/content/doi/10.1108/BFJ-04-2015-0148/full/html
- Pazzanese, C. (2020). Great promise but potential for peril. The Harvard Gazette. Last accessed May 4th 2021: https://news.harvard.edu/gazette/story/2020/10/ethical-concerns-mount-as-ai-takes-bigger-decision-making-role/
- Six, J. (2018). Testing Low-Fidelity Versus High-Fidelity Mockups. UXmatters. Last accessed May 4th 2021: https://www.uxmatters.com/mt/archives/2018/06/testing-low-fidelity-versus-high-fidelity-mockups.php
- Tomitsch, M., Ahmadpour, N., Borthwick, M., Frawley, J., Kocaballi, A. B., Loke, L., Nuñez-Pacheco, C., Straker, K., M., & Wrigley, C. (2018). Design. Think. Make. Break. Repeat. — A Handbook of Methods. BIS Publishers.
- Trunk, A., Birkel, H. & Hartmann, E. (2020). On the current state of combining human and artificial intelligence for strategic organizational decision making. Business Research. 13, 875–919 (2020). https://doi.org/10.1007/s40685-020-00133-x
- Yang, Q., Zimmerman, J., Steinfeld, A., & Tomasic, A. (2016). Planning adaptive mobile experiences when wireframing. DIS 2016
