Real-life case studies on how retailers can optimise on-shelf availability with machine learning.
How many strawberries should I ship to each of my grocery stores? What level of energy consumption should I expect at 7pm in a specific neighborhood? How do I know when there’s a need to upgrade my network capacity to cope with increased usage?
All these questions relate in one way or another to a type of forecasting based on estimating the level of demand for a product or service to better plan and manage supply chains or future investments. Moreover, models and predictions need to be tailored to specific situations. For example, demand for strawberries might be different to that of tomatoes, and sales patterns in a large rural store will be dissimilar from those of a small, inner-city one.
Transforming retail intelligence
Traditionally, statistical demand models have been built using either relatively simple techniques and limited data sources or aggregated data and manual processes, as the sheer number of models makes granular model tuning all but impossible. For instance, the number of combinations of products and stores can reach into the millions for a large national retailer. However, these limitations can be overcome. Here at Think Big Analytics, we are developing a scalable framework for modelling and forecasting that combines modern machine learning techniques with big data technologies.
Rather than relying on time-series techniques that estimate demand based on previous sales only, our methodology has been designed to use a variety of data sources: from pricing and competitive information to variables that capture cannibalization across different product lines, to external factors such as weather forecasts.
Figure 1: Modelling GUI. The variable inspection tab visualizes the relationship between product sales and other key features such as product price
The data gathering stage is followed by a variable intelligence engine that allows performing transformations to capture different types of impacts on demand. For example, demand might be dependent on a categorical distinction between promotion versus full price that does not depend on the level of discount. It could also be captured by a non-linear relationship between price and sales, whereby increasing the price is associated with the same level of sales up to a certain price point, after which we observe a dramatic drop in sales.
The purpose of the feature intelligence module is to generate all these variables for each model and pass them to the modelling engine. That’s where all the signals are combined and prioritized so that only the most relevant will be used in each situation. For example, the demand for a bottle of wine sold at a large store might be strongly dependent on whether the item is on promotion or not, whereas the sales pattern for a barbeque set will be highly reliant on the rain forecast in the area where it is sold. These relationships are automatically learnt from data while requiring minimal user intervention, and used to accurately predict future demand.
Real-life success stories
Not only have we used the framework for building general and robust demand models, but we have also combined its different components to tackle more specific problems, for example, reporting out of stock items for a large UK food retailer. Out of stocks occur for a variety of reasons linked to the running of operations in stores, and to the management of supply chains. It is estimated that between 3% and 6% of items in UK supermarkets are out of stock at any given time. This has a significant impact on missed sales opportunities, which can escalate to losses amounting to millions, as well as have a negative impact on customer experience. Furthermore, retailers spend on human workers who manually check the product levels on the shop floor, even if in the great majority of cases replenishments are unnecessary.
Figure 2: Out of stocks. It is estimated that between 35% to 6% of products are not in stock at any given time in the UK supermarkets
Both operational costs and missed opportunities could be significantly reduced if we had a reliable way of estimating when and for which products an out of stock occurred, enabling retailers to tackle the underlying root causes and plan ahead. For instance, the could order or bake more bread if it tended to be unavailable at the end of the day or only check products when they are likely to be missing, hence freeing up time previously invested in a repetitive stock count rota.
That’s where our modelling approach enters the picture: by combining stock data with historical sales patterns, we can estimate the expected demand for any given product at any given store and compare it with sales data and insights. Sometimes, products will not sell simply because they are what in retail jargon is referred to as “slow rotation.” For example, think about a bottle of expensive champagne at the end of an aisle in a small village shop that only sold when Jane got her job promotion. Other times, however, we might expect a high level of sales, but we observe zero in the cash register: if sandwiches are not selling at lunchtime at a busy city centre store, it might be because someone has forgotten to replenish the shelves.
During a trial involving a sample of products and stores at major UK grocery retailer, our model-factory reporting tool was accurate in discriminating in-stock from out-of-stock items in more than 97% of cases based on a manual scan of the products available on the shelves. As the tool is implemented across the entire range of products and stores, operational savings and reductions in missed sales opportunities are estimated to bring in tens of millions of pounds annually.
We are currently testing and refining our modelling engine and, going forward, we plan to use this powerful tool to tackle a wide range of supply chain challenges. Demand models can be enriched with new data sources such as weather forecasts. Promotions can be better designed through a data-driven approach that integrates with other functions and operations such as media and in-store support. Finally, price reductions can be optimized to reduce waste for highly perishable products, while improving lead times and increasing availability.
Pricing is a major focus area for our future development. We are making significant steps towards a framework that combines flexible and highly predictive models with a robust, business-validated optimization engine to enable better pricing decisions. Such a system can underpin multiple domains such as strategic pricing, promotional planning, markdown pricing and reduced to clear items by analyzing and optimizing different “levers” that the business can pull to maximize revenue, sales volume, and margins.
Leveraging and deploying machine learning at scale is what enables the most innovative organizations to build the products and services of the future. Our mission is to empower high impact business outcomes with cutting-edge data science and big data technologies.