You can see elements of Big Data everywhere. For us, this is interesting when transport companies want dynamic pricing models, insurance companies calculate insurance premiums, electricity companies want to estimate the likelihood of you changing supplier (customer churn) or media companies want data-driven decision-making. Sitting at home we also see Big Data at work when Netflix suggests our next film or when Google presents us with tailored ads and Amazon shows us related products.
Big Data describes a set of methods and tools that helps us to analyse, visualise and understand large datasets. In Norway, Knowit cooperates with several key players in the fields of transport, banking, finance and media in terms of big data, prediction and machine learning.
NSB – facing competition – wants strong control of pricing
You have almost certainly experienced how airlines use dynamic pricing. Thanks to a partnership with NSB, Norwegian state railways, we recently came to understand how, in fact, dynamic pricing for trains is far more challenging than in the aviation industry. Unlike when you fly, you can get off along the way when you take the train. For that reason, forecast work generates huge amounts of data. Forecasts for all departures in a given time period involve several thousand route combinations and usually hundreds of millions of observations.
“The use of all available data sources and good forecast work around capacity calculations and pricing still has a long way to go,” says Bjørn Inge Stalsberg, Business Developer at NSB. “Our aim is to be able to price journeys dynamically and achieve an increase in revenue with the same materiel and performance as today. A good prediction and algorithm for pricing would, of course, represent a significant competitive advantage.”
“We already had a promising prototype that together with Knowit we refined to develop an algorithm that we can launch on the market. Currently we have theoretical models that haven’t been tested in the real world, but the data we do have indicates good results. The prediction model is now much more precise and thorough and provides us with a high level of accuracy on the most heavily used stretches of the network. We can also reflect seasonal variations and different purchasing patterns.”
“The algorithm Knowit has developed is such an exciting technological breakthrough that Amazon wants to look at it as a potential reference case. Based on NSB’s tough demands for both accuracy and scope, Knowit has used leading machine-learning algorithms to deliver very thorough work in connection with the forecasts. We regard the investment risk as low at the moment, and we are planning to launch dynamic pricing in relation to market demand during 2019,” says Stalsberg.
The algorithm Knowit has developed is such an exciting technological breakthrough that Amazon wants to use it as a reference case.
Bjørn Inge Stalsberg, NSB
Commercial TV channel adapted to viewers’ new patterns of use
Most media companies naturally put a high priority on their insight work and employ both qualitative and quantitative methods to ensure that internal and external data sources deliver greater understanding. That was also the case when one of the first commercially financed TV stations in the Nordic region cooperated with Knowit in order to be able to make choices based on data-driven decisions. The media group was seeing viewers moving away from linear TV to streaming on demand. Reduced revenues and an increased risk in connection with content production that did not suit the needs of viewers triggered a call for action. Until that point, most of the content had been developed based on experience and gut feeling, and so there was a need to make decisions based to a greater degree on facts and real customer insight.
“We put together a team with industry experience, data scientists and developers, who together came up with exact descriptions of the problem and were then able to design relevant algorithms,” says Ingvar Larsson, CEO of Knowit Decision. “Eventually our customer had dashboards that continuously presented data on what viewers were watching and which channels they were doing it through. In addition, the dashboard has revealed other parameters that could form the basis for producing relevant content in a relevant format.”
The Nordic TV company now has an overview and understanding of how the relationship between different choices in production has impacted customer loyalty and the choice of platform. Predictions also indicated how many viewers they would get for different concepts and what content they would need to develop and launch to maintain high ratings. These predictions saved the company between fifty and one hundred million krone, and increased revenues as a result of optimising the advertising placement.
Prediction of customer churn increasingly in demand
A classic yet ultramodern concept used by many service providers within finance, energy and transport is customer churn. In other words, a probability estimate of whether or not a customer will leave you. The process for a bank might look something like this when using Big Data and machine learning:
- A large dataset that contains details by customer. This may be transaction frequency based on time, gender, age, marital status, loan and repayment details, tracking of clicks online and mobile banking, and more – the only limit is your imagination.
- Analysts who cleanse the dataset and review it to get an understanding of which correlations you need to look for and which can be ignored.
- Carefully selected algorithms that are used to construct a model. This model will be responsible for making any predictions.
Using relatively simple processes, a bank can use this approach to calculate the likelihood of any of its customers leaving the bank at a given time. This is critical information that can be used to communicate preventatively with these customers. Experience shows that churn prediction can produce substantial savings, particularly for companies where acquisition costs weigh heavily.
Is it just large international companies that can generate revenue from Big Data?
Big Data, AI, machine learning, autonomy, neural networks and churn prediction are all fascinating concepts, but they are still largely limited to fancy PowerPoint presentations. That’s why we’ve been looking at how ordinary Nordic businesses can achieve business value in this area. And the investment level needs to be affordable.
... and here comes the technical description
In order to construct Big Data models, the algorithms involve large amounts of mathematics and statistics. One of the most significant challenges in this area is how to convert technical metrics for algorithm performance into good goals and business value metrics. (A metric is a unit for defining qualitative and quantitative goals, for example a ranking of how attractive a quote is on a scale from one to ten. Here, the scale can be called a metric for how attractive the quotes are.) This requires sound domain knowledge combined with a good understanding of the potential of the dataset compared to its generated model. This is the tricky bit – building a bridge between two separate fields. Should the model have a high level of accuracy for the selected customers, or ensure it identifies as many customers as possible who are likely to leave the bank? So, maximum precision or the proportion of customers likely to leave the bank? These are metrics that are dependent on one another – if you increase one, the other is reduced. Both are popular within performance assessment of machine-learning algorithms, but how do you translate this into financial goals?
We recently faced this issue when we helped a bank to predict whether or not a customer would want to leave the bank, and so take initiatives targeted at the customer segments revealed.
Interdisciplinary cooperation needed for increased business value
Often, targeted initiatives involve ringing the customer or sending customised offers. In the above example, it may be considered better to go for a high level of accuracy. What if the company in reality earns more by focusing on one customer segment with a lower likelihood of customer drop-outs? This also proved to be the case in the example above, but it took a long time before this was discovered.
One essential aspect of a pipeline (procedure or protocol) is to preserve the human dimension; never before has customer experience and value to the user been as valued as it is today. Domain insight must be mapped among both consumers and providers, and through cooperation, a common understanding of needs and desires must be reached. This is a key step in any pipeline.
Because it isn’t the model alone, or the algorithm, or the dataset that creates value for a company. It’s the pipeline, the method, the strategy. With a well-designed ‘pipeline’ (procedure or protocol) assembled through interdisciplinary collaboration between data scientists, business developers and the customer’s knowledge of economic goals and KPIs, value creation can be substantial.
The rough diamond
Instead of underestimating such projects, good processes in these projects can provide you with great results by uncovering the potential available from the data that has piled up in the back room. In some cases, the datasets will not have the potential to lead to significant revenue, but even in cases like that, a good ‘pipeline’ will reveal the needs and opportunities for setting up systems that passively develop good datasets over time. This is a low-cost investment that we can see more and more customers coming to value.
A good pipeline increases the chances that the final product will be in harmony with the company’s ambition. Ideally you will want to construct a model using metrics optimisation that takes into account the domains and the value you are looking for. And in a good pipeline it is absolutely key that the customer is at the centre. If you only process customer data as numbers on paper, you lose the empathy and relevance required for a successful Big Data project.