Independent Evaluation of the Risk Technology driver scoring methodology

Introduction

Risk Technology is a company specializing in the provision of “next generation” automotive telematics. The services they provide are aimed at helping businesses manage all aspects of vehicle and driver behavior. Over the past few years they have been working to refine the data collection, scoring and interpretation routines they use for assessing driver and vehicle performance, both by improving the underlying concepts and testing these against real driving performance data.

This paper presents a summary of a much larger study carried out by CAS involving an independent evaluation of Risk Technology’s driver behavior scoring methodology and algorithms. The evaluation made use of the largest data set which Risk Technology has collected to date relating driver behaviours to insurance claims.

Telematics and Driving Behaviour

Automotive telematics have been in use for improving various aspects of driving for over 20 years. In their earliest form, they were used largely for tracking vehicle and driver movements and calculating mileage and fuel consumption. These are still the most common uses but in recent years there has been increasing interest in using telemetry to improve driver safety and performance.

There are three main approaches to improving driver safety:

• Crash detection and management
• Preventive vehicle maintenance and servicing
• Influencing driver behaviour and performance

Risk Technology provides services in all of these areas but the last is the focus of this paper. For this use, the telematics device usually provides a combination of GPS data and vehicle motion data. The motion data is usually provided by accelerometers within the telematics device but other sorts of motion information can be provided by other devices.

The key to influencing driver behaviour and performance, however, is not the nature of the devices that are used but the way the data from these devices is interpreted. To fully appreciate the interpretations that data from telematics can provide, you need to think about the human factors that contribute to crashes and other driving-related problems, such as breaking the rules of the road.

There are a number of ways of thinking about the human factors which influence driving behaviour. A common one, based on theories of human error, is to differentiate between errors and violations. Essentially, in respect of driving, errors are genuine mistakes, possibly as a result of failures of perception, attention, memory, decision making or skill. Violations are behaviours where the driver deliberately drives in a way which breaks the rules, endangers others or themselves, or is just thoughtless in so far as they do not consider the effect of their actions on other road users.

The majority of systems built around telematics fail to differentiate between these different factors. They tend to only look at certain sorts of deliberate violations (e.g. breaking the speed limit) and fail to take into account the full range of factors which may result in crashes. For this reason, they tend to adopt a very one dimensional approach to evaluating driver performance and so are inadequate for accurately predicting crash liability. Furthermore, they tend to approach the task of assessing driver behaviour as a set of descriptions of vehicle activity rather than thinking about the interpretation of these indicators in human terms that drivers will recognise in their own behaviour.

The Risk Technology Approach

The Risk Technology approach is to only use indicators of driving performance that are immediately meaningful to drivers. Currently, five indicators are produced but more are being added as ways are found to turn the raw data into meaningful measures. The five current indicators are:

Speed: This is measured using the proportion of time spent driving above the local speed limit and the severity of the speeding. The score is interpreted as the extent to which the driver exceeds local speed limits over a number of trips.

Braking: This is measured using the harshness of braking and the proportion of occasions when braking is harsher than would be expected given the speed of the vehicle at the time. The score is interpreted as the extent to which the driver brakes unusually hard.

Acceleration: This is measured using the rapidity of acceleration and the proportion of occasions when acceleration is more rapid than would be expected. The score is interpreted as the extent to which the driver accelerates more rapidly than is usual.

Urban: Crash risk is known to be higher on urban roads than other types of road. Therefore, this measure takes into account the proportion of time spent driving on urban roads. However, the way people drive on urban roads is also important, so this measure takes into account the speed, braking and acceleration scores while driving on urban roads. The score is interpreted as the extent to which the driver’s behavior is appropriate for driving in an urban environment.

Night Time: Crash risk is also known to be higher at night. As with the urban measure, this measure takes into account the proportion of time spent driving at night plus the speed, braking and acceleration scores while driving at night. The score is interpreted as the extent to which the driver’s behaviour is appropriate for night time driving.

All the measures are scored on a 0 – 100 scale where low scores indicate high risk, unsafe driving. The five scores can be combined to produce an overall score. The weighting of individual measures can be adjusted by users but Risk Technology has also produced weighting patterns based on analysis of drivers’ crash risk. It is important to note that it is possible to produce a number of different overall scores which can be used to predict different aspects of driver performance.

Accuracy of Prediction

The key question concerns how accurately the scores derived from the telematics data predict driving performance. The following results are from the independent study carried out by CAS on data provided by Risk Technology. The data was taken from 1,291 drives insured with a leading UK insurer who had had a telematics device installed in their vehicle. The device had been installed and the policies held for between one month and three years. Of these policy holders, 104 had had claims made against them for crashes where they had been assessed as having been at fault An overall score using weighting of the five measures, based on a previous analysis of insurance claim data, provided good prediction of who would be involved in crashes in the independent sample. The following graph shows that very low scorers (less than 10) had a 45% chance of being involved in a crash which was their fault. Those with scores greater than 70 had a less than 6% chance of being at fault for a crash.

Overall, the driving score identified 40% of the drivers who had been at fault for a claim. By chance you would only expect to identify 7.8%. However, this also means that the driving score did not identify 60% of those who had been at fault for a claim. Furthermore, predictions based on driving score resulted in a large number of false positives. That is, it predicted that 20% of policy holders who had not had an at fault claim should have. Why does this happen?

Driver Profiles

There are two main reasons. First, some of the poor scorers had not held their policy very long. They are likely to be involved in a crash in the next few years. Second, different predictors are better at predicting different types of crash. The following graph considers four types of reasons for crashes:

Hazard perception
Basic steering (mostly concerned with reversing errors)
Loss of control
Maintaining a safety envelope
(i.e. driving too close to other road users)

It shows that drivers who are involved in loss of control crashes tend to have very poor braking scores. Those involved in crashes where a poor safety envelope or hard perception were factors have quite poor braking scores. Those involved in crashes where basic steering errors were the main factor tend to have average braking scores. So, braking score is a very good predictor of loss of control crashes but not so good at predicting other sorts of crashes. Different patterns emerge for different driving scores.

Conclusions

The existing scoring methodology and algorithms have been shown to produce significant prediction of insurance claim rates and thus of crash risk. All six of the measures have been shown to have value. Furthermore, the driving performance scores are also significantly correlated with the length of the claim-free period for those who have had claims made against them. The scoring routines are already fit for purpose for an insurer to ‘start its trip’ of establishing its own linkages between driver score and risk. However, it is also recognized that there is scope to enhance the accuracy of prediction using more sophisticated approaches to assessing driver performance and additional telematics indicators. Risk Technology is committed to both improving its scoring methodology and accumulating more data so that these improvements can be validated over time.

PDF Download
  • "The existing scoring methodology and algorithms have been shown to produce significant prediction of insurance claim rates and thus of crash risk."
PDF Download