+44 (0)20 7148 7800
LinkedIn
Twitter
Email
PrintCYK Partner Philip Young considers the requests made of lawyers to offer probabilistic assessments of the outcome of commercial litigation. He points out the potential hazards, and the role that human biases and assumptions play in both making and interpreting these predictions. A version of this article also appears in the New Law Journal.
Nowadays, in modern commercial litigation, practitioners are frequently asked to predict the outcome of a case, issue or application in the form of a mathematical percentage. General Counsel often ask for percentages for their own internal purposes and the growth of the ATE and litigation funding markets has also required lawyers to offer percentages as many insurers and funders insist on a minimum percentage before agreeing terms. For any competent practitioner, offering a mathematical percentage is often an aspect of preparing any opinion that creates anguish, not least because it attempts to compress a very complex scenario (involving complex thinking and reasoning) into the arbitrary simplicity of one number.
In this article, I gently point out some hazards with this modern trend. I argue that the increasing prevalence of assigning percentages is unavoidable but can invite misfortune and risks being misleading. It should not obscure that the best way to convey the core information, namely the essence of the risk being run, is a combination of both language and a percentage, with language playing as much, if not more of, a part than the percentage.
So how do practitioners choose the percentage? If viewed objectively, a bystander – particularly one with a reasonable understanding of mathematics – might assume that the practitioner will draw on his or her knowledge of the case, and of prior cases, and then assign a percentage between 0% and 100% depending on the likelihood of success. Logically, this presupposes that there are cases that can and will fall anywhere in this range and that over time, if enough cases were assigned percentages, you would end up with a very wide range of percentages across numerous opinions ranging throughout the 0% to 100% spectrum. So, whereas one matter might be awarded a 17% prospect, then another might receive 53% and yet another 86% and so on.
In my experience, and also anecdotally having spoken to many other practitioners over the years (both solicitors and barristers), this is far from the reality of how percentages are assigned in commercial litigation. At least among practitioners in London, there seems to be a rough heuristic at play. Lawyers put the matter into a mental “bucket” and the percentage flows from there. The buckets might usefully be categorised as: “hopeless”;[1] “weak”; “toss up”; and “strong”. Whilst different practitioners use different numbers for each “bucket”, very broadly speaking, and admittedly as a generalisation, it seems these attract percentages respectively of: 0-10% (hopeless); 20-30% (weak); 50/50 or sometimes a figure varying between 40% to 60% depending on the practitioner (toss up) and finally 60% to maybe 80% (strong). It is very rare to find an experienced practitioner who would assign any matter, no matter the strength, with a percentage higher than the last range and many practitioners would never go above 70%. It is even rarer to see a percentage that is not a round number or one that ends in a 5, i.e.: 35% or 65%.
What this tells us is that the process of assigning percentages in commercial litigation is a long way from an objective mathematical process that a percentage number might otherwise imply. It is not the frequency or propensity of a phenomenon (i.e. the measurement of the real probability of something occurring). Rather, it is a highly subjective process. The number actually reflects the practitioner’s reasonable expectation and represents a state of knowledge or a quantification of a personal belief with the practitioner doing the best they can drawing on their specific experiences and knowledge. Putting it another way, it is a form of Bayesian probability.
This is a subtlety that is often overlooked. It can be illustrated by an example. A fair die has six sides and so on any roll there is a 1 in 6 chance of any particular side being uppermost. Thus, the odds are 1/6 and that is a property of the die itself. However, imagine the person who has never seen a dice but has been told the outcome of a series of rolls (with no number greater than 6 being rolled) and therefore based on that experience expresses the belief that there is a 1 in 6 chance of the number 2 being uppermost. The propensity of something and someone’s belief about that something are different things.
That a percentage for the merits of litigation is a Bayesian probability is unsurprising. Whereas there are areas of civil litigation that are conducted in such bulk that there is a reasonably large sample size from which meaningful statistical information could be drawn (such as slipping and tripping cases and other high volume PI cases), such as to lead to probabilities that can be given based on statistical propensity, large scale modern commercial litigation is so substantial that a practitioner may only act on a few cases a year and no two cases are alike. The practitioner therefore has a limited sample pool from which to assess likelihood of success or failure and therefore has to rely more on judgment than on statistical analysis.
Moreover, humans have a stronger belief that they can predict the future accurately than scientific evidence supports. Connected to this is plenty of evidence from behavioural science that humans think the future is more accurately predictable than it actually is. This is not least because humans suffer from hindsight bias (our tendency to remember the past as being more predictable than it actually was), optimism bias (which is self-explanatory) and salience bias (people’s tendency to focus on things – especially recent things – that seem more important while downplaying other things that, viewed objectively, are actually more pertinent). Humans also have a tendency to overestimate the likelihood that the future will resemble the past.
Furthermore, the heuristic buckets themselves are inevitably problematic for a host of other reasons. Whereas the practitioner broadly understands what they intend to convey, that is not to say the recipient necessarily does. They also come accompanied by a host of biases and assumptions, some hidden and some unspoken. Moreover, whilst the competent practitioner is attempting to explain chance, the recipient may have fallen into the fallacy of believing the practitioner has in fact mathematically predicted the future.
It is worthwhile unpacking this. First, say a practitioner assigns a percentage of 65% to a case. That sounds like a high number and rather deceptively sounds like the case is very likely to prevail. Consequently, many clients – and litigation funders – would regard that as a strong assurance of success. But is it? It is the same as saying the case has a 35% chance of failure. What does that mean where that case will only be tried for the client once and the client only gets one shot at it? One way to regard it is to say that it is no different than saying that if the same case was tried three times, on slightly less than two occasions it would likely win and on slightly more than one occasion it would likely fail. That is of scant comfort to a client who then takes that case to trial and then it fails, which on a 35% probability is a material risk. Indeed, consider that there may be three cases in the High Court at that time with the same predicted probability. If so, one of those claimants is almost certainly going to fail. You might be surprised to read that but the probability of that occurring is 96% (1 – (0.35 x 0.35 x 0.35)), which feels counter-intuitively too high because of the difficulties humans have in innately understanding probability. A yet further way of seeing it is that the probability of all three cases succeeding is only 27% (0.65 x 0.65 x 0.65) and so the inverse 63% is the entire range of possibilities of one or more of those three cases failing, in any combination.
This illustrates a deeper truth: by and large people are bad at understanding probability not least because of heuristics such as representativeness, availability, anchoring and confirmation bias. Humans are also very bad at understanding randomness and, as I have already noted above, suffer from their own inbuilt cognitive biases. Considerable research in the behavioural sciences in recent years shows that people routinely misjudge by minimalisation the likelihood of unlikely events (so-called “Black Swans”) occurring. We can all point to a number of such events happening in recent memory – the recent pandemic being a glaring one.
Because of this, it can help to contrast the percentages of success and failure with the percentage odds of ordinary life occurrences, to put the likelihoods into a more easily digestible context. A 35% chance of failure may not sound too likely weighed against a 65% chance of success. Yet most people have had the disagreeable experience of food poisoning from time to time and so would regard it as a not un-common experience. Statistically, about 1 in 6 Americans get ill from food poisoning each year, i.e. about 16%. The chance of losing the litigation is therefore twice the chance of getting food poisoning in a year. Suddenly, it doesn’t sound quite so unlikely. Indeed, returning to the example of three cases going through the Courts at the same time, the chance of all three cases losing is about 1 in 20, i.e. the inverse of the 96% odds referred to above. 1 in 20 odds are a lot more likely to occur in life than people may instinctively realise.
Furthermore, prospect theory, developed by Daniel Kahneman and Amos Tversky, shows that humans are more sensitive to losses than they are to gains and loss aversion is greater as the potential loss increases in size. Thus, the framing of a scenario often has an impact on someone’s decision making when mathematically it should not. This, too, can be illustrated by examples. Put yourselves in the shoes of a prospective client. Imagine that the client has a claim for £6m, its legal costs would be £1.5m (of which £500k will be irrecoverable) and its adverse cost risk amounts similarly to £1.5m. If you were the client, what would your instinctive reaction be if you are told that you have a 35% chance of losing £3m but you might win £5.5m? You probably recoil a little from that. Now try, as far as possible, to forget that. Do you have a different instinctive reaction if you are told you have a 65% chance of winning a net £5.5m? That probably feels more palatable. Now weigh it all up together and in the round – is your reaction a little bit different again?[2]
It bears saying that the Court is human too and at times is seduced by the artificial illusion of mathematical certainty as well as subject to the cognitive biases referred to above. In Levicom International Holdings BV & Anor v. Linklaters [2010] EWCA Civ 494, referring to the practitioner’s opinion that the prospects of success were “in the region of, but not less than 70 per cent” Stanley Burnton LJ (disagreeing with the Judge below) said: “I do not think it unrealistic to consider that lawyers do not advise that the prospects of success are that high unless they are very confident indeed.” and “I am not surprised that after this advice [the client] thought that their case on liability was a “home run””. Pausing here, the formulation of “not less than” 70% is a curious one and has the hall marks of a practitioner under considerable client pressure to deliver some positive news. That said, with all due respect to the learned Lord Justice, a 30% chance of losing is not a “home run” by any stretch of the imagination.
Next, any percentage is at best only a statement of opinion that is applicable at that time based on what is then known. This is often overlooked. Commercial litigation is very often a rollercoaster and no matter how rosy, or bleak, the future may look at one point in time, it may look very different at some later point. Cases that originally appear comparatively strong can be holed beneath the water-line by a new development and, by contrast, very weak cases can suddenly and unexpectedly be materially strengthened. Indeed, experienced practitioners know how often a case takes on a very different complexion in the run up to a trial.[3] Some experienced funders say regardless of what percentage view was taken earlier in a case, when a trial begins they always regard the case at that point as a 50/50 gamble.
It is not difficult to see what might be called the “judge effect” in action. You only have to look at case law to see the number of times cases are successfully appealed or appeals where the bench is split in order to see the frequency with which Judges disagree with each other. For example, UK Supreme Court statistics for 2020-21 show 22 out of 48 cases that went through to judgment were dismissed meaning that in 54% of cases the judgment below was overturned in some particular. Another example is that in 2019-20 the Court of Appeal (Criminal Division) allowed 39% of conviction appeals and a remarkable 60% of sentencing appeals. A yet further example is that across the Commercial Court, Chancery and the TCC, appeals to the Court of Appeal are upheld in some respect about 40% of the time.
Lastly, any percentage comes complete with a host of assumptions, some unspoken, some hidden, and some of which I doubt even practitioners stop to interrogate as those assumptions are unconscious.
Some assumptions are fairly obvious. For example, that important witnesses will be believed (and not die or become very ill before trial), that damaging “smoking gun” documents will not unexpectedly appear, etc. There are others, however, that are much deeper. For example, rarely will a practitioner know the identity of the trial judge much before the trial. Commercial litigation is a human process and that observation applies as much to the Judge as it does to the witnesses, the parties and their legal representatives. Practitioners tend to assume, largely without even thinking about it, that they will receive a hearing dispassionately and uninfluenced by matters extraneous to the case. But, in reality, some Judges have biases in their view of particular areas of law, i.e. what the law should or should not be. Some Judges may have their view of a point skewed by the particular pressures that they are working under (say, for example, if they are under intense time pressure and do not therefore pick up an importance nuance of a point) and even very extraneous factors, such as a particular dislike towards an advocate may colour the view taken of the case.
The process of assigning a percentage therefore involves a very high degree of subjectivity.
Does all of this mean that trying to assign a percentage is misconceived? An ill-fated attempt to place the artificial appearance of mathematical precision on a very messy human process that is inherently highly uncertain? I would not go that far. Percentages do not easily bear the weight that is very often (wrongly) placed on them and they are easy to misunderstand. They do, however, serve a much more limited purpose in giving an indication of the practitioner’s then present belief of the outcome of the case overall, in very broad terms. They also perform a valuable role in forcing the person giving the opinion to think more deeply about the opinion and what it really means (and what the person means to convey overall) than otherwise might be the case. Thus, they may increase the overall accuracy of the prediction (a point made by Philip Tetlock, Professor of Psychology and Management at the Wharton School, in his research on guessing the future). Moreover, as clients, insurers and funders often require percentages then we do, and must, all live with giving them and, as I have said, it is probably a good discipline to do so. All that said, a percentage never should be taken in isolation. Because of the fallibilities of the human mind in understanding probability, language is of equal importance in conveying the perceived merits.
But even here I must strike a note of caution. Very interesting recent research has shown that the meaning of the language practitioners sometimes use is not consistently understood even by the practitioners. Perhaps this is unsurprising because language is gloriously flexible. As Humpty Dumpty said: “When I use a word it means just what I choose it to mean – neither more or less.” It necessarily follows there must be a risk of it being misunderstood by the client.[4] Care and thought in language is as important as care and thought in percentage.
Taking account of this, and bearing in mind the heuristics above, I suggest that if a practitioner describes a case as “bound to fail”, “probably going to fail”, “a toss-up” or “probably going to succeed but nevertheless having a material risk of failing”, those are useful formulations, and may help to reduce the risk of an accompanying percentage being misleading.[5] Finally, if a percentage is being given, it is probably for the best that the issues inherent with it are explained – something that you could achieve by, for example, referring the recipient to this article!
