Host Gene Tunny chats with Dr. Nicholas Gruen about economic forecasting and what recommendations former US Fed Chair Ben Bernanke could make in his current review of forecasting at the Bank of England. Nicholas, the CEO of Lateral Economics, discusses the shortcomings of economic forecasting and shares his insights into how it can be improved. The conversation was inspired by Nicholas’s article in the Financial Times titled “How to Improve Economic Forecasting.” The episode is split into two parts, with the second part focusing on the feedback Nicholas received on his article.
Please get in touch with any questions, comments and suggestions by emailing us at contact@economicsexplored.com or sending a voice message via https://www.speakpipe.com/economicsexplored.
You can listen to the episode via the embedded player below or via podcasting apps including Google Podcasts, Apple Podcasts and Spotify.
About this episode’s guest: Nicholas Gruen
Described by the Financial Times’ Chief Economic Writer Martin Wolf as “a brilliant man who deserves to be better known”, and by former Finance Minister Lindsay Tanner as “Australia’s foremost public intellectual”, Dr Nicholas Gruen is a policy economist, entrepreneur and commentator on our economy, society and innovation.
What’s covered in EP207
- [00:02:13] Ben Bernanke’s review of economic forecasting at the Bank of England.
- [00:05:23] Hedgehogs and foxes.
- [00:09:36] Long-term issues with economic forecasting.
- [00:13:18] Improving economic forecasting techniques.
- [00:19:29] Forecasting accuracy.
- [00:24:30] Open sourcing economic forecasting.
- [00:26:29] Developing a forecasting market.
- [00:34:21] Tetlockian forecasting tournaments.
- [00:48:37] Wind in the Willows author Kenneth Grahame at the Bank of England.
Links relevant to the conversation
Video versions of the conversations featured in this episode on Nicholas’s YouTube channel:
Information on the Bank of England’s Citizens’ Panels/Forums:
https://www.bankofengland.co.uk/about/get-involved/citizens-panels
Mandarin column in which Nicholas declares former Bank of England Chief Economist Andy Haldane was “my favourite public servant in all the world”:
https://www.themandarin.com.au/87423-now-time-complacency-rba-vs-bank-england-edition/
Transcript: How Ben Bernanke can bring Superforecasting to the Bank of England w/ Nicholas Gruen – EP207
N.B. This is a lightly edited version of a transcript originally created using the AI application otter.ai. It may not be 100 percent accurate, but should be pretty close. If you’d like to quote from it, please check the quoted segment in the recording.
Gene Tunny 00:06
Welcome to the Economics Explored podcast, a frank and fearless exploration of important economic issues. I’m your host Gene Tunny. I’m a professional economist and former Australian Treasury official. The aim of this show is to help you better understand the big economic issues affecting all our lives. We do this by considering the theory evidence and by hearing a wide range of views. I’m delighted that you can join me for this episode, please check out the show notes for relevant information. Now on to the show. Hello, thanks for tuning in to the show. In this episode, I chat with Dr. Nicholas Gruen about economic forecasting. Nicholas is CEO of lateral economics. He’s been described by the Financial Times as Chief Economic writer Martin Wolf as a brilliant man who deserves to be better known, and by former Australian finance minister Lindsay Tanner, as Australia’s foremost public intellectual. This conversation was inspired by an article that Nicholas had published in late August in the Financial Times How to improve economic forecasting. The FTS one line summary of the article was myopia and groupthink mean this science is not as evolved as it could be. This episode is in two parts. The first was recorded prior to Nicholas’s article coming out, and in the second part, we reconvened to go over some of the feedback that he received on the article. The video version of the first part is available on Nicholas’s YouTube channel. I’ll include links in the show notes to the YouTube channel, and to material mentioned in the episode. Okay, let’s get into the conversation. I hope you enjoy my conversation with Nicholas Gruen. Nicolas, good to be catching up with you again on economic forecasting. Likewise, so Nicholas, last month, the Bank of England announced that Ben Bernanke, so the former chair of the Federal Reserve in the US, he is to lead a review into forecasting at the Bank of England. So the the court of the Bank of England’s pleased to announce Dr. Ben Bernanke has agreed to lead a review of the bank’s forecasting and related processes during times of significant uncertainty, or we’ve had plenty of those. And he’ll be supported by the bank’s Independent Evaluation Office. Now, Nicholas, you’ve had some thoughts on what Ben Bernanke could offer to the Bank of England regarding forecasting, haven’t you? So would you be able to give us an overview of what those thoughts are, please?
Nicholas Gruen 02:44
Sure. So their thoughts? I’m not terribly hopeful. And that’s an amazing thing to say about Ben Bernanke. I regard Ben Bernanke happens to have a Nobel Prize on his shelf. Ah, you’ll notice that I don’t. And I also think he’s a great guy. You know, he’s a very sensible, practical economist with a lot of understanding of empirical economics and happened to be a one of the world’s experts on the Great Depression at the time when boy, did we need an expert on the Great Depression in the Fed. So that’s all great. I fear that Ben Bernanke, like a really scandalously large proportion of economists are so caught up in their own discipline that they haven’t noticed what has happened in adjacent areas. And this is a little bit like, what’s been going on is something quite like what Daniel Kahneman and Danny and a must for Seversky. If I got that, right, we’re cooking up with behavioural economics. It’s happened a little since then. But a guy that many people will have heard of Philip Tetlock, he got tenure in about 1982 Or three. And he decided that he would now engage in a long term project that he always wanted to engage in, but you can’t if you don’t have tenure, because you get sacked before you could get a publication if there’s so long range thing. And what he wanted to measure was do geo political experts. You can call Tom Friedman. He certainly poses as a geopolitical expert, The New York Times columnist, but also intelligence analysts, academics, international relations academics. If you ask them to forecast events, do they add value? Do they the fact that it’s quite clear they know more than your average bear? Does that translate into actually having actionable better capacity to say what’s going to happen? And the answer was on average and barely. And then he divided that up into experts that did add something. And they didn’t add that much, and experts that actually were worse than ranked, or worse than a naive prediction, and he divided them up into hedgehogs and foxes, hedgehogs no one big thing. And that means that their forecasts are worse than yours or mine, Gene, because we’re just trying to doing our best, whereas the hedgehog will have one big thing, you’ll be anti communist or pro communist, or this or that. And that banks, their forecasts worse than a fox, I think of someone like the economist, John Maynard Keynes, or Paul Krugman, as a fox, someone who knows many things and is trying to balance all those things, and to work out how much this matters and how much that matters, and how much do I know and so on. Now, that’s pretty striking. But it doesn’t tell us exactly what to do. But there is one thing that the study showed us. And it didn’t, we didn’t actually need the study to show us. But it gives us a very concrete illustration of a problem, which is, and this goes on in economics, which is that if you don’t issue your forecasts in a form, that can be back tested, that we can revisit and say did was that a good forecast or a bad forecast? And how did it compare with your peers? You’re basically, you know, it’s a bit like fortune telling. And to do that. What Tetlock did was he forced analysts to say precisely what they were predicting would happen, or in fact, he would specify something like, my Mikhail Gorbachev will continue to be the secretary of the general Committee of the Communist Party, whatever it was called, then, by the end of 1988. What are the chances and then you would have to say, I think the chances are 88% or 23%? Not probably, which means somewhere between 51% and 100%. And not unlikely. And not you can’t rule this out the sort of things you read in a newspaper column. Now we need to do that with economic forecasts.
Gene Tunny 07:31
Yeah, yeah. So just for background, so Philip Tetlock is a Canadian American Political science professor at University of Pennsylvania. And yeah, he wrote that book, super forecasting, or super forecasters. I’m
Nicholas Gruen 07:46
just gonna get on to the talk about that’s the book for the people who can watch not the people who are listening. I’m holding it up to the microphone. Thank you.
Gene Tunny 07:54
Yeah, absolutely. And so he was looking at, you mentioned geopolitical forecasts. But we’re interested in economic forecasts. Now, we know and I guess the general public knows that economic forecasts have been had. there been some notable failures and Amin in Australia that they go way back. I mean, always remember the I mean, I guess I was young at the time was in high school with the Treasury. And was forecasting the soft landing during was it the 9091 recession? Yeah. And it was the worst recession since
Nicholas Gruen 08:30
then, you know, the problems. Yeah. And there are other notable examples. More recently, we’ve been expecting wages to pick up and abroad for about, well over a decade, it just goes on. And, and to their credit, the Treasury and the reserve, published these graphs, I might see if I can put one in the show notes or on screen, or the editor can put one on screen, where you see wage growth gradually trending down with every year, the forecast is to come back to the long term at what was the long term trend average, it’s no longer the long term trend average.
Gene Tunny 09:08
Yeah. And there are some charts like that in the latest intergenerational report that the Treasury has put out, Jim Chalmers launched today, which showed just how bad those long run projections have been. So you know, it’s a it’s a problem, both in the short term and the long term. With economics. Yes. So I suppose yeah, be good to sort of to diagnose I mean, what are the what’s the actual issue and the problem is that the the economy is fundamentally difficult to forecast but
Nicholas Gruen 09:41
no, but I mean, we’re not even trying so to try, we would nail economic forecast down to something that can be properly back tested so I we have a forecast. You may know what the Treasury’s forecast is for wages or growth. Next year, I don’t you just give us a number. Even if you don’t know, the sort of thing you think it should be around what for wages, wages or for growth it all for economic growth,
Gene Tunny 10:12
it’s probably around 2%, or two and a half percent or so and a
Nicholas Gruen 10:16
half, okay, two and a half percent. So first problem is that if the forecast is 2.5%, and it comes in at 2.62%, is that a success? Or is that a failure? So because 2.5%, we call it a point forecast, and the chances that it comes in exactly at that number are infinitesimally small, I just have to add decimal points. And eventually, it won’t won’t be 2.500000. It will be it will fall on one side or the other of 2.5. So we need if, if we’re going to back test, a forecast, we need a forecast that we can declare a success or a failure. And the next thing we need is we need the forecast to tell us how confident they are that it’s got that that event will happen. And that happens to be exactly how weather forecasters forecast. They give us an event it will rain which I’m sure has a media or logical definition of you know more than this amount of precipitation in 24 hours or in in an hour. It will rain and it will rain with this degree of probability. Now what’s beautiful about that is Daniel Kahneman says that there are places where he said this I think he’s a no doubt he’s been more circumspect in other places, but I’ve heard him say, all professions are overconfident? Well, weather forecasters are not overconfident. Because the confidence with which they express themselves becomes part of the metric by which we judge them. And so they make a point of being exactly the right degree of confidence. So I think of weather forecasting as one of the few Socratic areas of domain expertise, because it knows what it knows. And it knows the limits of that knowledge. So that’s what we need to start to try to do with economists. And I think it was you who sent me this thing in the last six months where some of the techniques that Philip Tetlock has perfected has developed, have started to show dividends in economic forecasting. Now, one thing we haven’t explained yet is that that in that book, super forecasting, Philip Tetlock took the ideas with which he demonstrated how little value was added, and how some types of people added more value than others. And he asked himself the question, could we identify the very best to the people who consistently add the most value? Can we understand more about how they do that? Could we get them together and get them to help each other? And the answer is that using these simple and common sensical techniques, you can actually start to get a lot better. Certainly, geopolitical forecasting. And now there’s some evidence that we may be able to get better at economic forecasting.
Gene Tunny 13:32
Right? So with weather forecasting, so in your you’ve been working on a, an article on this, and you’ve identified that weather forecasts are much better than they were 30 years ago. Yeah. Now, that’s because of an infant. My understanding is that’s because of the ingestion of so much new data. And I mean, we’ve seen with that integrated marine observing system, for example, the imass organisation that we’ve done some work for that there’s a whole bunch of data that comes from the ocean, and that helps with weather forecasts. They’ve got huge numerical models and their physical processes involved that they can actually model with economics is a lot, a lot more challenging. So yeah, weather I guess, it is embarrassing. How economic forecasting hasn’t hasn’t improved. And I suppose that does suggest we need to, we need to adopt a different approach is not necessarily going to be we’re not necessarily going to improve our forecast by building more complicated models or bringing in more data. Perhaps we do need to adopt a new approach along the lines of this super forecasting methodology. And you mentioned, yep, there was that evidence about how they’re forecasting the Fed rate decisions much more accurately than others their super forecasting approach. So I guess you are starting to unpack it. What do you see as the main elements of This super forecasting approach, Nicolas.
Nicholas Gruen 15:02
So one of the things that that I think is quite interesting and useful is that like Daniel Kahneman, who was the last person who really, I won’t say revolutionise because it’s not true, but he really he started a whole new way of thinking about things within economics and managed to get himself a Nobel Prize for his trouble. And he’s a psychologist. And so it was Philip Tetlock and Philip Tetlock is drawing our attention to something that’s incredibly important. But because it lies outside of economics, economists just ignore it. And what he’s saying is that if you want to be a good forecaster, you must forecast in a particular way, I’ll say you must have a certain kind of psychology. Now. In fact, in philosophy, there is a term for this, I don’t much fancy it, but the term is Virtue Epistemology. That is if you want to, if you want to be good at knowing if you want to be a good scientist, if you want to be good at mastering a domain and being useful to other people by not being overconfident. By actually knowing how much you know and making it count. Then you have to exhibit virtues, you have to exhibit actual virtues, you have to have the courage of your convictions, you have to have the humility to know when other people or events might be, make it time for you to revise your opinion. Is this reminding you of lots of economists? You’ve talked to Jim? And perhaps not so so the list that I put in this op ed that I’ve written for the Financial Times and may have been published by the time you people get to listen to this conversation? What qualities does he see in Super forecasters, as well as mastering the mesh necessary formal techniques, which we economists are very strong on. They’re open minded, careful, curious, and so critical. away like Socrates, of how little they know, they’re constantly seeking to learn from unfolding events, and from respected colleagues. So that’s how you forecast I would argue, that is how you do anything that is expert. And there’s a really important thing here. Because even if we can’t improve forecasting much, and one thing I do want to throw in, parenthetically on that question, is that when economists make for when a central bank or a treasury makes forecasts, this is a forecast of how certain economic aggregates are going to move that they plan to try to manipulate on on the way through. So it’s a very, it’s a very different kind of forecast, the, the forecasters of the weather don’t say, well, it’s going to be a 30% chance of rain on Tuesday, and we’re going to be trying to make it a 30% chance of rain on or we’re going to be making trying to make it a 20% chance of rain. So so it’s it’s a lot more complicated. But one of the things that are super forecaster might do person have that kind of temperament might do is they might say, well, our point forecasts much used to us. And the answer is I don’t think they I mean, quite apart from the fact that we can’t back test them. I think the most important thing I want to know as a business person doing planning of for something or as an employee, and I’m thinking should I buy a house or buy an investment property or whatever? Seen, I think the most important metric I want the most important thing I want forecast is what is the chance of a recession in the next six months or 12 months or two years? So I think we should be trying to forecast a lot more along those lines. Now there’s a problem and that is that well, firstly, let’s talk about the problem of forecasting at the moment. Because economists forecasts are not probabilistic because we don’t test an economist according to they don’t issue those forecasts like there is a 40% chance of recession or whatever. Almost all the time, even when a recession is more likely than most other times, it’s still unlikely that there will be a recession. And so now what we’ve got is we’ve got all the forecasters in the same situation as 40 tippers, which is I might want to say that the backmarker What do you call it the last of the non favourite in a horse race or a football am, I might want to say that I think the favourite has got an unusually large chance of losing. But I still think it’s more than 50%. So if people are just saying, How many times did you tip the right answer, then we’re not going hunting for who knows that this is the who’s got some extra information, which is that for some reason or other some some particular players not inform or something rather, that there’s a lower chance of the favourite winning than usual, no one has an incentive to do that if we’re going to give a prize out to the person at the end of the year, who tipped more winners than anyone else. And that’s real. And that’s what happens in economics. So of the last 18 recessions, economists pick, tipped about one or two of them. And if you’re competing with other economists, with how often you got it right or wrong, that’s actually quite a rational strategy. So what we need is, we need to find a way for economists to put their hand up and say, I think the chance of recession have gone from, let’s say, 10% per year or something like that, maybe a bit more, I think to the next year, it’s 35%, or whatever, and then at least you get an effective, you know, a number.
Gene Tunny 21:24
Right. So is this what Ben Bernanke should be recommending he should be recommending that the Bank of England provides percentage estimates of regarding its forecast, so how confident it is? I mean, to an extent it does that, I think, doesn’t it? It has Fein charts. It has fan
Nicholas Gruen 21:41
charts, it has fan charts. And I think, yeah, once you try to operationalize this in economics, you end up with a lot of fan charts. Now fan charts, we may or may be able to show those on the screen. And in the show notes, fan charts show you the point forecast through time, and then they say this, the 70% confidence interval is this fat. And the 90% confidence interval is this fat. In other words, if you want to know what were the the range within which we’re 90% Sure, that’s the range. Now the problem is that range isn’t helpful doing because the 90% range usually takes you from somewhat one of the most savage recessions you can possibly imagine through to boom conditions. So we do need to think about that. But what really, I think that there’s a few things here. One of the things is that we need to get, this is a good way to get different teams and different forecasters to compete with each other. It’s a good way to compare forecasters, so that you’re constantly getting feedback on who’s good and who’s not. The other thing that I think it does, well, it also enables us to surface you can have a different series, which is not in any central bank or Treasury that I know of, which is the chances of recession, you can have that series and you can have people trying to forecast that. Now there’s a further problem. And the problem is that we get feedback on what growth was every time we forecast it, because we can’t we get a growth number. We don’t get feedback on what the question was there a session will accept that the answer is no. It only varies once a decade or so. That’s a really big problem. Because if you want to ask who’s the best person at forecasting recessions, then you’ve got to wait 20 or 30 years to even start to short sort the sheep from the goats. Yeah. So Philip Tetlock has actually been working on this on a problem. It’s not in economics. It’s in his his the area that he manages to get the most funding from, which is in intelligence organisations and so on. But what he’s trying to ask is, can we leverage the credibility of forecasters of things we do get a lot of feedback from for these other areas where we get less feedback? And I think the answer is yes, we should be able to do that. And we must be able to do that in some areas, and maybe not in others. And then we don’t know about this area, but that’s the sort of thing that we should
Gene Tunny 24:28
be exploring. Okay, so for economics, so just to summarise, are you arguing for open sourcing for coal, that’s
Nicholas Gruen 24:36
a separate thing. That was what I was going to get to, which is that so what I want to see is that this is one area that given that we’ve outsourced all kinds of things in government that we shouldn’t have outsourced. Maybe we could outsource some of the things we should and we this is the sort of thing that we can outsource on I don’t even mean outsource we can’t what we should do the best Bank of England, the Reserve Bank of Australia can get with the programme and the programme is the smartest person is always outside the room. And in some areas, you can, in some sense, bring them in. And in other areas you can’t. But in the area of forecasting, you can and you can hold a Tetlock like forecasting competition, you can say, we’re trying to get forecast for this, and this and this and chances of recession in six months, one year and two years, and then everyone can participate. Now, the world or certainly the markets and the people in the different national countries, they want to know, what’s the reserve, what’s the central bank forecast, so that central bank has its own, I think that central bank should have its own teams, team or teams in these forecasts. But they should separate out the teams from the bank itself, and the bank should observe the forecast should observe the forecasting competition. And from that forecasting competition, say what it thinks is its best forecasts and those become signed with the imprimatur of the central bank. They might be produced by the central bank team, or one of them, they might be produced by somebody completely outside, they might be produced by some kind of hybrid. And all of this is visible to everyone. And so we’re starting to develop a market in which we can start to see who’s really good at this. And some people are going to surprise us on both the upside and the downside, by the way. So that’s what I’m suggesting.
Gene Tunny 26:46
Yeah, I mean, what, what I’d like to understand is, to what extent will it be teams, interdisciplinary teams of economists, and then some other non economists, may be busy people who are expert in business or maybe not even expert in business people who are just good forecasters. And when I was chatting with Warren hatch from good judgement, this is a organisation he set up with Philip Tetlock, he was telling me that it’s people with good pattern recognition skills, and then be in any discipline and people who are cognitively flexible, or they’re there. As you were saying before they actually they’re not caught up with their particular theory. They’re actually yeah, they’re evaluating everything. Yeah, that’s right. That’s
Nicholas Gruen 27:33
right. So the answer is, we don’t have to know the answer to that. But we Yes, you would expect that the teams that are going to perform best will be hybrid teams will have economists Well, technically excellent economists in them. They’ll have people who look at other kinds of things. And there will certainly be some surprises. And some people who’ve always had a fascination with, you know, certain kinds of things which turn out to be relevant to how you forecast. So that’s where I would expect it to, to end up. But maybe it’ll just be economic experts. If they win the if they win the competitions. All this Tetlock stuff will have proven itself to be relevant for economics, but both common sense and the evidence suggests that that that’s not the way it will turn out. And there aren’t that many areas where at the centre of government, you can improve performance and improve. And through that improve economic performance someone. This is this is one of those billion dollar bills on the pavement that we find ourselves talking about from time to time, Gene,
Gene Tunny 28:46
absolutely. Yeah. And I misremembered. What Treasury’s forecast is 2023 24 GDP forecasts for Australia at one and a half percent. So not Oh, is there any
Nicholas Gruen 29:01
memorable number or perhaps it is memorable, but not in a good
Gene Tunny 29:04
way? Just so many numbers out there? Harada? Yeah, exactly. Exactly. I feel sorry for these politicians, they get put on the spot about these different numbers from toe to toe? Oh, absolutely. Yeah, absolutely. fully on board with that suggestion. At the very least it’d be a good trial, a good pilot. Exactly that out, see how it will works?
Nicholas Gruen 29:23
Well, I’ll just say one other thing, which is that this is again, what we’re talking about here is convening power, not executive power. So anyone can run this. The Business Council could run this. It’s not it won’t be cheap, but it’s not very expensive. Having worked at the Business Council, I can tell you, their budget easily would easily accommodate this. You could do it for a few $100,000. Anyone can do this. So it’s it’s kind of extraordinary and pretty outrageous that we’ve really known this, that there are benefits here. We can do this better. And it just gets ignored again. And again, it got ignored in the review of the RBA that we had here. It’s pretty terrible that we’re not looking around and trying to grab hold of things that are in the ether, that it’s starting to work, and that we can benefit from.
Gene Tunny 30:21
Yeah, I suppose there’s a public benefit to it. It’s not necessarily in the interest of the people in the Treasury or the Reserve Bank or the Bank of England or their ministers. I think that’s one of the the issues.
Nicholas Gruen 30:32
Yes, but economists are pretty impatient with policy makers who don’t do the right thing, but that the economists have to figure this out themselves. And I would, I would have thought that it’s Well, time for this to be standard economic advice, and it’s very, very left field and economic advice at this stage.
Gene Tunny 30:55
Okay, we’ll see how your Financial Times I bet is received?
31:00
Well, let’s see. Let’s see what Ben says. Very good, he might be giving you a call. Let’s hope.
Gene Tunny 31:09
Okay, we’ll take a short break here for a word from our sponsor.
Female speaker 31:15
If you need to crunch the numbers, then get in touch with Adept Economics. We offer you Frank and fearless economic analysis and advice. We can help you with funding submissions, cost benefit analysis, studies, and economic modelling of all sorts. Our head office is in Brisbane, Australia, but we work all over the world. You can get in touch via our website, http://www.adepteconomics.com.au. We’d love to hear from you.
Gene Tunny 31:44
Now back to the show. So Nicholas, it’s been a few weeks now since your article was published in the Financial Times. So your article, how do we improve economic forecasting? And we chatted about that, in the previous conversation, the in the lead up to that coming out. So your ideas about how the Bank of England and other central banks or treasuries or finance ministers can improve economic forecasting? So it’s been a few weeks and says come out, you’ve had a bit of feedback. Yeah. How would you describe the reaction to your article?
Nicholas Gruen 32:22
I think it’s been the best reaction. I’ve published three pieces in the Financial Times of this kind, which is a sort of, hey, why don’t we do this? It’s reasonably out there kind of proposal and my judgement of the comments, and you’ve looked at them slightly more carefully than me that I looked at them, you know, in the first 24 or 48 hours, I thought they were more positive, and more constructive than most than is mostly the case in comment sections. It’s a pretty sad state of affairs. And nevertheless, the case that even in a really high quality newspaper, like the Financial Times, a lot of the people they’re not super ignorant, and, and just just totally dumb, but what they do is they sort of come on and they make a point. And the point is a perfectly okay, point, one of the points for instances, well, weather forecasting, which was full of praise for is different to economic forecasting, because the weather doesn’t decide to change its mind when it sees a forecast. And human beings do. It’s a very good point. It doesn’t completely obliterate all the points I was making. So if someone wants to come on and say that, that’s fine, I know that, but they’re not really participating in the spirit of things. Another person who wrote a letter to the Financial Times I think his name’s Tim Connington, or Contin, you might know his name. He said that really what mattered was having models that have proper allowance for monetary policy in them well, I’m not against having models that have the proper allowance for monetary policy in them, but it doesn’t really address the point. And then, and then there was some really quite good criticisms. The other thing was really good was that I was approached by a number of people, some of them well, one was a large corporate, which is doing Tetlock in forecasting tournaments internally. That was an interesting exercise. And I’ve been engaged with them. I’ve been they haven’t been paying me or anything, but I’ve been suggesting that they look further afield to the services of people like Warren hatch who you interviewed. On your podcast, he runs a thing called Well, it’s called good judgement. I don’t know whether it’s good judgement Inc. Or, anyway, it’s not the Philip Tetlock project which is run with inside University But it’s an offshoot of it, which is a commercial project. That was interesting. There was another economist, who was really quite pissed off, if I might say this, about the fact that forecasting prowess is not a very strong criterion of promotion inside government agencies that deal with economics include, including government agencies, in which forecasting is a very important matter. And he’s right. And I talked to him about Kaggle. And how Kaggle, the data science forecasting platform that I was involved in, when it started up, has changed the market to a substantial extent because people want data scientists who actually perform well. And you can see whether they performed well or not on Kaggle. And then another person who contacted me was actually from the Bank of England. Now, I’ve not had that experience in Australia, where someone from inside government you publish something. I mean, it wasn’t directly critical of the bank, I suppose you could say it was in a way. Anyway, he engaged me. And he said, Well, actually, we do, too, a little bit of what you’re suggesting. And it’s true, that the Bank of England, which is about my favourite Central Bank, I think they’ve done better than any other central bank in terms of their thinking. Not it turns out in terms of all the judgments about the about inflation, and so on, because we do we require a degree of clairvoyance for that. And they’ve had a recent spate of arguably bad luck in terms of working out the future. But he pointed out that the Bank of England does have a very, very simple in the form of seeking feedback from the community. It asks people for their own forecasts. Well, that’s a good beginning. And it’s better than any other bank that I know. I thought it was a terrific reaction.
Gene Tunny 37:06
Oh, that’s good. Yeah. Citizens panels, I think they call them so I’ll put a link in the show notes. I thought that was really good. And, and it really is heartening to see how open they are. And you’re right. I mean, I can’t remember anyone from a Australian government agency getting in touch or if they did get in touch, it would be all this has to be confidential, and it wouldn’t be an official email. So I think that’s good about the Bank of England. So that was great to see that. Now, just on some of those points, you raise you mentioned about modelling and that was it one comment that said I Okay, the issue was just the specification of the model. And I think you the way you reacted to that was, was was right. And one of the some of the comments I took out of the ft. Like there was some positive really positive comments in the comment section of the Financial Times. It was one about, ah, this sort of approach could have helped us in the early days of COVID. It could have avoided us from having some of your apocalyptic or Yeah, ridiculous, for ridiculous, and I think there was some criticism of the forecast room from his sage. I think they were sage forecasters. Yeah. That’s right,
Nicholas Gruen 38:20
sage, and was a guy who got himself briefly famous. And then arguably infamous. You put his name in these notes we have in front of us, Ferguson. Yeah, yeah. Yeah. Neil Ferguson. That’s it. And that, and you just had to look into that for a while to see that. The model was an immensely complex model. It wasn’t clear what it was useful for. But it wasn’t useful for quickly trying to understand, you know, ask quick, what if questions, it was an ornery monster of a model that produced a different result every time he ran it, because it was so common. Yeah. Just just not not built to certainly not in that situation. It was not built to help people make quick probabilistic decisions. But because it was a model, and because he was at a university Imperial College, as I recall, I hope correctly, then he had the stamp, you know, you had the brand. And so we spent a fair bit of our time with his model. It was pretty low grade stuff.
Gene Tunny 39:31
And so some of the negative comments or there were some people who are saying, Oh, well, look, you’re not you haven’t taken to account the fact that we’ve made all these advances in economic forecasting, and there are these new techniques and you’re unaware of them. I’m not sure that that’s true. And when I didn’t
Nicholas Gruen 39:47
mention any No, I didn’t mention any. So I mean, I’m sure I’m unaware of some of them, but he had no evidence that I was because what he’s or she is criticising me for is St. totally irrelevant. There is a state of the art of forecasting, the Bank of England or anyone else is either at the forefront or a bit back from the forefront. And the way to get to the forefront is to have a process of integrity, where people who are good at forecasting end up with better reputations than people who are not so good at
Gene Tunny 40:21
forecast. Yeah, yeah, exactly. And the point I would make, like when I read those comments, they were almost as I think they are assuming that it’s the model that gives the forecast that’s published in the Bank of England monetary policy statement or, or in any of these statements from economic agencies, it’s actually a forecast directly from a model. And it almost never is, there’s always an element of judgement, the model is one input into the the actual official forecast. And if you read the bank, the publication’s of the Bank of England, that’s very clear. And so your approach is about taking all of the the evidence out there or different views. I mean, you know, it could be in I think there’s something I was chatting with Warren hatch about, if I remember correctly, Warren was saying that, look, there can be value from having people in teams, like some people, someone has a model than there’s others who are more qualitative. And there are others who are looking at different bits of data you want. You don’t want a variety of approaches, I think and perspectives to get better forecasts.
Nicholas Gruen 41:27
I’d say some, I think that’s absolutely right. But I think you can say something more than that. We exist in a society in which governments and agents and organisations are performing for our entertainment, if I can put it that way, at least under the guise of the media, they’re doing stuff, they’re justifying them stuff. They’re got comms people coming out, saying, This is what we’re doing, and they’re putting over a plausible story. And then you get pundits, I would say, like us, except I try not to do this. But almost all pundits and almost all Twitter pundits, almost all instant experts, they come out. And they say what, really what you should do is x or y. But in fact, what you should do is a very complex and acculturated performance. So it will involve technical understanding and modelling. It will then involve judgments, as you say, but then how do you get the people with the best judgement to make the judgments? Well, we haven’t really solved that problem, we just get the most senior people to make those judgments. So it’s like me saying, I want a good COVID vaccine. And this is the process that we should go through to get the COVID vaccine. What I want is a process that has legitimacy, because I believe that if I looked into what that process was, it would add up it would have integrity. In the words of Charlie Munger, the highest form of civilization is a seamless web of deserved trust. In other words, there isn’t a clear line between the pundit class and what you do. If you’re doing anything difficult building a bridge or dare I say, a nuclear submarine pundits can can’t actually say very much, they can say a few things about what would be really dumb. But there’s so much that goes into this. And the public discussion isn’t had in that kind of way. But that, ultimately, is one of the reasons that I’m such a fan of Philip Tetlock stuff on forecasting and creating forecasting tournaments, because it’s one of the few areas where you can start to build some objective relation between reality. And as poor munchkins working away trying to work out what that reality is, and our social and political institutions have done? Well, the job they’ve done might be the best in history, but when you look at it, it’s not all that great, there are plenty of things wrong with it. So, this is a rare case where there is a better way you can see what it is you can understand its principles and we should really try to implement it and also learn from it how how we could extend that since making reality contacting function.
Gene Tunny 44:31
Yeah, absolutely, fully agree there. So, I mean, one other point I just wanted to make is on that, the forecasting the the whatever the you know, best practice or the in terms of technical forecasting. One of the articles there was, it was linked to in the in the comment section, the Financial Times it was an article that was by a number of forecasting experts and one of them was Jennifer Castle’s, who works with David Hendry. And Henry has been on the show. And if you’re interested in these issues, that would be a good conversation to go back to because David talks a lot about the ways that he tries to get his model based forecast as best as possible. Now, that’s, that can be an input into this a super forecasting approach. It’s not, these things aren’t mutually exclusive. But what he’s doing, he’s trying to build an econometric model that can be an input into the forecasting. For the point I’d like to emphasise is that the forecasts that end up in the reports and then end up influencing budget, so they’re never just the outcome of models, because we know that a model is useful. But you there’s always a judgement involved, you’re always going to be tweaking things to make it because there’ll be things in the model you go hang on that may not be realistic in the current circumstances. Yeah, exactly. Yeah, exactly. Right. Oh, so Nicholas. I just wanted that quick catch up. Because I thought, yeah, that was a great article of yours. And it’s got some excellent feedback. And I think it’s, it’s probably achieved what you wanted to achieve, I imagine.
Nicholas Gruen 46:08
Yeah, absolutely. Even though they told me I only had 650 words, and then they only allowed me 570 words. So my nice paragraphs about what a big fan I was of Andy Haldane, who was no longer at the Bank of England, they were all taken out the likes of fanboy helding while he was a civil servant, was my favourite civil servant in all the world. Very good. Yes.
Gene Tunny 46:36
I’ll put some links in to about Andy Hill died. Did you? Have you written this on your club dropout? Or Nicholas? Your? Um, I’m
Nicholas Gruen 46:44
not sure I have I’ve. Yeah, maybe I should. But But no, I have because I published some articles in the Mandarin, which is an Australian Public Policy Magazine, if you like, which is and they’re always backed up onto my blog, and one compared the Australian Reserve Bank, with the Bank of England and the and particularly the blog notes underground. I think it’s called always good to quote Dostoevsky. I suppose when Greg Clark isn’t quoting, isn’t quoting titles from Hemingway, the Bank of England can be can be paraphrasing Dostoevsky in the name of its blog notes underground, I think it’s called. And it has lots of really interesting think pieces. It’s not very standard academic stuff, although there’s some of that as well. I think it’s a very sad thing that government, certainly independent agent, government agencies around the world don’t do that a great deal more. I may be fondly imagined that Andy was one of the movers and shakers behind that. But certainly he did lead a lot of research showing the costs of too big to fail implicit subsidies for large banks and just did lots of use the, the US the independence of the central bank in a way that was very, very helpful in difficult times during the global financial crisis. And in the years after the financial crisis is people trying to work out what had gone wrong and how to fix things. Yeah, absolutely.
Gene Tunny 48:23
It’s, it’s interesting that Yeah, I agree about the Bank of England, probably being the best central bank certainly has the best museum. I guess there’s that literary connection. Yes. And I only learned about this when I went to the museum, Kenneth Graham work there, the author with the willows. Hmm, yeah, I work there. I mean, I have relatively senior position there in the Bank of England because they’ve got a little display about Kenneth grime in there.
Nicholas Gruen 48:53
I missed it. I missed it. I’m sorry that I missed it. Because I have seen that museum. It’s quite small. It’s just a few artefacts as I recall a room or 2am I
Gene Tunny 49:02
wrong. Yeah, it’s a maybe a few rooms, but there’s that great display where you can lift up a bar of gold, you stick your hand in a glass glass box, and you’re gonna lift up an actual gold bar, which I thought was pretty cool. And you know, they’ve got all the currency. Yeah, he got up to the rank of Secretary in 1908. So I don’t think he was he wasn’t the governor, but he got up to a senior position. Excellent. Very good. Okay, Nicholas, thanks. Again. That was such a it was good to catch up because, yeah, good. always interested in economic forecasting, because we’ve had such a, unfortunately a mixed record of it in Australia and around the world. So it’s, it’s good to talk about a new approach and well done for doing your best to advance one.
Nicholas Gruen 49:50
Thanks very much
Gene Tunny 49:53
rato thanks for listening to this episode of Economics Explored. If you have any questions, comments or suggestions, please get in touch match, I’d love to hear from you. You can send me an email via contact@economicsexplored.com Or a voicemail via SpeakPipe. You can find the link in the show notes. If you’ve enjoyed the show, I’d be grateful if you could tell anyone you think would be interested about it. Word of mouth is one of the main ways that people learn about the show. Finally, if you’re podcasting outlets you then please write a review and leave a rating. Thanks for listening. I hope you can join me again next week.
50:40
Thank you for listening. We hope you enjoyed the episode. For more content like this or to begin your own podcasting journey. Head on over to obsidian-productions.com
Credits
Thanks to Obsidian Productions for mixing the episode and to the show’s sponsor, Gene’s consultancy business www.adepteconomics.com.au. Full transcripts are available a few days after the episode is first published at www.economicsexplored.com. Economics Explored is available via Apple Podcasts, Google Podcast, and other podcasting platforms.
