When I was a little boy, my mother liked to quote the following quatrain (sometimes attributed to the New York wit Dorothy Parker): "See the happy moron, / He doesn't give a damn, / I wish I were a moron, / My God! Perhaps I am!"

I often think of the happy moron when I settle down to the read the International Monetary Fund's (IMF) biannual publication World Economic Outlook. Almost without fail, this publication acknowledges that its previous projections were too optimistic and need to be revised downwards.

In fairness to the IMF's economists, they do give a damn. Indeed, two years ago, the fund published an introspective paper on its forecasts, concluding that they "tended to be consistently over-optimistic in times of country-specific, regional and global recessions".

Quite. In the autumn of 2007, the IMF's forecast for global growth in 2009 was 5 per cent. Because of the financial crisis, the actual figure was minus 0.6 per cent.

The IMF's intrinsic optimism matters because if the organisation is pessimistic about something, it is very likely to be understating the problem. So I sat up when the newly released outlook warned that "a Brexit could do severe regional and global damage by disrupting established trading relationships".

"Heightened uncertainty ahead of the June referendum on European Union membership" was, the fund noted, already a "headwind" to UK growth. A vote to leave the EU "could pose major challenges for both the United Kingdom and the rest of Europe".

The divorce negotiations "would likely be protracted, resulting in an extended period of heightened uncertainty that could weigh heavily on confidence and investment".

You could, I suppose, dismiss all this as overcompensation. Perhaps, after years of Panglossian optimism, the IMF has decided to try Cassandra-style prophecies of doom. And maybe these will be no more accurate than its previous overly sanguine projections. But the IMF is scarcely a lone voice.

The Bank of England's monetary policy committee warned delicately last week that concerns about Brexit "might lead to some softening in growth during the first half of 2016" as businesses postponed capital spending decisions and commercial property transactions.

Goldman Sachs, JP Morgan and Deutsche Bank put it more bluntly: Brexit will hurt the UK economy.

Ah yes, you say, here we go again: Project Fear. The bankers are ganging up with the Old Etonians to put the frighteners on John Bull. Like Boris Johnson, you may feel like saying: "Knickers."

To allay my fears, I decided to read one of the most widely lauded pro-Brexit publications so far, Daniel Hannan's Why Vote Leave. It surprised me very much.

""Nothing will change" in the event of Brexit, according to Hannan. "The short-term effects will be slight." And: "No jobs will be lost." This, he explains, is because "on a technical level - certainly when it comes to trade - most existing structures will almost certainly be left in place".

The paradox at the heart of the Brexiteers' case lies here. First, we are told that nothing much will change. But then we are told a great deal will change, because the EU is an undemocratic superstate that is strangling British economic life with regulations. June 24, says the Mayor of London, will be our Independence Day .

The central claim of Why Vote Leave is that Brexit would liberate Britain from a federalist plot hatched in Brussels, yet leave it free to enjoy "the European free trade zone that stretches from non-EU Iceland to non-EU Turkey".

Though he does not make clear which model he prefers - Iceland, Norway, Switzerland or Guernsey - Hannan is pulling a fast one whichever it is. Access to the single market is limited for non-EU members unless, like Norway, they accept EU regulations.

You cannot have it both ways. Either, as Hannan promises, British bankers, car makers and herbal medicine producers - not to mention "art dealers, cheesemakers, temping agencies, slaughtermen, fund managers, trawlermen, steel workers and cider producers" - will be freed from EU regulations, in which case British exports to the EU will be reduced, or they will not, in which case all a Brexit vote achieves is to remove the UK entirely from the process whereby those regulations are made.

Having assured us that nothing much will change, Hannan offers a cornucopia of post-Brexit benefits: New trade agreements with other non-EU economies that are growing faster than the EU; cheaper food, gas and electricity; and emancipation from the crushing burden of our net contribution to the EU budget: £11 billion (S$21.2 billion) a year. He makes not the slightest attempt to estimate the value of the first two categories. And we should use the correct figure for the third, which - according to the Institute of Fiscal Studies - is closer to £8 billion. That is approximately 0.4 per cent of gross domestic product.

On the other side of the ledger, however, are some distinctly larger numbers. Let's leave aside the trade question, since the size of the hit to British exports - nearly half of which go to the EU - would depend on whether we opted to be Norway (accepting EU regulations) or not. Focus instead on capital flows.

A very awkward fact for the Brexit campaign is that in the last quarter of 2015, Britain's current-account deficit hit a record 7 per cent of GDP. That needs to be financed. But the risk of Brexit is already acting like a flashing red light to foreign investors.

According to the Centre for Economic Performance at the London School of Economics, leaving the EU could reduce foreign direct investment in the UK by 22 per cent and real income by 3.4 per cent.

Chancellor of the Exchequer George Osborne warned last week that Brexit could raise mortgage rates. That's possible, though probably not if we suffer a recession as a result of cancelled projects and capital flight. The most immediate effect of falling foreign investment would, of course, be on the sterling, already down 12 per cent against the euro since November. Could it fall further? You bet. Investment bank Nomura estimates sterling could depreciate by a further 10 per cent to 15 per cent after a vote to leave.

Project Fear? No. This is the consensus of the financial world, from the IMF down.

In recent years, a lot of national referendums on EU issues have produced negative results. But these votes were on treaty modifications or membership of the monetary union. No country has ever voted on whether or not to leave the EU. Maybe the public is just reckless enough to believe Mr Johnson when he says there is a "glorious alternative" to EU membership. But people need to understand the likely costs. The Brexiteers are just like the Scottish nationalists two years ago who wildly underestimated the costs of Scotland's independence.

Perhaps, as the old poem says, I am the one who is the moron. But I do give a damn about this country's economic future. And when I see the risks of Brexit being glossed over in ways that would disgrace an undergraduate essay, I feel anything but happy.

THE SUNDAY TIMES, LONDON

•Niall Ferguson is Laurence A. Tisch professor of history at Harvard and a senior fellow of the Hoover Institution, Stanford.