In the quarter-century since the end of the Cold War, profound grievances, misperceptions and disappointments have often defined the relationship between the United States and Russia. I lived through this turbulence during my years as a diplomat in Moscow, navigating the curious mix of hope and humiliation that I remember so vividly in the Russia of Mr Boris Yeltsin, and the pugnacity and raw ambition of Mr Vladimir Putin's Kremlin. And I lived through it in Washington, serving both Republican and Democratic administrations.
There have been more than enough illusions on both sides. The US has oscillated between visions of an enduring partnership with Moscow and dismissing it as a sulking regional power in terminal decline. Russia has moved between notions of a strategic partnership with the US and a later, deeper desire to upend the current international order, where a dominant US consigns Russia to a subordinate role.
The reality is that our relationship with Russia will remain competitive, and often adversarial, for the foreseeable future. At its core is a fundamental disconnect in outlook and about each other's role in the world.
It is tempting to think that personal rapport can bridge this disconnect and that the art of the deal can unlock a grand bargain. That is a foolish starting point for sensible policy. It would be especially foolish to think that Russia's deeply troubling interference in our election can or should be played down, however inconvenient.
Mr Putin's aggressive election meddling, like his broader foreign policy, has at least two motivating factors. The first is his conviction that the surest path to restoring Russia as a great power comes at the expense of an American-led order. He wants Russia unconstrained by Western values and institutions, free to pursue a sphere of influence.
The second motivating factor is closely connected to the first. The legitimacy of Mr Putin's system of repressive domestic control depends on the existence of external threats. Surfing on high oil prices, he used to be able to bolster his social contract with the Russian people through rising standards of living.
That was clear in the boom town Moscow I knew as the American ambassador a decade ago, full of the promise of a rising middle class and the consumption of an elite convinced that anything worth doing was worth overdoing.
But Mr Putin has lost that card in a world of lower energy prices and Western sanctions, and with a one-dimensional economy in which real reform is trumped by the imperative of political control and the corruption that lubricates it.
The ultimate realist, Mr Putin understands Russia's relative weakness, but regularly demonstrates that declining powers can be at least as disruptive as rising powers. He sees a target-rich environment all around him.
If he can't easily build Russia up, he can take the US down a few pegs, with his characteristic tactical agility and willingness to play rough and take risks. If he can't have a deferential government in Kiev, he can grab Crimea and try to engineer the next best thing, a dysfunctional Ukraine. If he can't abide the risk of regime upheaval in Syria, he can flex Russia's military muscle, emasculate the West, and preserve President Bashar al-Assad atop the rubble of Aleppo. If he can't directly intimidate the European Union, he can accelerate its unravelling by supporting anti-Union nationalists and exploiting the wave of migration spawned in part by his own brutality. Wherever he can, he exposes the seeming hypocrisy and fecklessness of Western democracies, blurring the line between fact and fiction.
So what to do? Russia is still too big, proud and influential to ignore and still the only nuclear power comparable to the United States. It remains a major player on problems from the Arctic to Iran and North Korea. We need to focus on the critical before we test the desirable. The first step is to sustain, and if necessary amplify, the actions taken by the Obama administration in response to Russian hacking. Russia challenged the integrity of our democratic system, and Europe's 2017 electoral landscape is the next battlefield.
A second step is to reassure our European allies of our absolute commitment to Nato. American politicians tell one another to "remember your base", and that's what should guide policy towards Russia. Our network of allies is not a millstone around America's neck, but a powerful asset that sets us apart.
A third step is to stay sharply focused on Ukraine, a country whose fate will be critical to the future of Europe, and Russia, over the next generation. This is not about Nato or European Union membership, both distant aspirations. It is about helping Ukrainian leaders build the successful political system that Russia seeks to subvert.
Finally, we should be wary of superficially appealing notions like a common war on Islamic extremism or a common effort to "contain" China. Russia's bloody role in Syria makes the terrorist threat far worse and despite long-term concerns about a rising China, Mr Putin has little inclination to sacrifice a relationship with Beijing.
I've learnt a few lessons during my diplomatic career, often the hard way. I learnt to respect Russians and their history and vitality. I learnt that it rarely pays to neglect or underestimate Russia, or display gratuitous disrespect. But I also learnt that firmness and vigilance, and a healthy grasp of the limits of the possible, are the best way to deal with the combustible combination of grievance and insecurity that Mr Putin embodies. I've learnt that we have a much better hand to play with Mr Putin than he does with us. If we play it methodically, confident in our enduring strengths, and unapologetic about our values, we can eventually build a more stable relationship, without illusions.
• William J. Burns is president of the Carnegie Endowment for International Peace and a former deputy secretary of state. He served as US ambassador to Russia from 2005 to 2008.