THE relationship between the United States and China has rightly been described as the most important relationship in world affairs. It is also the most complex and fraught one. These two titans are the world's two leading powers and are interconnected in numerous ways bilaterally, regionally and globally. It is therefore of vital importance to understand the dynamics that underlie and drive this relationship at present, which are shifting.

While Washington and Beijing cooperate where they can, there has also been steadily rising competition in the relationship. This balance has now shifted, with competition being the dominant factor. There are several reasons for it - but one is that security now trumps economics in the relationship.

The competition is not only strategic competition, it is actually comprehensive competition: commercial, ideological, political, diplomatic, technological, even in the academic world where China has banned a number of American scholars and is beginning to bring pressure to bear on university joint ventures in China.

Mutual distrust is pervasive in both governments, and is also evident at the popular level. The last Pew global attitudes data on this, in 2013, found distrust rising in both countries. Roughly two-thirds of the public in both countries view US-China ties as "competitive" and "untrustworthy" - a significant change since 2010 when a majority of people in both nations had positive views of the other.

One senses that the sands are fundamentally shifting in the relationship. Viewed from Washington, it is increasingly difficult to find a positive narrative and trajectory into the future. The "engagement coalition" is crumbling and a "competition coalition" is rising. In my view, the relationship has been fundamentally troubled for many years and has failed to find extensive common ground to forge a real and enduring partnership. The "glue" that seems to keep it together is the fear of it falling apart. But that is far from a solid basis for an enduring partnership between the leading powers.

Sliding downwards

THE macro trajectory for the last decade has been steadily downwards - punctuated only by high-level summits between the two presidents, which temporarily arrest the downward trajectory. This has been the case with the last four presidential summits. Occasionally, bilateral meetings like the Strategic and Economic Dialogue, which will convene in Washington in two weeks' time, provide similar stabilisation and impetus for movement in specific policy sectors. But their effects are short-lived, with only a matter of months passing before the two countries encounter new shocks and the deterioration of ties resumes.

The most recent jolts to the relationship, just a few months since Mr Xi Jinping and Mr Barack Obama took their stroll in the Zhongnanhai (the so-called Yingtai Summit), have been the escalating rhetoric and tensions around China's island-building in the South China Sea. Behind this imbroglio lie rising concerns about Chinese military capabilities, US military operations near China, and the broader balance of power in Asia.

But there have been a number of other lesser, but not unimportant, issues that have recently buffeted the relationship in different realms - in law enforcement (arrests of Chinese for technology theft and falsification of applications to US universities), legal (China's draft non-governmental organisation and national security laws), human rights (convictions of rights lawyers and the general repression in China since 2009), cyber-hacking (of the US Office of Personnel Management most recently) and problems in trade and investment. Hardly a day passes when one does not open the newspaper to read of more - and serious - friction.

This is the "new normal" and both sides had better get used to it rather than naively professing a harmonious relationship that is not achievable.

This has given impetus to an unprecedented outpouring of commentary and reports by Washington think-tanks in recent months. I have lived and worked there a long time, and cannot recall such a tsunami of publications on US-China ties - and they are all, with one exception (former Australian prime minister Kevin Rudd's Asia Society report), negative in nature, calling for a re-evaluation of US policy towards China, and a hardening of policy towards China across the board.

Shift in US thinking

A QUALITATIVE shift in American thinking about China is occurring. In essence, the "engagement" strategy pursued since Richard Nixon across eight administrations, that was premised on three pillars, is unravelling.

The American expectation has been, first, as China modernised economically, it would liberalise politically; second, as China's role in the world grew, it would become a "responsible stakeholder" - in Robert Zoellick's words - in upholding the global liberal order; and third, that China would not challenge the American-dominant security architecture and order in East Asia.

The first premise is clearly not occurring - quite to the contrary, as China grows stronger economically, it is becoming more, not less, repressive politically. There are any number of examples, but political repression in China today is the worst it has been in the 25 years since Tiananmen. With respect to the other two, we are not witnessing frontal assaults by China on these regional and global institutional architectures. But we are witnessing Beijing establishing a range of alternative institutions that clearly signal China's discomfort with the US-led post-war order. Make no mistake: China is methodically trying to construct an alternative international order.

This disillusion with China in America probably says much more about America than it does about China. One pattern has repeated itself over the past two centuries of the relationship: America's "missionary impulse" to transform China in its image has repeatedly been disappointed by not understanding the complexities on the ground in China and by China's unwillingness to conform to American expectations. So, once again, this seemingly has more to do with the US and its unrealistic expectations, than with China.

Despite this overall macro climate in the relationship, the US and China still have to coexist, and to do so peacefully if at all possible. We have business to do with each other - both commercial and diplomatic business.

Perhaps the most immediate opportunity - and one that would give an enormous boost to the relationship - would be the conclusion of a bilateral investment treaty. But negotiating this treaty is hung up in the queue behind the Trans-Pacific Partnership agreement. Given the difficulty the White House is having getting that agreement finalised and through Congress, there may be little appetite in Washington to conclude an investment treaty with China this year.

Also high on the agenda at present is the real need to forge practical cooperation on a number of so-called "global governance" issues, including North Korea, Iran, the Islamic State in Iraq and Syria, Afghanistan, counterterrorism, anti-piracy, climate change, maritime security, economic stability, energy security, sea-lane security, and setting global rules for cyber activity.

To date, China has been extremely reluctant to collaborate openly with the US on such global governance issues, but now it possibly seems more feasible. This is because President Xi has personally endorsed more "proactive diplomacy" by China in the global governance arena. This will not solve the problems in US-China relations, but it will help.

The upcoming Strategic and Economic Dialogue and Mr Xi's September state visit to Washington are golden opportunities to discuss these issues, try to forge tangible cooperation, and arrest the negative dynamic in the relationship. The question is whether it will be temporary again, or a real "floor" can be put beneath the relationship. If the past is any indicator, we should not expect too much.

What worries me is that in this increasingly negative and suspicious atmosphere, "tests of credibility" will increase. The best we can probably hope for over the next two to three years - as President Obama becomes a lame duck and the election cycle stimulates more heated rhetoric about China - is tactical management of the relationship, with sensitivity to each side's "red lines" and "core interests", while hoping that no "wild card" events occur. This could include another military incident in the air or at sea, or renewed tension over Taiwan.

Even the current situation in the South China Sea has real potential to haemorrhage, as China is not going to stop its island- building activities and hence will not meet American demands that it do so. Or if China, having fortified the islands, proclaims an air defence identification zone over the South China Sea. What is Washington to do then? The potential for military confrontation is not insignificant.

So, looking to the future, the key responsibility for both countries is to learn how to manage competition, keep it from edging towards the conflictual end of the spectrum, while trying to expand the zone of practical cooperation.

Neither country has any playbook to guide such a relationship. Henry Kissinger envisions what he calls "co-evolution" between the two powers, but even he concludes that this will require "wisdom and patience". But it is not at all clear to me that the respective political cultures and existing political systems, national identities, social values and world views will afford such a strategic grand bargain today.

Thus, these two great nations are likely to find it increasingly difficult to coexist - yet they must. However fraught, this is a marriage in which divorce is not an option. Divorce means war.

The writer is professor of political science and international affairs at George Washington University, and a senior fellow at the Brookings Institution. This is an excerpt from a lecture he gave at the American Chamber of Commerce in Hong Kong on Wednesday. The article was published in the South China Morning Post.