Russia's annexation of Crimea and intervention in Ukraine have frustrated Tokyo's efforts to strengthen its strategic partnership with Moscow.
Until March this year, Moscow and Tokyo were taking important steps to strengthen ties. President Vladimir Putin and Prime Minister Shinzo Abe had met five times within just 10 months. While some of the world's leaders boycotted the opening ceremony of the Sochi Winter Olympics in response to Russia's adoption of anti-gay laws last year, Mr Abe attended the ceremony to demonstrate his commitment to improving ties.
Behind the initiative were the two countries' shared interests. Japan needed energy and Russia wanted to sell it. After the 2011 nuclear disaster in Fukushima, all the nuclear power plants in Japan were shut. Japan's reliance on liquefied natural gas (LNG) for electric power generation jumped from 29.3 per cent in 2010 to more than 43.2 per cent last year as a result.
Russia has the second-largest natural gas reserves in the world, but natural gas demand in Europe has plateaued, while North America can now tap shale gas, undercutting the importance of Russian gas. That has made it imperative for Russia to diversify its gas export destinations. Despite their proximity, only 9.8 per cent of the LNG that Japan consumed last year came from Russia. But that will go up if the necessary infrastructure is put in place and Russian gas is made available at internationally competitive prices.
Moscow and Tokyo also share concerns over the rise of an assertive China.
Moscow worries about China's growing influence in the Russian Far East, Central Asia, the Sea of Okhotsk and the Arctic Ocean.
When the Chinese icebreaking research ship Xue Long entered the Sea of Okhotsk during its first trip to the Arctic Ocean in 2012, the Russian navy conducted an anti-ship missile exercise in the area. On the nuclear front, while Russia is bound by the Intermediate-Range Nuclear Forces Treaty, China is building up its intermediate-range nuclear arsenal.
It was significant in this context that Japan and Russia initiated "2+2" defence and foreign ministerial talks last year. As a result, Russia became the third country with which Japan has 2+2 meetings, after the United States and Australia. Japan is the fifth country that Russia holds such talks with, after the US, Britain, France and Italy.
Japan is also interested in resolving a longstanding territorial dispute with Russia over four of the Kuril Islands lying to the north of Hokkaido. These four islands were occupied by Russia two weeks after Japan surrendered to the Allied Powers in 1945. There were 16,874 Russian residents in the disputed islands last year, and Japanese former residents of the islands are ageing.
In the meantime, the Russian government is implementing an US$800 million (S$1 billion) socio-economic development project for the Kuril Islands for the target period of 2007 to next year. Airports, hospitals, schools and paved roads have been constructed as a result. As time goes by, the status quo will become more consolidated.
The Japanese government, in fact, has a level of flexibility on how to settle the territorial issue. While it demands sovereignty over the four disputed islands, it might be willing to accept an interim arrangement to let the Russians maintain administrative control over part of the islands for the time being. If the territory issue is resolved, it could have a positive impact on Japan-Russia relations.
While Japan is interested in energy and territories, Russia is interested in technology and investment. Russia's economy depends excessively on energy exports. Last year, sales of oil, petroleum products and natural gas constituted 66.6 per cent of Russia's exports.
With the advent of the shale gas/oil revolution and the global proliferation of nuclear power plants, an energy-dependent economy like Russia's will not be sustainable in the long run. Successful promotion of non-energy sectors is the key for Russia to become a true economic power.
The Ukraine crisis created multiple dilemmas for Japan. While Japan wishes to keep engaging Russia, turning a blind eye to the ongoing situation could seriously harm Japan's position on its own territorial claims, not only with Russia but also with China and South Korea.
The US remains strongly committed to the government in Kiev, and Japan is uncomfortably stuck in the middle of the US-Russia confrontation. When Japan decided to impose wide-ranging sanctions on Russia partly as a result of pressure from the US, Moscow criticised Tokyo for being unable to "pursue an independent foreign policy".
Despite the sanctions and the postponement of Mr Putin's planned visit to Tokyo this autumn, Mr Abe is determined to meet Mr Putin during the Asia-Pacific Economic Cooperation forum's summit next month and invite him to Tokyo next year.
Mr Abe's determination stems from two factors - one strategic and the other personal. Strategically, isolating Russia may drive the country into a tighter partnership with China.
While Japan imposed sanctions on Russia, Chinese President Xi Jinping reiterated that China would not participate in the "Western-led" sanctions on Russia. In their bilateral meeting in Shanghai in May, Mr Xi and Mr Putin celebrated the victory over "Japanese militarism" in World War II. They also attended the opening ceremony of the China- Russia joint military drill in the East China Sea, where China and Japan continue to confront each other over overlapping claims to the Senkaku/Diaoyu islands.
Moreover, China and Russia signed a US$400 billion natural gas deal that they had been negotiating since 2006. Based on the deal, Russia will provide 38 billion cu m of gas annually over 30 years. Desperate to defy Western pressure, Russia seemed to have made concessions to China on the gas price, which had long prevented the agreement.
In addition, Mr Abe has a personal commitment to the improvement of relations with Russia. His father, Mr Shintaro Abe, once a leading contender for the post of prime minister, worked hard to improve Japan's ties with Russia in the early 1990s.
After learning he had terminal cancer, he still went out and met Mr Mikhail Gorbachev in Tokyo in 1991. He died a month later. The young Shinzo Abe witnessed all this as his father's personal secretary and inherited his cause.
Mr Abe's commitment notwithstanding, dealing with Russia is a tricky business. Russia is using China as a counterweight to the West, and Japan as a counterweight to China. Despite some tension, Russia needs China as the most important growing energy consumer, a major investor and a strategic partner.
Some analysts think that the Ukraine crisis might have a long-term positive impact on the Japan-Russia relationship. Russia's isolation in the West may further accelerate its shift to the East, potentially enhancing Japan's bargaining position vis-a-vis Russia.
However, pessimists disagree. Facing difficult territorial issues in the West, Mr Putin will be in no position to make concessions on sovereignty over the disputed islands in the East. Japan's nuanced balancing act and Mr Abe's effort to realise his father's dream will have to continue.
The writer is professor at the National Graduate Institute for Policy Studies in Tokyo, where he is director of the Security and International Studies Programme.