To work in the Royal Archives at Windsor Castle in England is one of the most delicious prizes for a researcher. Climbing the steps to the Round Tower - where you can read centuries-old correspondence between monarchs and their ministers, untie ribbons binding intimate family records and feel the parchment crackling between your fingertips - is viscerally thrilling.

At every stage, you are reminded of the privilege you have been given. You must pass through layers of security, dress appropriately and use only "non-propelling" pencils. Disconcertingly, visitors must expect to be escorted to the bathroom and searched before departure (to prevent the theft of precious documents).

It's rarefied, elevated, lofty. And extremely frustrating.

Established by King George V in 1914, the archives are a private collection, with no public right of access. Records are exempt from freedom of information laws and rules covering Britain's National Archives that have traditionally allowed for the release of most government documents after 30 years.

Even highly qualified scholars have a hard time gaining entry to the Royal Archives, which cover 21/2 centuries of history and hold roughly 2 million documents. An unspecified number of boxes and files are off limits for no stated reason, and there is no public catalogue. And the process by which the keepers decide who may enter is mysterious and opaque. Researchers are left with the uncomfortable feeling that there could be material withheld, and that their quest for historical accuracy and completeness could be thwarted.

It is hard not to conclude that the gatekeepers regard their role, in part, as guarding the reputation of the British monarchy. Even today, nearly two decades after the death of Diana, Princess of Wales, it is deemed too risky to let slip the mask of monarchy and reveal that life for royalty, as for us mortals, can be a messy business involving illness, infidelity, complicated marriages, mistakes and prejudices.

Such discretion about the private lives of living members of the royal family is understandable, but why must those of monarchs long dead remain shaded and secret?

My own requests to study in the Round Tower were repeatedly rebuffed - despite my credentials as a writer working on a forthcoming biography of Queen Victoria and a clear commitment to good scholarship. After several attempts and many months, my request was rejected - on the grounds that I had not written a biography or a royal history before.

It was not until former governor-general Quentin Bryce (Queen Elizabeth II's representative in Australia) pressed my cause that I was finally admitted. Having been crushed previously, I was jubilant - but what of the many other worthy historians who happened not to have an influential supporter?

The secrecy and selectivity of the Royal Archives are well known among academics and historians, many of whom have encountered delay and censorship by the archives' custodians. Attempts to control what is published have often led to protracted disputes.

While the success of recent TV dramas such as The Crown and Victoria proves the enduring popular appeal of fictional accounts of the British royal family, historians are still fighting subterranean battles to tell the uncensored truth. And the censors can be capricious.

"On one occasion, something was taken out of my Queen Mother book," says biographer Hugo Vickers - only for the information to appear later in someone else's book. "That made me cross."

While writing a book about Victoria's daughter Louise, biographer Lucinda Hawksley was warned off even trying the archives by other authors, who told her she would "come up against a brick wall". An artist and a beauty, the princess married a man thought to be gay and enjoyed a storied love life. Her lovers are believed to have included her brother-in-law and a sculptor said to have died in her presence.

When Hawksley requested Princess Louise's file, she was simply told that it was closed. She thinks that this is pure censoriousness - and that the famously candid royal would have been "horrified".

The purpose of my book on Queen Victoria was to hack through the thicket of cliches around her: that she was an implacable puritan, a harsh mother who hated her children, a reluctant monarch, a puppet and a creature of the men around her, and a widow who refused to rule. But what I learnt through my interactions with the Royal Archives was that their control of vital records makes it hard for historians not to hew to the myths.

I have great respect for the archival librarians, who are careful, rigorous and exacting, and I was very grateful for the opportunity to study there. But after a senior archivist read my final manuscript, to check any references to material in the Windsor collection - a precondition of entry - I was asked to remove information for which I had uncovered evidence outside the archives. This concerned the queen's burial instructions and other evidence of her loving intimacy with her personal servant John Brown in the Scottish Highlands. My reference to an episode of postpartum depression was also queried.

After months of consultation with lawyers and rewriting to avoid any breach of "crown copyright", which lasts 125 years for unpublished documents, I decided not to take out the material. But it helped me understand the reason for the exclusion of such material from other approved works on Victoria.

Still, there is hope.

One historian who has recently been researching in the archives (but wished not to be named for fear of repercussions) reports signs of "more openness" at Windsor, since the appointment in 2014 of the respected Oliver Urquhart Irvine as royal librarian. But there is a long way to go.

The Times of London has been campaigning to see documents relating to correspondence between the royal family and the Nazi regime in the days before World War II. The Guardian fought a decadelong battle (which it won last year) for access to Prince Charles' secret "black spider memos", so named for his distinctive handwritten ink annotations, in which he lobbied government ministers.

A Times editorial argued: "Without compelling individual reasons otherwise, there should be a systematic presumption that royal documents will be made public. Royal history is British history."

And not just British history but the history of the Commonwealth and all the countries of the former British Empire. Australian Prime Minister Malcolm Turnbull wants to shed light on the 1975 constitutional crisis when then governor-general John Kerr dismissed one prime minister and appointed another. Mr Turnbull wants historians to be able to view the correspondence between the queen's representative and the palace.

It is as though the archives' keepers have taken too literally the words of Victorian constitutional historian Walter Bagehot: "We must not let in daylight upon magic." He meant only that the queen must be kept above politics, not that she should be kept a mystery.

By rationing access and suppressing evidence, the Royal Archives have accomplished the very reverse of their intention. In the absence of the full historical truth about the British monarchy, sensationalism, suspicion and spin have reigned for too long.

NYTIMES

• Julia Baird, a writer and broadcaster, is the author of the forthcoming Victoria: The Queen.