A lack of time and a lack of interest are the most common reasons for not reading, according to a National Arts Council (NAC) survey.
And it does not help that a large proportion of people prefer online pursuits to reading books.
Of the 1,015 Singaporeans and permanent residents surveyed, 56 per cent had not read a literary book between March 2014 and March 2015.
Almost half of these non-readers cited lack of time as a reason for not reading. The same proportion cited a lack of interest. (See chart.)
Other reasons given for not reading included work commitments - with one in seven people saying so - and family commitments, raised by one in nine. Over one in eight said they preferred other activities.
This is according to the first National Literary Reading and Writing Survey, released on Monday.
Literary books, according to the NAC, include fiction, poetry, drama, graphic novels, creative non-fiction such as biographies, critical writing and anthologies.
Asked why other written works such as newspapers were not included, NAC's acting director of sector development (literary arts) May Tan said: "Encouraging people to read fiction and poetry is a priority for the council as we believe that the literary arts is an avenue for self-expression, learning and reflection.
"Reading allows us to see the world through the lens of the different characters in a story, opens our minds to different opinions and perspectives and helps nurture a sense of empathy in us."
Older people were less likely to have read a book in the year before March 2015. Less than a quarter of those aged 60 and above had done so, compared with more than half of those in their teens, 20s and 30s.
Another factor distracting people from book reading could be the Internet. The survey found that two in five people said they preferred the Internet and social media to reading books.
Observers said the findings were not necessarily cause for concern.
"The results show just one part of the state of literary arts," said Mr William Phuan, managing director of non-profit arts company The Select Centre.
Other aspects tied to reading not fully reflected in the survey include the number of literary books borrowed from libraries, he added.
The survey did, however, find that about a third of respondents got their literary books by borrowing them from the library or other people, or as gifts.
Said Mr Phuan: "The next step is really for all the various stakeholders - from publishers to writers, from educators to policymakers - to figure out together how to use all this information to encourage people and boost the percentages."
One way to get students interested is to give them a say in what they read for class, said Assistant Professor Suzanne Choo from the National Institute of Education.
"I think we should encourage students to read good works of literature, but we should also attend to their... interests."
Popular teen novels could be brought into the classroom for discussion, she suggested.
"If we find, for example, that many students are reading John Green's popular teen fiction The Fault In Our Stars, why not bring this into our classroom?" she said.
Mr Kenny Leck, owner of independent bookstore BooksActually, said the focus should be on cultivating the joy of reading rather than on instrumental goals such as improving English proficiency.
"The emphasis on reading for education is killing it slowly," he said. "If reading is about broadening one's horizons, then maybe we will have a fighting chance."
But for some people, technology has come in to fill that role.
SIM-University of London student Ang Beng Heng, 24, said that he prefers scrolling through news apps and Facebook feeds on his phone to reading fiction books because it is a convenient and accessible way to keep himself updated on current affairs.
"Current affairs are more often used as a conversation topic," he said. "It is also more important and related to work and career."