But for some tourists, that is why they love it.
Those who do discover Koh Jum sit around beaches reading books, laze by pools sipping mango shakes or get countless massages.
Kayakers of varying abilities potter around the western coast with mask and snorkel on board, discovering new lunch venues.
Some tourists choose bicycles or motorbikes as their preferred transport and explore the sleepy villages and bays along the 12km coastline.
For the fit and adventurous, the 420m-high Mount Pu beckons. The steep and sometimes slippery track is almost impossible to follow.
For about 1,000 baht (S$40) a person, local guide Khun Nit expertly coaxes groups up the jungle-clad northern ridge, through banana orchids and luxurious palms and up to the home of majestic sea eagles. Puffing trekkers savour their picnic while taking in Krabi's glittering island gems and its verdant hinterland.
The hike can be booked through Jum Island Travel or any hotel.
For an underwater excursion, the best snorkelling and diving is farther offshore.
While snorkel tours that promise fast boats and four islands are available, those wishing to preserve their low-key holiday theme can charter their own long-tail boat, including lunch and snorkelling gear (Jum Island Travel, about 6,000 baht, depending on numbers).
At Bamboo Island, snorkellers show one another turquoise parrotfish and giant clams of every colour. Next door at Mosquito Island, thousands of fish charge at morsels of bread offered by boat drivers.
Be warned that mass tourism is not far away and day-tripping speedboats throng to these islands from nearby Phi Phi, Phuket and Ao Nang.
Arriving by 9am is the best way to enjoy the experience without having someone's flippers in your face.
After all the physical activity, food is bound to be a high priority. There are plenty of superb options on Koh Jum, featuring locally caught seafood and the curries for which southern Thailand is famous.
Most people stroll along the beach to their chosen restaurant, returning by the moonlight. A torch may be needed, but shoes, definitely not.
Do note that formal addresses are rarely used on the island and that these would not be displayed anywhere on the buildings. The villages are tiny and if you head to the right ones, you will not miss the restaurants as described below.
Right on the beach in the village of Ban Ting Rai, the rustic Golden Pearl Restaurant serves truly spicy Thai food the way locals like it. Its laab gai (minced chicken with mint and lime) is lip-stingingly delicious and affordable at 100 baht.
The eastern side of the island is home to the "city" (Ban Koh Jum). A sleepy street dotted with a couple of convenience stores and some children playing, the scene hardly fits the nickname.
On a pier right over mangroves, Koh Jum Seafood serves fresh local catch. The delicious stuffed squid, green curry prawn with pineapple and black pepper scallops, priced between 250 and 500 baht each, are a taste sensation.
Those looking for family food gravitate towards Ban Ban Cafe in Ban Ting Rai.
Khun Mali and her husband Somchai survived the 2004 tsunami on Phi Phi Island, although their restaurant jobs did not, so they returned home to Koh Jum to start their own business. While junior visitors dig into pizza and sticky banana roti, adults devour her traditional massaman curry and tangy seafood noodle salad, full of fresh chilli, coriander and lime. They cost between 100 and 180 baht.
Despite its sleepy vibe, this is not a place where people go thirsty. Just next door to Golden Pearl Restaurant is the kiosk-like Fu Bar, where the quirky and hip barman greets customers with: "Sawatdee krup, man", followed by an irrational, high-pitched giggle. With 150-baht cocktails in hand and toes in the sand, contented visitors sit mesmerised as the sun collapses behind Phi Phi islands.
And for those who think that sounds low-key, venture farther to the end of this beach to discover the best-hidden watering hole on the island. Rock Bar is constructed high over the rocky headland, seemingly entirely out of salvaged materials and driftwood, and held together by little more than its colourful prayer flags. Get in quick for a pre-dinner drink (80 to 120 baht) before the bar disintegrates into the ocean.
Hunger addressed and dehydration avoided, I amble along the kilometres of beach at the front of my villa. I have seen only three other people in the last hour. I have not been hassled into a tailor shop or observed anyone dangling from a parachute behind a speedboat. I have slept with my doors wide open and invited the sea breeze in without a single beat of party music finding my ears.
Koh Jum is wonderfully under the radar and, for those who make it here, that is the main attraction.
•Carolyn Beasley is a freelance travel writer, marine scientist and family adventure travel devotee.