If one looks at a satellite view of that Kanchanaburi area such as seen in the GOOGLE MAPS software, it is obvious that the rails run about 5 Km NW of the old walled city of Kanchanaburi to the point where the bridges were built. Once across the river, the tracks essentially double-back on themselves over that same 5 KM length. The obvious question is: why was the railway lengthen by some 10 Kms? The answer is found in a book written by one of the primary engineers who surveyed and directed the building of the TBR: Across the Three Pagodas Pass: The Story of the Thai-Burma Railway by Yoshihiko Futamatsu (as translated by British ex-POW Ewart Escritt).
He describes how there were three possible routes considered initially. One would have started much farther north in central Thailand in the province of Pitsanulok. This route had been previously explored by German engineers in the late 1800s but was rejected due mainly to the gradient that the railway would have had to cross in terms of the elevation of the mountains that formed the border with Burma in that area. In a straight line distance it would have been a shorter hop to Moulmein, but also would have had to cross the Ping and Moei rivers; the latter being the Thai-Burma Border. It would also have had to use the existing Thai rails from Bangkok to Pitsanulok.
Once that choice was ruled out, the route from Nong Pladuk up to Kanchanaburi posed two alternatives. One would have taken the rails about 10 Km beyond Kanchanaburi city to Lat Ya where it would have turned to cross the Mae Klong (Kwae Yai) River and the turn NW towards the Three Pagodas Pass which would have been necessary to avoid the ruggedly mountainous areas that run north and south in that area. That alternate route would have met the route of the railway as it was built somewhere near NamTok (before Hellfire Pass).
Futamatsu relates that the initial party of Japanese Railway engineers led by a MAJ Hikiji used aerial maps to survey the possible routes and rejected those first two resulting in the one that followed the Kwae Noi River from ChungKai to the Three Pagodas Pass. It then became Futamatsu’s task to oversee that construction beginning at Nong Pladuk.
The reason why the railway took a 5 Km swing up and back is simple geology. The shortest route would have been to turn near the walled city of Kanchanaburi and cross the two rivers at about the point where they converge. But the engineers took core samples and determined that the river bed in that area was took silted to be able to easily reach the bedrock that would be needed for a stable bridge. So they went a bit farther north and found a more suitable site at Tha Mar Kam where the footing was more stable. They also seemed to listen to the locals who described how the Kwae Noi tributary — more so than its larger neighbor — flooded reliably at least once a year. So a bridge near its course would have been subjected to immense water pressure. The Tha Mar Kam site was a bit wider and the banks higher, so there was less of a flooding danger. I’m also sure that the inhabitants of the city were happy that the Allied bombs intended for those bridges were falling 5 Km north of their homes!
The follow series of maps published on the AUS 2/4 MG Bn site [https://2nd4thmgb.com.au/soldiers-all/camps-and-maps/ ]shows the terrain that determined the precise route the railway took (particularly maps 7 & 10):