Choose the mode that doesn't work, then read your /var/log/Xorg.0.log file. Read through for the mode validation and selection processes. If there are issues it should list what the problem was there. You may find you either need to program the mode directly with a modeline to get the timings right (look up the modeline database on the mythtv web site as this has specific modelines for many TVs) or possibly find an option on the TV itself to "scan" the signal to determine the screen metrics (had to do this on my Samsung to get rid of overscan on 1080p50Hz, that worked perfectly as 1080p60Hz)