To evaluate the performance of autopilots in real scenarios for maritime autonomous surface ships (MASS), this article investigates four autopilots via the experimental study instead of the theoretical analysis and discusses their application capabilities. Free running model tests with four autopilot algorithms were performed using a scale model of a Liquefied Natural Gas (LNG) carrier in the towing tank at Flanders Hydraulics Research (FHR), and they were tested at different forward speeds and Under Keel Clearances (UKCs) focusing on the autopilot performance in shallow water. Furthermore, an innovative method combining the model evaluation mechanism and the grey relational decision-making theory was proposed to comprehensively evaluate the performance of autopilots. Finally, we discussed the application potential of the autopilots for controlling a surface ship in shallow water and presented shallow water effects on autopilots’ performance. The four autopilots presented in this research could successfully follow the desired trajectories with satisfactory control effects in shallow water, but their performance varied in different scenarios. Our results indicate the tested controllers can be applied in the towing tank, but for optimum selection of autopilot algorithms, we should take into consideration multiple factors including controller parameters, trajectory, and test condition, etc.