The heritage of temple gardens is an essential part of China’s cultural heritage and a precious historical and cultural asset. However, with the advancement of urban construction, China’s temple and garden heritage also face corresponding protection issues. This article takes the remains of the Giant Buddhist Temple at the southern foot of Baoshi Mountain in Xihu District, Hangzhou City, as an example. By organizing historical literature and painting materials, the historical development of the remains of the Giant Buddha Temple is summarized, and the surrounding environment, spatial layout, and garden activities are studied to explore their unique heritage value. Based on research on the existing problems of the site, three different design ideas are proposed: restoring the historical style of the relics, telling cultural stories of the relics, expanding the functions of the relics as a new type of urban public space, and combining the relics with contemporary art to continue the Buddha worship activities. The aim is to protect the remains of the Giant Buddha Temple while continuing the history it carries. At the same time, the public can experience and understand Buddhist culture and contribute to the garden heritage, especially for preserving Buddhist culture. The protection, utilization, and inheritance of temple garden heritage provide new directions for experimentation.