The identification of Yuyuan Garden’s heritage value has started since its initial destruction. In its ever-evolving history, while the value identification points to the space, the traceability and identification of its history and culture have also been continuously strengthened in different historical stages and garden tomography. Active interaction with urban space culture along with Shanghai’s urban development process, thus forming a rich connotation of Yuyuan Garden heritage value changes. Through the analysis and comparison of the Yuyuan Garden heritage value framework in different historical stages, this paper studies and sorts out the basic laws of Yuyuan Garden heritage value and its changes, especially under the historical background of Shanghai’s rapid urban renewal, the composition of the Yuyuan Garden value framework system, the interpretation of its structure and the evaluation of value assessment. The basic structure and value connotation of the contemporary Yuyuan Garden heritage value interpretation framework should and must be constructed as a whole based on the historical development of Yuyuan Garden. The ultimate goal is to integrate the “human” element into a contemporary understanding of heritage, in the way of “Ego in Garden”: based on and respecting the heritage cognition and value thinking of the contemporary people on traditional gardens, to balance various value conflicts and challenges in the process of contemporary conservation and development.