Huge-scale video surveillance systems have become essential in crime prevention and situation recording. Traditional surveillance systems relied on human monitoring of video streams, which often led to errors and difficulties in understanding events. Furthermore, locating specific scenes within recorded videos required extensive human investigation. To overcome these inefficiency, inconvenience, and potential risk challenges, we propose an intelligent analysis scheme that utilizes abnormal behavior recognition and metadata retrieval algorithms to replace human monitoring. The proposed method consists of three stages: i) basic metadata generation through object detection and tracking, ii) abnormal behavior recognition for event metadata, and iii) SQL-based metadata retrieval. By incorporating specific information such as object color and aspect ratio, our technique enhances retrieval usability. Moreover, our module for recognizing abnormal behavior demonstrates robust classification capabilities for activities such as pushing, violence, falling, and crossing barriers. Since our system considers the possible harsh scenarios that cause the computation limitation of edge devices, we choose the adoptable best algorithm for each edge device. In addition, analysis sever complements the detection results of edge cameras. As a result, the proposed method can robustly generate the metadata without any object exception and search the specific object by query. Therefore, the proposed method can be seamlessly deployed on both edge cameras and analysis servers, making it adaptable to various surveillance setups. This approach revolutionizes the traditional surveillance paradigm, enabling more efficient, reliable, and secure video monitoring and analysis.