An integrated heating, ventilation and air-conditioning (HVAC) system is one of the most important components to determine the energy consumption ofthe entire building. For commercial buildings, particularly office buildings and schools, the heating and cooling loads are largely dependent on the occupantbehavioral patterns such as occupant density and their activities. Therefore, if HVAC system can respond to dynamic occupancy profiles, there is a largepotential for reducing energy consumption. However, currently, most of existing HVAC systems are being operated without the ability to adjust supplyair rate in response to the dynamic profiles of occupants. Due to this inefficiency, much of the HVAC energy use is wasted, particularly when theconditioned spaces are unoccupied or under-occupied (fewer occupants than the intended design). The solution to this inefficiency is to control HVACsystem based on dynamic occupant profiles. Motivated by this, the research provided a real-time vision-based occupant pattern recognition system foroccupancy counting as well as activity level classification. The research was divided into two parts. The first part was to use an open source library basedon deep learning for real-time occupancy counting and background subtraction method for activity level classification with a static RGB camera. Thesecond part utilized a DOE reference office building model with dynamic set-point control and conventional HVAC control to identify the potentialenergy savings and thermal comfort. The research results revealed that the vision-based system can detect occupants and classify activity level in real timewith accuracy around 90% when there are not many occlusions. Additionally, the dynamic set-point control strategies indeed can bring about energysavings and thermal comfort improvements.