In this paper, we focus on the random forest based concept detection system, and we intend to improve the efficiency of the system in testing phase and to save memory and storage usages by reducing the total number of trees (classifiers). However, reducing the tree number often results in poor performance. In this article, we proposed a method called tree-sharing to cope with this issue. Unlike the traditional method that treats each concept independently, our work shares the trees among concepts, and leave the most important ones from the view of whole system. Experiments on different concept sets show tree-sharing can greatly reduce the number of total trees while the performance decreases slightly. Even in the worst case, we achieve 80% of original performance with only 5% of trees.