High demand for caisim in Indonesia’s main export commodity must be accompanied by a good planting process. The obstacle faced is that farmers are currently able to apply pesticides when the caisim plants have holes due to being eaten by pests. This control can be a good step to maximize the yield of caisim farming. However, many farmers have not implemented proper control of pests, one of which is farmers in Kebon Raya Dempo, South Sumatera, Indonesia. The obstacles faced such as not being able to detect pests correctly and provide pesticides with precision. Motivated by CNN’s success in image classification, a learning-based approach has been carried out in this study to detect the presence of pests in caisim. The experimental results show differences in accuracy in each experiment with a dataset of 1000, consisting of 500 image data with pests and 500 without pests. The accuracy of the experiment A – CNN from Scratch is 48.33%, precision 1, recall 0.48, F1-score 0.65, experiment B – CNN from Scratch is 73.00% precision 1, recall 0.64, F1- score 0.78, experiment C–CNN from Scratch experiment is 92.00% precision 0.88, recall 0.96, F1-score 0.92. Of the 3 trials, experiment A – CNN from Scratch experienced underfitting, experiment B – CNN from Scratch overfitting, and the C – CNN experiment from Scratch can be used for pest detection in ciasim.