Serangan path traversal (kerentanan No. 1 versi OWASP) terus membebani analis pusat operasi keamanan akibat tingginya volume peringatan dan analisis manual. Meski Large Language Model (LLM) potensial untuk otomasi respons insiden, keterbatasan domain spesifik dan halusinasi masih menjadi kendala. Penelitian ini mengevaluasi integrasi LLM, arsitektur retrieval-augmented generation (RAG), serta prompt engineering untuk mengatasi tantangan tersebut. Menggunakan design research methodology, tiga model LLM (Llama 3.1 8B, Mistral 7B Instruct, Qwen 2.5 7B) dievaluasi pada 166 entri log audit ModSecurity. Teknik prompting zero-shot, few-shot, dan chaining diuji melalui confusion matrix serta metrik RAGAS. Integrasi RAG dan prompt chaining terbukti konsisten menekan kesalahan klasifikasi. Kombinasi Qwen 2.5 7B mencapai performa sempurna pada accuracy, precision, recall, dan f1-score (skor 1.000). Namun, skor factual correctness RAGAS berada pada rentang moderat (0,0845–0,2115), mengindikasikan model belum konsisten menyertakan pemetaan standar keamanan pada narasinya. Temuan ini membuktikan klasifikasi serangan dan analisis teknis merupakan dua dimensi kualitas yang independen. Penelitian merekomendasikan Qwen 2.5 7B dengan RAG dan prompt chaining sebagai deteksi tahap awal, dengan catatan keluaran narasinya tetap memerlukan tinjauan analis manusia saat pelaporan.

Path traversal attacks (OWASP No. 1 vulnerability) burdening security operations center (SOC) analysts with high alert volumes and manual analysis. While Large Language Models (LLMs) show promise for automating incident response, domain specificity gaps and hallucination risks remain key obstacles. This study evaluates the integration of LLMs, retrieval-augmented generation (RAG), and prompt engineering to address these limitations. Using a design research methodology, three LLMs (Llama 3.1 8B, Mistral 7B Instruct, and Qwen 2.5 7B) were evaluated on 166 ModSecurity audit log entries across zero-shot, few-shot, and prompt chaining techniques, assessed via confusion matrix and RAGAS metrics. Integrating RAG with prompt chaining consistently reduced classification errors, with the Qwen 2.5 7B combination achieving perfect scores (1.000) in accuracy, precision, recall, and f1-score. However, the RAGAS factual correctness score remained moderate (0.0845–0.2115), indicating inconsistent explicit mapping to formal security standards within mitigation narratives. These findings demonstrate that attack classification capability and analytical depth are independent quality dimensions. The study recommends deploying Qwen 2.5 7B with RAG and prompt chaining as an early-stage detection engine to reduce analyst workload, while emphasizing that its narrative output still requires human analyst review during the reporting phase.