Natural Language to Bash Command Translation Using Large Language Models with Agentic External Tool Feedback

Yesilyurt M. E., TANRIÖVER Ö. Ö.

6th International Conference on Problems of Cybernetics and Informatics, PCI 2025, Baku, Azerbaycan, 25 - 28 Ağustos 2025, (Tam Metin Bildiri) identifier

  • Yayın Türü: Bildiri / Tam Metin Bildiri
  • Doi Numarası: 10.1109/pci66488.2025.11219789
  • Basıldığı Şehir: Baku
  • Basıldığı Ülke: Azerbaycan
  • Anahtar Kelimeler: large language models, natural language processing, nl2bash
  • Ankara Üniversitesi Adresli: Evet

Özet

In Unix-like systems, users can interact with the operating system through a command-line interpreter such as Bash. Although the command-line interpreter provides a powerful interface to the computer, these commands can often be verbose and difficult to write. In this work, we evaluate the performance of several large language models (LLMs) in translating natural language prompts to bash commands. We investigate the results of providing feedback to the models via external tools, such as a Bash syntax analyzer. We also investigate an agentic approach where the models can make requests to use the tools. We use the NL2SH-ALFA dataset, and InterCode-ALFA benchmark to evaluate the results. We also investigate the comparative performance of using a non-English language for natural language prompts by creating a Turkish version of the test set.