AI and VoIP Blog

Large Language Models (LLMs) are powerful, but using external tools can disrupt your Vim workflow. llama.vim, a dedicated Vim plugin from the ggml-org organization, offers a solution by integrating local LLM inference directly into Vim through llama.cpp server.

(Link: https://github.com/ggml-org/llama.vim)

What It Is & Why Use It

llama.vim is a Vim plugin that connects to a local llama.cpp server instance for LLM-assisted code/text completion. Key benefits include:

• Private: Your prompts stay on your machine
• Offline: Works without internet access
• Cost-Free: No API keys or usage fees
• Integrated: Seamless text completion within Vim
• Smart Context: Maintains context from open files and edited text
• Resource-Efficient: Works well even on low-end hardware

Setup Guide

1. Prerequisites

• llama.cpp server instance
• A compatible FIM (Fill-In-the-Middle) model
• Vim or Neovim
• Vim plugin manager

2. Installation

We need to perform the following steps:

• Install plugin manager if not already installed
• Add llama.vim plugin in the .vimrc
• Run plugin manager commands inside vim to install the plugin

I am using vim-plug plugin manager, if you use any other plugin manager then check this link for steps here

a. Download plug.vim and put it in ‘autoload’ directory

  # Vim
  curl -fLo ~/.vim/autoload/plug.vim --create-dirs \
    https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
  # Neovim
  sh -c 'curl -fLo "${XDG_DATA_HOME:-$HOME/.local/share}"/nvim/site/autoload/plug.vim --create-dirs \
    https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'

b. Add a vim-plug section to your ~/.vimrc (or ~/.config/nvim/init.vim for Neovim).
If you don’t have a .vimrc file then create one first.

vim ~/.vimrc

Then add the below lines at the top.

call plug#begin()

Plug 'ggml-org/llama.vim'

call plug#end()

c. Save the changes and exit, now you can install the plugin:
Open ~/.vimrc and run the following command to install the plugin

vim ~/.vimrc
:PlugInstall

Enter to run the command and install the plugin.Exit the vim (!q) and you are all set.
Here are some more useful commands to manage plugins.

:PlugUpdate // to update plugins
:PlugDiff // to review the changes from the last update
:PlugClean // to remove plugins no longer in the list

3. llama.cpp Setup

Mac OS:

brew install llama.cpp

Other OS: Either build from source or download the latest binaries from llama.cpp releases

4. Configuration (Optional)

Configure the plugin by setting g:llama_config in your vimrc. Skip this section for default settings:

" Example: Disable inline info
let g:llama_config = { 'show_info': 0 }

" Example: Disable auto FIM completion
let g:llama_config = { 'auto_fim': v:false }

" Example: Change accept line keymap
let g:llama_config = { 'keymap_accept_full': '' }

Model Selection

Choose your model based on available VRAM:
The models will be auto downloaded when you run the command.

• >16GB VRAM: Use llama-server --fim-qwen-7b-default
• <16GB VRAM: Use llama-server --fim-qwen-3b-default
• <8GB VRAM: Use llama-server --fim-qwen-1.5b-default

These are qwen 2.5 coder models, for example “ggml-org_Qwen2.5-Coder-7B-Q8_0-GGUF_qwen2.5-coder-7b-q8_0.gguf”

Also known as Fill-in-the-Middle models, FIM is a specific type of LLM.

• They predict missing spans of text (“fill in the middle”) rather than just the next word.
• They uniquely utilize bidirectional context, considering both the text before (prefix) and after (suffix) the gap.
• This differs from standard causal/autoregressive models (like GPT) which only use preceding text to predict the next token.
• FIM excels at tasks like advanced code completion (inserting code within existing lines/blocks) and text infilling, where understanding the surrounding context is crucial.
• Often trained using specific FIM objectives or Masked Language Modeling (MLM) techniques.

Features

• Auto-completion: Suggestions appear automatically during Insert mode
• Manual Control: Toggle suggestions with Ctrl+F
• Quick Accept: Use Tab to accept full suggestion or Shift+Tab for first line
• Context Awareness: Maintains ring buffer of context from open files and yanked text
• Performance Stats: Shows generation time and context usage
• Advanced Features:
  • Speculative FIM support
  • Speculative Decoding support
  • Smart context reuse
  • Configurable context scope

Usage Tips

1. The plugin automatically suggests completions as you type in Insert mode
2. Use Ctrl+F to manually trigger suggestions
3. Accept suggestions using:

• Tab for full suggestion
• Shift+Tab for first line only

1. Monitor performance through the inline info display (can be disabled)

Key Considerations

• Performance: Varies based on hardware and chosen model size
• Model Choice: Use FIM-compatible models for best results
• Server Instance: Ensure llama.cpp server is running before using the plugin
• Context Management: The plugin intelligently manages context from your editing session

Additional Links

llama.vim official repository.
Vim Plug repository

Conclusion

llama.vim provides a powerful, local-first approach to LLM-assisted coding in Vim. With its smart context management and support for various hardware configurations, it offers a practical solution for developers and Vim users wanting AI assistance while maintaining privacy and performance.

Akash Gupta
Senior VoIP Engineer and AI Enthusiast

AI and VoIP Blog

Thank you for visiting the Blog. Hit the subscribe button to receive the next post right in your inbox. If you find this article helpful don’t forget to share your feedback in the comments and hit the like button. This will helps in knowing what topics resonate with you, allowing me to create more that keeps you informed.

Thank you for reading, and stay tuned for more insights and guides!

• Mail
• LinkedIn
• X