AI in Neovim

It took a lot of trial and error to get the CodeCompanion Neovim plugin to work. I thought maybe it would help replace quick google searches or help to provide quick code snippets so that I wouldn’t have to go and look over as many samples before starting in on an unfamiliar task.

Instead it was really hit or miss. It was great at producing toy programs and translating them (more or less) between languages. And then it was able to produce nearly correct Nix package definitions for packaging up those toy programs.

However, it produced python with subtle, but common, bugs that took a long time to nail down.

It’s sort of handy at producing sentences in foreign languages.

I still don’t know how it picks which model it’s going to use on the remote side. (Edit from the future: it uses the first available model, unless one is configured locally to be the default.)

the setup

NixOS for the host system and (coincidentally) where neovim is running. I didn’t open a firewall port, so it only worked from the local system. So what follows will be the nix stanzas I put in my configuration.nix file.

I put these snippets into configuration.nix and ran a nixos-rebuild switch. But I couldn’t figure out how to pull a model from the open web ui, so I just did an ollama pull codellama at the CLI.

Ollama

  services.ollama = {
    enable = true;
    acceleration = "cuda"; # Or "rocm"
  };
  systemd.services.ollama.serviceConfig = {
    Environment = [ "OLLAMA_HOST=0.0.0.0:11434" ];
  };

Open Web UI

  services.open-webui = {
    enable = true;
    port = 3000;
    # note: add openFirewall = true; to share it with other hosts
    # also add host = "0.0.0.0"; or other appropriate value
    environment = {
      ANONYMIZED_TELEMETRY = "False";
      DO_NOT_TRACK = "True";
      SCARF_NO_ANALYTICS = "True";
      OLLAMA_API_BASE_URL = "http://127.0.0.1:11434/api";
      OLLAMA_BASE_URL = "http://127.0.0.1:11434";
    };
  };
  nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
     "cuda_cudart"
     "cuda_nvcc"
     "cuda_cccl"
     "libcublas"
     "open-webui"
  ];

Once the Web UI is running, API Keys must be enabled in Admin → Settings → Generarl, Enable API Keys.

Neovim

I think the sticking point here was realizing the /api had to be in the connection url. While other transports had those settings baked in, I had to include that in the configuration.

I’m using Lazy, so here’s the stanza I use inside my plugins:

{
    "olimorris/codecompanion.nvim",
    dependencies = { "nvim-lua/plenary.nvim", "nvim-treesitter/nvim-treesitter" },
    config = function()
        require("codecompanion").setup({
            adapters = {
                http = {
                    hal900 = function()
                        return require("codecompanion.adapters").extend(
                            "openai_compatible", {
                                env = {
                                    url =
                                    "http://127.0.0.1:3000/api",
                                    api_key = secrets['api_key'],
                                },
                                headers = {
                                    ["Content-Type"] =
                                    "application/json",
                                    ["Authorization"] =
                                    "Bearer ${api_key}",
                                },
                                parameters = {
                                    sync = true,
                                },
                            })
                    end,
                },
            },
            strategies = {
                chat = {
                    adapter = "hal900",
                },
                inline = {
                    adapter = "hal900",
                },
                cmd = {
                    adapter = "hal900",
                }
            },
            opts = {
                log_level = "INFO",
            },
        }
        )
        vim.keymap.set("n", "<Leader><Leader>", ":CodeCompanionChat Toggle<cr>",
            DESC("AI: Chat"))
    end,
},

I made a table to hold secret values that I load from an external file, so the above could just as easily be:

api_key="sk-some-value-from-the-open-web-ui"

I load this up and then double tap the space bar to start a chat session. Enter key (from normal mode) sends.

(Note: JWT keys don’t start with sk…)

keepers?

Nope. Neither free enough nor useful enough.

I’ll try again in the future with other models, maybe.

Tags: index, nvim, ai

Tags