Local LLMs

Learn how to run a Retrieval-Augmented Generation system locally using R2R

Introduction

To run R2R with default local LLM settings, execute r2r serve --docker --config-name=local_llm.

R2R supports RAG with local LLMs through the Ollama library. You may follow the instructions on their official website to install Ollama outside of the R2R Docker.

Preparing Local LLMs

Next, make sure that you have all the necessary LLMs installed:

# in a separate terminal
ollama pull llama3.1
ollama pull mxbai-embed-large
ollama serve

These commands will need to be replaced with models specific to your configuration when deploying R2R with a customized configuration.

Configuration

R2R uses a TOML configuration file for managing settings, which you can read about here. For local setup, we’ll use the default local_llm configuration. This can be customized to your needs by setting up a standalone project.

Local Configuration Details

The local_llm configuration file (core/configs/local_llm.toml) includes:

[completion]
provider = "litellm"
concurrent_request_limit = 1

  [completion.generation_config]
  model = "ollama/llama3.1"
  temperature = 0.1
  top_p = 1
  max_tokens_to_sample = 1_024
  stream = false
  add_generation_kwargs = { }

[database]
provider = "postgres"

[embedding]
provider = "ollama"
base_model = "mxbai-embed-large"
base_dimension = 1_024
batch_size = 32
add_title_as_prefix = true
concurrent_request_limit = 32

[ingestion]
excluded_parsers = [ "mp4" ]

This configuration uses ollama and the model mxbai-embed-large to run embeddings. We have excluded media file parsers as they are not yet supported locally.

We are still working on adding local multimodal RAG features. Your feedback would be appreciated.

For more information on how to configure R2R, visit here.

Summary

The above steps are all you need to get RAG up and running with local LLMs in R2R. For detailed setup and basic functionality, refer back to the [R2R Quickstart]((/documentation/quickstart/introduction). For more advanced usage and customization options, refer to the [basic configuration]((/documentation/configuration/introduction) or join the R2R Discord community.