Tools tagged with "Llm"

LLM covers repositories built around large language models: inference engines for running models locally (llama.cpp, Ollama, vLLM), application frameworks for building on top of LLM APIs (LangChain, LlamaIndex), and open-weight models themselves along with their fine-tuning tools. It also includes retrieval-augmented generation (RAG) tooling, prompt management libraries, and evaluation frameworks for testing model outputs.

The first decision in this tag is usually local inference versus calling a hosted API. Tools like llama.cpp and Ollama let a model run entirely on local hardware, which matters for privacy-sensitive use cases or offline environments, but performance and model quality are bounded by available GPU or CPU memory. vLLM and similar serving engines target higher-throughput self-hosted deployment, optimized for serving many concurrent requests rather than single-user local use.

For building applications on top of a model, LangChain and LlamaIndex both provide abstractions for chaining prompts, managing context, and connecting a model to external data sources, though they differ in emphasis: LlamaIndex focuses more on indexing and retrieval over documents, while LangChain covers a broader range of agent and chain patterns. Many teams find these frameworks most useful early on and later replace parts of them with direct API calls as requirements narrow.

RAG tooling and vector search integration matter for any application that needs a model to answer questions about specific documents rather than general knowledge, since the underlying model has no built-in access to that data otherwise.

Comparison points:

• Local inference versus API-based, and the hardware that implies
• Model license terms, separate from the surrounding tooling's license
• Context window size and retrieval strategy for long documents
• Latency and throughput for the expected request volume
• Evaluation and observability tooling for tracking output quality over time