AI Framework Selection & Orchestration Playbook

When building production Generative AI applications, choosing the right orchestration framework is one of the most critical architectural decisions. The ecosystem has evolved from simple prompt wrappers into complex engines for data-loading, stateful graphs, and multi-agent collaboration.

🧭 1. Framework Ecosystem Overview

• LangChain: A general-purpose library providing standardized interfaces for prompt templates, models, and chain execution. Best suited for simple chains and generic API wrappers.
• LlamaIndex: A data-centric framework designed to connect external data sources (RAG) to LLMs. It features ingestion pipelines, vector indices, and advanced query routers.
• LangGraph: A stateful orchestration framework built on top of LangChain. It uses a Directed Acyclic Graph (DAG) architecture to model agent execution loops, giving developers granular control over state transitions.
• CrewAI: A prescriptive framework designed for role-playing multi-agent systems. Agents are assigned roles, goals, and tasks, working collaboratively in structured workflows.
• AutoGen: Microsoft’s actor-based framework for building multi-agent systems. It allows agents to dynamically converse and collaborate to solve tasks.
• OpenAI Agents SDK: A lightweight, developer-focused SDK designed for building assistant-style agents with direct integration into OpenAI’s developer platform.
• Haystack: A modular, pipeline-based framework optimized for document search, NLP tasks, and custom RAG flows.

🌲 2. Framework Selection Decision Tree

The diagram below guides framework selection based on application requirements:

⚖️ 3. LangChain vs. LlamaIndex

While both libraries integrate models with data, they are built on fundamentally different paradigms.

Production Comparison Matrix

🤖 4. LangGraph vs. AutoGen vs. CrewAI

For multi-agent systems, developers choose between explicit graph state machines and dynamic conversational actors.

State & Orchestration Profiles

• LangGraph (Explicit Graphs): Models workflows as nodes (actions) and edges (transitions) over a shared, immutable state. It is highly deterministic, making it ideal for systems requiring strict compliance or business logic gates.
• CrewAI (Prescriptive Crews): Employs a role-playing paradigm. The developer defines a structured “crew,” assigning specific tools and sequential or hierarchical tasks to each agent.
• AutoGen (Conversational Actors): Employs an event-driven, actor-model architecture. Agents converse dynamically, selecting their next steps based on conversational context, which can introduce non-determinism.

Agent Platform Comparison Matrix

💻 5. Equivalent RAG Implementations

To illustrate syntax differences, the following code examples show the same RAG pipeline (load documents, chunk text, index, and query) implemented in both frameworks.

A. LangChain RAG Implementation (LCEL)

import os
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import ChatPromptTemplate
 
# 1. Load and Split
loader = TextLoader("docs.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)
 
# 2. Embed and Index
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
 
# 3. Build LCEL Chain
template = "Answer the question based strictly on the context:\nContext: {context}\nQuestion: {question}"
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)
 
rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | model
)
 
# 4. Invoke
response = rag_chain.invoke("What is the rate limit?")
print(response.content)

B. LlamaIndex RAG Implementation

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
 
# 1. Configure Global Settings
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.0)
Settings.embed_model = OpenAIEmbedding()
 
# 2. Ingest and Auto-Split
documents = SimpleDirectoryReader(input_files=["docs.txt"]).load_data()
 
# 3. Embed, Index, and Query
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=2)
 
# 4. Invoke
response = query_engine.query("What is the rate limit?")
print(response.response)

💾 6. Memory & State Management Patterns

Managing conversational state across stateless HTTP calls requires structured persistence engines:

• Window Memory: Keeps only the last $N$ turns of a conversation to prevent token inflation.
• Semantic Memory: Embeds past conversation turns and saves them in a vector database. At query time, semantic search retrieves relevant historical context, reducing prompt token footprint.
• Thread-Based Checkpointing: Persistence adapters (e.g., LangGraph’s MemorySaver) save the execution graph’s state at each node transition. This enables features like multi-session threads and rollback capabilities.

🙋 7. Human-in-the-Loop (HITL) Architectures

Enterprise workflows often require human verification (e.g., reviewing generated code or approving a bank transfer) before proceeding.

Node: Call Tool ➔ Edge: Pause (Interrupt) ➔ Wait for Human Approval ➔ Resume execution

In LangGraph, this is achieved by setting a compilation breakpoint on a node:

# Compile graph with thread checkpointer and breakpoints
app = workflow.compile(
    checkpointer=MemorySaver(),
    interrupt_before=["execute_wire_transfer"]
)
 
# Execution pauses automatically before hitting "execute_wire_transfer" node.
# To resume, the client sends a message containing the approval status.

🏛️ 8. Production Deployment Patterns

1. Stateless Serverless Gateways: Wrap framework pipelines inside a lightweight API (e.g., FastAPI running on AWS Lambda or GCP Cloud Run). Best for simple RAG pipelines with under 5-second execution times.
2. Stateful Agent Workers (Temporal / LangGraph Cloud): For long-running agents (lasting hours or days), deploy stateful orchestrators. These engines persist execution states, handle retries automatically, and survive server restarts.
3. Local Container Proxies (LiteLLM Gateway): Route framework model requests through a local LiteLLM gateway container to manage rate-limiting pool failovers and audit token spend.

⚠️ 9. Framework Anti-Patterns

• Overengineering Simple Workflows: Wrapping single-step prompt requests in complex framework chains. If a prompt can be handled with a simple model call, avoid the overhead of LangChain/LlamaIndex.
• Hidden State Side-Effects: Relying on frameworks to automatically manage global variables or historical context. This makes testing and debugging difficult.
• Framework Lock-In: Coding deep business logic directly into framework abstractions. Ensure your agent’s core capabilities are written in standard Python, wrapping them in frameworks only at the entry points.

🔌 10. OpenTelemetry Integration

Production telemetry tools (e.g., Arize Phoenix, LangSmith) trace every framework execution block to isolate slow nodes or failing API calls.

from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor
 
# 1. Register the OTel tracer endpoint
tracer_provider = register(
    project_name="genai-handbook",
    endpoint="http://localhost:6006/v1/traces"
)
 
# 2. Instrument the framework
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)

🚦 11. Cost & Latency Trade-offs

• Token Amplification Loop: Multi-agent architectures can trigger loops where agents repeatedly query each other. Implement loop-limit caps (max_iterations=10) to prevent runaway costs.
• Node Execution Latency: Every node transition in stateful frameworks adds serialization and scheduling latency (~10ms–100ms). Minimize graph depth in customer-facing applications.