LLM application quality requires a multi-layered evaluation strategy: offline evals (automated benchmarks using LLM-as-judge) online monitoring (latency cost error rates) and human evaluation for quality calibration. There is no single metric — you need task-specific criteria.

Evaluation layers: automated offline evals (run test cases through the system compare outputs against reference answers using another LLM as judge — e.g. GPT-4 scoring responses on accuracy relevance groundedness and format compliance) human evaluation (sample of outputs reviewed by domain experts to calibrate the LLM judge and catch systematic failures) production monitoring (latency per-call cost API error rates user feedback signals like thumbs up/down) and A/B testing (compare system versions on real user traffic). RAGAS framework evaluates RAG systems specifically: faithfulness (is the answer grounded in retrieved context?) answer relevancy (does the answer address the question?) context recall and context precision. For agents: task completion rate steps per completion tool error rate and cost per successful task completion.

Attention allows a model to directly reference any position in the input sequence when processing each output token regardless of distance. RNNs process sequentially and lose information about distant tokens. Attention solved this and enabled parallelization of training.

RNNs process sequences step by step maintaining a hidden state that compresses all previous context. This creates two problems: vanishing gradients (difficulty learning long-range dependencies) and sequential computation (cannot be parallelized — step N requires step N-1). Attention solves both. For each output position attention computes a weighted sum of all input positions — the weights (attention scores) are learned and indicate relevance. Self-attention attends to all positions in the same sequence. Multi-head attention runs multiple attention computations in parallel each learning different types of relationships (syntax semantics coreference). The Transformer architecture (2017) used only attention (no recurrence) enabling full parallelization of training which allowed training on massive datasets that were impractical for RNNs.

Python uses reference counting as the primary memory management mechanism supplemented by a cyclic garbage collector to handle reference cycles. Memory is allocated from private heaps managed by the Python memory manager.

Every Python object has a reference count. When you assign a variable or pass an object to a function the count increases. When a reference goes out of scope or is deleted the count decreases. When the count reaches zero memory is freed immediately. The problem is reference cycles: object A references B B references A — neither count reaches zero. Python's gc module handles this with a generational garbage collector that periodically identifies and clears cycles. Objects are sorted into three generations based on survival — most objects die young (generation 0) so the GC focuses there. You can trigger collection manually with gc.collect() and disable it in performance-critical code if you are certain there are no cycles.

A metaclass is the class of a class — it controls how classes themselves are created. Use them when you need to enforce constraints auto-register classes or modify class definitions at creation time.

In Python everything is an object including classes. The default metaclass is 'type'. When Python processes a class definition it calls the metaclass to build the class object. By creating a custom metaclass (inheriting from type and overriding __new__ or __init__) you can intercept class creation and modify or validate it. Practical uses include: enforcing that all subclasses implement certain methods automatically registering plugin classes in a registry adding logging to all methods automatically and implementing singleton patterns. Django's ORM uses metaclasses to convert class-level field declarations into actual database schema mappings.

Fine-tuning adjusts the model weights on domain-specific data to internalize knowledge or style. Use it when the task requires consistent behavior style or format the base model cannot achieve through prompting alone. RAG is better for factual grounding; prompt engineering first for most tasks.

Fine-tuning: continue training a pretrained LLM on a curated dataset of examples in your target format/domain. Changes the model weights permanently for that task. Types: full fine-tuning (expensive updates all parameters) parameter-efficient fine-tuning (PEFT — LoRA QLORA update a small fraction of parameters cheaply). When to fine-tune: consistent output format the base model keeps breaking (code generation with specific conventions) domain-specific style or tone (legal writing medical reports) task-specific behavior patterns (classification schema extraction) or reducing prompt length at inference (baking instructions into the model). When NOT to fine-tune: you need up-to-date information (use RAG) you are still exploring requirements (use prompting first) you have less than 1000 high-quality examples (insufficient for fine-tuning) or the base model already performs the task well with prompting.

An AI agent uses an LLM as a reasoning engine to autonomously plan use tools and complete multi-step tasks. Unlike a single LLM call that maps input to output an agent operates in a loop: observe think act observe again — until the task is complete.

The ReAct pattern (Reason + Act) describes the core agent loop: the LLM receives a task and available tools generates a thought (reasoning about what to do) selects an action (a tool call) receives the observation (tool output) and repeats until producing a final answer. Tools are functions the LLM can invoke: web search code execution database queries API calls file operations. Agent architectures range from simple (single LLM with tools) to complex (multi-agent systems where specialized agents collaborate with a planner/orchestrator agent routing tasks). Key engineering challenges: tool design (tools must have clear descriptions for the LLM to select them correctly) error handling (agents can get stuck in loops or make wrong tool calls) context management (the agent's action history grows and fills the context window) and cost control (multi-step agents can make many API calls).

RAG retrieves relevant documents from a vector database using semantic similarity search injects them into the LLM context and generates a response grounded in the retrieved content. Main failure modes are retrieval failures context window overflow and hallucinations about retrieved content.

RAG has three main components: indexing (documents are chunked embedded using an embedding model and stored in a vector database like Pinecone Weaviate or pgvector) retrieval (the user query is embedded and semantically similar chunks are retrieved using approximate nearest neighbor search) and generation (retrieved chunks are inserted into the LLM prompt as context and the model generates a response). Key design decisions: chunk size (too small loses context too large wastes context window and dilutes relevance) embedding model choice number of retrieved chunks (k) whether to use reranking to improve retrieved chunk ordering and metadata filtering to constrain retrieval. Advanced patterns include hybrid search (semantic + keyword/BM25) HyDE (hypothetical document embeddings) and multi-hop retrieval for complex questions.

Model drift is the degradation of model performance over time as the real-world data distribution changes after deployment. Detect with monitoring (input distribution prediction distribution and ground truth metrics). Handle with automated retraining triggers shadow deployments and champion-challenger frameworks.

There are two types of drift: data drift (input feature distributions change — customer demographics shift new product categories appear) and concept drift (the relationship between inputs and outputs changes — what predicts churn changes as customer behavior evolves). Detecting data drift: monitor statistical properties of input features using tests like KS test Population Stability Index (PSI) or Jensen-Shannon divergence. Detecting concept drift: monitor prediction distribution shifts and when labels are available track accuracy/AUC over time. PSI > 0.2 typically signals significant drift requiring investigation. Handling drift: trigger model retraining when drift metrics exceed thresholds use sliding window retraining on recent data implement champion-challenger deployment to safely test retrained models and maintain feature stores that can be queried at training and serving time to ensure consistency.

A feature store is a centralized repository for ML features that solves the training-serving skew problem by ensuring features computed at training time are computed identically at serving time. It also enables feature reuse across teams and models.

Training-serving skew is one of the most common and damaging production ML problems: features computed during training using the full historical dataset are computed differently at serving time using real-time data leading to performance degradation. A feature store has two components: offline store (historical feature values for training — typically a data warehouse like BigQuery or Redshift) and online store (latest feature values for low-latency serving — typically Redis or DynamoDB). Feature pipelines write to both stores ensuring identical computation logic. Feature engineering logic is defined once and shared — a 'user_30_day_purchase_total' feature computed for a recommendation model can be reused by a fraud model without re-implementation. Modern feature stores (Feast Tecton Hopsworks) also handle: feature versioning (audit trail) feature sharing across teams and point-in-time correct feature lookup (critical for preventing temporal data leakage in training).