Interview Questions& Model Answers
Real questions. Real answers. Built from 20 years of actual hiring and being hired.
Type hints are annotations that specify expected types for variables function parameters and return values. They are ignored at runtime by default but used by static analysis tools (mypy pyright). Runtime enforcement requires libraries like Pydantic or beartype.
Python's type system is gradual — you add hints progressively without breaking existing code. Basic syntax: def greet(name: str) -> str. Complex types: List[str] Dict[str int] Optional[str] (can be None) Union[int str] and in Python 3.10+ int | str. Generic types allow parameterized classes: class Stack(Generic[T]). TypeVar creates generic type variables. Protocol defines structural subtyping (duck typing with type safety). At runtime type hints are stored in __annotations__ and are just metadata — Python does not check them. mypy and pyright perform static analysis. Pydantic validates at runtime using type hints for data parsing and validation. beartype provides runtime type checking with minimal overhead.
FastAPI's entire API surface is type-annotated — function parameter types define API request validation response model types define OpenAPI documentation and return type serialization. SQLAlchemy 2.0 uses type annotations for ORM model definitions. Both use the same type hints for static analysis AND runtime behavior.
Adding type hints to existing code and then being confused when it still fails at runtime (hints are not enforced by default). Using complex Union types when Optional (Union[X None]) is the common case. Not using TypedDict for dict structures with known keys (makes static analysis much more useful). Mixing legacy typing module types (List Dict) with modern built-in generics (list dict) available from Python 3.9+.
A production data pipeline was passing incorrectly typed arguments silently for months because no type checking was in place. Adding mypy to the CI pipeline immediately surfaced 47 type errors. Fixing them prevented a class of bugs that had been causing occasional data corruption. Three of the errors would have caused production failures in the next quarter based on upcoming data changes.
Threading is for I/O-bound tasks with moderate concurrency. Asyncio is for I/O-bound tasks with high concurrency and fine-grained control. Multiprocessing is for CPU-bound tasks requiring true parallelism. The GIL makes threading unsuitable for CPU parallelism.
Threading: OS threads preemptive scheduling GIL limits CPU parallelism good for I/O-bound work where threads sleep during I/O (GIL released) moderate overhead race conditions possible. Asyncio: single-threaded cooperative concurrency a single thread switches between coroutines when they await I/O handles thousands of concurrent connections efficiently requires async/await syntax throughout (async code cannot call sync code without blocking the event loop) best for high-concurrency I/O (web servers API clients). Multiprocessing: separate OS processes each with own Python interpreter and memory true CPU parallelism high overhead (process creation IPC) no shared memory by default best for CPU-bound tasks (numerical computation image processing ML inference). Decision: high-concurrency I/O → asyncio. CPU parallelism → multiprocessing. Simple I/O parallelism with existing sync code → threading.
FastAPI uses asyncio for handling thousands of concurrent HTTP connections efficiently. A background task that processes images uses multiprocessing.Pool to distribute work across CPU cores. A legacy synchronous database library is called from a thread pool using asyncio's run_in_executor to avoid blocking the event loop.
Mixing asyncio and synchronous blocking calls — calling requests.get() in an async function blocks the entire event loop. Using multiprocessing for I/O-bound tasks (huge overhead for no benefit over threading). Using threading for CPU-bound tasks and wondering why there is no speedup. Not using asyncio.gather() for concurrent async operations calling them sequentially instead.
A FastAPI service was timing out under load despite appearing to handle requests correctly in development. Profiling revealed synchronous database calls (using the requests library instead of httpx) inside async route handlers blocking the event loop during every database query. Replacing with async database drivers (asyncpg databases library) resolved the timeouts.
Zero-shot uses the base model with only instructions (no examples). Few-shot includes examples in the prompt. Fine-tuned models are retrained on domain data. The tradeoff is cost and flexibility versus consistency and performance.
Zero-shot: just the task description in the prompt. Relies entirely on the model's pretraining. Fast to deploy requires no labeled data. Performance varies by task complexity. Best for: common well-defined tasks (summarization translation sentiment). Few-shot: include 3-10 task examples in the prompt. Dramatically improves consistency and format adherence. Cost: larger prompts = more tokens per call. Performance ceiling limited by context window and what can be communicated via examples. Best for: uncommon tasks new formats specific style requirements. Fine-tuned: domain-specific retraining. Bakes behavior into model weights instead of prompt tokens. Shorter prompts lower inference cost better consistency on trained tasks. Requires labeled data (minimum 100-1000 high-quality examples) compute for training. Not updatable without retraining. Best for: consistent structured output domain-specific terminology and behaviors classification with specific categories.
A legal clause extraction system evolution: zero-shot (78% accuracy) → few-shot with 5 examples (86% accuracy) → few-shot with 20 examples (89% accuracy) → fine-tuned on 3000 examples (96% accuracy lower latency lower cost per call). Each step required more investment but delivered better ROI at the production volume they were operating at.
Jumping to fine-tuning before exhausting prompt engineering (expensive and inflexible). Using few-shot examples that are low quality or inconsistent — few-shot examples teach the model a behavior; bad examples teach bad behavior. Not measuring whether the performance gain justifies the cost of fine-tuning. Fine-tuning on a narrow task and breaking general capabilities (catastrophic forgetting).
A startup building a document AI product started with zero-shot (fast prototype) discovered insufficient performance moved to few-shot (8 examples in prompt fixed 70% of failures) then fine-tuned only their highest-volume document type (processing 100K documents/month — fine-tuning ROI was clear) while keeping few-shot for lower-volume types. This staged approach minimized cost while maximizing quality where it mattered.
The most practically useful Python patterns are: Singleton (via module-level objects or metaclass) Factory (via functions not classes) Strategy (via first-class functions) Observer (via callbacks or event systems) and Decorator (using Python's native decorator syntax). Python's first-class functions make many GoF patterns simpler or unnecessary.
Python's features change how classic patterns are implemented. Singleton: in Java you implement a private constructor with a static instance. In Python a module-level instance is already a singleton — module state is shared across all imports. Factory Method: in Java a separate factory class. In Python a function or callable that returns the right type is sufficient — first-class functions eliminate the need for a factory class hierarchy. Strategy: in Java each strategy is a class implementing an interface. In Python pass the strategy function directly — no class needed. Decorator: Python has native decorator syntax making this pattern trivially implementable. Observer/Event: Python's callable objects and collections of callbacks implement this cleanly without interface boilerplate. The key insight: Python's dynamic typing first-class functions and duck typing make many patterns simpler and reduce the class hierarchy complexity required in statically typed languages.
Django's middleware system is a chain-of-responsibility pattern implemented as callable objects. Flask's signal system (blinker) is an Observer pattern. SQLAlchemy's session uses Unit of Work pattern. Python's built-in sorted() function's key parameter is a Strategy pattern using first-class functions — sorted(users key=lambda u: u.last_name) passes the sorting strategy as a function.
Implementing Java-style patterns verbatim in Python (creating unnecessary class hierarchies). Not leveraging Python's first-class functions to simplify Strategy Command and Factory patterns. Implementing Singleton as a class when a module-level instance or functools.lru_cache(maxsize=None) serves the same purpose more simply.
A Python service implemented a complex Factory class hierarchy (AbstractFactory ConcreteFactory AbstractProduct ConcreteProduct) in Java style. Code review replaced it with a registry dictionary mapping string keys to constructor functions — 5 lines instead of 50 with identical functionality and better extensibility.
Batch GD computes gradients on the entire dataset — slow but stable. Stochastic GD (SGD) computes gradients on one example — fast but noisy. Mini-batch GD computes on a subset (typically 32-256 examples) — balancing speed and stability. Mini-batch is the standard for deep learning.
Batch gradient descent: compute loss and gradients across all training examples then update weights. Advantage: stable convergence guaranteed direction toward minimum. Disadvantage: extremely slow for large datasets (must process all data before updating) cannot fit large datasets in memory. SGD: compute gradient on one random example update weights immediately. Advantage: fast updates can escape local minima due to noise. Disadvantage: noisy updates cause loss to oscillate even near minimum hard to parallelize. Mini-batch: compromise — compute gradient on a random subset (batch size). Advantages: vectorized computation uses GPU parallelism efficiently noise helps escape local minima more stable than pure SGD. Batch size is a key hyperparameter: smaller batches (16-32) more noise better generalization larger batches (512-2048) more stable faster wall-clock time but may generalize worse (sharp vs flat minima research). Modern optimizers (Adam AdaGrad RMSprop) adapt learning rate per parameter addressing many SGD limitations.
Training GPT-scale models: batch sizes of 2048-8192 tokens are used across hundreds of GPUs. The batch is distributed across GPUs (data parallelism) with gradients averaged across GPUs before weight updates. Learning rate warmup (gradual increase from 0) is used because large batch sizes are sensitive to initial learning rate choice.
Using batch size 1 (pure SGD) on modern GPU hardware — wastes parallelism. Not adjusting learning rate when changing batch size (linear scaling rule: if you double batch size double learning rate). Using a constant learning rate when training benefits from decay (use cosine annealing or linear decay). Not shuffling training data before each epoch causing the model to see data in the same order repeatedly.
A production deep learning model was trained with batch size 4 because the researcher was worried about memory. Training took 72 hours. Using gradient accumulation (accumulate gradients over 32 steps before updating) achieved effectively batch size 128 without exceeding memory limits reducing training time to 18 hours with better final performance.
A metaclass is the class of a class — it controls how classes themselves are created. Use them when you need to enforce constraints auto-register classes or modify class definitions at creation time.
In Python everything is an object including classes. The default metaclass is 'type'. When Python processes a class definition it calls the metaclass to build the class object. By creating a custom metaclass (inheriting from type and overriding __new__ or __init__) you can intercept class creation and modify or validate it. Practical uses include: enforcing that all subclasses implement certain methods automatically registering plugin classes in a registry adding logging to all methods automatically and implementing singleton patterns. Django's ORM uses metaclasses to convert class-level field declarations into actual database schema mappings.
Django's Model metaclass (ModelBase) reads the field attributes you declare on a model class and builds the database schema query interface and validation logic automatically. Without metaclasses Django's ORM syntax would require explicit registration calls for every model field.
Overusing metaclasses for problems that class decorators or __init_subclass__ solve more simply. Metaclasses from different libraries conflicting when a class inherits from both (metaclass conflict). Writing metaclasses that are so abstract they become impossible to debug.
An internal plugin system at a SaaS company used a metaclass to automatically register all subclasses of a BasePlugin class into a global plugin registry. This eliminated the need for manual plugin registration and prevented the recurring production bug where developers created a plugin class but forgot to register it.
Python uses reference counting as the primary memory management mechanism supplemented by a cyclic garbage collector to handle reference cycles. Memory is allocated from private heaps managed by the Python memory manager.
Every Python object has a reference count. When you assign a variable or pass an object to a function the count increases. When a reference goes out of scope or is deleted the count decreases. When the count reaches zero memory is freed immediately. The problem is reference cycles: object A references B B references A — neither count reaches zero. Python's gc module handles this with a generational garbage collector that periodically identifies and clears cycles. Objects are sorted into three generations based on survival — most objects die young (generation 0) so the GC focuses there. You can trigger collection manually with gc.collect() and disable it in performance-critical code if you are certain there are no cycles.
A long-running FastAPI service was growing in memory over days. Profiling with tracemalloc revealed a reference cycle in a caching layer where cached response objects held references back to the cache container. Explicitly breaking the cycle with weakref.ref() eliminated the memory growth.
Assuming memory is freed immediately after del (del only removes the reference the GC frees memory). Creating reference cycles in data structures without using weakref. Disabling the GC for performance without understanding the cycle risk. Not using __slots__ in high-volume object creation wasting memory on per-instance __dict__.
A Python-based IoT data collector crashed with OOM after running for several days. Memory profiling showed 50000 DataPoint objects that should have been freed were kept alive by a reference cycle between DataPoint and its parent DataStream. Using weakref.ref for the back-reference fixed the leak.
During backpropagation in deep networks gradients shrink exponentially as they propagate backward through many layers making early layers learn very slowly or not at all. Solutions include ReLU activations batch normalization residual connections and careful weight initialization.
In backpropagation gradients are computed by multiplying partial derivatives through each layer using the chain rule. If activation functions have derivatives less than 1 (sigmoid outputs derivatives between 0 and 0.25) multiplying many such small values causes exponential decay — a 20-layer network might have gradients 10^-10 times smaller at layer 1 than layer 20. Solutions evolved over time: ReLU activation (derivative is 1 for positive inputs 0 otherwise — no saturation in positive region). Batch normalization normalizes layer inputs keeping activations in a healthy range. Residual connections (ResNet) add shortcuts that allow gradients to flow directly backward without passing through activation functions. Careful initialization (He initialization for ReLU Xavier for tanh) sets initial weights so activations neither explode nor vanish from the first forward pass.
ResNet (Residual Network) solved the degradation problem where very deep networks (100+ layers) performed worse than shallower ones despite having more parameters. The residual connections allowed training networks with 1000+ layers that would have been completely untrainable with standard architectures.
Using sigmoid or tanh activations in very deep networks without understanding their gradient saturation behavior. Not using batch normalization in deep CNNs. Thinking the vanishing gradient problem only affects RNNs — it was originally identified in feedforward networks and RNNs face an even more severe version.
A production time-series forecasting LSTM model for financial data was not learning beyond the first few timesteps. Diagnosis showed vanishing gradients preventing the model from learning long-range dependencies. Switching to a Transformer architecture with attention mechanisms and positional encoding resolved the long-range dependency problem entirely.
Attention allows a model to directly reference any position in the input sequence when processing each output token regardless of distance. RNNs process sequentially and lose information about distant tokens. Attention solved this and enabled parallelization of training.
RNNs process sequences step by step maintaining a hidden state that compresses all previous context. This creates two problems: vanishing gradients (difficulty learning long-range dependencies) and sequential computation (cannot be parallelized — step N requires step N-1). Attention solves both. For each output position attention computes a weighted sum of all input positions — the weights (attention scores) are learned and indicate relevance. Self-attention attends to all positions in the same sequence. Multi-head attention runs multiple attention computations in parallel each learning different types of relationships (syntax semantics coreference). The Transformer architecture (2017) used only attention (no recurrence) enabling full parallelization of training which allowed training on massive datasets that were impractical for RNNs.
Translation quality: an RNN translating a 100-word sentence compresses the entire source into a fixed-size vector losing detail about early tokens. An attention-based model when generating each target word directly attends to the most relevant source words — when translating 'bank' in a financial context it attends to financial terms in the source to disambiguate meaning.
Confusing self-attention with cross-attention (cross-attention attends between two different sequences as in encoder-decoder translation). Thinking attention has O(n) complexity — it is O(n2) in sequence length which is why very long sequences are computationally expensive and why efficient attention variants (Flash Attention sparse attention) were developed.
A document classification system for a legal tech company was using an LSTM that performed poorly on contracts longer than 1000 words — important clauses near the beginning were forgotten by the time the model reached the end. Switching to a transformer-based model (BERT fine-tuning) that could attend to any position simultaneously improved accuracy by 18%.
Model drift is the degradation of model performance over time as the real-world data distribution changes after deployment. Detect with monitoring (input distribution prediction distribution and ground truth metrics). Handle with automated retraining triggers shadow deployments and champion-challenger frameworks.
There are two types of drift: data drift (input feature distributions change — customer demographics shift new product categories appear) and concept drift (the relationship between inputs and outputs changes — what predicts churn changes as customer behavior evolves). Detecting data drift: monitor statistical properties of input features using tests like KS test Population Stability Index (PSI) or Jensen-Shannon divergence. Detecting concept drift: monitor prediction distribution shifts and when labels are available track accuracy/AUC over time. PSI > 0.2 typically signals significant drift requiring investigation. Handling drift: trigger model retraining when drift metrics exceed thresholds use sliding window retraining on recent data implement champion-challenger deployment to safely test retrained models and maintain feature stores that can be queried at training and serving time to ensure consistency.
A credit scoring model deployed in January showed 0.81 AUC. By September AUC had dropped to 0.71. PSI analysis of input features revealed significant drift in employment status and income features — COVID-19 had fundamentally changed the distribution. Emergency retraining on recent data restored AUC to 0.79.
Not monitoring model performance after deployment — treating deployment as the end of the ML lifecycle. Retraining on all historical data including outdated periods instead of using a recent sliding window. Not having rollback capability when a retrained model performs worse than the current champion. Ignoring the feedback loop where model predictions affect future training data.
A fraud detection model at a payment processor declined from 89% recall to 74% recall over 6 months as fraudsters adapted their behavior patterns. Monthly retraining on recent fraud cases and implementing a fast-response challenger model that retrained weekly restored recall to 86% while reducing false positives.
RAG retrieves relevant documents from a vector database using semantic similarity search injects them into the LLM context and generates a response grounded in the retrieved content. Main failure modes are retrieval failures context window overflow and hallucinations about retrieved content.
RAG has three main components: indexing (documents are chunked embedded using an embedding model and stored in a vector database like Pinecone Weaviate or pgvector) retrieval (the user query is embedded and semantically similar chunks are retrieved using approximate nearest neighbor search) and generation (retrieved chunks are inserted into the LLM prompt as context and the model generates a response). Key design decisions: chunk size (too small loses context too large wastes context window and dilutes relevance) embedding model choice number of retrieved chunks (k) whether to use reranking to improve retrieved chunk ordering and metadata filtering to constrain retrieval. Advanced patterns include hybrid search (semantic + keyword/BM25) HyDE (hypothetical document embeddings) and multi-hop retrieval for complex questions.
A legal research assistant RAG system at a law firm used chunk sizes of 512 tokens for case documents. Attorneys complained answers lacked context. Investigation showed important legal reasoning spanned across chunk boundaries. Implementing larger overlapping chunks (1024 tokens with 200 token overlap) and a reranker (Cohere Rerank) improved answer quality significantly.
Chunking documents arbitrarily without considering semantic boundaries (splitting mid-paragraph). Using cosine similarity retrieval without reranking causing less relevant chunks to appear in context and confuse the model. Not handling the case where no relevant documents are retrieved — the model hallucinates instead of saying it does not know. Embedding the entire document instead of chunking exceeding context limits.
A production customer support RAG system was giving confidently wrong answers about product return policies. Investigation revealed the retrieval was returning chunks from old policy documents because they had higher semantic similarity scores than newer updates. Implementing date-based metadata filtering to prefer recent documents and adding a retrieval confidence threshold solved the problem.
An AI agent uses an LLM as a reasoning engine to autonomously plan use tools and complete multi-step tasks. Unlike a single LLM call that maps input to output an agent operates in a loop: observe think act observe again — until the task is complete.
The ReAct pattern (Reason + Act) describes the core agent loop: the LLM receives a task and available tools generates a thought (reasoning about what to do) selects an action (a tool call) receives the observation (tool output) and repeats until producing a final answer. Tools are functions the LLM can invoke: web search code execution database queries API calls file operations. Agent architectures range from simple (single LLM with tools) to complex (multi-agent systems where specialized agents collaborate with a planner/orchestrator agent routing tasks). Key engineering challenges: tool design (tools must have clear descriptions for the LLM to select them correctly) error handling (agents can get stuck in loops or make wrong tool calls) context management (the agent's action history grows and fills the context window) and cost control (multi-step agents can make many API calls).
A customer onboarding agent at a SaaS company replaces a 12-step manual process: it receives a new customer email calls the CRM API to create a contact queries the provisioning API to set up an account generates and sends a personalized welcome email creates a Jira ticket for account review and posts a Slack notification to the account manager — all autonomously from a single trigger.
Building agents without observability — impossible to debug why an agent made wrong decisions without logging the full thought-action-observation trace. Not implementing maximum step limits — agents can loop indefinitely on ambiguous tasks. Giving agents too many tools — LLMs struggle to select from large tool sets. Not handling tool failures gracefully in the agent loop.
A document processing agent for an insurance company was processing claims autonomously. Without a step limit it entered an infinite loop trying to resolve a document parsing error making 10000 API calls in 8 minutes and generating a $400 API bill before being detected. Implementing a 20-step maximum and exponential backoff on tool errors fixed the runaway behavior.
Fine-tuning adjusts the model weights on domain-specific data to internalize knowledge or style. Use it when the task requires consistent behavior style or format the base model cannot achieve through prompting alone. RAG is better for factual grounding; prompt engineering first for most tasks.
Fine-tuning: continue training a pretrained LLM on a curated dataset of examples in your target format/domain. Changes the model weights permanently for that task. Types: full fine-tuning (expensive updates all parameters) parameter-efficient fine-tuning (PEFT — LoRA QLORA update a small fraction of parameters cheaply). When to fine-tune: consistent output format the base model keeps breaking (code generation with specific conventions) domain-specific style or tone (legal writing medical reports) task-specific behavior patterns (classification schema extraction) or reducing prompt length at inference (baking instructions into the model). When NOT to fine-tune: you need up-to-date information (use RAG) you are still exploring requirements (use prompting first) you have less than 1000 high-quality examples (insufficient for fine-tuning) or the base model already performs the task well with prompting.
A financial services company needed an LLM to consistently extract structured data from loan applications into a specific JSON schema. Prompt engineering achieved 78% schema compliance. RAG did not help (the schema was fixed not document-dependent). Fine-tuning with 5000 labeled examples achieved 97% schema compliance with shorter prompts reducing inference cost.
Fine-tuning with low-quality or insufficient examples — produces a model worse than the base model. Fine-tuning when prompt engineering would suffice — expensive and inflexible. Forgetting that fine-tuned models still hallucinate and still need RAG for factual grounding. Not evaluating catastrophic forgetting — fine-tuning on a narrow dataset can degrade performance on general tasks.
A customer service company fine-tuned an LLM on 2000 examples of customer conversations expecting it to handle all intents. In production the model lost general language capabilities and failed on intents not well-represented in the training data. Rebuilding with a larger curated dataset (15000 examples across all intents) with proper evaluation resolved the regression.
LLM application quality requires a multi-layered evaluation strategy: offline evals (automated benchmarks using LLM-as-judge) online monitoring (latency cost error rates) and human evaluation for quality calibration. There is no single metric — you need task-specific criteria.
Evaluation layers: automated offline evals (run test cases through the system compare outputs against reference answers using another LLM as judge — e.g. GPT-4 scoring responses on accuracy relevance groundedness and format compliance) human evaluation (sample of outputs reviewed by domain experts to calibrate the LLM judge and catch systematic failures) production monitoring (latency per-call cost API error rates user feedback signals like thumbs up/down) and A/B testing (compare system versions on real user traffic). RAGAS framework evaluates RAG systems specifically: faithfulness (is the answer grounded in retrieved context?) answer relevancy (does the answer address the question?) context recall and context precision. For agents: task completion rate steps per completion tool error rate and cost per successful task completion.
At a legal document AI company: automated evals used a curated set of 500 document-question pairs with reference answers GPT-4 as judge scored faithfulness and accuracy monthly human review by paralegals calibrated the automated judge real-time dashboards showed per-endpoint latency and cost and a thumbs-down button collected user feedback that triggered human review for systematic issues.
Using only automated LLM-as-judge evaluation without human calibration — the judge model has its own biases and blind spots. Not evaluating on adversarial cases (edge cases failure modes). Measuring only technical metrics (latency cost) and not quality metrics. Not separating evaluation of the retrieval step from the generation step in RAG systems.
A customer service AI showed consistently positive automated evaluation scores but had a growing volume of user complaints. The disconnect was because the LLM judge was evaluating response quality in isolation while users were frustrated by the system's failure to resolve their issues (task completion rate was not measured). Adding task completion as a primary metric revealed the real problem.
A feature store is a centralized repository for ML features that solves the training-serving skew problem by ensuring features computed at training time are computed identically at serving time. It also enables feature reuse across teams and models.
Training-serving skew is one of the most common and damaging production ML problems: features computed during training using the full historical dataset are computed differently at serving time using real-time data leading to performance degradation. A feature store has two components: offline store (historical feature values for training — typically a data warehouse like BigQuery or Redshift) and online store (latest feature values for low-latency serving — typically Redis or DynamoDB). Feature pipelines write to both stores ensuring identical computation logic. Feature engineering logic is defined once and shared — a 'user_30_day_purchase_total' feature computed for a recommendation model can be reused by a fraud model without re-implementation. Modern feature stores (Feast Tecton Hopsworks) also handle: feature versioning (audit trail) feature sharing across teams and point-in-time correct feature lookup (critical for preventing temporal data leakage in training).
At a major e-commerce company the customer lifetime value model the recommendation model and the fraud model all needed 'user_purchase_frequency_last_30_days'. Before the feature store each team computed it differently with subtle differences (timezone handling business day vs calendar day) producing inconsistent results. The feature store defined one authoritative computation shared across all three models.
Implementing offline-only features (fast to build but creates training-serving skew when serving). Computing features in the model serving code itself (no reuse performance overhead). Not handling point-in-time correctness in the offline store (using features from after the label timestamp in training data — a form of feature leakage). Building a feature store before having more than 2-3 models (premature optimization).
A churn prediction model performing at 0.84 AUC in offline evaluation dropped to 0.71 AUC in production. Investigation revealed that customer engagement features were computed using UTC timestamps in training but local time in the serving API — a seemingly minor difference that caused dramatic feature value shifts for users in non-UTC timezones. Centralizing feature computation in a feature store with explicit timezone handling fixed the skew.