Debasis Bhattacharjee

A generator produces items one at a time using lazy evaluation — it only computes each item when requested. A list computes and stores all items immediately. Generators use far less memory for large sequences.

Deep Dive: Generators are created using generator functions (functions with yield instead of return) or generator expressions (like list comprehensions but with parentheses). When you call a generator function it returns a generator object without executing the body. Each call to next() on the generator executes until the next yield pauses execution and returns the value. The generator remembers its state between next() calls. Key advantage: memory. A list of 1 million items stores all 1 million in memory. A generator that yields 1 million items stores only the current item and the execution state. Generators are also composable — you can chain generators to build processing pipelines without intermediate memory allocation.

Real-World: Processing a 10GB log file: reading the entire file into a list would require 10GB of RAM. A generator that yields one line at a time uses constant memory regardless of file size. In data pipelines: file_lines → filter_errors → parse_timestamps → aggregate — each step is a generator passing items to the next without intermediate storage.

⚠ Common Mistakes: Forgetting that a generator is exhausted after iteration — you cannot iterate over it twice. Not recognizing that for loops and many Python builtins (sum list map) accept any iterable including generators. Using a list comprehension when a generator expression would suffice (when you only need to iterate once). Confusing generator functions (use yield) with regular functions that return lists.

🏭 Production Scenario: A data export API was timing out for large datasets because it built a complete list of 500000 records before streaming. Refactoring to yield records one at a time from a generator allowed streaming the response immediately and eliminated the memory spike and timeout.

Follow-up questions: What is the difference between yield and return in a generator? What is yield from and when do you use it? How do you convert a generator to a list and back?

// ID: PY-BEG-009  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

Training set is used to fit the model. Validation set is used to tune hyperparameters and select the best model. Test set is held out completely and used only once to report final performance. Using only train/test leads to overfitting on the test set through repeated evaluation.

Deep Dive: Without a separate validation set developers tune hyperparameters (learning rate tree depth regularization strength) by evaluating on the test set. Each evaluation leaks information about the test set into the model selection process — the final reported test accuracy is optimistically biased. A proper split: 70% training (model learns from this) 15% validation (used during development for hyperparameter tuning and model selection) 15% test (locked away evaluated exactly once to report final performance). For small datasets k-fold cross-validation replaces the validation set by rotating which portion of training data is held out. The test set must never be touched during any development decision.

Real-World: An ML competition showed that teams who repeatedly submitted to the public leaderboard (which used the test set) were effectively overfitting to the test set through hundreds of submission cycles. Teams who maintained a strict held-out final test set reported the more realistic performance numbers.

⚠ Common Mistakes: Using the test set during hyperparameter tuning then reporting test set performance as if it were unbiased. Not stratifying the split for classification — random splits of imbalanced data can put almost no positive examples in the validation set. Time-series data: splitting randomly instead of chronologically leaks future information into training.

🏭 Production Scenario: A production recommendation system was developed with 50 rounds of hyperparameter tuning each evaluated on the same test set. Deployed performance was 15% lower than the reported test AUC. Post-mortem confirmed the test set had been evaluated 50 times during development causing effective test set overfitting.

Follow-up questions: What is k-fold cross-validation and when do you use it? How do you handle time-series data splitting? What is nested cross-validation?

// ID: ML-BEG-004  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

A neural network is a series of connected layers of mathematical functions (neurons) that transform inputs into outputs. It learns by adjusting the connection weights using backpropagation — computing how much each weight contributed to the error and updating it to reduce the error.

Deep Dive: A neural network has an input layer (receives features) hidden layers (learn representations) and an output layer (produces predictions). Each neuron computes a weighted sum of its inputs adds a bias and applies an activation function (ReLU sigmoid tanh) to introduce non-linearity. Learning happens through: forward pass (compute prediction) loss computation (measure how wrong the prediction was using a loss function like cross-entropy or MSE) backpropagation (use chain rule to compute gradient of loss with respect to each weight) and gradient descent (update weights in the direction that reduces loss). This cycle repeats for many iterations (epochs) over the training data. The learning rate controls how large each weight update is.

Real-World: Image classification: the input layer receives pixel values early hidden layers learn to detect edges and colors middle layers detect shapes and textures later layers detect object parts and the output layer assigns class probabilities. This hierarchical feature learning happens automatically through training — no hand-engineering required.

⚠ Common Mistakes: Using too high a learning rate causing the loss to oscillate or diverge. Not normalizing inputs (neural networks are sensitive to input scale). Not enough data — neural networks need more data than traditional ML algorithms to generalize. Using too many layers for a simple problem when a shallower network would suffice.

🏭 Production Scenario: A production image recognition model for quality control on a manufacturing line was failing to converge during training. Investigation showed input images were not normalized — pixel values ranged 0-255 instead of 0-1. Adding a normalization layer as the first layer stabilized training and the model converged in 50 epochs.

Follow-up questions: What is the vanishing gradient problem? What is the difference between SGD Adam and RMSprop optimizers? What is batch size and how does it affect training?

// ID: ML-BEG-005  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

Hallucination is when an LLM generates confident-sounding but factually incorrect or fabricated information. It happens because LLMs are trained to produce plausible next tokens based on patterns — not to retrieve verified facts.

Deep Dive: LLMs learn statistical patterns from training data and generate text that sounds fluent and coherent — but they have no mechanism for verifying that what they generate is factually true. The model predicts the most probable next token given context which may not correspond to reality especially for: obscure facts (low representation in training data) recent events (after training cutoff) precise numerical information (dates statistics) citations and URLs (commonly fabricated) and complex multi-step reasoning (errors compound). Hallucination is not a bug it is an inherent property of the probabilistic text generation approach. Mitigation strategies: RAG (ground the model in retrieved documents) chain-of-thought (forces the model to reason explicitly) output validation (verify claims against reliable sources) and citation requirements (ask the model to quote source text supporting claims).

Real-World: A legal AI assistant was generating case citations that did not exist — fabricated case names and citations that looked completely plausible. Lawyers who did not verify sources submitted briefs with non-existent precedents. Implementing a verification layer that checked all citations against a legal database before displaying them eliminated the problem.

⚠ Common Mistakes: Believing LLM outputs are inherently factual. Not validating LLM outputs before acting on them especially for medical legal or financial decisions. Using LLMs to recall specific numbers dates or citations without verification. Thinking that larger models do not hallucinate — they hallucinate less but still hallucinate.

🏭 Production Scenario: A medical information chatbot was confidently providing incorrect drug dosage information that contradicted official guidelines. The information sounded authoritative and patients followed it. This resulted in a product recall and regulatory action. The fix required implementing RAG against official medical databases for all drug-related queries.

Follow-up questions: What is grounding in AI and how does it reduce hallucination? What is the difference between closed-book and open-book question answering? How do you measure hallucination rate in a production system?

// ID: AI-BEG-004  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

A module is a single .py file containing Python code. A package is a directory containing multiple modules and an __init__.py file. Packages allow organizing related modules into a hierarchical namespace.

Deep Dive: Any .py file is a module — it can be imported with 'import filename'. A package is a directory with an __init__.py file (can be empty) that tells Python to treat the directory as a package. The __init__.py can import from submodules to define the package's public API. Modern Python (3.3+) supports namespace packages — directories without __init__.py — but explicit __init__.py is still preferred for clarity. Import paths follow the directory structure: in a package 'myapp' with a subpackage 'utils' containing 'helpers.py' you import with 'from myapp.utils.helpers import my_function'. The __init__.py content controls what 'from myapp import *' exports.

Real-World: Django is structured as a package: the top-level 'django' directory contains __init__.py and subpackages like 'django.db' 'django.http' 'django.contrib' each have their own __init__.py. This allows clean imports like 'from django.db import models' while keeping the codebase organized across hundreds of files.

⚠ Common Mistakes: Forgetting __init__.py in package directories (causes ImportError in Python 2 sometimes works as namespace package in Python 3 but can cause confusing behavior). Circular imports between modules in the same package. Relative imports (from . import module) vs absolute imports — relative imports can cause issues when running scripts directly.

🏭 Production Scenario: A production Django application was growing to 50+ Python files in a single directory. Refactoring into packages (api/ models/ services/ utils/) with __init__.py files and clean public APIs reduced import statement complexity and made it possible to see the application structure at a glance.

Follow-up questions: What is the __all__ variable in Python modules? How does Python's import system search for modules (sys.path)? What is the difference between 'import module' and 'from module import name'?

// ID: PY-BEG-008  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

'try' runs code that might fail. 'except' catches specific errors. 'finally' always runs regardless of whether an error occurred — used for cleanup.

Deep Dive: The try block contains the risky code. If an exception occurs Python looks for a matching except clause. You can catch specific exception types (except ValueError) or use a bare except to catch everything (not recommended). The else clause (optional) runs only if no exception occurred. The finally clause always executes even if there was an exception or a return statement inside try — making it essential for releasing resources like file handles database connections or locks. Multiple except clauses can handle different exception types differently.

Real-World: In a database write operation: the try block executes the INSERT query the except block catches IntegrityError for duplicate keys and returns a meaningful error message the finally block always closes the database connection regardless of success or failure — preventing connection pool exhaustion.

⚠ Common Mistakes: Using a bare 'except:' that catches everything including KeyboardInterrupt and SystemExit making the program impossible to stop. Not closing resources in finally causing memory or connection leaks. Catching too broad an exception type and hiding real bugs.

🏭 Production Scenario: A production API server ran out of database connections after 6 hours because a developer forgot to close connections in a finally block. The try block opened a connection an exception occurred the connection was never closed and the pool was exhausted within hours under normal traffic.

Follow-up questions: What is the difference between except Exception and bare except? When does finally NOT execute? How do context managers (with statement) relate to try-finally?

// ID: PY-BEG-006  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

*args collects extra positional arguments as a tuple. **kwargs collects extra keyword arguments as a dictionary. Both allow functions to accept a variable number of arguments.

Deep Dive: When you define a function with *args any positional arguments beyond the explicitly defined ones are packed into a tuple called args. With **kwargs any keyword arguments not explicitly defined are packed into a dictionary called kwargs. The names args and kwargs are just convention — the * and ** operators are what matter. You can use *args and **kwargs together and you can also use them when calling functions to unpack sequences and dictionaries into arguments. This pattern is heavily used in decorators, class inheritance, and API wrappers.

Real-World: Django's class-based views use **kwargs extensively to pass URL parameters captured by the router into view methods. FastAPI uses *args and **kwargs in middleware to forward requests without knowing the exact signature of the next handler.

⚠ Common Mistakes: Confusing *args (tuple) with a list. Forgetting that *args must come before **kwargs in the function signature. Trying to access args by keyword or kwargs by position. Mutating args thinking it is a list.

🏭 Production Scenario: A logging decorator in a production Flask app broke when a new endpoint added a keyword argument. The fix was changing the decorator to use *args and **kwargs so it would transparently forward any arguments to the wrapped function without needing updates every time a new parameter was added.

Follow-up questions: How does ** unpacking work when calling a function? Can you have both *args and explicit keyword arguments? How are *args and **kwargs used in class __init__ with inheritance?

// ID: PY-BEG-003  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

'==' checks value equality. 'is' checks identity — whether two variables point to the exact same object in memory.

Deep Dive: The == operator calls the __eq__ method and compares values. The 'is' operator compares object identity using id(). Two objects can be equal in value but be different objects in memory. Python caches small integers (-5 to 256) and interned strings which can make 'is' return True unexpectedly for these values leading to subtle bugs if misused. You should almost never use 'is' to compare values — reserve it for None checks (if x is None) where it is both correct and idiomatic.

Real-World: In a user authentication system: 'if user_role == admin_role' correctly compares role names as strings. Using 'is' instead works on small test data due to string interning but silently fails in production when role strings come from a database and are different objects with the same value.

⚠ Common Mistakes: Using 'is' to compare strings or integers expecting value equality. Being confused by small integer caching making 'is' appear to work correctly during testing. Not using 'is None' — using == None instead which is slower and less Pythonic.

🏭 Production Scenario: A production bug was caused by comparing user permission strings with 'is' instead of '=='. Tests passed because short strings were interned but in production with database-fetched strings the comparison always returned False locking all users out of admin features.

Follow-up questions: What is object identity in Python? How does Python intern strings? Why is 'is None' preferred over '== None'?

// ID: PY-BEG-002  ·  DIFFICULTY: 3/10  ·  ★★★☆☆☆☆☆☆☆

Cross-validation trains and evaluates a model multiple times on different subsets of data giving a more reliable estimate of generalization performance especially for small datasets. The most common form is k-fold cross-validation.

Deep Dive: In k-fold cross-validation the dataset is split into k equal parts (folds). The model is trained k times each time using k-1 folds for training and 1 fold for validation. The final performance metric is the average across all k evaluations and you also get a standard deviation showing how stable the model is. Common choices: k=5 (20% validation each time) or k=10 (10% validation). Benefits over single split: uses all data for both training and validation (important for small datasets) provides confidence intervals on performance (single split gives one number — is it lucky or representative?) and reveals if the model is sensitive to which data is in training vs validation (high variance = potential overfitting). Stratified k-fold maintains class proportions in each fold — essential for imbalanced classification.

Real-World: A medical ML model for rare disease diagnosis had only 800 labeled examples. A single 80/20 split would train on 640 examples and validate on 160 — too few for either. 10-fold cross-validation trained 10 models each on 720 examples and validated on 80 giving a reliable performance estimate with confidence intervals and using all data for both training and evaluation.

⚠ Common Mistakes: Using k-fold cross-validation for hyperparameter tuning and reporting those scores as test performance (data leakage — use nested cross-validation instead). Not using stratified folds for imbalanced classification. Ignoring the standard deviation across folds — high variance means the model is sensitive to data splits which is itself a problem. Applying cross-validation to time-series data without using TimeSeriesSplit.

🏭 Production Scenario: A production model selection process used 5-fold cross-validation to compare 20 candidate models. The winning model had a mean AUC of 0.87 with standard deviation 0.02 — indicating stable performance across folds. The runner-up had mean AUC 0.86 with standard deviation 0.09 — highly variable and less trustworthy. The stable model was selected and performed as expected in production.

Follow-up questions: What is nested cross-validation and when do you need it? What is TimeSeriesSplit and why can't you use standard k-fold for time-series? What is sklearn Pipeline and why does it matter for cross-validation?

// ID: ML-BEG-006  ·  DIFFICULTY: 4/10  ·  ★★★★☆☆☆☆☆☆

A list comprehension is a concise way to create lists using a single line expression. Avoid them when the logic is complex enough that a regular loop is more readable.

Deep Dive: List comprehensions follow the syntax [expression for item in iterable if condition]. They are faster than equivalent for loops because they are optimized at the C level in CPython. However they are not always the right choice. Avoid them when: the logic requires multiple nested conditions you need to handle exceptions inside the loop the comprehension spans more than two lines when formatted or you are consuming a large dataset where a generator expression would be more memory-efficient. Nested list comprehensions (list comprehensions inside list comprehensions) are almost always a readability mistake.

Real-World: In a data processing pipeline: [user.email for user in users if user.is_active and user.verified] is clean and appropriate. But building a matrix transformation with three nested comprehensions is a maintainability trap — a regular loop with clear variable names is better for the next developer.

⚠ Common Mistakes: Nesting comprehensions three levels deep making code unreadable. Using list comprehensions when you actually need a generator (you are iterating once over a large dataset). Adding side effects inside comprehensions (modifying external state) which is a major anti-pattern.

🏭 Production Scenario: A memory crash in a production data export service was traced to a list comprehension processing 2 million records at once loading everything into memory. Replacing it with a generator expression fixed the memory issue without changing any other code.

Follow-up questions: What is the difference between a list comprehension and a generator expression? How do dict comprehensions and set comprehensions work? What is the performance difference between a comprehension and a map() call?

// ID: PY-INT-001  ·  DIFFICULTY: 4/10  ·  ★★★★☆☆☆☆☆☆