Overview
Based on "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari. The core principle: features are the interface between raw data and model performance. The right transformation of an existing variable outperforms adding more data or tuning hyperparameters in most real-world settings. Feature engineering is domain knowledge encoded into math - and it is the highest-leverage step in the ML pipeline.
Workflow
Step 1: Audit raw columns for engineering opportunity
Before transforming anything, catalog what you have and what problems each column has.
For each column, note:
- Type: numeric continuous, numeric discrete, ordinal categorical, nominal categorical, datetime, text, ID
- Problem: skewed, high cardinality, missing, mixed type, free text, leaky (contains target signal from the future)
- Engineering opportunity: log transform, binning, encoding, extraction, embedding
Create a feature engineering plan as a table before writing any code:
| Column | Type | Problem | Planned Transform |
|-------------|-----------|------------------|---------------------------|
| revenue | numeric | Right-skewed | log1p transform |
| country | nominal | 150 unique vals | Frequency encoding |
| signup_date | datetime | Raw timestamp | Extract day-of-week, hour |
| description | text | Unstructured | TF-IDF, 500 features |
Step 2: Apply numeric transformations
Skewed distributions: Apply log transform to right-skewed variables. Use np.log1p to handle zeros.
df["log_revenue"] = np.log1p(df["revenue"])
Scaling: Standardize for distance-based models (KNN, SVM, neural nets). Tree-based models (XGBoost, Random Forest) do not require scaling.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df["revenue_scaled"] = scaler.fit_transform(df[["revenue"]])
Binning: Convert continuous to ordinal when the relationship to the target is non-linear.
df["age_bin"] = pd.cut(df["age"], bins=[0, 18, 35, 55, 100], labels=["<18", "18-35", "35-55", "55+"])
Capping outliers: Cap at a percentile rather than removing rows.
p99 = df["session_time"].quantile(0.99)
df["session_time_capped"] = df["session_time"].clip(upper=p99)
Step 3: Encode categorical variables
Match the encoding method to the cardinality and the model type.
| Cardinality | Model Type | Method |
|---|
| Low (< 10) | Any | One-hot encoding |
| Medium (10-50) | Tree-based | Ordinal / Label encoding |
| High (> 50) | Any | Frequency or target encoding |
| High (> 50) | Neural net | Embedding layer |
# One-hot
df = pd.get_dummies(df, columns=["color"], drop_first=True)
# Frequency encoding
freq = df["country"].value_counts() / len(df)
df["country_freq"] = df["country"].map(freq)
# Target encoding (use cross-validation to avoid leakage)
from category_encoders import TargetEncoder
encoder = TargetEncoder(cols=["city"])
df["city_encoded"] = encoder.fit_transform(df["city"], df["target"])
Step 4: Extract features from dates and text
Datetime columns: Raw timestamps carry no signal. Extract structured components.
df["signup_hour"] = df["signup_ts"].dt.hour
df["signup_dow"] = df["signup_ts"].dt.dayofweek # 0=Mon, 6=Sun
df["signup_is_weekend"] = df["signup_dow"].isin([5, 6]).astype(int)
df["days_since_signup"] = (pd.Timestamp.now() - df["signup_ts"]).dt.days
Text columns: Start with TF-IDF for structured text. Move to embeddings when semantic meaning matters.
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=200, ngram_range=(1, 2), stop_words="english")
text_features = tfidf.fit_transform(df["description"])
Step 5: Create interaction and aggregate features
Zheng and Casari's insight: the most predictive features are often combinations of existing ones, not raw columns.
Interaction features: Multiply or divide two numeric columns when their ratio or product has domain meaning.
df["revenue_per_session"] = df["revenue"] / (df["session_count"] + 1)
df["cost_per_click"] = df["spend"] / (df["clicks"] + 1) # +1 avoids division by zero
Aggregate features (group-level statistics): Capture behavior relative to a group.
group_stats = df.groupby("user_id")["purchase_amount"].agg(["mean", "std", "count"]).reset_index()
group_stats.columns = ["user_id", "user_avg_purchase", "user_std_purchase", "user_purchase_count"]
df = df.merge(group_stats, on="user_id", how="left")
Lag features for time series:
df = df.sort_values(["user_id", "date"])
df["revenue_lag_7d"] = df.groupby("user_id")["revenue"].shift(7)
df["revenue_rolling_30d"] = df.groupby("user_id")["revenue"].transform(lambda x: x.rolling(30).mean())
Step 6: Validate features and check for leakage
Before training, run these checks:
- Leakage check: Can any feature be computed from the target or from future data? Flag and remove.
- Correlation check: Drop features with correlation > 0.95 to another feature.
- Importance check: Train a simple Random Forest and check feature importances. Drop zero-importance features.
- Missing rate check: Any feature > 30% missing should be dropped or imputed before inclusion.
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
importance = pd.Series(model.feature_importances_, index=X_train.columns).sort_values(ascending=False)
print(importance.head(20))
Document the final feature set with a name, source column(s), transform applied, and rationale.
Anti-Patterns
1. One-hot encoding high-cardinality categoricals
Bad: One-hot encoding a "city" column with 500 unique values - creates 500 sparse binary columns.
Good: Use frequency encoding or target encoding. Reserve one-hot for columns with fewer than 10 unique values.
2. Using the full dataset to fit encoders
Bad: Fitting a TF-IDF or TargetEncoder on the entire dataset before splitting into train/test.
Good: Fit only on training data. Apply (transform) to test data. This prevents leakage from test labels.
3. Adding features without checking importance
Bad: Engineering 50 new features and feeding all of them to the model.
Good: Run a quick feature importance check. Remove zero-importance features before final training.
4. Ignoring domain knowledge
Bad: Letting the model figure out that revenue/sessions is meaningful by including both raw columns.
Good: Create the ratio feature explicitly. Domain-derived features outperform raw features in most tabular settings.
Quality Checklist