Bank Account Fraud Detection

Problem

Data description

Dataset source

High-level overview

Common preprocessing steps

• Train/test split: first 6 months for training, last 2 months for testing (794989 / 205011 split).

• Drop the month column.

• Drop the device_fraud_count column; it is constant-valued and therefore useless for prediction.

• One-hot encode categorical columns.

Methodology

Logistic Regression

Additional data preprocessing

• Replace all -1 values in the dataset (which indicate missing values) by setting them to 0.

• Normalize all numerical variables by subtracting their mean and dividing by their standard deviation.

• We obtain 49 features.

Final model training and evaluation

K-Nearest Neighbor

Data preprocessing

• Train/test split: for training data, we kept all fraudulent samples but downsampled non-fraudulent samples to have a ratio of 1:10, resulting in a training size of 89661 (fraud = 8151, non_fraud = 81510).

• For columns prev_address_months_count, bank_months_count, and session_length_in_minutes, we added missingness indicators for each to handle -1 as a missing value and replaced -1 with NaN to impute numerically. For all NaN values, we used SimpleImputer(strategy="median").

• All features were standardized using the training set statistics and applied to both training and test sets.

Cross-validation and hyperparameter tuning

• Cross-validation: we used StratifiedKFold with n_splits = 5 to preserve class proportions in each fold.

• Hyperparameter grid:
• k ∈ {3, 5, 7, 9, 11, 21, 31, 41, 51, 61, 81, 101, 121}
• weights ∈ {"uniform", "distance"}
In total, 26 hyperparameter combinations were evaluated.

• For each fold:
• Train kNN with the specific (k, weights) combination.
• Choose the best threshold from threshold_grid = {0.01, 0.02, …, 0.50} that yields the best F2 score and record the corresponding threshold and hyperparameter pair.

• After all 5 folds, we average the best F2 score and select the corresponding k value and weights with the optimal threshold value.

Final model training and evaluation

• We trained the final kNN model with the selected hyperparameters (k = 81, weights = "distance") on a class-balanced training subset of the full training set from months 0–5.

• Evaluation was done on the test set using the optimal threshold τ = 0.134, and we reported F2, recall, precision, and accuracy.

Linear-Kernel Support Vector Machine

Additional data preprocessing

• Create indicator columns for columns with missing values.

• Standardize columns.

• Mean imputation of missing values.

After preprocessing

• Training data count: 794989.

• Test data count: 205011.

• Number of features: 49.

Modelling

• Model: LinearSVC(C=C, dual=False, class_weight={0: 1, 1: R}) from sklearn.svm.

• Hyperparameters: C, R.

• Coarse-grained, stratified 3‑fold logspace search for C ∈ {0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0} and R ∈ {1.0, 10.0, 100.0, 1000.0}, using F2 as the evaluation metric.

• Fine-grained, stratified 3‑fold linespace search in the best-magnitude neighborhoods for C and R, again using F2.

• Train the final model on all of the training data using the best (C, R) pair.

RBF-Kernel Support Vector Machine Ensemble

Additional data preprocessing

• Identical to the linear-kernel SVM.

Modelling

• Fix N = 7.

• Divide the training dataset into stratified folds of size ≈ 10000. Create 3 splits of 8 stratified folds each. For each split, 7 of the folds are used to train 7 SVM models, and the remaining fold is used as a validation set. This requires 24 folds; we have approximately folds available.

• Hyperparameters and ranges:
• C ∈ [10^-2, 10]
• R ∈ [10, 1000]
• gamma ∈ [10^-3, 1] (RBF kernel)
• K ∈ {1, …, 7} (the number of learners needed to make a positive prediction)

• Because an exhaustive grid search is infeasible, we use random search: we randomly sample 10 different (C, R, gamma) combinations from their logspaces and combine each with K ∈ {1, …, 7}, producing 4‑D candidates (C, R, gamma, K).

• After finding the optimal (C, R, gamma, K) combination, we create 7 stratified folds of size ≈ 20000, train 7 RBF SVM models with (C, R, gamma), and define the final ensemble to predict 1 if at least K of the 7 models predict 1.

Tree-Based Methods – Decision Trees

Additional data preprocessing

• Replace all -1 values in the dataset (which indicate missing values) by setting them to 0.

• Normalize all numerical variables by subtracting their mean and dividing by their standard deviation.

Cross-validation

• Implement a 10‑fold stratified cross-validation using recall to find the best model.

• Use the gini criterion for impurity and set the number of features to consider at each split to sqrt(n_features).

• The best depth of the decision tree was max_depth = 90.

Tree-Based Methods – Random Forest

Data preprocessing

• Train/test split: all observations from months 0–5 as the training set and months 6–7 as the test set.

• For prev_address_months_count, bank_months_count, and session_length_in_minutes, add missingness indicators, replace -1 with NaN, and use median SimpleImputer.

• Standardize all features using the training set statistics.

Cross-validation and hyperparameter tuning

• Cross-validation: StratifiedKFold with n_splits = 3.

• Hyperparameter grid:
• n_estimators ∈ {100, 300, 500}
• max_depth ∈ {None, 10, 20}
• min_samples_leaf ∈ {1, 5}
In total, 18 hyperparameter combinations.

• For each combination and fold, train a Random Forest, predict with threshold 0.5, compute F2, and average F2 across folds.

• Best setting: n_estimators = 500, max_depth = 10, min_samples_leaf = 1.

Final model training and evaluation

• Train the final Random Forest model with the best hyperparameters on the full training set (months 0–5).

• Evaluate on the test set and report F2, recall, precision, and accuracy.

Tree-Based Methods – XGBoost

Additional data preprocessing

• No additional preprocessing steps beyond the common ones.

After preprocessing

• Number of features: 45.

Modelling

import xgboost as xgb

params = {
    "learning_rate": learning_rate,
    "max_depth": max_depth,
    "min_child_weight": min_child_weight,
    "subsample": subsample,
    "colsample_bytree": colsample_bytree,
    "scale_pos_weight": scale_pos_weight,
    "objective": "binary:logistic",
    "eval_metric": "aucpr",
    "tree_method": "hist",
    "verbosity": 0,
}

model = xgb.train(
    params=params,
    dtrain=dtrain,
    num_boost_round=best_iteration_int,
)

y_test_prob = model.predict(dtest)
y_test_pred = (y_test_prob >= best_threshold).astype(int)

• learning_rate (range: logspace from 0.03 to 0.2)

• max_depth (range: integers in [3, 8])

• min_child_weight (range: [1, 10])

• subsample (range: [0.7, 1.0])

• colsample_bytree (range: [0.6, 1.0])

• scale_pos_weight (range: logspace [10, 200])

• best_threshold (range: linspace [0, 1])

Neural Networks

Data preprocessing

• Replace all -1 values (missing values) with 0.

• Normalize all numerical variables by subtracting their mean and dividing by their standard deviation.

• One-hot encode categorical variables.

• Drop device_fraud_count as it is constant-valued.

• Train/test split: first 6 months for training, last 2 months for testing.

• Number of features: 49.

Model architecture and loss

• Fully connected feed-forward neural network with three hidden layers, with dropout on the first and last layers and ReLU activations.

• Output layer: single neuron with linear output, followed by a sigmoid to convert logits to probabilities.

• To address class imbalance, we tried combinations of sampling and loss functions:
1. Oversampling with surrogate loss.
2. Oversampling with BCEWithLogitsLoss.
3. No oversampling with BCEWithLogitsLoss.

• The best configuration was no_oversampling + BCEWithLogitsLoss with positive-class weight pos_weight = (# of negatives / # of positives).

Hyperparameter tuning and training

• Small grid search with 27 combinations:
• Hidden layer sizes: (64, 32, 16), (128, 64, 32), (256, 128, 64)
• Dropout probability: 0.2, 0.3, 0.5
• Learning rate: 1e-4, 1e-3, 3e-3

• For each configuration:
• Optimizer: Adam with weight_decay = 1e-4.
• Batch size: 4096.
• Up to 20 epochs with early stopping based on validation F2.
• After each epoch, compute validation probabilities, sweep a threshold grid between 0.01 and 0.99, and record the threshold that maximizes F2.

• Best setting: hidden_sizes = (256, 128, 64), dropout = 0.3, learning_rate = 0.001 with optimal threshold τ = 0.84.

Final evaluation

Final Results

Interpretable feature analysis

Positive signals for fraud

• Missing values in prev_address_months_count (months in previous registered address) are a relatively strong signal for fraud. One potential reason is that fraudsters tend to avoid providing a traceable address history.

• The device OS being "Windows" is a relatively strong signal for fraud. One possible explanation is that fraudsters often operate from cheap and widely available Windows environments.

Negative signals for fraud

• Keep_alive_session = 1 (the user keeping the session alive on logout) is a negative indicator of fraud. Fraudsters may prefer not to stay connected longer than necessary.

• Higher name_email_similarity is a negative indicator of fraud. Fraudsters often use randomly generated or non-identifying email handles.

• phone_home_valid = 1 (valid home phone numbers) is a negative indicator of fraud, as fraudsters may provide fake or burner numbers.

• Has_other_cards = 1 (applicant has other cards with the same bank) is a negative signal of fraud, possibly because multiple cards under the same identity increase detectability.

• Certain housing_status categories (for example, "BE", "BB", "BC" in the anonymized encoding) are negative indicators of fraud, but since these are anonymized, we cannot interpret them semantically.

Conclusions

Contributions

• Jerry Bao (@bao-jerry)

• Saim Zafar (@s29zafar)

Source