Model Performance

How well the trained model detects fraud on unseen test data

Selected model

Random Forest

Highest F1 score (tie-break: recall) · trained 2026-10-21 17:38:05

19,961 rows · 4.0% fraud
15,968 train / 3,993 test
Accuracy 98.4%

Share of all predictions that were correct. Misleading when fraud is rare.

Precision 81.9%

Of transactions flagged as fraud, how many really were fraud.

Recall 76.2%

Of all real fraud, how much the model caught. Most important for banks.

F1 score 79.0%

Balance of precision and recall. Used to pick the best model.

ROC-AUC 98.0%

How well the model ranks fraud above genuine (1.0 is perfect).

Confusion matrix

Predicted legit Predicted fraud
Actual legit 3,806True negatives 27False alarms
Actual fraud 38Missed fraud 122Fraud caught

Missed fraud (false negatives) costs the bank money. False alarms annoy genuine customers.

Top features (what the model relies on)

  • amount to avg ratio 21.9%
  • txn count last 24h 18.3%
  • account age days 12.2%
  • amount 9.1%
  • transaction hour 8.6%
  • location = home city 4.8%
  • device type = new device 4.0%
  • avg transaction amount 3.7%
  • is night 2.7%
  • location = international 2.2%
  • location = unknown 2.0%
  • merchant category = gift cards 1.4%

Model comparison

ModelAccuracyPrecisionRecallF1ROC-AUC
Logistic Regression 0.931 0.357 0.894 0.511 0.968
Random Forest Best 0.984 0.819 0.762 0.790 0.980

Why accuracy is misleading

Only about 4% of transactions are fraud. A useless model that always answers “legitimate” would score about 96% accuracy while catching zero fraud.

That is why we compare models by recall (fraud caught) and F1 (balance with false alarms), and handle imbalance with class_weight="balanced".