A statistical evaluation of machine learning classifiers for credit risk prediction

Introduction

The significance of assets in banking is reflected in the profitability derived from issuing credit cards to customers. Credit scoring endeavors focus on identifying the impact of various applicants' characteristics associated with illicit behavior and payment defaults. The central aim of any banking system is to distinguish valuable stakeholders from whom they can achieve optimal returns on investments in assets (Kumar, 1998) [1] . The banking sector significantly influences credit card holders through its myriad services. Financial institutions issue credit cards only after thorough verification and validation process. However, this rigorous scrutiny does not always guarantee that the most deserving candidates receive these credit cards (Bhattacharya, 2021) [2] . Credit scoring model predictions have become integral part of the commercial sector. Credit scoring involves a suite of decision models and underlying techniques that aid lenders in issuing consumer credit (Thomas et al., 2002) [3] . This method, primarily utilized in consumer credit, assists credit granters in making informed lending decisions (Abdou and Pointon, 2011) [4] . It functions as a crucial decision-making tool for lenders in allocating consumer credit (Halima and Humira, 2012) [5] . Credit scoring, a traditional decision-making model, assesses the risk associated with credit products, such as credit cards and loans, by analyzing the historical data of applicants. This process aids credit lenders in making well-informed decisions regarding the issuance of credit products (Altman and Saunders, 1998) [6] .

In contemporary practice, financial institutions employ a variety of risk assessment tools and techniques to mitigate risk. The analysis of customer credit data is crucial in identifying potential defaulters. The effectiveness of a credit scoring model is paramount for managing and establishing credit risk, which is essential for the institution's profitability. Research indicates that ensemble-based approaches are particularly effective in credit score assessment (Dastile et al., 2020) [7] . Credit scoring systems are predominantly used to estimate the probability of loan default. The primary objective of this paper is to assess and compare the predictive power of different classifier models in credit scoring. This involves analyzing the strengths and weaknesses of these models. We also provide recommendations for financial institutions regarding model selection and the automatic identification of defaulters using advanced machine learning methods. Thus, the purpose of this paper is to identify key characteristics of customers to support credit decisions, compare various variable selection methods, make credit decisions using the generalized linear model (GLM), specifically Logistic Regression model, Linear discriminant analysis & support vector classification and compare the performance of GLM with machine learning methods (Hand and Henley, 1997) [8] .

Methodology

The data collection process varies depending on the type of data. Datasets can be sourced from files, databases, sensors, and other sources. For this study, we sourced our data from the Kaggle website: Kaggle Credit Risk Dataset. Data preprocessing is a crucial step in machine learning, significantly impacting the accuracy of model building. According to the 80/20 rule in machine learning, 80% of the time is spent on data pre-processing, and 20% on analysis. Data preprocessing involves cleaning raw data to produce clean data suitable for training models. Effective data pre-processing is essential for achieving good results from machine learning models. To train the best-performing model using the pre-processed data, we utilized supervised learning techniques (Tsai and Wu, 2008) [9] .

Supervised Learning

Supervised learning involves training an AI system with labeled data, where each data point is tagged with the correct label. Supervised learning can be divided into two primary categories: "Classification" and "Regression." Classification: A classification problem occurs when the target variable is categorical, meaning the output can be classified into distinct classes, such as "default" or "nondefault." The commonly used classification algorithms are:

Gaussian (Normal) Naive Bayes

The Gaussian (Normal) Naive Bayes can be adapted to handle real-valued attributes by assuming a Gaussian (Normal) distribution. This classification method simplifies the model by requiring only the estimation of the mean and standard deviation from the training data (Saunders and Allen 2010) [10] . Where

, n is the number of instances

Logistic Regression

Logistic regression is similar to multiple linear regression, with the primary difference being that the outcome is binary and a structured model approach. It is a popular method due to its fast computational speed and the ability to produce a model that allows for rapid scoring of new data.

Data and Pre-processing:

The dataset used is the Credit dataset from the machine learning data archive, containing 12 covariates (8 numerical, 4 categorical) and 32,581 observations. Each observation represents an individual customer, with the response variable indicating their classification (1 = "Bad" or 0 = "Good"), and the covariates providing various attributes related to the customer's personal or financial information.

A binary logistic model is fitted to the data using the logit link function. This models the classification of the 𝑖 𝑡ℎ customer as either good or bad using a Bernoulli random variable.

Where 𝑥 𝑖 is a vector of covariates associated with this customer.

The conditional expectation is then given by:

and this is associated to a linear predictor via the logit function,

i.e., 𝑙𝑜𝑔𝑖𝑡 𝜋 𝑖 = 𝑙𝑜 𝑔 ( 𝜋𝑖

where 𝛽 denotes the vector of parameters to be estimated. This estimation process utilizes iterative weighted least squares (IWLS), a method thoroughly explained in Davison (2003). The conditional joint probability of 𝑌 1 , 𝑌 2 … … . 𝑌 𝑛 , assuming conditional independence, can be expressed as:

This indicates that the probability distribution belongs to the exponential family. The choice of link function is primarily driven by the desire for straightforward interpretation of model parameters. The linear predictor model, we are fitting is given by: 𝜂 = 𝛽 0 + 𝛽 1 . 𝑝𝑒𝑟𝑠𝑜𝑛𝑒𝑙 + 𝛽 2 . 𝑒𝑑𝑐𝑎𝑡𝑖𝑜𝑛 + 𝛽 3 . ℎ𝑜𝑚𝑒 𝑙𝑜𝑎𝑛 + 𝛽 4 . 𝑚𝑒𝑑𝑖𝑐𝑎𝑙 + 𝛽 5 . 𝑣𝑒𝑛𝑡𝑢𝑟𝑒 + 𝛽 6. 𝑑𝑒𝑏𝑖𝑡 𝑐𝑜𝑛𝑠𝑜𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛 To assess which variables influence creditworthiness and their impact, we start by evaluating the significance of variable groups. This involves performing a likelihood ratio test by comparing the deviance of a full model with a reduced model where one group of variables is omitted. The test statistic is then compared to a 𝜒 2 distribution, where 𝑘 is the number of excluded parameters [4] .

Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA), a traditional technique for classification based on assumptions that, the data for each variable is normally distributed and all attributes have the same variance. LDA estimates the mean and variance for each class from the data. In a univariate scenario with two classes, the mean for each class is computed as the average of values in that class.

Here, 𝑋 𝑘 ̅̅̅ represents the mean value of 𝑥 for class 𝑘, and 𝑛 𝑘 denotes the number of instances in class 𝑘. Variance is computed across all classes as the average of the squared deviations of each value from the mean.

Support Vector Machine (SVM)

The core concept of a Support Vector Machine (SVM) is to identify the optimal hyperplane that maximizes the margin between two classes. In a two-dimensional space, this hyperplane is a line; in three dimensions, it is a plane; and in higher dimensions, it is a hyperplane. The objective is to find the best boundary that separates two classes.

Extreme Gradient Boost (XGBoost)

The XGBoost, or Extreme Gradient Boosting, typically uses decision trees as base learners. These trees are constructed iteratively until a stopping criterion is met. XGBoost is an ensemble method that employs Classification and Regression Trees (CART), where each tree contains real-valued scores at the leaves, which can be used for classification if necessary.

K-Nearest Neighbor's (KNN)

The k-nearest neighbors (KNN) algorithm is a nonparametric, supervised learning method used for classification and regression. It classifies or predicts based on the proximity of data points. For classification, a class label is assigned based on the majority vote among the nearest neighbors.

While "majority vote" is commonly used, it does not strictly require more than 50% of the votes, especially in multi-class problems. Precision: The percent (proportion) of predicted 1s that are actually 1's.

Random Forest

2.2 F1-Score / F1-Measure Classification accuracy is a popular metric because it provides a straightforward summary of a model's performance in a single value. However, to get a more nuanced view of a model's effectiveness, especially in the context of imbalanced datasets, the F-Measure (or F1 Score) is used. The F-Measure combines both precision and recall into a single metric. While precision measures the accuracy of positive predictions, and recall measures how well all positive instances are identified, neither metric alone provides a complete picture. A model might show high precision but poor recall, or vice versa. In summary, the F-Measure integrates both precision and recall into a comprehensive metric, providing a more holistic evaluation of a model's performance in scenarios where both metrics are important.

ROC Curve

The ROC curve is a graphical representation of a classifier's performance, showing the trade-off between recall (sensitivity) and specificity. The ROC curve illustrates how the balance between recall and specificity shifts as the classification threshold changes. A good classifier will achieve high recall while maintaining high specificity. In other words, it should correctly identify as many positive cases as possible without misclassifying too many negatives as positives. The AUC is a summary metric derived from the ROC curve. AUC measures the overall performance of the classifier:  An AUC of 1 indicates a perfect classifier that correctly classifies all positive and negative instances.  An AUC of 0.5 signifies a classifier with no discriminative power, equivalent to random guessing.

Variable Importance and the Dalex Package

The Dalex package is a valuable tool for interpreting machine learning models.

Results and Discussion:

We consider Kaggle's credit risk dataset for the analysis. We focused on two types of analysis (univariate and bi-variate analysis).

Univariate Analysis:

Univariate analysis is the simplest form of data analysis, focusing on a single variable. Its primary purpose is to describe the data and uncover patterns within it, without examining relationships between variables.

In this analysis, we utilized libraries such as Pandas and NumPy for data manipulation. We started by checking the shape, size, and information about the data, followed by obtaining summary statistics. We visualized the target variable using a pie chart, which illustrates the distribution of the two classes (0 and 1) in terms of percentage. Additionally, we analyzed the frequency of unique values in each categorical variable to better understand the dataset.

Bivariate Analysis

Bivariate analysis investigates the relationship between two variables to understand their interaction and the nature of their association. This analysis aims to uncover whether there are any significant relationships, discrepancies, or causes of differences between the two variables.

Key Objectives

 Explore Relationships: Determine how two variables are related.  Identify Discrepancies: Spot any inconsistencies or variations between the variables.  Assess Relationship Depth: Gauge the strength and significance of the relationship.

In our analysis, we used histograms to evaluate the distribution of the data. This visualization helps in understanding how values are spread across different ranges, providing insights into the data's overall distribution. From the Figure 3 we observed that neither feature of the graphs follows a normal distribution. Consequently, we decided to fill the missing values (NaN's) with the median values for both the loan interest rate and employment length features. Additionally, we identified some unrelated data entries, which were cleaned using cross-tabulations and median values. Notably, clients who had not defaulted on loans had a median interest rate that was 4% lower than those who had defaulted. Given that issuing loans to high-risk clients can have adverse outcomes for both the lender and the client, our goal is to enhance credit risk modeling using machine learning algorithms to mitigate these risks. Interpretation: There is no association between variable loan status and all other categorical variables. Interpretation: There is no significant difference between loan status and all other variables. The difference between train accuracy and test accuracy is less for LR, LDA and SVC classifier methods as compared to all other classifier methods. But SVC showed better performance on both the training and test datasets. Among these classifier methods, SVC is considered as the best classifier due to its consistent performance across all evaluation metrics. Checking Account Balance: The odds of creditworthiness increase with higher checking account balances. Compared to those with a zero balance, the odds are higher by factors of 𝑒 𝛽 0-1 = 1.662, 𝑒 𝛽 1 = 3.015, and 𝑒 𝛽 0 = 6.25 for various account balances. Notably, those with no checking account have odds 𝑒 𝛽 0 +𝛽 0-1 = 3.76 times higher than the 0-1 balance group, which suggests an unexpected trend where no checking account is associated with higher creditworthiness.

Chi-Square test for independence

Student's t-test for significant difference Population 1: Defaulted vs Population 2: Non defaulted

Personnel Variable: Consumers whose personnel category is "purchase of a personnel" have odds of creditworthiness 𝑒 𝛽(𝑝𝑒𝑟𝑠𝑜𝑛𝑛𝑒𝑙)-𝛽(𝑒𝑑𝑢𝑐𝑎𝑡𝑖𝑜𝑛) = 5.497 times higher compared to those whose personnel category is "education."

Loan Duration: Each additional month of loan duration reduces the odds of creditworthiness by a factor of 𝑒 𝛽(𝑒𝑑𝑢𝑐𝑎𝑡𝑖𝑜𝑛) = 0.958 reflecting the increased default risk associated with longer loans.

Credit History

Consumers with a "critical" credit history show significant increases in creditworthiness odds. 4 and 5 together, gives that drop-out loss against the variables of importance. We see that in both figures; the top four variables of importance are the same. It indicates loan percent income, loan interest rate, loan amount and person home ownership own are the most significant and contributing variables towards sanction of the loan.

Conclusions

The credit scoring enables lenders to create a scorecard where each characteristic is assigned a weight, and the aggregate score determines an individual's creditworthiness. The decision to approve or reject an applicant is made by setting a cut-off level (threshold) corresponding to a specific value of the estimated probability of default (PD). Applicants with a PD above this threshold are not granted credit. Loan percent income, person home ownership RENT, loan int rate, loan amount these are the key characteristics of a customer to support credit decision. Based on the analysis, we conclude that, for SVC, LR, and RF, the bold values indicate higher variables of importance. We also conclude that, the loan percent income is most significant variable and home ownership, the other is the least significant. Among statistical models (LDA, LR, GNB), Gaussian Naïve Bayes has higher ROC AUC based on LR_TOP_S1 variable set. The SVC classifier is considered as the best classifier with consistent evaluation metrics across train and test split. Drop out loss has been calculated for support vector classification (SVC) and logistic regression for LR_TOP_S1 variable. The four variables such as loan_percent_income, person_home_ownership_RENT, loan_int_rate, and loan_amnt are the most significant variables.

References

1. Kumar K. Banking profitability and asset management. International Journal of Banking and Finance. 1998;22(2):89-102.
2. Bhattacharya P. The impact of rigorous scrutiny in credit card issuance. Journal of Financial Services Research. 2021;50(1):123-139.
3. Thomas LC, Edelman DB, Crook JN. Credit scoring and its applications. Philadelphia: SIAM; 2002.
4. Abdou H, Pointon J. Credit scoring, statistical techniques and evaluation criteria: A review of the literature. Intelligent Systems in Accounting, Finance and Management. 2011;18(2-3):59-88.
5. Altman EI, Saunders A. Credit risk measurement: Developments over the last 20 years. Journal of Banking and Finance. 1998;21(11-12):1721-1742.
6. Dastile X, Celik T, Potsane M. Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing. 2020;91:106263.
7. Hand DJ, Henley WE. Statistical classification methods in consumer credit scoring: A review. Journal of the Royal Statistical Society. Series A, Statistics in Society. 1997;160(3):523-541.
8. Tsai CF, Wu JW. Using neural network ensembles for bankruptcy prediction and credit scoring. Expert Systems with Applications. 2008;34(4):2639-2649. doi:10.1016/j.eswa.2007.05.019
9. Saunders A, Allen L. Credit risk management in and out of the financial crisis: New approaches to value at risk and other paradigms. Hoboken, NJ: Wiley; 2010.