Algorithm-Sensitive Feature Weighting using Mutual Information and ReliefF for Heart Disease Classification

Abstract

Heart disease remains a critical global health issue requiring reliable early prediction for clinical decision-making. This study evaluates the effect of feature selection and feature weighting on the performance of machine learning models with different learning mechanisms, namely K-Nearest Neighbors (KNN), Naïve Bayes (NB), and Logistic Regression (LR), for heart disease prediction. An algorithm-sensitive comparative framework was applied using global feature relevance estimation (Mutual Information), local feature relevance estimation (ReliefF), feature selection, and feature weighting, with raw features as baseline. Experiments were conducted on the Cleveland Heart Disease dataset using stratified 5-fold cross-validation. The results show that ReliefF-based feature weighting achieves the best performance for Naïve Bayes (accuracy = 0.8085; F1-score = 0.7774), while Logistic Regression attains the highest overall performance under the baseline (accuracy = 0.8547; F1-score = 0.8324). Feature selection improves KNN performance due to reduced sensitivity to irrelevant features. These findings indicate that the effectiveness of feature importance strategies depends on model-specific learning behavior, where feature weighting benefits probabilistic models, while feature selection is more effective for distance-based models. This study contributes an algorithm-sensitive evaluation perspective for aligning feature importance strategies with machine learning model characteristics to improve heart disease prediction performance.