NLP-Cyber-Harm-Detection

NLP Fraud/Scam Detection Baseline Models

GitHub Repository

A comprehensive implementation for fraud and scam detection using advanced Natural Language Processing techniques. This project leverages state-of-the-art transformer-based models for accurate fraud classification with explainable AI-powered reasoning. The latest BART Joint Model provides simultaneous classification and detailed contextual explanations in a single unified architecture.

📁 Repository

🔗 GitHub Repository: https://github.com/RockENZO/NLP-Cyber-Harm-Detection.git

🎯 Project Overview

This project implements advanced models for detecting fraudulent content (scams, phishing, spam) in text data using state-of-the-art transformer architectures. Key approaches include:

Unified Classification + Reasoning (FLAN-T5)

We added a unified, text-to-text approach using FLAN-T5 that produces both the class label and a concise explanation in a single generation. This works directly with your existing CSV (final_fraud_detection_dataset.csv) that has text and detailed_category columns. If your dataset also has rationale fields such as explanation or rationale, the trainer will use them; otherwise, it auto-synthesizes short label-specific explanations.

Key files:

Install/update dependencies:

Quick start (macOS, zsh): 1) Train the unified model (adjust args as needed):

Apple Silicon acceleration: the trainer auto-selects CUDA, MPS, or CPU. You can force MPS with --device mps.

Notes:

🌟 BART Joint Model - Enhanced Classification + Contextual Reasoning (LATEST)

NEWEST: The project now features a state-of-the-art BART joint model (unified-bart-joint-enhanced) that combines classification and contextual reasoning in a single unified architecture:

Key Features:

Architecture Highlights:

Key Files:

Sample Output:

Input: "Congratulations! You've won a $1000 gift card. Click here to claim now!"

Predicted: reward_scam
Confidence: 0.987 (98.7%)
Enhanced Reasoning: The message contains typical reward scam indicators including 
unsolicited prize announcement, urgency ("claim now"), request for immediate action 
via suspicious link, and promises of high-value rewards without prior participation. 
These are classic tactics used to lure victims into providing personal information 
or making fraudulent payments.

Performance:

Kaggle Deployment Ready:

The notebook demos/test-unified-bart-joint-enhanced.ipynb is fully Kaggle-compatible:

  1. Upload unified-bart-joint-enhanced folder as a Kaggle Dataset
  2. Attach dataset to notebook
  3. Run evaluation with automatic environment detection
  4. Get comprehensive metrics, visualizations, and export capabilities

⚡ DistilBERT Model Highlights

NEW: The project now includes a production-ready DistilBERT model with significant advantages:

The DistilBERT model is trained for multiclass classification, providing granular fraud type detection rather than just binary fraud/legitimate classification.

📁 Project Structure

├── README.md                           # This comprehensive documentation
├── requirements.txt                    # Python dependencies
├── final_fraud_detection_dataset.csv  # Training dataset (Git LFS)
├── models/                            # Saved trained models
│   ├── bert_model/                    # Trained BERT model files
│   ├── bert_tokenizer/               # BERT tokenizer files
│   ├── distilbert_model/             # Trained DistilBERT model files
│   ├── distilbert_tokenizer/         # DistilBERT tokenizer files
│   ├── unified-bart-joint-enhanced/  # BART joint classification + reasoning model
│   │   ├── config.json               # Model configuration
│   │   ├── model.safetensors         # Trained model weights
│   │   ├── generation_config.json    # Generation parameters
│   │   ├── tokenizer files           # BART tokenizer
│   │   └── checkpoint-*/             # Training checkpoints
│   ├── flan-t5-base/                 # FLAN-T5 model files
│   └── unified-flan-t5-small/        # Unified FLAN-T5 model
├── training/                          # Training scripts and notebooks
│   ├── baseline_fraud_detection.py   # Traditional ML baseline models
│   ├── bert_fraud_detection.py       # BERT-based classifier
│   ├── fraud_detection_baseline.ipynb # Interactive Jupyter notebook
│   ├── kaggle_fraud_detection.ipynb  # Kaggle-optimized training notebook
│   ├── unified-bart-joint-enhanced-reasoning.ipynb # BART joint model training
│   ├── unified_t5_fraud_kaggle.ipynb # FLAN-T5 training notebook
│   └── unified_t5_fraud.py           # FLAN-T5 training script
├── demos/                             # Demo and testing tools
│   ├── fraud_detection_demo.py       # Full-featured demo script
│   ├── fraud_detection_demo.ipynb    # Interactive demo notebook
│   ├── quick_demo.py                 # Quick verification script
│   ├── test-unified-bart-joint-enhanced.ipynb # BART joint model testing (Kaggle-ready)
│   ├── unified_flan_t5_demo.ipynb    # FLAN-T5 demo notebook
│   └── unified_reasoning_demo.py     # Unified reasoning demo script
├── reasoning/                         # AI-powered reasoning pipeline  
│   ├── GPT2_Fraud_Reasoning.ipynb    # GPT2-based reasoning analysis
│   └── KaggleLLMsReasoning.ipynb     # Local reasoning notebook
├── docs/                              # Documentation
│   └── nlp_terms_explanation.md      # NLP concepts explanation
├── runs/                              # Training run outputs and analysis results
│   ├── fraud_analysis_results_20250916_155231.csv
│   ├── fraud-detection-kaggle-training-bert-run.ipynb
│   ├── gpt2_fraud_analysis_20250917_034015.csv
│   ├── LLMsReasoningResultVisualization.ipynb
│   ├── MultipleLLMsReasoning(small-models).ipynb
│   └── LLMsStats/                     # LLM performance comparison charts
│       ├── llm_category_heatmap.png
│       ├── llm_comparison_table.csv
│       ├── llm_performance_comparison.png
│       ├── llm_quality_radar.png
│       ├── llm_size_performance.png
│       ├── llm_speed_quality_scatter.png
│       ├── llm_model_size_comparison.png    # Model size vs performance charts
│       └── llm_speed_quality_bubble.png     # Speed vs quality bubble chart
├── .gitattributes                     # Git LFS configuration
├── .gitignore                         # Git ignore rules
└── .git/                             # Git repository

🚀 Quick Start

Option 1: Use BART Joint Model (Latest & Best) ⭐

Test the enhanced BART model with classification + contextual reasoning:

  1. Install Dependencies 
    pip install torch transformers pandas numpy matplotlib seaborn jupyter
  2. Test BART Joint Model 
    # Open the comprehensive testing notebook
jupyter notebook demos/test-unified-bart-joint-enhanced.ipynb

    The notebook includes:

    • Model loading and configuration
    • Sample predictions with detailed reasoning
    • Batch evaluation with metrics
    • Confusion matrix visualization
    • Confidence analysis
    • Interactive testing interface
    • CSV export for results
  3. Kaggle Deployment
    • Upload models/unified-bart-joint-enhanced/ as a Kaggle Dataset
    • Upload demos/test-unified-bart-joint-enhanced.ipynb to Kaggle
    • Update MODEL_DIR path in the notebook
    • Run with GPU accelerator for best performance

Option 2: Use Traditional Pre-trained Models

If you have already trained BERT/DistilBERT models:

  1. Install Dependencies 
    pip install torch transformers pandas numpy matplotlib seaborn jupyter
  2. Quick Test Your Model 
    python demos/quick_demo.py
  3. Interactive Demo Notebook 
    jupyter notebook demos/fraud_detection_demo.ipynb
  4. Full Demo Script 
    python demos/fraud_detection_demo.py
  5. Local AI Reasoning 
    # Upload KaggleGPTReasoning.ipynb to Kaggle
# Enable GPU accelerator
# Run all cells for fraud detection + AI explanations
# Download results - no API costs

    📊 LLM Performance Analysis: Check runs/LLMsStats/ for performance comparisons.

Option 3: Train from Scratch

  1. Install Dependencies 
    pip install -r requirements.txt
  2. Run Traditional ML Baselines 
    python training/baseline_fraud_detection.py
  3. Run BERT Baseline (requires more computational resources) 
    python training/bert_fraud_detection.py
  1. Upload final_fraud_detection_dataset.csv to Kaggle
  2. Create a new notebook and copy the code from training/fraud-detection-kaggle-training-bert-run.ipynb
  3. Enable GPU accelerator for fast BERT training
  4. Download the trained models from Kaggle output
  5. Use the demo scripts to test your trained model

Note: The dataset is stored with Git LFS due to its size (~158MB). Clone with git lfs pull to download the full dataset. Large model files like model.zip are excluded from git to keep the repository size manageable.

📊 LLM Performance Analysis Results

The runs/LLMsStats/ directory contains LLM model analysis for fraud reasoning tasks.

📊 Models Implemented

1. Traditional ML Baselines (training/baseline_fraud_detection.py)

2. BERT-Based Classifier (training/bert_fraud_detection.py)

3. DistilBERT-Based Classifier (training/kaggle_fraud_detection.ipynb)

4. BART Joint Classification + Contextual Reasoning (training/unified-bart-joint-enhanced-reasoning.ipynb)

5. Unified FLAN-T5 Classification + Reasoning (training/unified_t5_fraud.py)

6. Kaggle Training Notebook (runs/fraud-detection-kaggle-training-bert-run.ipynb)

7. AI-Powered Reasoning Pipeline (reasoning/)

🤖 LLM Model Selection for Reasoning

LLM Model Size vs Performance

LLM Speed vs Quality

🎮 Demo and Testing Tools

Once you have a trained model, use these tools to test and demonstrate fraud detection capabilities:

2. fraud_detection_demo.py

3. quick_demo.py

🎯 Fraud Types Detected

Your trained model can detect these 9 classes:

  1. legitimate - Normal, safe messages
  2. phishing - Attempts to steal credentials/personal info
  3. tech_support_scam - Fake technical support
  4. reward_scam - Fake prizes/lottery winnings
  5. job_scam - Fraudulent employment opportunities
  6. sms_spam - Unwanted promotional messages
  7. popup_scam - Fake security alerts
  8. refund_scam - Fake refund/billing notifications
  9. ssn_scam - Social Security number theft attempts

💡 Demo Usage Examples

BART Joint Model (Classification + Contextual Reasoning) ⭐ LATEST

# Load the enhanced BART joint model
from pathlib import Path
from transformers import AutoTokenizer, AutoConfig
import torch

# Load model and tokenizer
MODEL_DIR = Path('models/unified-bart-joint-enhanced')
tokenizer = AutoTokenizer.from_pretrained(str(MODEL_DIR))
config = AutoConfig.from_pretrained(str(MODEL_DIR))
joint_model = BartForJointClassificationAndGeneration.from_pretrained(str(MODEL_DIR), config=config)
joint_model.eval()

# Single prediction with reasoning
def predict_with_reasoning(text):
    instruction = (
        'Analyze this message and classify it into one of these categories: '
        'job_scam, legitimate, phishing, popup_scam, refund_scam, reward_scam, '
        'sms_spam, ssn_scam, tech_support_scam. '
        'Then explain your reasoning by identifying specific suspicious elements.\n\n'
        f'Message: {text}'
    )
    
    inputs = tokenizer([instruction], return_tensors='pt', truncation=True, max_length=256)
    
    with torch.no_grad():
        # Get classification
        enc_out = joint_model.model.encoder(**inputs, return_dict=True)
        pooled = joint_model.pooled_encoder(enc_out.last_hidden_state, inputs['attention_mask'])
        cls_logits = joint_model.classifier(pooled)
        pred_label = ID2LABEL[cls_logits.argmax(-1).item()]
        confidence = torch.softmax(cls_logits, dim=-1).max().item()
        
        # Generate reasoning
        gen_ids = joint_model.generate(
            **inputs,
            max_new_tokens=128,
            num_beams=5,
            length_penalty=1.0,
            no_repeat_ngram_size=3
        )
        reasoning = tokenizer.decode(gen_ids[0], skip_special_tokens=True)
    
    return {
        'label': pred_label,
        'confidence': confidence,
        'reasoning': reasoning
    }

# Example usage
result = predict_with_reasoning("Congratulations! You've won $1000. Click to claim now!")
print(f"🎯 Predicted: {result['label']}")
print(f"📊 Confidence: {result['confidence']:.3f}")
print(f"💡 Reasoning: {result['reasoning']}")

Output Example:

🎯 Predicted: reward_scam
📊 Confidence: 0.987
💡 Reasoning: The message contains typical reward scam indicators including unsolicited 
prize announcement, urgency ("claim now"), request for immediate action via suspicious 
link, and promises of high-value rewards without prior participation. These are classic 
tactics used to lure victims into providing personal information or making fraudulent payments.

Traditional Models (Classification Only)

Single Prediction

from demos.fraud_detection_demo import FraudDetectionDemo

demo = FraudDetectionDemo()
result = demo.predict_single("Your account has been compromised! Click here now!")
print(f"Prediction: {result['predicted_class']}")
print(f"Confidence: {result['confidence']:.4f}")
print(f"Is Fraud: {result['is_fraud']}")

Batch Prediction

texts = [
    "Meeting at 3 PM tomorrow",
    "URGENT: Verify your SSN now!",
    "You won $10,000! Send fee to claim"
]

results = demo.predict_batch(texts)
for result in results:
    print(f"{result['predicted_class']}: {result['text']}")

Interactive Jupyter Demo

# In demos/fraud_detection_demo.ipynb
your_text = "Your Netflix subscription has expired. Update your payment method to continue watching."
result = predict_fraud(your_text)
display_prediction(result)

BART Joint Model - Jupyter Notebook Testing

See demos/test-unified-bart-joint-enhanced.ipynb for comprehensive testing including:

📈 Expected Performance

Based on training results and baseline implementations:

Model Type Expected Accuracy F1-Score Reasoning Notes
Simple Rule-Based 60-70% 0.6-0.7 ❌ None Quick prototype
TF-IDF + LogReg 80-90% 0.8-0.9 ❌ None Good baseline
TF-IDF + SVM 80-90% 0.8-0.9 ❌ None Robust to noise
BERT Fine-tuned 90-95% 0.9-0.95 ❌ None Best classification only
DistilBERT Fine-tuned 89-94% 0.89-0.94 ❌ None 60% faster, 97% of BERT
FLAN-T5 Unified 85-90% 0.85-0.90 ✅ Compact Text-to-text with short reasons
BART Joint Enhanced 91-93% 0.91-0.93 ✅ Rich Contextual Best overall: classification + detailed explanations ⭐

Model Comparison Highlights:

BART Joint Enhanced stands out as the most comprehensive solution:

Demo Troubleshooting

Model Not Loading

Low Performance

Memory Issues

Customization Tips

🔧 Configuration

Traditional ML Parameters

# In training/baseline_fraud_detection.py
vectorizer = TfidfVectorizer(
    max_features=5000,    # Vocabulary size
    stop_words='english', # Remove common words
    ngram_range=(1, 2)    # Use unigrams and bigrams
)

BERT Configuration

# In training/bert_fraud_detection.py
classifier = BERTFraudClassifier(
    model_name='bert-base-uncased',  # Or 'distilbert-base-uncased' for faster training
    max_length=128,                  # Maximum sequence length
    num_classes=2                    # Binary classification
)

# In training/kaggle_fraud_detection.ipynb
model = DistilBertForSequenceClassification.from_pretrained(
    'distilbert-base-uncased', 
    num_labels=10                    # Multiclass classification (9 fraud types + legitimate)
)
batch_size = 16      # Can use larger batches due to lower memory usage
max_length = 128     # Maximum sequence length
epochs = 3          # Faster training allows more epochs
learning_rate = 2e-5 # DistilBERT learning rate

Kaggle Training Configuration

# In runs/fraud-detection-kaggle-training-bert-run.ipynb
batch_size = 16      # Adjust based on GPU memory
max_length = 128     # Maximum sequence length
epochs = 3          # Training epochs
learning_rate = 2e-5 # BERT learning rate

📊 Sample Results

Traditional ML Output:

LOGISTIC_REGRESSION:
  Accuracy: 0.889
  F1-Score: 0.889
  AUC Score: 0.944

SVM:
  Accuracy: 0.889
  F1-Score: 0.889
  AUC Score: 0.944

BERT Output:

BERT Evaluation Results:
              precision    recall  f1-score   support

      normal       0.92      0.92      0.92        38
       fraud       0.92      0.92      0.92        37

    accuracy                           0.92        75
   macro avg       0.92      0.92      0.92        75
weighted avg       0.92      0.92      0.92        75

DistilBERT Output (Multiclass):

DistilBERT Multiclass Evaluation Results:
                    precision    recall  f1-score   support

         job_scam       0.89      0.94      0.91        32
       legitimate       0.95      0.91      0.93        45
         phishing       0.92      0.90      0.91        41
       popup_scam       0.88      0.92      0.90        38
      refund_scam       0.91      0.88      0.89        34
      reward_scam       0.90      0.93      0.91        36
         sms_spam       0.93      0.89      0.91        43
         ssn_scam       0.87      0.91      0.89        35
tech_support_scam       0.94      0.89      0.91        37

         accuracy                           0.91       341
        macro avg       0.91      0.91      0.91       341
     weighted avg       0.91      0.91      0.91       341

📊 DistilBERT Overall Metrics:
Accuracy: 0.9120
F1-Score (Macro): 0.9088
F1-Score (Weighted): 0.9115

📋 Data Requirements

Current Implementation

  1. SMS Spam Collection - Classic spam detection
  2. Phishing URL Dataset - URL-based fraud detection
  3. Enron Email Dataset - Email fraud detection
  4. Social Media Scam Data - Social platform fraud

Data Format Expected

data = pd.DataFrame({
    'message': ['text content here', ...],
    'label': ['fraud', 'normal', ...]
})

🛠️ Extending the Models

Adding New Features

# Add sentiment analysis
from textblob import TextBlob

def add_sentiment_features(text):
    blob = TextBlob(text)
    return {
        'polarity': blob.sentiment.polarity,
        'subjectivity': blob.sentiment.subjectivity
    }

Custom Preprocessing

def custom_preprocess(text):
    # Add domain-specific preprocessing
    text = remove_urls(text)
    text = normalize_currency(text)
    return text

Ensemble Methods

# Combine multiple models
def ensemble_predict(text):
    lr_pred = lr_model.predict_proba(text)[0][1]
    svm_pred = svm_model.predict_proba(text)[0][1]
    bert_pred = bert_model.predict(text)['probabilities']['fraud']
    
    # Weighted average
    final_score = 0.3 * lr_pred + 0.3 * svm_pred + 0.4 * bert_pred
    return 'fraud' if final_score > 0.5 else 'normal'

🌐 Deployment Options

Using the Demo Framework

# Production-ready integration using the demo class
from demos.fraud_detection_demo import FraudDetectionDemo

detector = FraudDetectionDemo()

# Single prediction
result = detector.predict_single(user_message)
if result['is_fraud'] and result['confidence'] > 0.8:
    alert_user(result['predicted_class'])

Flask Web App

from flask import Flask, request, jsonify
from demos.fraud_detection_demo import FraudDetectionDemo

app = Flask(__name__)
detector = FraudDetectionDemo()

@app.route('/predict', methods=['POST'])
def predict():
    text = request.json['text']
    result = detector.predict_single(text)
    return jsonify({
        'prediction': result['predicted_class'],
        'is_fraud': result['is_fraud'],
        'confidence': result['confidence']
    })

Streamlit Dashboard

import streamlit as st
from demos.fraud_detection_demo import FraudDetectionDemo

st.title("🛡️ Fraud Detection System")
detector = FraudDetectionDemo()

text_input = st.text_area("Enter message to analyze:")

if st.button("Analyze"):
    result = detector.predict_single(text_input)
    
    if result['is_fraud']:
        st.error(f"🚨 FRAUD DETECTED: {result['predicted_class']}")
    else:
        st.success("✅ Message appears legitimate")
    
    st.write(f"Confidence: {result['confidence']:.2%}")
    
    # Show probability distribution
    st.bar_chart(result['all_probabilities'])

🔍 Evaluation Metrics

The models are evaluated using:

For fraud detection, Recall is often most important (don’t miss actual fraud).

🚧 Known Limitations

  1. Sample Data: Currently uses synthetic data; real datasets needed for production
  2. Class Imbalance: Real fraud data is typically very imbalanced
  3. Context: Simple models may miss contextual nuances
  4. Adversarial Examples: Sophisticated scammers may evade detection
  5. Language Support: Currently English-only

🔮 Next Steps

For Model Development

  1. Data Collection: Gather real fraud/scam datasets
  2. Feature Engineering: Add metadata features (sender, timestamp, etc.)
  3. Advanced Models: Experiment with RoBERTa, DistilBERT (already implemented), or domain-specific models
  4. Active Learning: Implement feedback loop for continuous improvement
  5. Multi-modal: Combine text with image analysis for comprehensive detection

For Production Deployment

  1. Performance Optimization: Optimize for low-latency inference using the demo framework
  2. A/B Testing: Compare model performance in production using demo tools
  3. Real-time Processing: Integrate demo classes into streaming systems
  4. Monitoring: Use demo tools to validate model performance over time
  5. User Interface: Build on the Streamlit demo for user-facing applications

For Demo Enhancement

  1. Interactive Web App: Extend the Streamlit demo with more features
  2. API Development: Use the demo classes to build REST APIs
  3. Batch Processing: Implement large-scale batch prediction capabilities
  4. Model Comparison: Add functionality to compare multiple model versions
  5. Feedback Collection: Integrate user feedback mechanisms for continuous learning

📚 References

Based on analysis of existing projects including:

🔗 Quick Reference

Training Files

Demo Files

Model Files (after training)

Commands

# Test BART joint model (LATEST)
jupyter notebook demos/test-unified-bart-joint-enhanced.ipynb

# Quick test traditional models
python demos/quick_demo.py

# Interactive demo
jupyter notebook demos/fraud_detection_demo.ipynb

# Full demo
python demos/fraud_detection_demo.py

# Train from scratch
python training/baseline_fraud_detection.py

📅 Recent Updates

October 2025 - BART Joint Model (Latest)

Previous Updates

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Add your improvements
  4. Submit a pull request

Note: This is a baseline implementation. For production use, consider: