FPB Dataset Evaluation with O3-Mini

This tutorial demonstrates how to evaluate the O3-Mini model on the Financial PhraseBank (FPB) dataset for sentiment analysis. The evaluation script provides a complete pipeline for testing, storing results, and enabling reproducibility for future research.

Overview

The FPB evaluation script (FPB_TestByChatGPT_o3_mini.ipynb) is designed to:

  • Load the FPB test dataset from HuggingFace

  • Query the O3-Mini model for sentiment predictions

  • Store raw responses for future analysis

  • Calculate evaluation metrics (accuracy, F1-score)

  • Provide a reproducible framework for model comparison

File Structure

After completing the evaluation, you need to organize the generated files into three main directories in the FinLLM-Leaderboard repository:

FinLLM-Leaderboard/
├── text_responses/     # Raw LLM outputs (QA pairs)
├── tutorials_code/     # Evaluation scripts
└── evaluation/         # Computed metrics and results

Directory Purposes

text_responses/

Contains the raw question-answer pairs generated by the LLM. This allows researchers to:

  • Recompute metrics with different evaluation criteria

  • Analyze response patterns and failure modes

  • Compare outputs across different models

  • Perform qualitative analysis of model behavior

tutorials_code/

Stores the evaluation scripts and notebooks:

  • FPB_TestByChatGPT_o3_mini.ipynb - Main evaluation notebook

  • Helper functions and utilities

  • Data preprocessing scripts

evaluation/

Contains computed results and metrics:

  • Accuracy scores

  • F1-scores (macro, micro, weighted)

  • Confusion matrices

  • Performance breakdowns by sentiment class

Prerequisites

Before running the evaluation, ensure you have:

  1. API Access: Valid OpenAI API key for O3-Mini model access

  2. Python Environment: Python 3.8+ with required packages

  3. HuggingFace Access: Account for dataset access (if required)

Required Packages

pip install datasets>=2.19.0
pip install huggingface_hub>=0.24
pip install openai==1.40.2
pip install pandas>=2.2.2
pip install tqdm>=4.66.4
pip install scikit-learn
pip install requests>=2.31.0

Running the Evaluation

Step 1: Open Google Colab

  1. Go to Google Colab

  2. Sign in with your Google account

Step 2: Download the Evaluation Script

  1. Navigate to the FinLLM-Leaderboard repository: https://github.com/Open-Finance-Lab/FinLLM-Leaderboard

  2. Go to the evaluation script: https://github.com/Open-Finance-Lab/FinLLM-Leaderboard/blob/main/tutorials_code/FPB_TestByChatGPT_o3_mini.ipynb

  3. Click “Raw” to view the raw notebook file

  4. Copy the URL of the raw file

  5. In Google Colab, click “File” → “Open notebook” → “GitHub” tab

  6. Paste the repository URL or directly open: https://github.com/Open-Finance-Lab/FinLLM-Leaderboard/blob/main/tutorials_code/FPB_TestByChatGPT_o3_mini.ipynb

Step 3: Setup Environment

  1. Set up your OpenAI API key in Colab:

import os
os.environ['OPENAI_API_KEY'] = 'your-api-key-here'

  1. Alternatively, use Colab’s secret management: - Click the key icon (🔑) in the left sidebar - Add a new secret named OPENAI_API_KEY - Set your API key as the value

Step 4: Load Dataset

The script automatically loads the FPB test dataset:

from datasets import load_dataset

ds_raw = load_dataset("TheFinAI/flare-fpb", split="test")
print(f"Loaded {len(ds_raw)} test samples")

Step 5: Run Evaluation

Execute the notebook cells in sequence:

  1. Data Loading: Loads and preprocesses the FPB dataset

  2. Model Setup: Configures O3-Mini API connection

  3. Batch Processing: Sends queries and collects responses

  4. Result Storage: Saves raw responses to text_responses/

  5. Metric Calculation: Computes and saves evaluation metrics

Step 6: Organize Results

After completing the evaluation in Google Colab, you need to manually organize the generated files into the appropriate directories in the FinLLM-Leaderboard repository:

  1. Download Results from Colab: - Download the generated CSV files, JSON responses, and any output files from your Colab session

  2. Organize Files: - Place raw LLM responses in text_responses/ directory - Place evaluation metrics and results in evaluation/ directory - The notebook is already in tutorials_code/ directory

  3. File Placement Example:

# Place your generated files in the repository structure:
FinLLM-Leaderboard/
├── text_responses/
│   └── o3mini_fpb_responses.json    # Your generated responses
├── evaluation/
│   ├── o3mini_fpb_metrics.json      # Your computed metrics
│   └── o3mini_fpb_results.csv       # Your evaluation results
└── tutorials_code/
    └── FPB_TestByChatGPT_o3_mini.ipynb  # Already provided

Reproducibility Notes

To ensure reproducible results:

  1. Version Control: Pin all package versions as specified

  2. Random Seeds: The script sets random seeds where applicable

  3. API Consistency: Use the same model version (O3-Mini)

  4. Data Integrity: Verify dataset version and splits

Customization

The evaluation framework can be adapted for:

  • Different Models: Modify the model configuration

  • Custom Datasets: Replace the dataset loading logic

  • Additional Metrics: Extend the evaluation functions

  • Batch Sizes: Adjust processing parameters for efficiency

Troubleshooting

Common Issues

API Rate Limits

  • Implement exponential backoff

  • Reduce batch size

  • Add delays between requests

Memory Issues

  • Process data in smaller chunks

  • Clear intermediate variables

  • Use streaming for large datasets

Inconsistent Results

  • Check API model version

  • Verify random seed settings

  • Ensure identical preprocessing

Next Steps

After running the evaluation:

  1. Compare Results: Use stored responses to compare with other models

  2. Error Analysis: Examine misclassified examples

  3. Metric Extensions: Calculate additional evaluation metrics

  4. Visualization: Create charts and graphs from the results

For questions or issues, refer to the project documentation or create an issue in the repository.