AI in Mainframes: Real-Time Transactional Inference on IBM Z

Historically, running deep learning models on enterprise mainframe data required exporting transactional records off-host to cloud or x86 server clusters. This off-host integration pattern introduces significant latency overhead, network bandwidth costs, and PII data security egress risks.

Modern mainframe engineering resolves this by integrating Real-Time On-Host Inferencing directly into core transaction flows using specialized processors like the IBM Telum.

📐 1. IBM Telum Processor On-Chip AI Accelerator

The IBM Telum processor (introduced in the z16 mainframe family) features a dedicated, on-chip hardware AI accelerator designed for high-throughput, low-latency deep learning inference.

┌────────────────────────────────────────────────────────┐
│ Telum Processor Core Chip                              │
│                                                        │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐ │
│  │ Core 1  │   │ Core 2  │   │ Core 3  │   │ Core 4  │ │
│  └────┬────┘   └────┬────┘   └────┬────┘   └────┬────┘ │
│       │             │             │             │      │
│  ─────┴─────────────┴──────┬──────┴─────────────┴───── │
│             Low-Latency Cache Interconnect Ring        │
│  ─────┬────────────────────┬────────────────────┬───── │
│       │                    │                    │      │
│  ┌────┴────┐          ┌────┴────┐          ┌────┴────┐ │
│  │ Core 5  │          │ Core 6  │          │ Core 7  │ │
│  └─────────┘          └─────────┘          └─────────┘ │
│                                                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │ 🚀 Integrated AI Accelerator (6 TFLOPS)          │  │
│  │ - Matrix Multiplication & Tensor Math engines    │  │
│  │ - Activation function units (ReLU, Sigmoid, etc) │  │
│  └─────────────────────────▲────────────────────────┘  │
│                            │                           │
└────────────────────────────┼───────────────────────────┘
                             │ Shared ring access

Key Architectural Primitives:

• Co-Processor Design: Rather than deploying external GPU cards over PCIe lanes, each Telum chip includes an on-chip AI accelerator. All 8 processor cores on the chip share access to this accelerator via a low-latency cache interconnect fabric.
• Hardware Compute capacity: The accelerator delivers up to 6 TFLOPS of compute power per chip, optimized for tensor math, matrix multiplications, convolutions, and activation operations (e.g. ReLU, Sigmoid, Tanh).
• Virtual Cache Fabric: The processor utilizes a virtual Level 4 (L4) cache structure. This allows cores to feed data to the AI accelerator directly from adjacent core caches without writing to system memory (RAM), keeping execution latency at sub-millisecond scales.

📊 2. Inference Latency & Security Comparison Matrix

Deploying on-chip mainframe AI inference provides crucial advantages over off-host architectures:

🔄 3. Transactional On-Chip Inference Flow

In high-throughput transactional environments (such as credit card processing or banking registries), AI inference must run within the database transaction lock boundary. If the AI evaluation takes too long, the transaction will time out.

The sequence diagram below maps the execution flow of a real-time fraud scoring check running inside a CICS/COBOL transaction using IBM Watson Machine Learning for z/OS (WMLz):

💻 4. Programmatic Model Execution: ONNX Runtime on z/OS

Mainframe models are typically trained using standard open-source frameworks (such as PyTorch or TensorFlow) on GPU/CPU clusters, exported to the ONNX format, and deployed on z/OS.

To invoke these models locally on z/OS, engineers use z/OS Container Extensions (zCX)—a built-in Linux virtualization layer running on z/OS—integrated with the IBM-optimized ONNX Runtime engine. The runtime automatically translates ONNX operations into hardware-level instructions targeting the Telum processor.

Below is a Python implementation demonstrating how to load a fraud-detection model and run inference inside a z/OS Container Extensions (zCX) container:

import os
import numpy as np
import onnxruntime as ort
 
def run_mainframe_fraud_check(transaction_features: list[float]) -> float:
    """
    Executes deep learning inference on z/OS targeting the Telum AI Accelerator.
    Requires the IBM-optimized ONNX Runtime for z/OS configuration.
    """
    model_path = "/var/models/fraud_detection_v12.onnx"
    
    if not os.path.exists(model_path):
        raise FileNotFoundError(f"Fraud model not found at {model_path}")
        
    # Configure session options to utilize IBM hardware acceleration
    # The IBM provider hooks into the z/Architecture NNPA hardware instructions
    opts = ort.SessionOptions()
    opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    
    # Initialize the ONNX Runtime inference session
    # WMLz/zCX configuration maps the hardware provider automatically
    session = ort.InferenceSession(model_path, sess_options=opts)
    
    # Prepare input tensor arrays matching the model's signature
    # Shape: (1, 14) representing a single transaction with 14 normalized features
    input_name = session.get_inputs()[0].name
    input_data = np.array([transaction_features], dtype=np.float32)
    
    # Run the model execution block
    # Under the hood, this invokes the Telum on-chip coprocessor ring
    raw_outputs = session.run(None, {input_name: input_data})
    
    # Extract prediction probability output
    # Model returns: [Probability of Safe, Probability of Fraud]
    prediction_scores = raw_outputs[0][0]
    fraud_probability = float(prediction_scores[1])
    
    return fraud_probability
 
# Example Execution inside a z/OS Container Extensions microservice:
# mock_features = [450.00, 1.0, 0.0, 0.23, 1.1, -0.4, 0.0, 12.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.5]
# prob = run_mainframe_fraud_check(mock_features)
# print(f"Mainframe On-Chip Inference Result - Fraud Probability: {prob:.6f}")

🛠️ 5. Assembly Level Instruction: NNPA

At the lowest hardware execution layer, the operating system (z/OS or Linux on IBM Z) and compiler frameworks talk to the Telum AI Accelerator using the Neural Network Processing Assist (NNPA) instruction.

• Instruction Set Integration: NNPA is a machine-level hardware instruction introduced in the z/Architecture instruction set.
• Zero Kernel Switching: Cores execute the NNPA instruction directly. This prevents standard CPU context switches into kernel space or GPU driver hops, keeping execution times deterministic.
• Compiler Automation: Enterprise compilers (such as Enterprise COBOL v6.4+ or Enterprise PL/I v6.1+) automatically optimize compile outputs. If a developer calls deep learning functions or Watson API schemas in high-level COBOL code, the compiler compiles those directly into optimized NNPA assembly macros:

* Mock z/Architecture assembly showing NNPA instruction call
LG      1,INPUT_TENSOR_ADDR   * Load input tensor coordinate
LG      2,OUTPUT_TENSOR_ADDR  * Load target output buffer
NNPA                          * Trigger on-chip hardware execution

🔗 6. Related Sections

• GenAI Integration Patterns — Detailed guide on synchronous API gateways, asynchronous message queues, and event publish-subscribe.
• Production GenAI Gateway — In-depth reference implementation of FastAPI routing proxies, token monitoring, and Pydantic validators.
• Evaluation Tools — Frameworks and metrics for testing model accuracy, biases, and latency.