Why TCAD Convergence Fails — and Why ML Is the Fix

Traditional TCAD simulation fails to converge primarily because iterative numerical solvers — particularly Newton-Raphson loops coupling Poisson’s equation with carrier transport equations — accumulate errors across thousands of iterations, and this problem intensifies sharply as device geometries shrink to advanced technology nodes. As documented by Samsung Electronics (2021), the core challenge is twofold: prohibitive computational cost and recurrent convergence errors in three-dimensional TCAD simulation. Critically, the same research explicitly states that complete replacement of TCAD with deep learning has not yet been achieved, framing ML instead as a convergence stabilizer operating within a restructured hybrid system.

The earliest dataset-verified instance of data-driven convergence acceleration comes from a 2014 patent by Peking University Shenzhen Research Institute. The disclosed method records M historical input-output iteration pairs and uses weighted linear combinations to predict the next iteration input, directly reducing iteration count within semiconductor nanodevice solvers. This approach anticipated what has since become a broad research agenda: using prior simulation history to warm-start iterative solvers rather than initialising from scratch.

The scaling problem has made convergence failures progressively more expensive. At advanced nodes, a single three-dimensional TCAD simulation of a FinFET or gate-all-around structure can require hours of compute time on high-performance clusters, and a failed run — one that diverges before reaching a self-consistent solution — wastes that investment entirely. According to research published by IEEE, iterative solver stability in coupled partial differential equation systems is a well-established numerical challenge, and semiconductor device physics presents some of the most stiff variants of these systems due to the exponential carrier concentration dependencies in Boltzmann statistics.

The dataset spanning 2014 to 2026, drawing on more than 20 patents and peer-reviewed publications from institutions including Samsung Electronics, Synopsys, Gwangju Institute of Science and Technology (GIST), Tsinghua University, Seoul National University, Korea University, and Lam Research, reveals a consistent architectural philosophy: ML as a convergence mechanism, not a replacement solver.

Surrogate Models and Active Learning: Doing More with Fewer Simulations

The fundamental data acquisition problem in ML-TCAD integration is that training a predictive model requires TCAD simulation results, yet each simulation is the expensive resource being minimised. Active learning solves this directly by selecting only the most informative simulation points — and the evidence across the dataset is that this strategy substantially reduces the number of TCAD calls required to train accurate models.

Alsemi (주식회사 알세미), a Korean startup, has filed a cluster of patents (2024–2025) disclosing an active learning framework in which an agent queries TCAD simulations only for voltage-current or voltage-capacitance pairs that maximise model informativeness, then trains a neural network compact model on the returned results. This loop eliminates redundant simulations that would otherwise be required by exhaustive grid sampling. A related 2024 Alsemi patent extends the concept to reinforcement learning agents that optimise semiconductor process variables using the trained neural compact model.

Tsinghua University has extended active learning into multi-objective transistor design optimisation (2025), using Gaussian process regression as a surrogate model and differential evolution to generate candidate process parameter sets. Only candidates with high predicted performance and high uncertainty are forwarded to TCAD for simulation, with results iteratively fed back to update the surrogate. This approach directly targets the Pareto front of transistor performance optimisation, substantially shortening the development cycle.

Taiwan Semiconductor Research Institute (2022) contributed a complementary Bayesian meta-learning approach specifically for semiconductor laser annealing — a process where experimental iteration is prohibitively expensive. The proposed meta-learner adjusts the hybridisation ratio between TCAD and ML when selecting the next sampling point, achieving significantly lower mean square error during the first 100 sampling steps compared to pure ML approaches. This TCAD-assisted meta-learned sampling strategy is directly relevant to fabrication processes where each physical experiment carries substantial cost, as acknowledged by standards bodies including SEMI.

For process simulation model calibration, Lam Research has filed multiple patent generations on optimising process simulation model parameters by minimising cost functions derived from CD-SEM, optical scatterometry, and transmission electron microscopy measurements. The optimisation loop adjusts pre-process profiles to match post-process metrology, representing a data-driven calibration approach that reduces manual tuning iterations. The company’s sustained investment across multiple patent generations (2020 and 2025) confirms this as a strategic priority in advanced etch and deposition process control.

Neural Architecture Choices and Their Convergence Impact

The choice of neural network architecture determines which specific convergence bottleneck is addressed — and the dataset documents at least four distinct architectural paradigms, each targeting a different failure mode in the TCAD simulation pipeline.

Graph Neural Networks: Better Starting Points for Iterative Solvers

The most direct lever for Newton-Raphson convergence acceleration is the quality of the initial solution provided to the solver. Gwangju Institute of Science and Technology (GIST) holds patents (2021, 2023) on graph neural network architectures that derive an area graph from the device structure file, classify device type, and generate a compact-model-derived initial solution. This physically plausible starting point is passed to the TCAD simulator, dramatically reducing the number of Newton-Raphson iterations required to reach convergence. A hierarchical variant employs a higher-level deep neural network to provide the initial solution hint to a lower-level device simulator, creating a coarse-to-fine initialisation hierarchy.

CNNs for Quantum Transport: Accelerating the Most Expensive TCAD Kernel

The nonequilibrium Green’s function (NEGF) method is among the most computationally intensive kernels in TCAD, essential for quantum transport simulation in nanoscale FETs. Kobe University (2020) demonstrated that a convolutional autoencoder trained to predict carrier density and local quantum capacitance distributions from input potential distributions can substitute the NEGF computation within the self-consistent loop alongside Poisson’s equation. The result is accurate potentials across a range of gate lengths in significantly shorter computation time than conventional NEGF — making quantum transport simulation tractable within standard TCAD workflows. This approach is particularly relevant as gate lengths approach and fall below 5 nm, where quantum effects dominate device behaviour, a regime increasingly relevant to roadmaps published by the Semiconductor Industry Association.

LSTM Networks: Preventing Pseudo-Transient Analysis Divergence

Pseudo-transient analysis (PTA) is a widely used continuation algorithm for DC convergence in TCAD-linked circuit simulation. Its most common failure mode is time-step selection: too large a step causes divergence, while an overly conservative step wastes computation. Huada Empyrean Software (2023) proposes an LSTM-based method for controlling the time step in PTA, exploiting the LSTM’s temporal memory to predict future time steps from the history of the simulation trajectory. A coarse-and-fine-grained hybrid sampling strategy identifies the optimal time step, and the LSTM network predicts forward to avoid both divergence and excessive conservatism. This directly addresses one of the most common convergence failure modes in both TCAD and SPICE simulation engines.

“Neural network surrogate models for FinFET simulation achieve R² = 0.99 accuracy while delivering speedups exceeding 122,000× compared to traditional simulators — a benchmark that establishes the practical viability of ML-TCAD hybrid deployment.”

Recurrent Networks for Sequential Process Emulation

Samsung Electronics (2021) disclosed a simulation system built around a recurrent neural network consisting of process emulation cells arranged in time series. Each cell receives the previous output profile, target profile, and process condition information, generating the current output profile using time-series causal relationships encoded as prior knowledge. This architecture mirrors the sequential physics of multi-step semiconductor manufacturing — each process step depends on the output state of the preceding step — enabling step-by-step profile prediction that bypasses TCAD at each individual process step while preserving the causal dependencies that make the predictions physically meaningful.

Optimising Simulator Settings and Compact Model Convergence

Beyond replacing TCAD kernels, machine learning has been applied to optimising the configuration of TCAD simulators themselves — mesh density, solver tolerance, stepping strategy — parameters that critically affect both convergence and accuracy without changing the underlying physics equations being solved.

Synopsys (2022) filed the most direct patent expression of this concept: an ML framework that predicts optimised TCAD simulator system settings for future simulation executions. By learning from past simulation runs — which settings led to convergence and which did not — the system preemptively configures the solver for new simulations. This transforms simulator configuration from a manual expert task into a learned, data-driven process, and represents a natural extension of the company’s broader EDA toolchain strategy.

Samsung Electronics (2024) addressed the specific problem of regression model prediction failure through an adaptive retraining loop. When the consistency of a trained neural network model falls below a target threshold, the system identifies consistency reduction factors — regions of the input space where the model fails — and retrains on additional sample data targeted at those regions. This is conceptually equivalent to active learning but applied to model consistency rather than output uncertainty, and it ensures that deployed neural models remain reliable across the full operating range of device parameters.

The impact of activation function selection on SPICE convergence is a finding with direct practical implications for how ML compact models should be architected. The ISRU activation function maintains smooth, bounded derivatives across the full input range, which prevents the gradient discontinuities that cause Newton-Raphson divergence in SPICE engines — a property that sigmoid and tanh share in principle but that ReLU lacks entirely due to its non-differentiability at zero. This analysis, published by researchers at National Yang-Ming Chiao-Tung University and consistent with convergence theory reviewed by ACM, has direct implications for compact model standardisation efforts.

Xiangtan University (2024) reported a machine learning-based MOS transistor simulation model calibration method that combines a classifier, regressor, and surrogate model to calibrate TCAD parameters, reporting computation speed approximately 10⁷ times faster than direct TCAD simulation. The method trains a random forest model to replace expensive TCAD calls during the iterative calibration loop, demonstrating that ensemble methods — not just deep learning — can deliver dramatic acceleration for specific calibration tasks.

Key Players, Patent Landscape, and Innovation Trajectory

Samsung Electronics and Synopsys are the most prolific patent filers in ML-TCAD convergence, while academic contributors from Korea, Taiwan, and China dominate the peer-reviewed literature — a bifurcation that reflects the different incentives driving industrial IP protection versus academic publication.

Samsung Electronics is the dominant patent filer, with contributions spanning semiconductor design automation systems that integrate ML with TCAD simulation data, neural network-based semiconductor characteristic prediction, recurrent process simulation, and adaptive retraining loops for neural network model consistency. Its 2021 foundational paper on 3D TCAD restructuring establishes the hybrid complementarity thesis that the rest of the field has largely adopted.

Synopsys contributes both the ML-optimised simulator settings patent (2022) and an ML-driven IC design prediction framework (EP, 2024), as well as a 2019 machine learning-assisted DTCO (Design-Technology Co-Optimisation) modeling framework — establishing it as the leading EDA vendor in this space and consistent with its broader strategy of embedding intelligence into simulation toolchains, as tracked by PatSnap’s IP intelligence platform.

Alsemi has filed a cluster of active learning patents for neural compact model generation (2024–2025), including a reinforcement learning agent for process variable optimisation. This represents a focused Korean startup effort targeting the TCAD-to-compact-model pipeline — a commercially valuable bottleneck between process simulation and circuit design.

From a trend perspective, the dataset shows clear progression: from simple neural network surrogates trained offline (2019–2021), through active learning and adaptive retraining loops (2022–2023), toward multi-agent reinforcement learning and large language model integration for parameter optimisation (2024–2025). This trajectory indicates rapid expansion of the ML-TCAD solution space, with each generation addressing limitations identified in the prior one. The PatSnap Insights blog tracks emerging innovation across the semiconductor design automation landscape.