Guest Blog: Learning Cats via Proximal Policy Optimization (PPO)

Cat qubits come with a built-in advantage: their errors don’t play fair. They lean heavily toward one type over the other. In other words, one type of error (bit-flip) will almost never occur, leaving us to deal only with the second type of error (phase-flip). This property is exactly what makes them so useful for error correction. But keeping that advantage intact in real hardware, against noise and drift, is a different problem entirely. So for Y-Quantum 2026 (Yale), we handed participants that exact challenge: can you preserve a cat qubit’s bias when conditions refuse to cooperate? No fixed method, just the problem itself. They could use quantum control, optimization, error correction, whatever route gets them there.

The team that impressed us most were a group of students from Harvard: Batu Yalcin, Tymofii Baranenko, Leon Katarzynski, Jack Ploof, and Alex Wong. They used two learning algorithms, PPO and CMA-ES, to automatically tune a cat qubit’s control parameters and extend how long it holds its state under realistic noise and drift.

What follows is their write-up, in their own words. We’re happy to host it.

Optimization of Dissipative Cat-Qubit Stabilization

Batu Yalcin, Tymofii Baranenko, Leon Katarzynski, Jack Ploof and Alex Wong Y-Quantum Hackathon 2026 · Alice & Bob Challenge

Abstract. We got the opportunity to explore the stability and limitations of cat qubits through the Alice & Bob Challenge at the Y-Quantum hackathon. Our work investigates how environmental drift impacts cat-qubit coherence, and how researchers can stabilize the system in the face of drift. We begin by defining two optimization strategies: Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and Proximal Policy Optimization (PPO) to gain an understanding of the terms of the Hamiltonian modeling standard cat qubits. We then model moon/crescent cat qubits to explore a new degree of freedom of cat qubits and reinforce our own understanding of the cat qubit terms. Then we introduce noise/drift sources into our system and see how our optimization algorithms respond. Finally, we observe the effect of gates on the system, and conclude with a physical interpretation of our results.

Introduction

The challenge introduced a Hamiltonian system for stabilizing a cat qubit, given by

$H = g_2 a^2 b^\dagger + g_2^* (a^\dagger)^2 b - \epsilon_d b - \epsilon_d^* b^\dagger$

where

Two-Photon Loss: $g_2^* a^2 b^\dagger$
Two-Photon Gain: $g_2 (a^\dagger)^2 b$
One-Photon Loss: $-\epsilon_d b$
One-Photon Gain: $-\epsilon_d b^\dagger$

In terms of intuition, these terms correspond to two photon registers, one controlled by $b$ and the other by $a$ . For example, the Two-Photon Loss operator corresponds to two photons annihilated by $a^2$ in the latter register while one photon being created in the former register by $b^\dagger$ , representing a form of two-to-one photon conversion. Such parametrizations allow us to fight against drift by tuning parameters available to us, $g_2$ and $\epsilon_d$ , and engineer well-built cat states by utilizing quantum resources.

As suggested by the challenge notebook, we wanted to create optimization algorithms that would maximize the $T_x$ bit flip and $T_z$ phase flip logical lifetimes of cat qubits. To maximize these times and maintain a target bias $\eta$ defined by

$\eta = \frac{T_z}{T_x} \approx 320$

we wanted to tune the real and imaginary values of $g_2$ and $\epsilon_d$ . Through the four tuned values of $Re(g_2), Im(g_2), Re(\epsilon_d), Im(\epsilon_d)$ , we hoped to extract some physical meaning from the parameters.

Optimization Framework

Since the mapping from $\vec{x}$ to $(T_X, T_Z)$ is available only through simulation, we treated this as a black-box constrained optimization problem. Our objective was to maximize $f(\vec{x}) = \log(T_X \cdot T_Z)$ subject to $\eta \approx 320$ , enforced by a penalty reward function:

$reward = \frac{1}{2}\left(\log T_X + \log T_Z\right) - \lambda_\eta \cdot \left(\log \eta - \log 320\right)^2$

CMA-ES

CMA-ES is a derivative-free algorithm that maintains a multivariate Gaussian over the parameter space, iteratively shifting this distribution towards higher reward regions. Figure 1 shows convergence of $T_X$ , $T_Z$ , $\eta$ , control parameters, cat size $\alpha$ , and the joint objective across 60 epochs.

Figure 1: CMA-ES optimization progress over 60 epochs. Top row: T_X, T_Z, noise bias η (dashed target at 320). Bottom row: control parameters, cat size α, and joint objective.

CMA-ES converged to $T_X = 0.212\ \mu s$ , $T_Z = 70\ \mu s$ , $\eta = 324$ , with $\epsilon_d = 4.89$ MHz and $g_2 = 1.76$ MHz.

Proximal Policy Optimization (PPO)

PPO is a policy-gradient RL algorithm that learns a stochastic policy and updates it via descent gradient steps, preventing large and unstable updates. Figure 2 shows the PPO results at $\eta = 320$ .

Figure 2: PPO optimization results for η = 320. T_X, T_Z, noise bias η, and control parameters all converge by epoch ~60.

PPO achieved $T_X = 0.261\ \mu s$ , $T_Z = 78.3\ \mu s$ , $\eta = 325$ . This outperforms the CMA-ES model on both lifetimes. Table 1 summarizes the full comparison.

Table 1: CMA-ES vs. PPO: final optimized parameters and lifetimes.

Quantity	CMA-ES	PPO
$\epsilon_d$ (MHz)	4.89	4.69
$g_2$ (MHz)	1.76	1.43
$\kappa_2 = 4g_2^2/\kappa_b$ (MHz)	0.50	0.82
$\alpha^2 = \epsilon_d/g_2$	2.78	3.28
$T_X\ (\mu s)$	0.212	0.261
$T_Z\ (\mu s)$	70	78.3
$\eta = T_Z/T_X$	324	325

Figure 3: Optimization landscape for autonomous cat-qubit stabilization in the (ε_d, g_2) parameter space…

Figure 3: Optimization landscape for autonomous cat-qubit stabilization in the $(\epsilon_d, g_2)$ parameter space. Left: Contour map of the resulting cat size $\alpha^2$ as a function of the buffer drive amplitude $\epsilon_d$ and nonlinear two-photon coupling strength $g_2$ . Larger $\epsilon_d$ injects more energy into the system, increasing the coherent-state separation, while larger $g_2$ enhances nonlinear stabilization and suppresses runaway growth. The dashed white curve denotes the target $\eta = T_Z/T_X = 320$ manifold, corresponding to operating points that maintain the desired bit-flip/phase-flip lifetime asymmetry. The cyan horizontal line indicates the experimentally motivated reference coupling $g_2^* = 1.58$ MHz. Markers compare different optimization strategies (baseline, CMA-ES, PPO, and brute-force grid optimum). Right: Stabilization ratio $\kappa_2/\kappa_a$ over the same parameter space, indicating the competition between engineered two-photon dissipation and intrinsic single-photon loss. The horizontal black line marks the threshold $\kappa_2 = \kappa_a$ : above this boundary ( $\kappa_2 > \kappa_a$ ), the cat manifold is autonomously stabilized, while below it the system becomes under-stabilized and susceptible to decay. The dashed black contour again traces the $\eta = 320$ manifold, illustrating the narrow region where strong stabilization, large cat size, and favorable error bias coexist simultaneously.

We can see from that $g_2$ alone sets the stabilization rate, while $\epsilon_d/g_2$ sets the cat size.

Drift Analysis

Real hardware is subject to perturbations from the environment that shift parameters over time. We described three drift mechanisms.

Crescent/Moon Cat Qubits

We wanted to extend our understanding of cat qubits to the deformed crescent/moon qubit model. Since the phase space of moon qubits are deformed, they can be engineered to affect the $\eta$ as well as suppress errors in a given direction. To ensure we had a good grasp of the Hamiltonian, we tried using our optimization models to simulate crescent qubits based on the Hamiltonian

$H = g_2^* a^2 b^\dagger + g_2 (a^\dagger)^2 b - \epsilon_d b - \epsilon_d b^\dagger + g_2 \lambda a^\dagger a b$

under the parameters

$0 \le Re(\epsilon_d) \le 8$
$0 \le Im(\epsilon_d) \le 8$
$0 \le Re(g_2) \le 8$
$-1 \le Im(g_2) \le 1$
$-1 \le \lambda \le 1$

yet ended up with a qubit that looked like a standard cat qubit

Figure 4: Our first attempt at realizing a moon qubit.

after much deliberation, stress testing our optimization models, and reading papers, we realized that we had in fact missed our Hermitian conjugate $g_2^* \lambda^* a^\dagger a b^\dagger$ , giving us our actual Hamiltonian

$H = g_2^* a^2 b^\dagger + g_2 (a^\dagger)^2 b - \epsilon_d b - \epsilon_d b^\dagger + g_2 \lambda a^\dagger a b + g_2^* \lambda^* a^\dagger a b^\dagger$

Modeling this Hamiltonian with our PPO optimization model yielded us

Figure 5: Moon Qubit modeled with the correct Hamiltonian and our PPO optimization model

confirming both the accuracy of our PPO model and Hamiltonian. Seeing we could model a system given the correct Hamiltonian with our optimization model, we were confident enough to move onto using the model to analyze other forms of drift.

Time-Dependence of Drift

We first observed the effects of added drift during simulation, allowing $g_2$ to rotate with $g_2 \to g_2 e^{i\phi(t)}$ where the time dependence of $\phi(t)$ is linear. This resulted in oscillations within optimization that correspond to the rotation period, as seen in the figure.

Figure 6: Expectation value of Z vs. time with rotating g.

Then, we wanted to approach the idea of correcting such drift behaviour in various situations.

Detuning Noise

One of the most significant areas to explore drift was detuning, where the transformation $H \to H + \Delta a^\dagger a$ takes place. We first saw that this detuning indeed does hurt the coherence time, as seen in the figure below:

Figure 7: Decay of coherence time T_Z with detuning Δ

Then, however, the way to counteract this detuning is more straightforward. Adding a counteracting parameter $\Delta_d a^\dagger a$ , which we learned to be experimentally feasible through our discussions, we can try and reduce the effects of such detuning. We were, in the end, successful, as can be seen from the graph below:

Figure 8: Optimal counteracting coefficient Δ_d vs. detuning parameter Δ

Kerr Effect Nonlinearity

Adding $H \to H + \delta(a^\dagger a)^2$ reveals non-monotonic behavior in both $T_Z$ and $T_X$ , with a possible resonance near $\delta \approx 0.4$ (Figure 9). Unlike detuning, Kerr Effect drift cannot be corrected by a linear term and requires real-time optimization.

Figure 9: T_Z and T_X vs. Kerr nonlinearity intensity δ.

Correcting Drift with Moon Cat States

We then wanted to bridge our separate efforts in building moon cat states and correcting for drift. For this, we added a two-level system (TLS) interaction as the present drift, with loss operator $L_\gamma = \gamma_{tls}\sigma$ where $\gamma_{tls} = 1/2$ and interaction term being:

$H \to H + \delta a \sigma^\dagger + \text{h.c.} \tag{1}$

This led to the following comparison between moon cat-corrected and circular cat-corrected results: We observe that moon cat qubits provide significantly better correction, yielding a promising avenue in cat state error correction.

Figure 10: Moon and Circular Cat-Corrected Statistics

Single Qubit Gates

Beyond passive stabilization, we implemented single-qubit logical gates on the optimized cat qubit. The Z gate is expressed by $H_Z = \epsilon_z(a + a^\dagger)$ , mapping $|0\rangle \to |0\rangle$ and $|1\rangle \to -|1\rangle$ . The X gate uses $H_X = \Delta_x(a^\dagger a)$ , inducing a phase-space rotation of angle $\pi$ at $t = \pi/\Delta_x$ . Gate simulations confirm analytic predictions, with small deviations arising from photon stabilization terms.

Conclusion

Our results demonstrate three key findings. First, different drift mechanisms affect cat qubits differently. Detuning can be compensated for analytically, Kerr effect nonlinearity requires real-time adaptation, and TLS interactions require modification at the state-level. Second, the exponential decay pattern of observables can identify which drift type is active. Third, moon cat states offer a promising route to improved robustness without sacrificing the biased-noise structure.

PPO outperformed CMA-ES on both lifetimes while achieving the same bias target, with final values $T_X = 0.261\ \mu s$ , $T_Z = 78.3\ \mu s$ , $\eta = 325$ . Future work could explore multi-qubit extensions, real-time adaptive optimization under live drift, and combining moon cat states with full gate sets for fault-tolerant operation.

Bibliography

For any and all information used here, please refer to the bibliography, references, and files in our challenge-related github repository: Piqasso-YQuantum