Ubiquitous in biomanufacturing, fermentation processes are complex and unpredictable when compared to traditional chemical-processing operations. From changing cell-growth rates to drifts in environmental conditions, typical control approaches can only react to predetermined setpoints rather than reliably maximize product output. A group of researchers from Iowa State University (ISU; Ames; www.iastate.edu) and Novonesis A/S (Lyngby, Denmark; www.novonesis.com), with support from BioMADE (www.biomade.org) and Schmidt Sciences (www.schmidtsciences.org), have applied reinforcement learning (RL) to maximize production yields in fed-batch fermentation, moving beyond setpoint-based constraints on process optimization.
Source: Shutterstock
“This work is leveraging sensing technology and machine-learning (ML) control systems to improve bioreactor performance to get better performance out of existing bioreactors — as opposed to changing the bioreactor itself,” explains David Nathan, a technical program director at BioMADE, a federally sponsored Manufacturing Innovation Institute focused on bioindustrial manufacturing.
The ISU team focused on developing RL-supported sensors that could adapt to the unpredictable environmental variables that impact industrial fermentation and created an array of small bioreactors where the sensors could be trained. “Reinforcement learning applies well to anything that needs autonomous control where there are a lot of environmental variables that you can’t account for in a defined model. Fermentation fits into that space, because even though you might have a very well-defined model, there are still things that are different every time you’re running the reactor,” explains Nigel Reuel, professor of chemical and biological engineering at ISU. For example, he notes how slight variances in cell freezing and thawing, or even the position of cells within a freezer, can impact cell growth and subsequent reaction output.
The team has demonstrated the RL sensors at a Novonesis production plant as a proof-of-concept. “When it was tested at our site, we actually used broth pulled directly from a fermenter, so everything was representative of what we would be doing at scale. It proved that this technology worked out of the box in an industrially relevant scenario, and that the model can basically self-learn and stay current,” comments Mike Hess, senior manager of regional technical strategy at Novonesis. A partnership with Craft Biosolutions (Chicago, Ill.; www.craftbiosolutions.com) is now underway to commercialize the technology for a full-scale bioreactor that can actively acquire data to facilitate autonomous control.
“Obviously, something that measures real-time enzyme activity is of real interest commercially because right now, we rely on at-line measurements, meaning you get the results after the culture is finished. You then try to tie different conditions to better or worse performance and iterate on that information. These sensors speed this up so that it happens within minutes inside the reactor, and then if you couple that with the RL model, it can learn and make the improvements itself. It’s a step towards advanced-control, hands-free fermentation,” says Hess.
The team started their work on simple hydrolases (high-volume starch-degrading enzymes), but has now expanded the research into sensors for protease and phytase classes of enzymes.
“We have simulated real-world problems, such as what happens if a pump stops working and the feed stops, looking at how the autonomous program responds to that, based on its learning from all the data that you fed it before. We have actually shown how this system is observing something in real time and can change it. In this case, it was changing feeding strategies to then optimize the amount of enzyme you get out of a run. We’re really focused on not only how to generate that data, but how to harness it so that we can create a real ‘self-driving’ autonomous reactor,” adds Reuel.
A cornerstone of this work is practically showing that RL can go beyond setpoint control and dynamically adjust process inputs over the course of a batch to maximize final enzyme yield, even under process variations that make real-world fermentation operations difficult to control. Details of this project were published in the journal Biotechnology and Bioengineering. ■