Optimizations on the first AI model achieved a DR of 99.2%, which outperformed the baseline by 4.2%. However, the FRR of 6.2% needed to be improved during the following weeks. Adding images, improving data label quality, and the recall optimization approach of the AI model—which is the performance metric targeted to decrease the FRR—were the primary goals in the improvement process. Finally, 3 months after implementing the deevio AI-Box, a second AI model was deployed that outperformed the baseline with a DR of 98.0% and a significant drop of the FRR from 4.9% to 1.3%, which is 0.4% better than the target FRR.
The performance of the AI model can still be further improved by optimizing the recall of the AI model and providing more well-labeled data in the future to decrease the FRR value close to 0%. The consequence might be that the DR value gets slightly worse. However, the addition of further data can solve this problem. All decisions taken in the optimization cycle are risk based and consider quality and business aspects. Creating a benchmark of the two systems not only confirmed the positive impact of AI, but also transformed the added value into tangible and comparable data that can be useful for business case considerations.
Use Case 2: Develop the Process Behind the Technology
In addition to the performance of new technology, its integration into an existing pharmaceutical environment is important, including its challenges. Therefore, processes were developed to integrate AI into a GxP environment. An important consideration was the evaluation of the most suitable algorithms to solve the specific problem statement. The deep learning (DL) method seemed to be a promising approach for our use case. DL is based on neural network architectures with multiple layers and has been successfully used to solve multiple problems in image recognition. The visual inspection process illustrated in Figure 3 identified AI algorithms as an algorithm set complementary to the traditional machine vision algorithms. Therefore, the process to deploy AI models according to Figure 5 was developed and adjusted to the AVI standard continuous improvement process to enable a smooth integration into the existing AVI process.
Previous experiences have taught us to use a supervised learning approach because of the possibility of freezing a model before the deployment on an AVI system. This enabled controlled learning and was an essential outcome for the qualification concept of an AI algorithm in a GxP environment. Furthermore, in the pharmaceutical validation processes, it is crucial to understand how and why AI models work. Therefore, integrated gradients were introduced to answer these questions and increase the understanding of AI models. This function aims to explain the relationship between a model’s predictions and its features by creating a colored mask that is overlaid on the original image. Based on the model’s decision, all important areas and pixels are then highlighted according to a color scheme that is instantly and intuitively visible to the user, which ultimately provides a better understanding of the model’s predictions.
Use Case 3: Business Insights from Implementing AI Algorithms into Existing Equipment
This specific use case explored the business perspective derived from the MVP. The development of an AI solution that was relatively easy and fast to implement and test as well as being cost efficient solved a real-world challenge. Further, it ultimately increased the awareness and understanding of AI technology, specifically the importance of image acquisition, capabilities of the hardware components, integration in current system architecture, and handling of large data sets. All these outcomes were consolidated and summarized as user requirements or implementation guidelines to support future AI projects. Finally, important lessons were learned to support a data-driven and AI-enabled organization, such as exploring required infrastructure, capabilities, interfaces, and processes; defining an AI governance; and establishing other valuable inputs to the AI strategy.
Key Takeaways
The impressive performance achieved by the AI MVP outperformed the existing system in only 3 months and highlighted the powerful potential of AI solutions. The AI retrofit project not only confirmed an improvement in the inspection performance but also enabled valuable insights into implementing AI technology in the manufacturing environment. On a technical level, user requirements—such as the importance of high-quality image acquisition, powerful hardware, and software components—were identified. One key success factor was establishing and maintaining a sustainable database, which set the foundation for other AI solutions.
The implementation of AI into an existing AVI machine also allowed us to develop guidelines and concepts to cope with existing processes and pharmaceutical regulations. Another significant outcome was using critical thinking and a risk-based approach when relying on AI algorithms. It is never a one-size-fits-all solution. Finally, the AI retrofit approach started with a clear value proposition, but it also leveraged additional business insights to generalize further AI implementations and enabled the organization to explore and improve AI capabilities for future projects.
Next Steps
An essential next step for the project team is to tackle the technology challenges identified during this project, such as the feasibility of updating the hardware components in the existing machine, equipping the vision system with the needed computing power, and integrating the AI solution in the AVI system, including different communication layers and user interactions (e.g., visualization or recipe and user management). The project’s successful outcome was also directly linked to the ability of the project team to develop new out-of-the-box ideas beyond the AI retrofit project, which consequently triggered new AI initiatives within the network and identified opportunities to improve AI capabilities. Finally, the project team is interested in fostering discussions around AI topics with other pharmaceutical companies, AI suppliers, and communities to exchange best practices and lessons learned to drive AI solutions in the pharmaceutical industry.