Remote Acceptance Testing of Automation Projects

Before the COVID-19 pandemic, acceptance testing of automation projects was conducted in person, with multiple subject matter experts attending. Factory acceptance tests (FATs) were conducted on the system integrator’s (SI) premises, executed by a representative of the SI and witnessed by a representative of the client. FATs could be a simulated operation of the automation system on a bench or the full operation of a machine, skid (prefabricated modular process), or system. Site acceptance tests (SATs) were conducted on client sites in situ, after initial commissioning and before operations. During SATs, all field equipment is connected and operational.

Acceptance test protocols are executed during FATs and SATs. They are approved before execution and during execution; the tester and reviewer/witness initial and date each test and sign that the tests were conducted correctly. The aim of the tests is to verify that the system is compliant with the requirements and the design specification. The granularity of the requirements can vary between industries. In the pharmaceutical sector, the ISPE Good Automated Manufacturing Practice (GAMP)International Society for Pharmaceutical Engineering. GAMP® 5 Guide: A Risk-Based Approach to Compliant GxP Computerized Systems. North Bethesda, MD: International Society for Pharmaceutical Engineering, 2008. is used as a guide.

Measures had been put in place to reduce the close contact of engineers during COVID-19 restrictions, including working from home, and this presented a unique set of challenges for those working in the sector. The writing of requirements, design documentation, and test scripts, and the configuration of hardware and software, can be conducted from home. However, conducting remote factory acceptance testing requires a methodology to allow projects to proceed to the site for site acceptance testing for validation before handover. The objective of our research: What techniques and procedures are required to enable reliable remote acceptance testing of automation projects?

This was a qualitative field study conducted on five automation projects across three SIs and four clients. One client is a medical device manufacturer in Ireland. Another two clients are pharmaceutical manufacturers, also in Ireland. The fourth client is a waste recycling company in Australia. Seven interviews with automation testers and client reviewers were conducted. Qualitative data analysis using the thematic analysis methodology was performed. As reported by Pazhayattil and colleagues, remote testing of biopharmaceutical manufacturing facilities has been successfully executed.Pazhayattil, A. B., K. Kotini, M. Kodidela, and M. Scribner. “Accelerating Biopharmaceutical Virtual FATs in a Pandemic.” Pharmaceutical Engineering 32, no. 3 (2022): 60–68.

Due to the COVID-19 pandemic and associated restrictions, remote acceptance testing was the only way these tests could be conducted. In these scenarios, the tester is present with the equipment and the reviewer is remote, either at work or at home. Computer screens are shared via remote viewing software; equipment is viewed using webcams, smartphones, or wearable cameras; and participants communicate using telephone, voice over internet protocol (VOIP), or text message. Despite the prevalence of remote acceptance testing in the COVID-19 pandemic, a search for literature on remote acceptance testing yielded no appropriate results.

Research Objective

The research objective is to determine the techniques and procedures that are required to enable reliable remote acceptance testing of automation projects while maintaining compliance with regulatory requirements and industry norms. During the COVID-19 pandemic, the US FDA issued guidelines for remote evaluations of drug manufacturing and bioresearch facilities.US Food and Drug Administration. “Remote Interactive Evaluations of Drug Manufacturing and Bioresearch Monitoring Facilities During the COVID-19 Public Health Emergency—Guidance for Industry.” April 2021. https://www.fda.gov/media/147582/download This is preliminary research based on a limited number of participants with a view to prompt publication.

Research Questions

The following research questions serve to achieve the research objective:

• What techniques and technologies are employed to facilitate remote testing?
• What are the advantages and disadvantages of remote testing compared to in-person testing?
• How are projects affected from commercial, quality, and schedule points of view?

Research Design

As a sociotechnical system, research in the field of information systems needs to be defined in a number of ways in order to establish the best approach, methodology, data collection, and analysis choices.

Philosophical Overview

The philosophy chosen, and hence the methodology, depends primarily on where the research sits on the objectivism–subjectivism continuum. Without going into the detail of how it was established, the author chose a value-free axiology, a conventional ontology, and an opinion-based epistemology.Benbasat, I., D. K. Goldstein, and M. Mead. “The Case Research Strategy in Studies of Information Systems.” MIS Quarterly 11, no. 3 (1987): 369–86. doi:10.2307/248684 The combination of regulation and subjectivism leads to an interpretivist philosophy of research.Saunders, M. N. K., P. Lewis, and A. Thornhill. Research Methods for Business Students, 7th ed. Harlow, Essex, England: Pearson Education Limited, 2016. In this article, the phenomenologist paradigm approach is taken, i.e., using the lived experience of the participants to understand the phenomena. Trustworthiness and authenticity are measures of quality in interpretivism.Denzin, N. K. and Y. S. Lincoln. The SAGE Handbook of Qualitative Research, 5th ed. Los Angeles, CA, US: Sage Publications, 2018.

Theory Development and Methodology

In this research, themes emerge from the data collected, which in this case were interviews with practitioners. This is the definition of the inductive approach to theory development. The methodology, strategy, and time horizon were chosen with the development of a timely report in mind. Forgoing the luxury of triangulation, due to limited access because of COVID-19, a monomethod was chosen. As the research lent itself to a qualitative study, the quantitative strategies of experimentation and survey were discounted. It was decided that a case studyEisenhardt, K. M. “Building Theories from Case Study Research.” The Academy of Management Review 14, no. 4 (1989): 532–50. doi: 10.2307/258557 of recently executed acceptance tests would be the most practical approach. As the scope and depth of the investigation is narrow, it has been considered an exploratory field study. In order to produce a timely report, it was decided to take a cross-sectional approach. Future research could be conducted after the COVID-19 pandemic for comparative results.

Data Overview

Access was limited to existing contacts including work colleagues and clients of the researcher. This enabled short lead times to conduct interviews and had the further advantage of a familiarity with the context. The personal credibility of the researcher and the goodwill between the researcher and the participants aided the access to participants and data collection.

The sampling was of a nonprobability nature, purposive and typical case, selecting engineers who had performed remote acceptance testing since the start of the pandemic. Participants were asked to recommend other participants and one extra contact was found using this snowball method.

Seven remote video conference interviews were conducted with participants of remote acceptance tests using Microsoft Teams. Four were testers, three were reviewers. All were male. The range of experience varied, with two having over 20 years of industry experience, three having approximately 10 years of experience, and two having less than 5 years in the industry. Seven companies were represented: two automation and control SIs, four clients, and a project management company. The client companies were two pharmaceutical manufacturers, a medical device manufacturer, and a waste recycling company. One reviewer worked for a project management company on behalf of the pharmaceutical client. The remote testing discussions were conducted on five projects. Three were in Ireland, one was in Australia, and one was in Germany. For the German project, the reviewers were based in Ireland and for the Australian project, the tester was based in Ireland.

The interviews ranged from just under 10 minutes to just over 20 minutes, resulting in transcripts with word counts from approximately 1,200 words to over 2,500 words. In total, approximately 2 hours of recorded interviews resulted in 12,150 words, transcribed (Table 1).

Qualitative Data Analysis

There are a number of qualitative data analysis techniques available: grounded theory,Glaser B. G. and A. L. Strauss. The Discovery of Grounded Theory: Strategies for Qualitative Research. Hawthorne, NY, US: Aldine de Gruyter, 1967.^, Charmaz, K. Constructing Grounded Theory, 2nd ed. Los Angeles, CA, US: Sage Publications, 2014. thematic analysis,Braun, V. and V. Clarke. “Using Thematic Analysis in Psychology,” Qualitative Research in Psychology 3, no. 2 (2006): 77–101. doi: 10.1191/1478088706qp063oa narrative analysis,Riessman, C. K. Narrative Analysis, Vol. 30. Newbury Park, CA, US: Sage Publications, 1993. discourse analysis,Harris, Z. S. “Discourse Analysis: A Sample Text,” Language (Baltimore) 28, no. 4 (1952): 474–494. doi: 10.2307/409683 content analysis,Smith, B. L., H. D. Lasswell, and R. D. Casey. Propaganda, Communication, and Public Opinion: A Comprehensive Reference Guide. Princeton, NJ, US: Princeton University Press, 1946. and interpretative phenomenological analysis (IPA).Smith, J. A., M. Jarman, and M. Osborn, “Doing Interpretative Phenomenological Analysis.” Qualitative Health Psychology: Theories and Methods (1999): 218–40 The chosen one should suit the research design. Thematic analysis was chosen because of its flexibility and adaptability, being independent of any research philosophy. Thematic analysis as outlined by Braun and Clarke uses six phases, as shown in Table 2.

All six phases were used in this field study. The interviews were transcribed and read to gain familiarity with the data. The transcriptions were imported into the computer-assisted qualitative data analysis software (CAQDAS) NVivoQSR International, NVivo. 2021. https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home to generate initial codes from the transcripts. Once generated, the codes were collated into themes and the themes were again categorized and a thematic map was created.

CAQDAS is a tool for transparent display of codified data but the researcher still uses the data analysis method of choice and sets up the CAQDAS to accommodate that method.O’Flaherty, B., and J. Whalley, “Qualitative Analysis Software Applied to IS Research-Developing a Coding Strategy,” European Conference on Information Systems (ECIS) 2004 Proceedings (2004): 123. The CAQDAS provides a transparent and portable audit trail of the researcher’s efforts.

Ethics

The research was submitted to the University College Cork (UCC) Social Research Ethics Committee (SREC) for approval. The participants are either work colleagues or clients of the researcher/interviewer, which could be perceived as a conflict of interest in that the results could be manipulated to benefit the company. A perception of coercion to participate could also arise. These considerations were mitigated by giving full transparency of the nature of the research to all participants and receiving signed informed consent forms.

Confidentiality concerns were addressed by the anonymization of all persons and organizations in the transcripts, the deletion of interview record-ings after transcription, and the plan to destroy the interview transcripts after publication. All emails scheduling the interviews were deleted. Interview recordings and anonymized written transcripts are stored on the university’s secure server before destruction. There should be no General Data Protection Regulation (GDPR) concerns, as anonymized data do not fall under GDPR regulations.

Results

Codes

After the initial search, 46 codes emerged. These are shown in Table 3.

The codes were then distributed across nine subthemes, and the subthemes were further analyzed and organized under three main themes, as shown in Table 4.

Three Pillars of Remote Acceptance Testing

The requirements to enable remote acceptance testing were collected into the subthemes of technology and communication requirements, procedural requirements, and human factors. These were collected into the theme “requirements/enablers.”

Technology and communication requirements

Essential to the operation of remote testing, the technology and communication requirements include the ability of the participants to see and hear each other, to see graphical screens, to see equipment installed and operational, and to view documents being completed and signed. This is achieved by off-the-shelf technology. Zoom, MS Teams, and Webex are some of the video conferencing systems used successfully. Webcams, smartphone cameras, and sports action cameras were also used to allow equipment to be inspected remotely. This was used to confirm that the correct hardware was installed, that it was constructed correctly, and that it used the specified techniques.

The objective of our research: What techniques and procedures are required to enable reliable remote acceptance testing of automation projects?

Procedural requirements

During in-person testing, test protocols are completed (pass/fail, initial and date) by the tester, and the reviewer signed that they are satisfied that all testing is complete and successful. For remote testing, several procedural changes have been made to accommodate the capturing of document approval. For some, the protocols were signed on camera. In other cases, the tester signed the protocol and the document was scanned and emailed to the reviewer daily, weekly, or on completion. In some cases, the tester was granted the authority to sign on the reviewers’ behalf once the reviewer was satisfied.

Human factors

The participants described various ways in which the lack of personal interaction hindered the testing. The rapport established during in-person testing is missing.

• Reviewer 3: “In a real FAT, you go have dinner, you have a few beers, and the next day, you’d be on first name terms. When you’re doing a remote, that doesn’t happen.”

The lack of the firsthand experience meant that more explicit explanation was required.

• Tester 2: “In some ways, you needed longer to explain things. If a client had been there, sitting there, doing it, interacting with it themselves, they would have seen it more clearly and you might not have had to spend more time explaining it then.”

With the lack of rapport, more effort had to be made to establish trust between the participants. This was made all the more difficult by the perceived ease with which mistakes could be overlooked in the remote scenario.

• Tester 2: “You do need understanding from customers that will have that level of trust and give you credit for being honest. When you build relationships with people, it’s easy for them to do that.”
• Tester 2: “It was the customer who mentioned it in this case, that there has to be a degree of trust. And they trusted that I was being honest in what I was saying.”
• Reviewer 1: “I’ve been surprised about the professionality and honesty of the people involved.”

Sentiment

Codes were organized into the subthemes of positive, negative, and ambiguous sentiment. These subthemes were collected into the theme “sentiment.”

Some examples of negative sentiment include the lack of visibility.

• Tester 1: “Maybe from the point of view of seeing what he wanted to see or experiencing what he would have wanted to experience, it probably wasn’t ideal. Certainly not as effective as in-person testing.”

Another disadvantage was the lack of the personal connection.

• Reviewer 2: “It’s a disadvantage that I couldn’t actually meet the clients in person or the people I’ve been working with. The personal communication, the personal touch.”

Positive sentiment included the ability to multitask and the ability to have subject matter experts attend for short productive spells:

• Reviewer 2: “Yes, so maybe with the likes of online, we didn’t have to waste time. While he was doing that, I was able to do other work.”
• Reviewer 2: “Like my supervisor could have dialed in. All I had to do was IM him, ‘could you dial in for a minute?’ and he came in for 10 minutes to figure out the problem and he could go away then again with his day’s work and he’s not tied up all day at a FAT.”

The remaining codes were collected into subthemes of future, time, and miscellaneous. These subthemes were collected under “miscellaneous.”

Time saved traveling to and from the test site was seen as an advantage.

• Reviewer 3: “That would normally be a full-day exercise, a flight over the night before. And we got it done in 2.5 hours.”
• Tester 3: “Not so much for us, but long term if there was another FAT and it had to be at the client side or abroad, you’re looking at time saved traveling, hotels, money, all that.”

Some ambiguity was expressed.

• Reviewer 1: “I would say that pros and cons were really 50/50.”
• Tester 2: “Plusses and minuses in that way, really for me.”

Tensions

Once coding was complete, the data were queried using a matrix coding query to establish agreements and tensions between the testers and the reviewers. A matrix coding query creates a table to find intersections of codes, to discover patterns [15]. Tester sentiment was split evenly between positive and negative sentiment, but reviewers had an overwhelmingly negative sentiment about the experience (Figure 1).

When asked if remote testing took more or less time than in-person testing (not including time saved on travel), both reviewers and testers were divided in their opinions on an almost equal basis (Figure 2).

Testers and reviewers overwhelmingly agree that they would prefer not to continue using remote testing when the COVID-19 restrictions are lifted. Both groups would prefer to return in-person testing (Figure 3).

Conclusion

The ability to perform remote acceptance testing has been implemented as a workaround in the period of travel and meeting restrictions imposed by the COVID-19 pandemic. It has allowed automation projects to progress to site. Methods, procedures, and techniques that would have been rejected as unacceptable before the COVID-19 pandemic are now de rigueur.

However, after analyzing the data collected in the seven interviews, it is clear that the engineers involved would prefer to conduct the testing in person. They felt that the rapport between colleagues working together builds trust which, along with the unrestricted visibility available in situ, allows for a more efficient and effective test. The major advantage identified was saving time and cost associated with travel to test sites. When one or more parties is required to travel internationally, the savings can be considerable. The savings may outweigh other considerations after COVID-19 restrictions are lifted.

Avenues for future research include investigating the savings to be made by reducing international travel and accommodation costs, studying the dichotomy between the commercial savings and the inconvenience of the testers, and researching the impact of reduced communication and relationship building resulting from remote testing.