Quality Agreements for SaaS Solutions Intended for GxP Use

Software, whether delivered as a purchased product or via a SaaS model, should be developed and managed according to a formal process, including specifications governing the software content and documented testing verifying that the software is fit for purpose. The process should be formally documented as a quality system. The quality system should align with industry best practices and standards1^, 2^, 3^, 4^, 5^, 6^, 7 to help ensure quality, compliance with quality obligations, and IT security. Many regulatory requirements have a foundation in good engineering practices for IT controls.

SaaS is the capability provided to the consumer to use the provider’s applications on a cloud infrastructure. The applications are accessible from various client devices through either a thin client interface such as a web browser (e.g., web-based email) or a program interface. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.8 A SaaS service provider is the vendor providing the SaaS application.

Quality Agreements

A GxP-regulated organization, referred to here throughout as a regulated company, must have clear roles and responsibilities for using an SaaS solution, interacting with the SaaS provider, and ensuring the application meets the intended GxP use. Further, a quality agreement needs to be in place to ensure that applicable requirements are met in the most pragmatic way, as this will allow the life sciences industry to use as wide an array of suppliers as possible while leveraging their services.

The regulated company must ensure that the requirements to meet the GxP intended use are understood internally, evaluated against the SaaS provider's quality system for equivalencies, and fit for purpose. The regulated company should also leverage the SaaS provider’s IT services used to deliver the application to support its role in achieving GxP regulatory compliance. The regulated company should clearly document how it will leverage deliverables from the SaaS provider, the mechanism for maintaining documentation as current, and the retrieval process for the documentation (e.g., portal).

It is important to note that the SaaS provider is not subject to the same GxP regulations as the regulated company and that ultimate accountability for GxP requirements resides with the regulated company. The SaaS provider is responsible for ensuring that it has integrated quality controls and industry best practices into its software development life cycle and operational processes.

A quality agreement should define and document the key responsibilities for both the SaaS provider and the regulated company for confirming an application’s intended use, fit for purpose, and associated services, including required controls and measures to ensure data integrity is maintained. The quality agreement must not delegate GxP accountabilities to the SaaS provider.

Although it is possible that the key elements of a quality agreement are listed in other types of agreements, such as service level agreements, enterprise agreements, and master services agreements, it is important that the quality agreement is the one document that contains all of the agreed-upon quality responsibilities and defined roles of the regulated company and the SaaS provider. SaaS-appropriate terminology may be used within the quality agreement, so long as there is a common understanding of the intended meaning between the parties involved.

SAAS Application Quality Pillars

Many elements contribute to the overall quality of a SaaS solution. These elements may be grouped into the following key pillars: infrastructure quality, software quality, and service quality.

Infrastructure Quality

Of the three key pillars, infrastructure quality may be the most familiar to a quality professional because of the necessity that it resembles the standard quality agreement used in the pharmaceutical development and commercialization outsourcing paradigm. The quality of physical infrastructure components including data center facilities, computer hardware, and environmental and security controls will typically be verified using normal hardware management activity following good IT practice as part of the service provider’s quality system. These services should be fully leveraged in support of the regulated company’s regulatory requirements.

Software Quality

From a holistic point of view, software quality is the degree to which the software meets predetermined specifications, specified requirements, and/or user needs and expectations. It implies that the software is designed and built according to best practices for software engineering, providing both functional quality and structural quality attributes to a software product.

Software users will perceive functional quality when software operates as intended, providing expected functionality without errors. Given that some performance aspects of the application will depend on the user’s systems, it is important that the minimum specifications needed to connect to the application are well communicated and understood.

It can be challenging to measure software quality because different people may interpret the term differently. Software users will not perceive quality in the same way as software developers, corporate executives, legal representatives, or other stakeholders. Different perspectives will prioritize different criteria. For this reason, following existing standards and/or frameworks, which are well defined and widely accepted, is recommended.

As depicted in Figure 1, the ISO/IEC 25010 standard9 contains a framework to evaluate software product quality, which includes a set of eight software quality characteristics, with corresponding subcharacteristics. The regulated company and SaaS provider can use this tool as part of the software development process and/or selection of the SaaS application to ensure the suitability of the product for its intended use.

Service Quality

Service quality is likely to be the most impactful of the three quality pillars because this is what users and administrators will experience from an operational perspective as they interact with the SaaS application. The critical facets of service quality include ensuring (a) security processes and tools are fit for intended use; (b) accurate, timely, and comprehensive dissemination of information that may impact the regulated company’s intended use of the application (i.e., support, release management, etc.); (c) customer data are accessible only to authorized individuals (i.e., data classification, governance, etc.); (d) the accuracy and completeness of customer data and processing methods are safeguarded (i.e., security, change management, etc.) and the application and customer data remain accessible (i.e., resilience, reliability, maintenance, monitoring, capacity planning, backup, disaster recovery, etc.).

Need for a Quality Agreement

As stipulated by several industry regulations and guidance documents, 10^, 11^,12^, 13 formal agreements should be in place when leveraging computerized systems and services managed by third-party service providers. The nature and level of risk associated with the GxP processes supported by the SaaS application should be evaluated by the regulated company to help establish the areas of the quality agreement that will require the most attention.

To establish what should be included in a quality agreement, it is important to understand the specific challenges that can affect the quality of the hosted application from the regulated company’s perspective. These challenges include:

• Potential lack of visibility over the technology stack that supports the SaaS application, including other subservice providers and/or suppliers. For example, the SaaS application may be hosted by a separate infrastructure as a service (IaaS) provider.
• Functional changes made to the SaaS application that are not controlled by the regulated company.
• Delineated and defined responsibilities of the controls and processes between each party that maintain the SaaS solution in a state of control and may have a potential impact on data confidentiality, availability, and integrity.

By outlining specific activities and reporting requirements performed by the SaaS provider and the regulated company, the quality agreement can be used to mitigate the risks associated with these challenges. To accomplish this, it is necessary to understand the partition of roles and responsibilities combined with the regulated company’s determination of what quality attributes are important in the context of cloud-based GxP applications.

Recommended Quality Agreement Contents

The quality agreement serves as a binding document that lists the actions and commitments that the SaaS provider agrees to accept to meet the industry standards and quality requirements deemed relevant by the regulated company. The regulated company will need to ensure the SaaS provider can fulfill the quality requirements, and both parties must agree on the responsibilities for meeting these requirements. The following sections include considerations and recommended controls that should be included in the quality agreement to ensure the SaaS solution meets the agreed-upon quality standards.

The quality agreement should also specify the key service metrics or key performance indicators (KPIs) measured. Shared dependencies should be clearly stated in the quality agreement. Examples of key metrics and measures are included where applicable in the following sections.

Roles and Responsibilities

In any contractual relationship for the provision and delivery of cloud-based services impacting GxP processes, ultimate accountability lies with the regulated company, but the roles and responsibilities are allocated between the regulated company and the SaaS provider.

The roles and responsibilities section of the quality agreement should delineate the responsible party as either the regulated company or SaaS provider, or both, for a given quality requirement. For each responsibility listed, the regulated company is establishing the controls to support the quality obligations of their GxP requirements. The SaaS provider is agreeing to fulfill these requirements by using controls to support the regulated company’s quality obligations.

The responsibilities associated with managing subcontracted activities should be defined if any subcontractors or subservice providers are used for services that are critical to the quality of the SaaS application.

Quality System

The SaaS provider’s quality system will direct and implement quality objectives and policies that are intended to provide a consistent level of quality and service, outlining business and management philosophies, mission, and goals. This requirement is met with any document or group of policies and/or procedures that provide the direction and expectations generally associated with the regulated company quality manual.

Standard Operating Procedures

Standard operating procedures (SOPs) should be in place to cover all critical processes and be approved according to the SaaS provider’s quality system. These procedures must be current and reviewed periodically for continued relevance and accuracy. There must be a process in place to ensure that SOPs are updated, reviewed, approved, distributed, and trained on by the affected staff following their quality system.

Based on the intended use of the SaaS application and the applicable regulations, the following is a list of recommended topics that should be included in the service provider’s processes:

• Physical and environmental security
• Logical security
• System monitoring and maintenance
• Data retention
• Data classification
• Data access policy (ensure data are not deleted or altered by the service provider without permission)
• Data protection and confidentiality
• Software development
• Computer system verification
• Change management
• Incident management
• Risk management
• Documentation management
• Asset/inventory management
• Training management
• Data backup
• Disaster recovery
• Business continuity
• Vendor management

The SaaS provider should have a system in place to ensure training on internal policies, procedures, and technical aspects of the respective roles within the company. Examples of key training metrics include:

• Number of training gaps (incomplete training)
• Frequency of refresher training and training review

Software Development Life Cycle

The SaaS provider should follow well-defined processes and industry best practices for software development. Use of quality-driven coding and design practices generally results in software that is fit for the intended use and easy to maintain and update, which will be reflected in how well the software complies with or conforms to a given design (based on functional requirements or specifications) and aligns with user expectations. Failure to meet functional requirements or specifications would be managed as deficiencies or bugs, which can be prioritized and classified based on risk and impact. The risk must be addressed with an appropriate mitigation strategy.

The SaaS provider’s development process should follow good engineering practices using a software development life cycle (SDLC) process, incorporating the appropriate controls for software development and testing. The accountability for GxP compliance and validation remains with the regulated company and thus must be verified. This verification may include monitoring of key quality metrics and measures and adherence to activities specified within the quality agreement. Examples of key metrics and measures include:

• Number of critical feature or user requirement gaps or errors
• Number of critical bugs

The SaaS provider should have a system in place to ensure training on internal policies, procedures, and technical aspects of the respective roles within the company.

Security

Security measures should be implemented to minimize the risk of potential security breaches and unauthorized penetration of the software that results in stolen information, altered records, or other forms of accidental or malicious behavior. Security arrangements should comply with all local legislative requirements for data privacy, for example, General Data Protection Regulation (GDPR) in the EU. One example of an industry standard in the area of cloud security is The Consensus Assessment Initiative Questionnaire (CAIQ) v3.1.14

These measures may be broken down in terms of the people, processes, and technology aspects of security. From a technical perspective, the quality agreement should specify expectations surrounding the conduct of periodic vulnerability analysis and evidence of penetration testing. In terms of processes, the SaaS provider must implement the necessary controls that describe how access to the backend of the system is provided and how privileged access is managed. With respect to people, the SaaS provider and regulated company should maintain vetting processes for any personnel with privileged access, ensuring there is a limited number of personnel with administrative access supporting the application.

A strategy for communicating and escalating security incidents must be defined. The SaaS provider must notify the regulated company of any known security breaches within a specified, agreed-to timeframe. The quantity and severity of vulnerabilities found in the software system need to be reported. Additional consideration may be required if parts of the system are managed by a third party. Examples of key metrics and measures include:

• Frequency of periodic penetration testing and vulnerability scanning
• Number of known security breaches
• Time taken to investigate and resolve security incidents
• Maximum time before a security breach is reported

Data Integrity and Record Management

For SaaS applications that manage GxP electronic records under the definition of 21 CFR Part 1115 and EudraLex Volume 4 Chapter 4,16 the quality agreement should identify the expected technical and procedural control objectives that should be in place to ensure the integrity of data stored within the application. The SaaS provider should identify how these control objectives are met with a balance of technical, procedural, and behavioral controls.

Given that several of the controls for ensuring data integrity are shared between the SaaS provider and the regulated company, it is important to identify each party’s responsibilities.

Shared Data Integrity Responsibilities

Activities should be clearly delineated and specified in a roles and responsibilities table; see the Appendix that covers all generation, processing, review, reporting, archiving, and retrieval of GxP data in a process that ensures data integrity at all steps.

The quality agreement should also indicate whether there are any specific technical dependencies or assumptions with regard to how the regulated company can generate accurate and complete copies of the records contained within the application, including the corresponding audit trails for each record.

Audit Trail Considerations

For applications that manage electronic records under the definition of US FDA 21 CFR Part 1115 and EudraLex Volume 4 Chapter 416 and that are required to maintain audit trails, the quality agreement should identify the expected technical and procedural controls that should be in place to ensure the data integrity requirements of the audit trail data.

The main purpose of the audit trail is to provide assurance concerning the integrity of the electronic record; therefore, a properly implemented au-dit trail should have the following key characteristics:

• Technical: The audit trail entries are generated by the computer system when an electronic record is created, modified, or deleted by a user.
• Secure: Audit trail data must be stored in a secure manner and must not be editable by any user.
• Contemporaneous: Each audit trail entry must be time-stamped according to a controlled clock that cannot be altered by users once the time has been set. The time should be based on either central server time or local time, so long as it is clear in which time zone the entry was performed.
• Traceable: Updates made to records must not obscure previous values and where required by regulations, the reason for changing the data and the person making the change must also be recorded.
• Archived: The audit trail must be retained as long as the electronic record is required to be stored.
• Available: The audit trail must be available for review and copying.

Software Release Cycle

In most cases where significant customer configuration of the SaaS application is needed, the SaaS provider will be expected to provide the SaaS application users access to a preproduction environment where they can assess the impact of upcoming changes, perform any necessary regression testing, and train users before these releases are pushed to the production environment.

The quality agreement should indicate the types of environments the users will have access to and provide as much clarity regarding the release process as possible, including (a) agreed-upon release frequency; (b) publication and extent of associated release notes; (c) impact assessments identify-ing the key features/functions of the system that were updated; and (d) time allotted for the regulated company to have access to a new release in a preproduction environment where they can perform impact evaluation, testing, and/or user training on upcoming features/functionality prior to release into the production environment. Examples of key metrics and measures include:

• Frequency of updates
• Time taken to deploy updates
• Number of bugs found in new releases
• Maintenance downtime

Testing

The SaaS provider should be able to document and present the methods used to ensure that the processes/systems are operating as intended and that the output is valid. The SaaS provider should also provide documented evidence to demonstrate that their processes/systems have met agreed/specified user requirements.

The testing and verification by the regulated company should be commensurate with the risk to the GxP data, patient, and processes covered. This testing must provide evidence demonstrating data integrity and security and strict confidentiality of all data. The regulated company needs to evaluate the SaaS provider’s documentation package in view of their own risks. If additional controls or extended testing is in order, it is documented in the quality agreement. Examples of key metrics and measures include the following:

• Number of incidents, problems, and changes aligned to failures
• Percentage of requirements/test coverage

Documentation

It is acceptable practice for a service provider to maintain their system documentation within a tool providing traceability to the components within their software development life cycle. System documentation including requirements specifications, configuration and design specifications, and architecture and data flow diagrams should be developed and maintained. The specifications should be current such that they demonstrate control and reflect the current version of the application and are readily retrievable, if required. Changes to system specifications and requirements should be implemented in a controlled manner following the service provider’s internal change management processes.

Change Control

The SaaS provider will ensure quality and the contractually agreed-upon level of service regarding release management and the cadence for delivery. Releases must be implemented by a formal change control process ensuring a state of control through management of changes to prevent unintended consequences.

All changes are executed per internal procedures in a controlled manner to evaluate and mitigate any potential negative impact to the regulated company’s data. In the unlikely event that a planned change cannot eliminate a negative impact or there are any changes that could impact regulated company data, the regulated company should be notified prior to the change. Example of key metrics and measures include:

• Number of major (customer-impacting) changes
• Number of emergency changes
• Time taken to perform changes
• Number of times a change has caused issues in the system

Service Support and Communications

The quality agreement should also specify the key information to be provided by the SaaS provider. This could include but is not limited to:

• Release notes with functional impact assessments
• Public-facing product roadmap with enough detail to allow the customer to evaluate new functionality and impact of potential changes to existing functionality
• Third-party audit reports, attestations, and/or certifications (e.g., SOC 2 Type II, ISO 9001, ISO 27001, FedRAMP)
• Statement or policy affirming commitment with respect to maintaining a quality-controlled environment through which the service is delivered, including a description of SOPs and verification methods used to establish that the service functions in the manner intended
• Statement or policy governing customer data destruction upon contract termination and how data will be transferred back to the customer upon contract termination and/or an interruption in service

Examples of key metrics and measures include:

• Time taken to resolve critical bugs
• Notification lead time for planned system maintenance that could impact system availability
• Notification lead time for addition of new functionality and change or removal of existing system functionality (e.g., release notes)
• Notification of a change in services including mergers, acquisitions, and divestitures
• Allotted time for data repatriation in case of contract termination

The SaaS provider is responsible for delivering a quality solution following industry and good engineering best practices.

Monitoring

Monitoring processes and tools should be deployed to identify issues that affect the performance, reliability, and security of the application. Metrics or KPIs used to measure performance and reliability should be defined and used to identify issues in real time and as a predictive indicator of potential issues that could then be resolved before user experience is affected. How and when these metrics and indicators need to be communicated to impacted users should be included in the quality agreement. Examples of key metrics and measures include:

• At a minimum, quarterly application availability/up time percentage commitment levels
• Performance and reliability monitoring
• Vulnerability management of access to customer data
• Process capacity and usage monitoring
• Network connectivity issues

Service/Application Review

Service review and/or equivalent process by the SaaS provider of the SaaS application/process should be conducted and documented following an IT industry standard. This approach will ensure continued quality and the ability to meet the requirements for the contracted services. The goal of this documented review is to provide evidence of the continuing controlled state for the product/process and continuous improvement. Examples of key metrics and measures include:

• Frequency of service and application reviews
• Incidents and problems
• Component failures
• Database performance

Supplier Assessment

Depending on the intended use and the outcome of the risk assessment, the structure of the assessment could consist of an on-site audit, remote audit, postal audit, or a review of available documentation and certifications such as SOC 2 Type 2 reports, ISO 9001, and ISO 27001. 1^, 5^, 6

Independent assessments of the groups, systems, and processes must be conducted to ensure conformance with the quality and regulatory items listed in the quality agreement. The type and depth of the assessment would be directed by the quality unit of the regulated company using risk management to determine how to structure the assessment commensurate with the risk associated with the intended use.

For additional information and guidance regarding the potential use of SOC 2 Type 2 to support the assessment process, please refer to the Pharmaceutical Engineering® article, “Application of SOC 2+ Process to Assessment of GxP Suppliers of Services.” 17. Examples of key metrics and measures include:

• Number of audit observations
• Responsiveness to address observations

System Retirement/Contract Termination

The quality agreement should include provisions for transfer of data back to the regulated company upon system retirement and/or contract termination. In this context, it is important to ensure there is a commonly understood definition of the term “user data.” For example, does it include audit trails, electronic signature manifestations, and previous versions of records?

It may also be important to specify the format of the extracted data to be provided and to identify the facilities and services that will be available to extract the data at the end of the contract. For example, some SaaS vendors may provide an application programming interface that allows users to retrieve their data programmatically.

Conclusion

The accountability for GxP requirements, including the integrity and use of the data, resides with the regulated company. The SaaS provider is responsible for delivering a quality solution following industry and good engineering best practices. Many tools are available to the regulated company to assist with the decision to use a specific SaaS provider. ISO certifications and SOC2 reports are just a few examples. The SaaS provider should have established processes and documentation following industry standards in place that can be utilized by the regulated company.

Automated IT services and monitoring tools provide services that can be directly leveraged by the regulated user in a proactive approach. Security vulnerabilities, database performance, component failures, application, and platform errors can be routinely monitored to provide real-time feedback on the status of the IT services and supporting infrastructure. These items should be outlined in a detailed quality agreement that delineates the specifics between the companies’ management of the quality product life cycle considerations.

By implementing a quality agreement, both parties acknowledge and accept their responsibilities and commit to meeting quality objectives that can be measured via agreed-upon quality metrics and KPIs. The appendix identifies and considers the responsibilities shared between the regulated company and the SaaS provider when establishing the quality agreement. The specific activities and acceptance criteria may differ from case to case and are based on who is responsible for each activity, the regulated company or the SaaS provider.I^, II^, III

Acknowledgments

The authors thank the following for their support and contribution to this article: Judy Samardelis, Director Quality Assurance, Syneos Health, and Michael Osborne, Head of Quality, Cornerstone OnDemand.