add metrics section, issue #14 and #16

Author: mariojmdavid

content/06.quality_criteria.md

@@ -52,10 +52,10 @@ functionality.
 
 * **[SvcQC.Int01]** Whenever a new functionality is involved, integration
 testing MUST guarantee the operation of any previously-working interaction
-with external **Services**. [SQA-QC.Int01]
+with external **Services**. [SQA-QC.Int01].
   * When using APIs, contract testing MUST detect any disruption in the
   communication between provider and consumer endpoints, through the validation
-  of the API specification [SvcQC.Api01]
+  of the API specification [SvcQC.Api01].
 
 * **[SvcQC.Int02]** Integration testing SHOULD be automated.
 
@@ -74,14 +74,14 @@ design analysis or side-effects to external systems.
 **Service** claims to provide. [SQA-QC.Fun01].
   * **[SvcQC.Fun01.1]** When using APIs, contract testing MUST detect any
   disruption in the features exposed by the provider to the consumer, through
-  the validation of the API specification. [SvcQC.Api01]
+  the validation of the API specification. [SvcQC.Api01].
   * **[SvcQC.Fun01.2]** Functional tests SHOULD include the Web interface of
   the **Service**.
 
 * **[SvcQC.Fun02]** Functional tests SHOULD be checked automatically.
 
-* **[SvcQC.Fun03]** Functional tests SHOULD be provided by the developers of the underlying
-  software. [SQA-QC.Fun04].
+* **[SvcQC.Fun03]** Functional tests SHOULD be provided by the developers
+  of the underlying software. [SQA-QC.Fun04].
 
 ### Performace tests [SvcQC.Per]
 
@@ -121,6 +121,10 @@ to variable demand or workload for those service(s) [@doi:10.1186/s13677-019-013
 
 ### Documentation [SvcQC.Doc]
 
+Documentation is an integral part of any Software or Service. For example,
+it describes how and what users can use and interact with it, or how operators can
+deploy, configure and manage a given Software or Service.
+
 * **[SvcQC.Doc01]** Documentation MUST be available online, easily
 findable and accessible. [SQA-QC.Doc03].
 
@@ -159,23 +163,6 @@ The identified types of documentation and their RECOMMENDED content are:
     * Public API documentation (if applicable).
     * Command-line (CLI) reference (if applicable).
 
-### Policies [SvcQC.Pol]
-
-* **[SvcQC.Pol01]** The **Service** MUST include the following policy documents:
-  * **[SvcQC.Pol01.1]** Acceptable Usage Policy (AUP): Is a set of rules applied
-  by the owner, creator or administrator of a network, website, or service, that
-  restrict the ways in which the network, website or system may be used and sets
-  guidelines as to how it should be used. The AUP can also be referred as: Acceptable
-  Use Policy or Fair Use Policy,
-  * **[SvcQC.Pol01.2]** Access Policy or Terms of Use: represent a binding legal
-  contract between the users (and/or costumers), and the Provider of the **Service**.
-  The Access Policy mandates the users (and/or costumers) access to and the use of
-  the Provider’s **Service**.
-  * **[SvcQC.Pol01.3]** Privacy Policy: Data privacy statement informing the users
-  (and/or costumers), about which personal data is collected and processed when they
-  use and interact with the **Service**. It states which rights the users (and/or costumers)
-  have regarding the processing of their data.
-
 ### Security [SvcQC.Sec]
 
 Security assessment is essential for any production **Service**. While an 
@@ -243,46 +230,49 @@ good practices.
 
 * **[SvcQC.Sec07]** IaC testing, from [SvcQC.Aud02] criterion, MUST cover the
 security auditing of the IaC templates (aka _Security as Code_ or _SaC_) in
-order to assure the deployment of secured **Services**.
-  * For all the third-party dependencies used in the IaC templates (including
-  all kind of software artefacts, such as Linux packages or container-based
-  images):
-    * **[SvcQC.Sec07.1]** SaC MUST perform vulnerability scanning of the
-    artefact versions in use.
-    * **[SvcQC.Sec07.2]** SaC SHOULD verify that the artefacts are trusted and
-    digitally signed.
+order to assure the deployment of secured **Services**. For all the third-party
+dependencies used in the IaC templates (including all kind of software artefacts,
+such as Linux packages or container-based images):
+  * **[SvcQC.Sec07.1]** SaC MUST perform vulnerability scanning of the
+  artefact versions in use.
+  * **[SvcQC.Sec07.2]** SaC SHOULD verify that the artefacts are trusted and
+  digitally signed.
   * **[SvcQC.Sec07.3]** SaC MUST scan IaC templates to uncover misalignments
   with widely-accepted security policies from [SvcQC.Sec06] criteria, such as
   non-encrypted secrets or disabled audit logs.
   * **[SvcQC.Sec07.4]** SaC MAY be used to seek, in the IaC templates, for
   violations of security requirements outlined in the applicable regulations
   from criterion [SvcQC.Sec05].
 
-### Automated Deployment [SvcQC.Aud]
-
-The automated deployment of **Services** implies the use of code to install and
-configure them in the target infrastructures. Infrastructure as Code (IaC)
-templates allow operations team to treat service provisioning and deployment in
-a similar fashion as developers manage the software code.
-
-Consequently, IaC enables the paradigm of immutable infrastructure deployment
-and maintenance, where **Services** are never updated, but deprovisioned and
-redeployed. An immutable infrastructure simplifies maintenance and enhances
-repeteability and reliability.
+### Policies [SvcQC.Pol]
 
-* **[SvcQC.Aud01]** A production-ready **Service** SHOULD be deployed as a
-workable system with the minimal user or system administrator interaction
-leveraging IaC templates.
-  * **[SvcQC.Aud01.2]** Any future change to a deployed **Service** SHOULD be
-  done in the form of a new deployment, in order to preserve immutable
-  infrastructures.
+Policy documents describe what are the users expected behaviour when using the
+**Service**, how they can access it and what they can expect regarding privacy
+of their data.
 
-* **[SvcQC.Aud03]** IaC SHOULD be validated by specific (unit) testing
-frameworks for every change being done.
-  * IaC (unit) tests MUST be idempotent.
+* **[SvcQC.Pol01]** The **Service** MUST include the following policy documents:
+  * **[SvcQC.Pol01.1]** Acceptable Usage Policy (AUP): Is a set of rules applied
+  by the owner, creator or administrator of a network, website, or service, that
+  restrict the ways in which the network, website or system may be used and sets
+  guidelines as to how it should be used. The AUP can also be referred as: Acceptable
+  Use Policy or Fair Use Policy.
+  * **[SvcQC.Pol01.2]** Access Policy or Terms of Use: represent a binding legal
+  contract between the users (and/or costumers), and the Provider of the **Service**.
+  The Access Policy mandates the users (and/or costumers) access to and the use of
+  the Provider’s **Service**.
+  * **[SvcQC.Pol01.3]** Privacy Policy: Data privacy statement informing the users
+  (and/or costumers), about which personal data is collected and processed when they
+  use and interact with the **Service**. It states which rights the users (and/or costumers)
+  have regarding the processing of their data.
 
 ### Support [SvcQC.Sup]
 
+Support is the formal way by which users and operators of the **Service**
+communicate with other operators and/or developers of the **Service**,
+in case of problems, be it operational problems or bugs in the
+**Service** or underlying Software. Reporting of enhancements, improvements
+and even documentation issues.
+
 * **[SvcQC.Sup01]** The **Service** MUST have a tracker or helpdesk
 for operational and users issues.
 
@@ -311,18 +301,69 @@ where the **Service** is foreseen to be integrated.
   functional tests of criteria [SvcQC.Fun01.2].
   * **[SvcQC.Mon01.3]** The **Service** Web interface MAY be monitored. Use
   functional tests of criteria [SvcQC.Fun01.3].
+
 * **[SvcQC.Mon02]** The **Service** MUST be monitored for security-related
   criteria:
   * **[SvcQC.Mon02.1]** The **Service** MUST be monitored for public endpoints
   and APIs secured and strong ciphers for encryption. Use Security tests of
   criteria [SvcQC.Sec02] and [SvcQC.Sec05].
   * **[SvcQC.Mon02.2]** The **Service** SHOULD be monitored with DAST checks.
   Use Security tests of criteria [SvcQC.Sec06].
+
 * **[SvcQC.Mon03]** The **Service** MUST be monitored for infrastructure-related
   criteria:
   * **[SvcQC.Mon03.1]** IaC (unit) tests [SvcQC.Aud02] SHOULD be reused as
   monitoring tests, thus avoiding duplication.
 
+### Automated Deployment [SvcQC.Aud]
+
+The automated deployment of **Services** implies the use of code to install and
+configure them in the target infrastructures. Infrastructure as Code (IaC)
+templates allow operations team to treat service provisioning and deployment in
+a similar fashion as developers manage the software code.
+
+Consequently, IaC enables the paradigm of immutable infrastructure deployment
+and maintenance, where **Services** are never updated, but deprovisioned and
+redeployed. An immutable infrastructure simplifies maintenance and enhances
+repeteability and reliability.
+
+* **[SvcQC.Aud01]** A production-ready **Service** SHOULD be deployed as a
+workable system with the minimal user or system administrator interaction
+leveraging IaC templates.
+
+* **[SvcQC.Aud02]** Any future change to a deployed **Service** SHOULD be
+  done in the form of a new deployment, in order to preserve immutable
+  infrastructures.
+
+* **[SvcQC.Aud03]** IaC SHOULD be validated by specific (unit) testing
+frameworks for every change being done.
+  * **[SvcQC.Aud03.1]** IaC (unit) tests MUST be idempotent.
+
 ### Metrics [SvcQC.Met]
 
-* **[SvcQC.Met01]** The **Service** SHOULD have metrics.
+A metric is a quantifiable measure that is used to track and assess
+the status of a specific process.
+
+In the case of **Services**, some relevant metrics are the number
+of users registered in the **Service**, or using it actively. Also
+accounting is important to track resource usage per user or group
+of users, either or both computing and storage resources.
+
+Although the the metrics maybe published in external services, this
+is a common case in federated infrastructures such as EOSC.
+
+* **[SvcQC.Met01]** The **Service** SHOULD implement the collection
+of metrics.
+  * **[SvcQC.Met01.1]** The collection of metrics SHOULD be comulative
+  over time and timestamped, so that the values can be queried per
+  time interval.
+  * **[SvcQC.Met01.2]** The metric *Number of registered users* SHOULD
+  be collected.
+  * **[SvcQC.Met01.3]** The metric *Number of active users over a given period of time*
+  MAY be collected.
+  * **[SvcQC.Met01.4]** The metric *Amount of computing resources per user or per group*
+  MAY be collected. The metric unit depends on the type of service and infrastructure.
+  An example is CPU x hours.
+  * **[SvcQC.Met01.5]** The metric *Amount of storage resources per user or per group*
+  MAY be collected. The metric unit depends on the type of service and infrastructure.
+  An example is GByte x hours.