Difference between revisions of "Training:PhD Course on Computational Ontologies @ University of Bologna 2011/Ontology Testing:Method-based testing" < Training:PhD Course on Computational Ontologies @ University of Bologna 2011

Revision as of 13:51, 9 May 2011

Method 1: Unit testing through SPARQL queries

Parts of the ontology requirements are usually expressed as competency questions (CQs), i.e. natural language questions that the ontology should be able to provide answers to. One way to practically allow the ontology to answer such questions is to reformulate them as SPARQL queries, in order to retrieve the appropriate facts from the knowledge base. This also introduces a way of testing your ontology module!

Assuming that your ontology attempts to solve the two CQs, CQ1 and CQ2. Then each of those should be possible to formulate as a SPARQL query, Q1 and Q2, over the ontology module, such that the retrieved instances and/or literal values constitute the answer to the CQs. Q1 and Q2 could be considered as unit tests for the module, since they assure that the requirements CQ1 and CQ2 are actually met.

This method of testing contains the following steps:

Retrieve (and read) all the CQs that are realized by the module in question.
For each CQ reformulate it as a SPARQL query over the ontology module, using the classes, properties, and instances that the module provides.
Add test data to your module, by creating a new "test case"-module importing the module to be tested and adding facts to it. The test data can come from the initial customer story, it can be provided by a domain expert, or it has to be "invented" by the developer. Make sure the test data covers the classes and properties that are present in the SPARQL query(-ies) from step 2.
For each SPARQL query developed in step 2 decide, based on the test data added in 3, what a successful test should return, i.e. for each SPARQL query, what instances and/or literal values should be returned?
Run the SPARQL queries over your "test case"-module and record the results.
Compare the actual results (from step 5) to the expected ones (from step 4), and decide if the test was successful or not.
For any unsuccessful tests, analyze the reason for the failure by inspecting the ontology. Document the error/problem, so that it can be resolved by you or other ontology developers.

Task:

Download the following ontology module xxx.owl, and put it in your NeOn Toolkit workspace folder. Access the ontology through the NeOn toolkit. Start the SPARQL plugin. The requirements for this module are the following:

CQs:

...
...
...

Contextual statements:

...

Reasoning requirements:

...

Now, use the method described above to test the ontology module. While you are working, document what you are doing, e.g. are you following the steps exactly or not, how are you performing your task in practice, how long time does it take, and what errors are you able to find. When you feel satisfied with the testing, please answer the questionnaire for method 1.

Method 2: Performing inferences

Although CQs express what information the ontology should be able to provide, they do not say how this information is produced, i.e. if it is input into the ontology explicitly as assertions or produced from other facts through some inference mechanism. Therefore an ontology can have additional requirements, describing some desired inferences that the ontology should support. For instance, in an ontology about people we may define the class "parent" and say that any person who has at least one child is a parent. If we are not expecting the information about being a parent or not to be explicitly entered into the ontology's knowledge base, we may instead expect it to be derived from the presence of "hasChild" relations. This is a reasoning requirement that requires the ontology to include the appropriate axioms to make this inference.

To test the desired inferences of the ontology module, follow these steps:

Inspect the explicit reasoning requirements that you can find among the requirements of the module, if any. Also, determine if there are any implicit reasoning requirements, i.e. reasoning requirements that should be added to the set of requirements. This may include some interaction with the customer or a domain expert, if the requirements are ambiguous.
For each such requirement (in the list from step 1), list what is the input and expected output, e.g. what facts (triples) need to be present before actually running the inference engine, and what facts (triples) can be produced by the inference procedure.
Create a new "test case"-module, by importing the module to be tested and adding facts to it. For each reasoning requirement in the list from step 1, add some facts that can be considered as the "input" to the reasoning process (according to the result of step 2). Add both facts that will produce the inference, and facts that should not.
List the expected inferences that should be made based on your test data, i.e. the expected test output.
Run an OWL reasoner over the "test case"-module and record the results.
Compare the actual results (from step 5) with the expected ones (from step 4), and decide if the test of each reasoning requirement was successful or not. A test is successful if it produces the expected inferences, and no harmful side-effects, i.e. no additional inferences that add undesirable or incorrect facts to the knowledge base.
For any unsuccessful tests, analyze the reason for the failure by inspecting the ontology. Document the problem, so that it can be solved by you or by other ontology developers.

Task:

Load the same ontology as you used for Method 1 (you see the requirements above).

Now, use the method described above to test the ontology module. While you are working, document what you are doing, e.g. are you following the steps exactly or not, how are you performing your task in practice, how long time does it take, and what errors are you able to find. When you feel satisfied with the testing, please answer the questionnaire for method 2.

Method 3: Performing "stress tests"

Although the ability to perform correct inferences, and provide the resulting information as a result of queries, allow us to see that the ontology actually realizes its requirements, another important characteristic of an ontology is to allow as few erroneous facts and/or inferences as possible. A high-quality ontology allows exactly the desired inferences and queries, while avoiding to produce irrelevant or erroneous side-effects. It may also be desirable to be able to check input data against some constraints and business rules.

This category of testing can be compared to software testing, when usually a system is fed random data, or data considered as "end conditions" for the data, i.e. the extremes of value ranges, in order to check its robustness and capability to handle unexpected input. When dealing with ontologies, we are not expecting error messages and recovery strategies, as for software, but rather that erroneous facts and data is detected in the first place. One way to detect such problems is by using the consistency checking facilities of a reasoning engine. A high-quality ontology facilitates the reasoner to detect inconsistencies when inappropriate or erroneous facts are entered.

For instance, in a user model ontology with the classes "female user" and "male user" where this information is going to be collected form a form with a radio button the classes should be disjoint, i.e. since no user can select both alternatives from the form any given user can only be instance of one class. After detecting such implicit requirements, or common sense constraints, they can be tested by entering obviously inconsistent facts, and checking that the reasoner is able to detect the inconsistency.

To test this ability of the ontology, the following steps can be used:

List the explicit requirements of the ontology, i.e. CQs, reasoning requirements and other constraints that have been explicitly expressed.
Determine if there are any implicit requirements, in the form of additional constraints that can be derived from the context but were not previously expressed as requirements.
For each of the requirements, determine if there is a range of values that constrain the acceptable input or output that should be allowed by the ontology. For each constraint, determine at least one case (literal value or fact) that lies outside of that range, and what response would be expected to get from the reasoner given that input.
Create a new "test case"-module, by importing the module to be tested and adding facts to it (see next steps).
Add each of the cases listen in 3) in turn to the "test case"-module, and for each one, run the reasoner and record the result.
For each run in 5) check the result against the expected result determined in 3), and decide if the test was successful or not. A test is successful if it produces the expected inferences, e.g. an inconsistency, and no harmful side-effects.
For any unsuccessful tests, analyze the reason for the failure by inspecting the ontology. Document the problem, so that it can be solved by the ontology developers.

Task:

Download the following ontology module xxx.owl, and put it in your NeOn Toolkit workspace folder. Access the ontology through the NeOn toolkit. Start the reasoning plugin. The requirements for this module are the following:

...
...
...

Now, use the method described above to test the ontology module. While you are working, document what you are doing, e.g. are you following the steps exactly or not, how are you performing your task in practice, how long time does it take, and what kind of errors are you able to find. When you feel satisfied with the testing, please answer the questionnaire method 3.

@@ Line 37: / Line 37: @@
-Although CQs express what information the ontology should be able to provide, they do not say how this information is produced, i.e. if it is input into the ontology explicitly or produced through some inference mechanism. Therefore an ontology can have additional requirements, explicit or implicit, describing some desired inferences that the ontology should support. For instance, in an ontology about people we may define the class "parent" and say that any person who has at least one child is a parent. If we are not expecting the information about being a parent or not to be explicitly entered into the ontology's knowledge base, we are instead expecting it to be derived from the presence of "hasChild" relations. This is a reasoning requirement that requires the ontology to include the appropriate axioms to make this inference.
+Although CQs express what information the ontology should be able to provide, they do not say how this information is produced, i.e. if it is input into the ontology explicitly as assertions or produced from other facts through some inference mechanism. Therefore an ontology can have additional requirements, describing some desired inferences that the ontology should support. For instance, in an ontology about people we may define the class "parent" and say that any person who has at least one child is a parent. If we are not expecting the information about being a parent or not to be explicitly entered into the ontology's knowledge base, we may instead expect it to be derived from the presence of "hasChild" relations. This is a reasoning requirement that requires the ontology to include the appropriate axioms to make this inference.
 To test the desired inferences of the ontology module, follow these steps:
-# List the explicit reasoning requirements that you can find among the requirements of the module, if any.
+# Inspect the explicit reasoning requirements that you can find among the requirements of the module, if any. Also, determine if there are any implicit reasoning requirements, i.e. reasoning requirements that should be added to the set of requirements. This may include some interaction with the customer or a domain expert, if the requirements are ambiguous.
-# Determine if there are any implicit reasoning requirements, and list those as well. This may include some interaction with the customer or a domain expert, if the requirements are ambiguous.
+# For each such requirement (in the list from step 1), list what is the input and expected output, e.g. what facts (triples) need to be present before actually running the inference engine, and what facts (triples) can be produced by the inference procedure.
-# For each such requirement (in the joint list from 1) and 2) above), list what is the input and expected output, e.g. what facts need to be present before actually running the inference engine, and what facts can be produced by the inference procedure.
+# Create a new "test case"-module, by importing the module to be tested and adding facts to it. For each reasoning requirement in the list from step 1, add some facts that can be considered as the "input" to the reasoning process (according to the result of step 2). Add both facts that will produce the inference, and facts that should not.
-# Create a new "test case"-module, by importing the module to be tested and adding facts to it. For each reasoning requirement in the joint list from 1) and 2), add some facts that can be considered as the "input" to the reasoning process (according to the result of step 3). Add both facts that will produce the inference, and facts that will not.
+# List the expected inferences that should be made based on your test data, i.e. the expected test output.
-# List the expected inferences that should be made based on your test data.
+# Run an OWL reasoner over the "test case"-module and record the results.
-# Run an OWL reaosner over the "test case"-module and record the results.
+# Compare the actual results (from step 5) with the expected ones (from step 4), and decide if the test of each reasoning requirement was successful or not. A test is successful if it produces the expected inferences, and no harmful side-effects, i.e. no additional inferences that add undesirable or incorrect facts to the knowledge base.
-# Compare the actual results with the expected ones (from step 5), and decide if the test of each reasoning requirement was successful or not. A test is successful if it produces the expected inferences, and no harmful sideffects, i.e. no additional inferences that add undesirable or incorrect facts to the knowledge base.
+# For any unsuccessful tests, analyze the reason for the failure by inspecting the ontology. Document the problem, so that it can be solved by you or by other ontology developers.
-# For any unsuccessful tests, analyze the reason for the failure by inspecting the ontology. Document the problem, so that it can be solved by the ontology developers.
 '''Task:'''
-Download the following ontology module xxx.owl, and put it in your NeOn Toolkit workspace folder. Access the ontology through the NeOn toolkit. Start the reasoning plugin. The requirements for this module are the following:
+Load the same ontology as you used for '''Method 1''' (you see the requirements above).
-*...
-*...
+Now, use the method described above to test the ontology module. While you are working, document what you are doing, e.g. are you following the steps exactly or not, how are you performing your task in practice, how long time does it take, and what errors are you able to find. When you feel satisfied with the testing, please answer the questionnaire for ''method 2''.
-*...
-Now, use the method described above to test the ontology module. While you are working, document what you are doing, e.g. are you following the steps exactly or not, how are you performing your task in practice, how long time does it take, and what kind of errors are you able to find. When you feel satisfied with the testing, please answer the questionnaire for ''method 2''.
 == Method 3: Performing "stress tests" ==