Background#
The development of large knowledge bases is a long-term process that requires careful quality management. A very important method of quality management is the analysis of key performance indicators. For a knowledge base a number of indicators (variables) is identified and their outcomes and values are checked over a number of real life or at least realistic cases. The performance of the knowledge base can then be analyzed over a big data sample, for example by comparing the averages of the indicators with expected or desired values.Definition#
%%KPIAnalysis @column: <Symptom> // column using a smptom and its value @column: <Column-Name> = <expression> // column using a temporal expression @case: <Regex> // evaluates all matching cases @purge: <enabled|disabled> // enables usage of the purge strategy
Configuration#
To conduct a new KPI-Analysis for a d3web knowledge base, we recommend to create a new wiki page “KPI”. This wiki page can be used to define KPIs in the KPI markup and to extend the knowledge base to calculate special KPIs if needed.
In the following section a common way to define a KPI analysis is described. For most use cases a more plain configuration may be sufficient, as described in section 3.2
General KPI Configuration#
Extending the knowledge base for a KPI analysis#
Not all KPIs will be available in the knowledge base as variables right away. Also, often the KPIs cannot simply be added to the regular knowledge base, because the knowledge base should stay unchanged for its integrity. Also the speed of the reasoning could be influenced negatively.
To work around this, the KPI analysis should be done in its own extended knowledge base, which will be compiled on a newly created page, for example with the name “KPI”.
As a first step, a new package will be set and used on this page. Again, for example “KPI“ can be used as the package name.
Next, the new knowledge base used for analyzing the KPIs has to be defined. It has to compile (@uses) the package used for the original knowledge base (in most cases “default”) and also the package used to extend the knowledge base with the KPI variables and calculations (“KPI”). This new knowledge base will then automatically be used in the generation of the KPI results. The original knowledge base will not be effected through this new knowledge base.
Using the new knowledgebase, as the next step the actual variables used in the KPI analysis can be collected and defined using the
Calculating useful KPIs#
In this section, useful ways to define and calculate KPIs variables for later analysis are introduced. First derivation of a SolutionIn many use cases, a KPI will be the date of the first derivation of a solution. How long did it take for the knowledge base to recognize and derive that a problem, e.g. in a machine or patient, has occurred. To calculate this, the following arrangement of formulas can be used as a template:
%%Variable KPI_Date = firstChange(filter(Solution[], established))After execution, the history of a solution is filtered for the entries where the Solution was established. In combination with the function firstChange, the date of entry will be selected, where the solution was first set to established.
Last derivation of a solution
The last derivation of a solution can be calculated analogous to the calculation of the first derivation:
%%Variable KPI_Date = latestChange(filter(Solution[], established))
Minimum or maximum of a variable
A frequently needed KPI is the minimum or maximum of a variable during the execution of a case. The calculation can here be done simply with the following formula:
%%Variable KPI_Num = min(QuestionNum[]) %%Variable KPI_Num = max(QuestionNum[])
Counting occurrences
A use case for a KPI could be the number of times, a solution was established or a variable was, for example, over a certain threshold. To calculate this, the following arrangement of formulas can be used as a template:
%%Variable KPI_Num = count(filter(QuestionNum[], '>', 25))The KPI variable KPI_Num will contain the amount of history entries of the variable QuestionNum, that are greater than 25.
Limitations#
In some cases or for certain KPIs, changing the original knowledge base cannot be avoided to effectively perform an analysis. For example, it may not be possible to count how many times a certain edge or loop in a flowchart was cycled, if the loop does not contain a note performing an action in each pass. To count the cycles, such a note has to be added. This can only be done in the original knowledge base and not in the one extended using the KPI page.If the integrity of the knowledge base is important, we suggest to clone the wiki content of the knowledge base and to perform the KPI analysis in the clone.
Specifying the KPI Markup#
After all KPIs are defined and calculated, the actual %%KPIAnalysis}} markup needs to be added to KPI page. In the markup, the different KPIs to be analyzed and also the cases to be executed have to be specified. The KPIs will be evaluated for every knowledge base that compiles the package the KPI markup is part of.To add a particular KPI, the annotation @column can be used, defining a column in the excel analysis result file. Any question or solution in the knowledge base can be set as a KPI. The added KPIs will be displayed as the columns in the resulting excel table (see section 4.1).
To add cases, the annotation @case can be used. All cases or test cases defined in the knowledge base can be set. Auto completion will help to use the right name/identifier for the case. Additionally, a regular expression can be used to define all cases that should be included in the result.
By default, all purging is disabled for KPI analysis. This may lead to high memory consumption if analyzing huge cases with very frequent value changes. To avoid this, purging can be enabled by using the annotation @purge, which can be set to enabled or disabled.
