70% 70%

60% 60%

Density

50% 50%

40% 40%

30% 30%

20% 20%

10% 10%

Fig. 3. Distributions of actors steps-sequences lengths in analysed use cases, a) histogram

presents the length of the steps sequences performed by main actor, b) histogram

presents length of the steps sequences performed by secondary actor – e.g. System

observed. The most common one is to use extensions to represent alternative system behaviour in case of failure of the verification process (41.3 % of extensions have this kind of nature). The second one, and less frequent, is to incorporate validation actions into steps (e.g. System verifies data). This kind of actions are observed only in 3.4 % of steps.

Analysed properties can be classified into two separate classes. The first one contains properties which are observed in nearly all of the projects, with comparable intensity. Those properties seem to be independent from the specification they belong to (from its author’s writing style). The second class is an opposite one, and includes properties which are either characteristic for a certain set of use cases only, or their occurrence in different specifications is between two extremes – full presence or marginal/none. Such properties are project-dependent. More detailed description of analysed requirements specifications is presented in Table 2.

4 ANALYSIS OF USE-CASES-BASED SPECIFICATIONS DEFECTS

Use cases seem simple in their structure, and easy to understand; however, it is easy to introduce defects, which might influence the communiation between analyst, customer and project stakeholders. Different research [1, 19] indicated that vague, inconsistent or incomplete requirements are the main reasons for project failures. Defects make specification cumbersome, tedious and hard to read, analyse and change, hence they can lead to misunderstandings and finally to higher costs of software projects. Based on patterns and guidelines presented by Adolph et al. and Cockburn [2, 7] a set of defect types has been distinguished. Projects stored in UCDB have been analysed in order to collect statistical data about defects found in the requirements specification. Table 3 presents the results of this analysis.

Building Benchmarks For Use Cases

Table2.Resultsofthestructureanalysisoftheusecasesstored in UCDB

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

Table3.Resultsofthequalityanalysisoftheusecasesstored in UCDB

One of the main observations is that almost 86 % of the analysed use cases contain at least one defect, which proves that it is not easy to provide requirements specification of good quality. For researchers who try to analyse use case quality the knowledge of average defects density can be used to tune their methods and tools in order to make them more effective and efficient.

Large number of different defects can be found in most of the projects. For example steps without explicitly specified actor can be found in 13 projects, 11 projects have some extensions missing alternative scenarios, the same number of projects have use cases with less than 3 steps. Some of the defects are unique only to some specifications. For instance langauge mistakes were found mainly in project B (over 47% of steps have language mistakes), while most of the projects did not have such problems. Scenarios longer than 9 steps is another example of a project-specific defect (it mainly relates to Project K, which has over 30% of use cases with this defect). The simple explanation for this is that different authors have different writing styles and probably they are not aware of some problems and defects.

Another interesting finding is that one defect can lead to another defect. The analysis shows that steps without actor tend to use passive voice (projects C, N, O, P). A similar situation can be observed in case of language mistakes and complex-steps structure which often lead to different possible interpretations (projects B, C and N).

5 BUILDING REFERENTIAL SPECIFICATION

Combining use-cases model and average values coming from the analysis, a profile of the typical use-cases-based requirements specification can be derived. In such specification all properties would appear with the typical intensity. In other words, analysis performed on such document could be perceived as an every-day task for use-cases analysis tools.

There might be at least two approaches to acquire instance of such specification. The first one would be to search for the industrial set of use cases, which would fulfil given criteria. Unfortunately, most of industrial documents are confidential, therefore they could not be used at large scale. The second, obvious approach would be to develop such specification from scratch. If it is built according to the obtained typical specification profile, it might be used instead of industrial specification. Since it would describe some abstract system it can be freely distributed and used by anyone.

Therefore we would like to propose the approach to develop an instance of the referential specification for the benchmarking purpose. The document is available at the web site [8] and might be freely used for further research.

5.1 Specification Domain and Structure

It seems that large number of tools for the use-cases analysis have their roots in the research community. Therefore the domain of the developed specification should be

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

easy to understand, especially for the academia staff. One of the well-recognised processes at university is the students admission process. Although the developed specification will describe hypothetical process and tools, it should be easy to understand for anyone who has ever participated in a similar process.

Another important decision concerns the structure of the specification (number of actors, use cases and business objects). Unfortunately the decision is rather arbitrary, because the number of use cases, actors and business objects may depend on the size of the system and on the level of details incorporated into its description. In this case a median number of use cases and actors from the UCDB set has been used as the number of use cases and actors in the constructed specification. The first version of specification consisted of 7 actors (Administrator, Candidate, Bank, Selection Committee, Students Management System, System, User), 37 use cases (3 business, 33 user, and 1 sub-function level), and 10 business objects. After extending UCDB with data from additional 5 projects the median number of use cases decreased to 20. Therefore we decided to prepare two versions of the specification. The larger one has 37 use cases and is labeled as FULL, while in case of smaller specification (SMALL) the number of use cases was reduced to 20.

5.2 Use Cases Structure

A breadth-first rule has been followed in order to construct referential specification. Firstly, all use cases were titled and augmented with descriptions. Secondly, all of them were filled with main scenarios and corresponding extensions. During that process all changes made in specification were recorded in order to check conformance with the typical profile. This iterative approach was used until the final versions of both SMALL and FULL quantitative specifications were constructed. Their profiles are presented in Table 4, in comparison to the corresponding average values from the UCDB projects analysis The most important properties are those which are observed in all of the specifications (specification independent). In case of those metrics most values in the referential document are very close to those derived from the analysis. This situation differs in case of properties which were characteristic only for certain projects (or variability between projects were very high). If the constructed specification is supposed to be a typical one, it should not be biased by features observed only in some of the industrial specifications. Therefore, dependent properties were also incorporated into the referential use cases; however, they were not a subject of the tuning process.

In addition, based on each of the quantitative specifications, a corresponding qualitative specification was proposed. In this case we focused on introducing changes which led to obtaining a qualitative profile close to the typical profile.

6 BENCHMARKING USE CASES – CASE STUDY

Having the example of the typical requirements specification, one can wonder how it could be used. Firstly, researchers who construct methods and tools in order

Table 4. Referential specifications profiles in comparison to the average profile derived from the analysis of the use cases stored in UCDB

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

to analyse use cases often face the problem of evaluating their ideas. Using some specification coming from the industrial project is a typical approach; however, this cannot lead to the conclusion that the given solution would give the same results for other specifications. The situation looks different when the typical specification is considered; then researchers can assume that their tool would work for most of the industrial specifications. Secondly, analysts who want to use some tools to analyse their requirements can have a problem to choose the best tool that would meet their needs. With the typical specification in mind it can be easier to evaluate the available solutions and choose the most suitable one. To conclude, the example of the typical requirements specification can be used:

• to compare two specifications and assess how the given specification is similar to the typical one;

• to assess the time required for a tool to analyse requirement specification;

• to assess the quality of a tool/method;

• to compare tools and choose the best one.

In order to demonstrate the usage of the constructed specification we have conducted a case study using the qualitative reference specification Admission system 2.0 FULL. As a tool for managing requirements in a form of use cases we chose the UC Workbench tool [17], which has been developed at Pozna´n University of Technology since the year 2005. In addition, we used the set of tools for defects detection [6] to analyse the requirements. The aim of the mentioned tools is to find potential defects and present them to analyst during the development of the requirements specification. The tools are based on the Natural Language Processing (NLP) methods and use Standford parser [9] and/or OpenNLP [20] tools to perform the NLP analysis of the requirements.

6.1 Quality Analysis

In order to evaluate the quality of the tools, a confusion matrix will be used [12] (see Table 5).

	Expert answer
System answer		Yes	No
	Yes	TP	FP
	No	FN	TN

Table 5. Confusion matrix

The symbols used in Table 5 are described below:

• T(true) – tool answer that is consistent with the expert decision;

• F(false) – tool answer that is inconsistent with the expert decision;

• P(positive) – positive system answer (defect occurs);

• N(negative) – negative system answer (defect does not occur).

• On the basis of the confusion matrix, the following metrics [16] can be calculated:
• Accuracy(AC) – proportion of the total number of predictions that were correct. It is determined using Equation (1).

TP + TN

AC = (1)

TP + FN + FP + TN

• True positive rate (TP rate) – proportion of positive cases that were correctly identified, as calculated using Equation (2).

TP

TP rate = (2)

TP + FN

• True negative rate(TN rate) is defined as the proportion of negative cases that were classified correctly, as calculated using Equation (3).

TN

TN rate = (3)

TN + FP

• Precision (PR) – proportion of the predicted positive cases that were correct, as calculated using Equation (4).

TP

PR = (4)

TP + FP

The referential use-case specification was analysed by the mentioned tools (to perform the NLP processing Stanford library was used) in order to find 10 types of defects described in [6]. Aggregated values of the above accuracy metrics are as follows:

• AC =0.99

• TP rate =0.97

• TN rate =0.99

• PR =0.80.

This shows that the developed tools for defects detection are rather good, however the precision should be better, so the researchers could conclude that more investigation is needed in this area.

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

If the researchers worked on their tool using only defect-prone or defect-free specifications the results could be distorted and could lead to some misleading conclusions (e.g. that the tool is significantly better or that it gives very poor outcome). Having access to the typical specification allows the researchers to explore the main difficulties the tool can encounter during working with the industrial specifications.

6.2 Time Analysis

One of the main characteristics of a tool being developed is its efficiency. Not only developers are interested in this metric, but also the users – e.g. when analysts have to choose between two tools which give the same results, they would choose the one which is more efficient. However, it can be hard to asses the efficiency just by running the tool with any data, as this may result with distorted outcome. Use-cases benchmark gives the opportunity to measure the time required by a tool to complete its computations in an objective way. It can be also valuable for researchers constructing tools to consider using different third-party components in their applications.

For instance, in case of the mentioned defect-detection tool there is a possibility of using one of the available English grammar parsers. In this case study two of them will be considered – Standford parser and OpenNLP. If one exchanges those components the tool will still be able to detect defects, but its efficiency and accuracy may change. Therefore, the time analysis was performed to asses how using those libraries influences the efficiency of the tool. The overall time required to analyse the typical specification² for the tool using Stanford parser was 53.11 seconds and for the one with OpenNLP it was 29.46 seconds. Although the first time value seems to be large, the examined tool needed 280 ms on the average to analyse a single use-case step, so it should not be a problem to use it in industrial environment. Of course, when comparing efficiency, one has to remember that other factors (like the quality of the results, memory requirements or the time needed for the initialization of the tools) should be taken into account, as they can also have significant impact on the tool itself. Therefore, time, memory usage and quality analysis is presented in Table 6. This summarised statistics allow researchers to choose the best approach for their work.

Although this specification describes some abstract system, it shows the typical phenomenon of the industrial requirements specifications. The conducted case study shows that the developed referential use-case specification can be used in different ways by both researchers and analysts.

Tests were performed for the tools in two versions: with Stanford and OpenNLP parsers. Each version of the tools analysed the referential specification five times, on the computer with Pentium Core 2 Duo 2.16GHzprocessor and2GBRAM.

Table6.Case-study results(tools are writteninJava, so memory was automatically managed – garbage collection)

7 CONCLUSIONS

In the paper an approach to create a set of reference use-cases-based requirements specifications was presented.

In order to derive a profile of a typical use-cases-based requirements specification 524 use cases were analysed. It has proved that some of the use-case properties are project-independent (are observed in most of the projects). They have been used for creating the referential specification. On the other hand, there is also a number of properties which depend very much on the author.

In order to present potential usage of the benchmark specification, a case study was conducted. Two sets of tools for defect detection in requirements were compared from the point of view of efficiency and accuracy.

In this paper a second release of UCDB is analysed, which was recently extended from 432 to 524 use cases. A positive observation is that despite the number of use cases increased by 21 % the typical profile did not change a lot. Therefore it seems that the typical profile derived based on the use cases stored in the database is stable and should not differ much in the future releases.

Acknowledgments

We would like to thank Piotr Godek, Kamil Kwarciak and Maciej Mazur for supporting us with industrial use cases.

The research presented at the CEE-SET ’08 [3] and being part of this paper has been financially supported by the Polish Ministry of Science and Higher Education under grant N516 001 31/0269.

Additional case studies and analysis have been financially supported by Foundation for Polish Science Ventures Programme co-financed by the EU European Regional Development Fund.

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

REFERENCES

[1] Chaos: A Recipe for Success. 2004, Technicalreport, Standish Group, 2004.

[2] Adolph, S.—Bramble, P.—Cockburn, A.—Pols, A.: Patterns for Effective Use Cases. Addison-Wesley, 2002.

[3] Alchimowicz, B.—Jurkiewicz, J.—Nawrocki, J.—Ochodek, M.: Towards Use-Cases Benchmark. In: 3^rd IFIP Central and East European Conference on Software Engineering Techniques CEE-SET 2008, 2008.

[4] Anda, B.—Sjøberg, D. I. K.: Towards an Inspection Technique for Use Case Models.In: Proceedings of the14^th international conference on software engineering and knowledge engineering, 2002, pp. 127–134.

[5] Bernardez, B.—Duran, A.—Genero, M.: Empirical Evaluation and Review of aMetrics-BasedApproachforUseCaseVerification.Journal ofResearch andPractice in Information Technology, Vol. 36, 2004, No. 4, pp. 247–258.

[6] Ciemniewska, A.—Jurkiewicz, J.—Nawrocki, J.—Olek, �L.: SupportingUse-Case Reviews. In: 10^th International Conference on Business Information Systems, Lecture Notes in Computer Science, Vol. 4439, April 2007, pp. 424–437.

[7] Cockburn, A.: Writing Effective Use Cases. Addison-Wesley Boston, 2001.

[8] Use Case Database. Availaible on: http://www.ucdb.cs.put.poznan.pl.

[9] de Marneffe, M.-C.—MacCartney, B.—Manning, Ch. D.: GeneratingTyped Dependency Parses from Phrase Structure Parses. In: Proceedings of the EACL workshop onLinguisticallyInterpretedCorpora(LINC),2006.

[10] Denger, C.—Paech, B.—Freimut, B.: Achieving High Quality of Use-Case-Based Requirements. Informatik-Forschung und Entwicklung, Vol. 20, 2005, No. 1, pp. 11–23.

[11] Diev, S.: Use Cases Modeling and Software Estimation: Applying Use Case Points. ACM SIGSOFT Software Engineering Notes, Vol. 31, 2006, No. 6, pp. 1–4.

[12] Fawcett, T.: Roc Graphs: Notes and Practical Considerations for Data Mining Researchers. HP Labs Technical Report HPL-2003-4, 2003.

[13] Hurlbut, R.: A Survey of Approaches for Describing and Formalizing Use Cases. Expertech, Ltd., 1997.

[14] Jacobson, I.: Object-Oriented Development in an Industrial Environment. ACM SIGPLAN Notices, Vol. 22, 1987, No. 12, pp. 183–191.

[15] Jacobson, I.—Christerson, M.—Jonsson, P.—Overgaard, G.: Object-Oriented Software Engineering: A Use Case Driven Approach. 1992.

[16] Krawiec, K.—Stefanowski, J.: Uczenie Maszynowe i Sieci Neuronowe. Wydawnictwo Politechniki Pozna´nskiej, 2004.

[17] Nawrocki, J.—Olek, �A Tool for Writing Use Cases. In: L.: UC Workbench –

₆th

International Conference on Extreme Programming and Agile Processes, Lecture Notes in Computer Science, Vol. 3556, Jun 2005, pp. 230–234.

[18] Neill, C. J.—Laplante, P. A.: RequirementsEngineering: TheState of thePractice. Software, IEEE, Vol. 20, 2003, No. 6, pp. 40–45.

[19] Niazi, M.—Shastry, S.: Role of Requirements Engineering in Software Deve

lopement Process: An Empirical Study. In: Multi Topic Conference, 2003, INMIC, ₇th

International, 2003.

[20] OpenNLP. Availaible on: http://opennlp.sourceforge.net.

[21] Shull, F. J.—Carver, J. C.—Vegas, S.—Juristo, N.: TheRole ofReplications in Empirical Software Engineering. Empirical Software Engineering. Vol. 13, April 2008, No. 2, pp. 211–218.

[22] Som´ S. S.: Supporting Use Case Based Requirements Engineering. Information e,

and Software Technology, Vol. 48, 2006, No. 1, pp. 43–58.

Bartosz is aPh.D. student working attheFa

���������

culty of Computing and Management, Poznan University of Technology in Poland. His main fields of interest are requirements engineering, software configuration management andquality of documentation.

Jakub ��������� is aPh.D. student at Pozna´nUniversity of

Technology,Poland. He receivedhisM.Sc.degreein computer science in the Faculty of Computer Science and Management of Poznan Univerity of Technology in 2007. His research interests are requirements engineering,quality requirements, software measurement and natural language processing.

Miros�law ������ is aPh.D. studentworking attheInstitute

of Computing Science at the Poznan University of Technology. He is mainly working in the domain of requirements engineering, software metrics, functional size measurement, and software effort estimation.

B. Alchimowicz, J. Jurkiewicz, M. Ochodek, J. Nawrocki

Jerzy received his M.Sc. degree(1980), Ph.D. de

�������

gree(1984),andDr.hab.degree(1994) –allininformaticsand allfromthePoznanUniversity ofTechnology(PUT),Poznan, Poland. Currently he is the Dean of the Faculty of Computing and Management at PUT, and a Secretary of IFIP Technical Committee 2: Software Theory and Practice.