ANNEX 2       Digital library users: results of a survey on needs, expectations and skills*

Abstract

This section presents an evaluation of the results of a survey to identify users’ needs, carried out by Fondazione Rinascimento Digitale. The aim of the survey was to stimulate a culture of excellence in the different cultural institutions involved. The primary objective was to establish and test a methodology with which to evaluate digital libraries. A secondary objective was to obtain feedback from the users on their satisfaction level, to find out what their needs and expectations are and to give them an opportunity to make suggestions.

Questionnaires and interviews were used to gather the data. A questionnaire was also distributed to the managers of the cultural institutions involved.

The survey’s results indicated that different users have different needs and that they tend to use the services of more than one cultural institution. Overall, there is a positive attitude towards digital libraries, the survey also underlines that users often don’t know how to use the libraries and are unaware of all of the services offered. The accessibility of the interface was considered important, but as it becomes more sophisticated to offer more services it will require more staff assistance. The survey also served to experiment with quality indicators and enquiry methodologies that focus on library users.

In conclusion, this paper evaluates the implications of these results for digital libraries in general, and, specifically, the value of a cooperative approach to the identification and evaluation of digital library users.

Introduction and background

The Fondazione Rinascimento Digitale1 was established to encourage the cooperation of various organisations with experience and know-how in the digital domain, in order to promote the use of new technologies in cultural institutions by establishing a high standard of quality. The Digital Libraries Applications Project, begun by the Foundation, aims to evaluate the services currently offered by digital libraries in Italy, to identify the actual state of the art and any obstacles to improving their services in order to stimulate greater cooperation between different cultural institutions. To realize this aim, a Study Group was established in the summer of 2005. It was composed of a wide group of experts, representing different cultural institutions that offer digital services or make their collections accessible digitally, as well as projects dealing with digital library themes.

The approach chosen by the Study Group was to evaluate the complexity of the digital library from the user’s point of view. Surely the user has a primary role in every digital library project; nonetheless it is not always easy to know what users really need and whether the user is satisfied with the digital resources and services available. The assumption on which the survey is based is that a digital library, to justify the effort invested to establish and manage it, must offer significantly superior user value when compared to a traditional library. This added value must be measured not only quantitatively, for example by the number of uses, but also qualitatively, based on research on the users themselves. The goal is to stimulate a culture of excellence with the user as the main focal point.

To accomplish this goal, the Study Group set the following obiectjves:
                        •     Create and test an evaluation method to identify and measure the expectations, service                                perceptions and user satisfaction with available digital resources and services
                        •     Launch a user survey comparing a range of case studies in the area of humanities.

                        Specifically, the Study Group posed these questions:
                        •     What needs are considered to be priorities by which group of users?
                        •     In relationship to these needs which services and resources are essential and which are considered                                desirable?
                        •     How can digital libraries be useful to their public?

Among the analyses already carried out there were two useful surveys, by the Istituto e Museo di Storia della Scienza, (Museum of the History of Science - IMSS) and the other by the Biblioteca Nazionale di Firenze (National Library of Florence - BNCF) from which the study group took useful methodology suggestions. From literature and documentation on the subject, the Study Group noted that many of the evaluation experiences had been implemented by developing quantitative indicators, but that few of these surveys studied the users’ opinions. The Study Group took as their reference points some of the most important studies, including the Project E-measures of SCONUL2, Project by ARL called E-Metrics3 and Project COUNTER4.

The Project eVALUEd5 proved particularly interesting to the Study Group as it had developed a toolkit to facilitate the assessment of digital libraries. The aim of the Project eVALUEd was to simplify a first reading of qualitative data, without abandoning the gathering of quantitative statistics. The Study Group used a methodological approach that integrated these international experiences with the experimental approaches of the IMSS and BNCF.

Methodology

The Fondazione Rinascimento Digitale survey has been divided into three phases. The first phase was dedicated to gathering information on existing digital libraries, and at the same time gathering the contributions of experts to define a theoretical context and reference model for digital libraries. These results were used in the second phase to develop measurement tools to gather data proposed for three areas:
                        1.   Content, services and their uses
                        2.   User satisfaction with digital resources and services
                        3.   Impact measurement

Finally, in the third phase a User Survey Subgroup carried out the survey and analysed the data.

From the beginning of the survey, the Study Group had to allow for its limited resources and time considering the scope of the survey, and therefore chose to carry out various case studies, in order to compare the final results, instead of a broad quantitative survey. The survey had other limitations as well:
                        •     It was limited to users on site at the institutions and therefore did not consider remote users
                        •     The results would have been more useful if they also included non-library users.

Normally the evaluations of user satisfaction, use measurement and service impact with digital libraries are done separately. Nonetheless, the Study Group believed that the three measuring and evaluation processes should be complementary, and that the comparative results wouldn’t be in conflict. Therefore a methodological toolkit was developed, and the methodological results are probably among the most interesting of the user survey.

The context chosen by the Study Group was that of three humanist cultural institutions and the survey was repeated in participating institutions with the same methodology. The case studies included the Mediateca of the Tuscany Region, the Humanities Library of the University of Florence and the Library of the Museum of the History of Science; the results were later compared to those obtained by the National Central Library of Florence.

Collections, digital services and their uses

Traditionally the most common measurements, because they are easily obtainable, are concentrated on numbers and data like the budget employed, the number of titles in a digital collection, etc. Because these types of statistics don’t give data about the user and their normal activities in a digital library, the Study Group did not take them into consideration.

The questions were aimed instead at identifying the users’ expectations of the services offered such as: available hardware, on line catalogues, access from home, portal/site, users’ educational background, promotion, and staff assistance. The resources were of the following types: electronic journals, e-books, databases, CD-ROM-s, learning materials, audiovisuals and multimedia, theses and students’ work.

Additionally, the survey attempted to identify cultural institutions other than the surveyed institutions, where the users regularly go virtually. The Study Group wondered if enlarging the size of the collection is necessarily correlated to meeting users’ needs, but they did not arrive at a final conclusion; moreover it is difficult to understand if a single research session in a digital library is truly useful to (or if it has had an impact on) the user. Therefore it was decided to evaluate the perception that users have of how resources are employed as well as the digital services available.

User satisfaction

Despite the fact that it is important to develop a tool to measure how digital libraries services contribute to the user’s success, this is very difficult to assess. Therefore the Study Group chose to define user success as closely tied to the success of the institution that the digital library belonged to, as expressed in their mission or in other project documents.

This necessitated finding a tool capable of identifying the critical criteria of the specific mission of each individual digital library, one that preferably included a definition of user activities. To identify and measure the impact, the Group limited itself to evaluating the specific digital library services in the case study, which were provided in such a way as to be a support or useful to the research activities of the users.

The impact, therefore, is not a function of the resources or of the services themselves, but more pragmatically, a measure that identifies activities that would be impossible to accomplish without the use of the digital library. Impact measurement defined in this way was researched in the comments section of the survey and in specific questions asked during the interview. Of particular interest to the Study Group were qualitatively negative or neutral  comments.

The tools employed in information gathering were a questionnaire and structured interviews. The following data represent the results:

The methodology employed could be used periodically to correlate the results obtained from user surveys and improvement in services.

The following table synthesizes the model chosen for the evaluation:

Results

Here we analyse the data for the comparison between the case studies, while the full survey is available in the online report.

Who are the users of the cultural institutions?

Users were classified by age, gender, nationality, profession and hobbies. In the Humanities Library of the University of Florence the users chosen were undergraduate and graduate students, with an age that ranged between 25 and 40; they have an average knowledge of the Internet and they frequently use the online library system, but they rarely use the Ateneo’s Digital Library. The users at the Library of the Institute and Museum of the History of Science are professionals and employees with a post bachelor degree and an age that ranges from 32 to 76; they have a good or excellent knowledge of the Internet and they frequently use the Digital Library. Mediateca users are primarily students and the youngest in age, from 19 to 25; they use the Mediateca weekly, also from home. All of the users replied that they also use the services of other libraries: the users of the Library of the Institute and Museum of the History of Science and the Mediateca users access other national and international institutions with the same specialisation from remote locations; the University students use primarily local services of the Florentine Public Library system.

The result that was discovered from this first part of the survey is the importance of clearly defining the user because upon it depends the choice of service and digital resource priorities. Even within the limits that have already been underlined, we can say that the users of different institutions have different service priorities.

Comparision of the results

Even with a wide range of different uses some results were common to all groups and can be compared.

 What are the service expectations?

Through a careful examination of the service expectations, correlated to the user satisfaction with those same services, it was possible to identify which services users consider to be unsatisfactory and therefore know where it is necessary to concentrate our efforts to improve. These are:
                        •     The promotion of resources and services
                        •     On line tutorials
                        •     User education
                        •     Staff assistance.

It should be noted that need for promotion of available services was expressed by all users.

The services that received the highest level of satisfaction were:
                        •     Remote access
                        •     On line portal.

University students prefer remote access, but contradictorily they also prefer the help of staff and information literacy courses with an actual teacher, to online tutorials. The IMSS users prefer remote access and the portal, together with on line tutorials, but they see staff assistance as their first priority. Mediateca users, also due to the unique characteristics of the current service that is mainly local, decidedly prefer a local access and are the users that demonstrate the greatest appreciation of staff assistance even though remote access, the portal and the tutorials are also regarded as important.

What are the priorities for digital resources?

By repeating, for the digital resources section, the correlation between the satisfaction with individual resources it was possible to identify the resources that are not considered priorities. They are listed in inverted order, starting from the less used:
                        •     E-books;
                        •     Audiovisual materials
                        •     Learning materials
                        •     Theses.

The resources that were listed as priorities are:
                        •     The OPAC catalogue
                        •     On-line databases
                        •     Electronic journals.

Regarding resources, the priorities demonstrate the biggest differences between different types of users. For example, University students mainly use the on-line catalogue and the databases; the IMSS users are the ones that prefer the e-book and CD-ROMs more than the others; Mediateca users tend to prefer the audiovisual material.

Impact

The impact was expressed primarily in terms of the advantages of the digital library. These included the speed of access to digital resources, the great number of resources available (even if this is not yet considered to be sufficient), and personalization.

What interventions are possible for improvement?

It can be stated that new users want to be independent to do their research and they want remote access: this is demonstrated by the general expectation of a good orientation through a portal, even in the case that the user regularly goes to a physical library.

Databases and on-line catalogues are areas that need particular attention, in order to meet the users’ expectations. A service that users view as particularly important is information retrieval. In the suggestions that were made it seems particularly relevant to underline the demand for enhanced OPAC functionality. Users expect to find and locate digital resources quickly and easily.
                       
In answer to the question: which services would you like to find in a digital library? most of the users interviewed answered: a greater number of digital resources available. Other answers mentioned the possibility of integrating the different databases available, with for example a link from the OPAC to a preview of the cover, copyright page, and contents of the book. The personalization of the service was also viewed as important, as well as the possibility of having functions available, to manage a personal digital collection.
                       
In answer to the question on the need for greater cooperation between cultural institutions, the indication was that the current situation is definitely unsatisfactory and insufficient. The answers also underlined the need to improve the user skills with digital resources and the need for more promotion of their existence.

Conclusions

The results of the survey have made clear that users have different needs, which correlate to the different goals of the digital libraries’ institutions. Nonetheless, users regularly use the services of more than one cultural institution and they share some common priorities. Users view the services offered by digital libraries in a positive light, but there is a lack of knowledge on how to use them. Users are often unaware of all the services that are available to them. The accessibility of the interface is considered important, but the more sophisticated it is, the greater the need for assistance from the staff.

In conclusion, it is important to give users the capability to say where the services should be improved, so that their expectations can be better met. Moreover, digital libraries can try to improve their services through a cooperative approach. In fact, with periodic user surveys, the individual institutions could compare their own results with other digital libraries, that are positively evaluated by their users.

1 Panel composed of: a high school teacher, a communication agent of a tourism office, a research director, three librarians (one from a municipal library with a regional mandate, a head of the digital library in a large public library, a head of unit at the ministry of culture), a researcher at the department of books and publishing at ministry of Culture, a professional genealogist, an artist (representing the “general public”).
2 Only 2 questionnaires contained this question: the n°1 general public questionnaire and the n°2 institution questionnaire.
3 Développement culturel, n. 137, October 2001.
4 Développement culturel, n. 151, January 2006.

*  Edited by Anna Maria Tammaro, University of Parma, Fondazione Rinascimento Digitale.

1 The project Digital Libraries Applications is part of the activities of Fondazione Rinascimento Digitale entitled: Management of and Access to digital libraries. A report on the state of digital libraries and other documentary material is available on line: http://www.rinascimento-digitale.it/.
2 The goal was to produce a set of statistics to estimate the use of digital services in the university libraries of the UK. The project is based on the periodic survey achieved in the all English university libraries. After two years of research the Project suspended the data-gathering, because they had no significant outcomes to understand the real performance of digital libraries. The Society of College, National and University Libraries (SCONUL) implement periodically the Annual Library Statistics, where the criteria developed within the Project during its existence are recorded. Cf Town Stephen, 2004. E-measures: A comprehensive waste of time, “VINE”, 34 (4): 190-195.
3 Miller Rush, Schmidt Sherrie, 2001. E-metrics: Measures for electronic resources, Keynote Delivered at the 4th Northumbria International Conference on Performance Measurement in Library and Information Services, Pittsburgh, 12-16 August, ARL http://www.arl.org/, http://www.arl.org/stats/newmeas/emetrics/miller-schmidt.pdf (30 November 2006).
4 Working Group on Database Vendor Statistics enquired how to gather data from the data-base vendor statistics. From the Group’s outcomes started up COUNTER project Cf Blixrud Julia C., 2002. Measures for electronic use: The arl e-metrics project, IFLA Satellite Conference “Statistics in practice - Measuring & managing”, Loughborough, UK, 13-25 August, Loughborough University http://www.lboro.ac.uk/, http://www.lboro.ac.uk/departments/dils/lisu/downloads/ statsinpractice-pdfs/blixrud.pdf (30 November 2006); Blixrud, Julia C., 2003. Assessing library performance: New measures, methods, and models, IATUL Proceedings “Libraries and Education in the Networked Information Environment”. Ankara, Turkey. 2-5 June, IATUL http://www.iatul.org, http://www.iatul.org/conference/proceedings/vol13/papers/BLIXRUD_ fulltext.pdf (30 November 2006). COUNTER project established the standard criteria that rules the availability of digital resources managed by the vendor statistics.
5 Began in 2001 and finished in 2004, eVALUEd sought to go over the measure of the performance indicator to tackle the evaluation outcomes related to the supply of electronic information services. Thebridge Stella, 2003. Evaluating electronic information services: A toolkit for practitioners, “Library and Information Research”, 27 (87): 38-46, E-LIS http://eprints.rclis.org/, The English is ok here, but the “list of axioms” presentation may be criticised.