Skip to Main Content

The University of Tennessee

School of Information Sciences

Frequently Used Tools:

INSC 554

Main Navigation:

Lindsay Pearce
Assignment 5: Developing an Evaluation Action Plan


The library used for this assignment is the Knox County Public Library. This project will be evaluating the main branch, Lawson Mcghee, which is located in downtown Knoxville, Tennessee. This project will specifically evaluate the reference service.

5.1 Develop a rationale to propose user-centered assessment/evaluation of the selected PL service. Assess the selected PL service carefully and explain your proposal in relation to participation of primary, secondary, and tertiary users identified in the earlier assignment.

In today's digital world, people frequently assume they can access any information through Google. This leads to concern that reference services are becoming outdated or obsolete. However, reference services can help users determine whether or not the sources they find online are reliable, and lead them to sources that they may otherwise not be able to find through Google, such as articles in databases. In addition, the reference service can provide assistance for people who may not be able to afford a computer or Internet connection at home.
According to previous deductions, the primary users of the Knox County Public Library are low-income adults. It would be beneficial for the library to conduct a user evaluation of the reference service to see if these adults feel they are benefitting from the existing service. These users could potentially get a large amount of use out of the reference service, since they may not be able to afford an internet connection at home, and may rely on the library for information.
The secondary users of the library are the elderly. It would be valuable to get their evaluation of the reference service, because it could help the library determine whether or not the level of digital or technological literacy among the users matches the materials in use. Currently, the library offers computer classes for users (Knox County Public Library, n.d.). However, if the elderly population at the library doesn't understand how to use digital resources the reference service recommends, the library may need to increase digital literacy classes.
Finally, the tertiary users of the Knox County Public Library are teenagers. Obtaining user evaluations from this user group could be beneficial to the library because teens are a new generation of library users. They can help the library to determine what aspects of the reference service are outdated or not useful. They could also suggest innovative ideas that haven't been considered yet by members of older generations.

Identify the problem (related to your selected service) in terms of: production bottlenecks; tasks that are performed frequently; activities that require frequent movement; and declining budgets.

A potential problem with evaluating a reference service is that all patrons are looking for different information. In fact, in 2014, the reference desk received 321,148 questions (J. Drumheller, personal communication, March 4, 2015). For this reason, it may be difficult to determine which aspects of the reference service should be changed or improved upon. It would also be difficult to decide upon new or different resources for the library to purchase.
Another possible problem with user evaluation is the library's budget. If the reference service does not have enough of a budget available to continue to update database subscriptions, it may lose valuable resources for patrons. The library has several databases that are heavily used, particularly those on ancestry (J. Drumheller, personal communication, March 5, 2015). If these databases are no longer available, library use may suffer. Another potential problem with a shrinking budget is that the library may have outdated or broken computers that can't be repaired or replaced.
Finally, repetitive questions may impact the level of service given to patrons. If a librarian is asked the same question over and over again, he or she may become tired of answering it, or annoyed with the patrons. This could affect the level of customer service given, and alienate patrons.

Determine the scope of analysis in terms of what specific information/aspects about the selected PL service do you hope to gain by eliciting feedback from the PL users. In other words, what is the purpose for doing the evaluation? Is the library attempting to improve its operational efficiencies (internal focus), or is the study being done to better understand the effectiveness of a library service? Explain in detail.

The purpose of user evaluation of the reference service is to determine how successful the reference service is. This will include both internal and external analysis. For example, it is vital to understand if the library service is the best it can be, but also if the materials provided are useful. It is important to understand if users feel satisfied with different aspects of the service. For example, it can help improve the reference service if the library knows that users feel satisfied with the databases and reference materials available. If there are materials that routinely do not bring up relevant results, the library could cancel the subscription in order to free up the budget for other purposes.
The feedback will also help determine if the level of customer service is what patrons expect. If librarians at the reference desk are not friendly, patrons may be hesitant to use the service. This feedback can help the library determine whether or not they need customer service training for their employees. This can help improve the service, because if librarians are friendlier, people may use the service more often.
Finally, the user evaluation will also help determine why users may not be using the reference service. If a majority of the user feedback states that users do not use the reference service, or do not plan to use it in the future, the library can make adjustments, such as increasing advertising, that can help expand user numbers and satisfaction. Understanding why people are not using the reference service may be the most successful way for the library to improve. If the library understands why people don't want reference help, they may be able to counter act these anxieties.

Determine the kinds of data that you hope to gather (be specific). What evaluation methodology and design will be used (quantitative and/or qualitative methods: be very specific in the types of tools that will be employed)? How will data be collected?

The user feedback will be collected via survey. This survey will help determine how often users frequent the reference desk, if they are satisfied with the answers and service they received, and what materials located were most useful. The survey will also demonstrate whether or not patrons actually use the reference desk. If the answer is no, the survey will help uncover reasons the patron does not use the reference service. It will also be beneficial to determine how long the entire reference interaction lasted. If patrons feel the reference interaction was too long, it may deter them from returning.
Qualitative and quantitative data will be gathered. First, demographic information will be gathered, such as gender and age. Some quantitative data that the survey will collect includes frequency of use at the reference service, the length of the reference interaction, and how often the patron uses the library in general, not just the reference service. Qualitative data will include what materials were most useful in answering user queries, if the customer service was adequate, why users may not use the reference service, and what improvements could be made to the service to make it more effective.

5.5 Describe the user-centered assessment/evaluation strategies that you are proposing to conduct community analysis and gather feedback from users about the selected service at your PL. Identify and analyze what methods you will use to gather feedback from various kinds of users as a part of your efforts. How do you propose getting access/permissions to select users? Provide a timeline for implementing your proposed methods.

The user assessment will be done via survey. This survey will be distributed via Survey Monkey. These online surveys help save the user time by simply clicking on a link and answering questions. This ease of access ensures that even those users that do not have digital literacy skills can easily answer the survey questions. Users at the library can also receive assistance with any technological difficulties they may encounter. Online surveys also provide anonymity and save materials such as paper and pen. In order to get permission for the survey the head of the reference department will be consulted, as well as each patron asked to participate. Participation will be completely voluntary. The survey will be distributed by April 6, 2015. It will be implemented and completed by April 10, 2015. This will allow adequate time for statistical evaluation.

5.6 Share the data collecting instrument that you propose to use. Identify the qualitative and/or quantitative measures you propose to use (what aspects are you trying to measure). Provide strategies to gather consent (consent forms, assent forms as appropriate) and your data-collecting tool(s) with listed questions/prompts, etc. You can explore multiple research methods to gather user-centered assessment/evaluation of the selected service.

The survey for this project will be conducted via Survey Monkey. It will attempt to measure overall use of the reference service, the average age of users, and how pleased users are with the help they received. It will also determine any suggestions users may have to improve the reference service. In order to get consent, the head of the reference department, Janet Drumheller, will be consulted. Individual participant consent will be given verbally before taking the survey. If a participant refuses to take the survey, no negative consequences will occur.
The survey questions are as follows:
1. How old are you?
2. Are you male or female?
3. Have you used the reference service at the Knox County Public Library?
4. If yes, was the customer service you received satisfactory?
5. If yes, were the materials presented adequate in answering your query?
6. If yes, how long did the entire reference interaction last?
7. If no, why have you not used the reference service?
8. How often do you use the public library?
9. What other services at the library do you use?
10. Do you have any suggestions to improve the reference service?

5.7 Provide strategies to maintain anonymity and confidentiality of users participating in your study (if appropriate).

An online survey company, such as Survey Monkey can ensure that no one knows who took the survey. Another method is to have someone unfamiliar with the library hand out paper surveys. This will ensure that the survey giver does not recognize library regulars. However, for the purposes of this evaluation, no sensitive information should be communicated. The survey will not ask what questions patrons have asked at the reference desk, only if they have used the service.

5.8 Create a budget in terms of investment of resources (e.g., money, time, efforts, human input, etc.) to orchestrate user-centered assessment/evaluation of various kinds. Identify stages for orchestrating a plan for the PL (e.g., previous efforts made, current efforts, future efforts) and describe what will be feasible as part of your efforts in this course during this semester.

The survey will require minimal resources. Paper isn't necessary, as online resources such as Survey Monkey can be employed. This will help keep costs down. The survey will be developed to take between 5-10 minutes of patron time. This will ensure that patrons provide meaningful answers without becoming distracted or tired of the survey. Therefore, if human input is charged at minimum wage, which is $7.25, it would cost about 12 dollars in patron time to complete the survey, assuming ten patrons complete the survey in ten minutes each. Since Survey Monkey is a free service, that will not factor into the budget. Therefore, the entire budget for the project will cost less than 30 dollars, after factoring in time to analyze the results.
The public library has previously implemented user surveys (J. Drumheller, personal communication, March 3, 2015). However, the last user survey that was conducted was in 2009 (J. Drumheller, personal communication, March 3, 2015). These results simply demonstrate whether or not patrons used the reference desk. It did not examine which resources were used. The survey did investigate whether or not users were satisfied with the service, which is important (J. Drumheller, personal communication, March 3, 2015). Since this last survey effort was so long ago, it should be fairly simple to get patron feedback, as users won't feel as though they are constantly being asked to complete evaluation. In fact, users may be more willing to participate in the survey if they have not voiced their opinion on the service before.
Due to the limited time resources this semester, a survey will be the most feasible type of research. The survey will not take long to complete, and can display results quickly and efficiently. Although larger numbers of surveys taken would yield better results, this survey will aim to get ten-twelve excellent survey results. That will allow for a glimpse of user insight into the reference service.

5.9 Determine the kinds of analysis you will do with the data collected. How do you plan to report the findings based on the data collected?

It would be beneficial to do both qualitative and quantitative data with the information collected. Charts and graphs will be utilized to display quantitative data, such as how often users frequent the reference desk and how long reference interactions lasted. A report will be created that displays the qualitative data collected. The report will be divided into sections based on the questions in the survey. These sections will display the results from the survey. For example, user suggestions for improvements for the reference service will be displayed in a user suggestion section of the report. This report can also be provided in PowerPoint form for presentation purposes.


Knox County Public Library. (n.d.). Computer workshops. Retrieved from

Spring 2015

Contact K.C. Williams

451 Communications Building, 1345 Circle Park Drive, Knoxville, TN 37996-0341