Skip to Main Content

The University of Tennessee

School of Information Sciences

Frequently Used Tools:



INSC 554


Main Navigation:

Developing an Evaluation Action Plan

The focus of this study is on the Bristol Public Library (BPL) which serves approximately 44,000 residents of Bristol, Tennessee and Bristol, Virginia. More specifically this study seeks to assess and evaluate BPL's Adult Learning Lab (A.L.L.) in terms of the library's strategic plan and the needs of its community. This paper explores the purpose and rationale for assessing and evaluating the A.L.L. Current problems related to the service are identified. The type of data to be collected and the method used to collect the data are examined. The design of the collection instrument, a web-based survey, are also discussed. In addition, this paper will discuss the estimated budget for the study and the type of analysis to be performed.

5.1 Develop a rationale to propose user-centered assessment/evaluation of the selected PL service. Assess the selected PL service carefully and explain your proposal in relation to participation of primary, secondary, and tertiary users identified in the earlier assignment.

The A.L.L. provides a very important service to the Bristol community. Its primary purposes are to support lifelong learning and assist in bridging the adult digital literacy gap. It is important to consistently evaluate the library services to assure they are fulfilling the goals set by the library as well as meeting the needs of the customers. According to Matthews (2007), "the perception of the customer is much more important than any other objective measure of the quality of the service being delivered" (p. 25). Therefore, it is of utmost importance to gather input on the satisfaction of the community when evaluating the services provided by the A.L.L.

BPL understands the importance of user-centered assessment and are currently in the process of assessing the A.L.L. to determine how it can best be utilized to meet the needs of the community (Powers, J., personal communication, 2015, January 27). The lab began life as a GED tutoring center and has continuously added new services throughout the years, such as computer assistance, career readiness certifications, and math reading and writing tutoring. Due to recent changes to the GED process the A.L.L. staff are no longer administering the tutoring services but instead the Mount Rogers Adult Education host GED preparation classes in the A.L.L. two nights a week (Gaylor, K., personal communication, 2015, March 9). This change in structure has led the library to turn to the community for input on new Adult Learning Lab programming (Stickley, D., personal communication, 2015, March 26). During the months of January and February of 2015, the library hosted five focus groups (McCaffrey, 2015). The focus groups were successful at generating a list of programming the community is interested in the library adding to the A.L.L. when it moves to a larger location as part of an upcoming renovation project (Stickley, D., personal communication, 2016, March 26).

The primary participants in the focus groups were non-users. To supplement these findings, this study will administer a satisfaction survey to assess how well the A.L.L.'s current offerings are satisfying the needs of the current users. This would provide insight to determine which offerings are meeting the needs and which need to be improved. The primary, secondary, and tertiary users of the library were previously identified as teen and young adults, elderly adults, and children under the age of 12. According to Gaylor, because the A.L.L. is an adult program it does not serve the tertiary user group of children under the age of 12, therefore they will not be a part of the assessment. The A.L.L. has slightly different user groups broken down as follows: primary-middle-aged adults, secondary-elderly adults, and tertiary-young adults. With the majority of users being low or extremely low income.

5.2 Identify the problem (related to your selected service) in terms of: production bottlenecks; tasks that are performed frequently; activities that require frequent movement; and declining budgets.

The main production bottleneck impacting the A.L.L. is the lack of staffing. Due to the lack of the staffing the lab is only open 40 hours a week, Monday through Friday. Ideally, Gaylor would like the see the lab staying open for evening and weekend hours to accommodate community members who work during weekdays (personal communication, 2015, March 9). The lack of staffing also limits how many individuals can be served at any given time during the lab's walk in hours. This bottleneck stems from a lack of funding. As was mentioned previously in this study, the lab is funded mostly by grants and fundraising. The lab is not part of the libraries budget. If the A.L.L. was able to obtain additional funding, the first thing Gaylor would do is hire more qualified staff to assist the users in meeting their educational and professional goals (personal communication, 2015, March 9). Additionally, funding would also allow the lab to keep the computers up to date and to license additional and update current software necessary to satisfy identified community needs and interests.

5.3 Determine the scope of analysis in terms of what specific information/aspects about the selected PL service do you hope to gain by eliciting feedback from the PL users. In other words, what is the purpose for doing the evaluation? Is the library attempting to improve its operational efficiencies (internal focus), or is the study being done to better understand the effectiveness of a library service? Explain in detail.

Currently, there is no standard way for users to express their opinions about the service they received. The scope of analysis for this study focuses on the satisfaction of the A.L.L. users with the current services provided. The satisfaction survey seeks to gain feedback from the users regarding their experiences and perceptions of the A.L.L to determine the effectiveness of the lab at meeting user needs. More specifically, the survey will help the library to gain information on what participants like and don't like about the service and any suggestions they have to help the lab to better meet their needs. This insight will allow BPL to better tailor the services to their target audience as well as gain knowledge on the user's perceptions of the service and their vision for the lab.

This information is especially important right now as the lab prepares to move to a new larger location in the library. It will also help to guide the lab as it transitions to its new role as the libraries Creativity Center. By asking about what the users would change, what they would like included, and what times work best for them the A.L.L. will be able to assure its service aligns with user's needs. This will help to enhance the users experience with the service as well. This data could also be used when applying for grants to demonstrate the importance of the service to the community users.

5.4 Determine the kinds of data that you hope to gather (be specific). What evaluation methodology and design will be used (quantitative and/or qualitative methods: be very specific in the types of tools that will be employed)? How will data be collected?

This evaluation hopes to gather both qualitative and quantitative data through a satisfaction survey. The survey will primarily be distributed online. The first section will ask close-ended questions utilizing a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) to allow participants to easily express their perception of the A.L.L. services. The second section will ask the participants to respond to a series of open-ended questions expressing their opinions on the service provided and suggestions for improvement. These open-ended questions will allow the participants to fully express their perceptions of the service. The overall goal of this evaluation is to provide the library with user input to assist in improving the service provided by the A.L.L and gather ideas for new services to offer as the lab transitions to the Creativity Center.

5.5 Describe the user-centered assessment/evaluation strategies that you are proposing to conduct community analysis and gather feedback from users about the selected service at your PL. Identify and analyze what methods you will use to gather feedback from various kinds of users as a part of your efforts. How do you propose getting access/permissions to select users? Provide a timeline for implementing your proposed methods.

The user-centered assessment strategy proposed is an eight-question web-based satisfaction survey. The web-based survey will be advertised on the A.L.L.'s Facebook page and during sessions from April 22nd through April 24th, 2015. The survey design has been approved by A.L.L. Coordinator, Karen Gaylor. Gaylor has also granted approval for the survey to be posted on the lab's Facebook page and promoted during the sessions. The web-based survey will remain open until 11:59 p.m. on Friday, April 24th. The data will be analyzed on April 25th through 26th and the results will be reported on April 27th. The target population for the survey is Bristol community members who have experience using the Adult Learning Lab services provided at BPL.

5.6 Share the data collecting instrument that you propose to use. Identify the qualitative and/or quantitative measures you propose to use (what aspects are you trying to measure). Provide strategies to gather consent (consent forms, assent forms as appropriate) and your data-collecting tool(s) with listed questions/prompts, etc. You can explore multiple research methods to gather user-centered assessment/evaluation of the selected service.

The survey will be administered using Google Forms, this survey tool was selected because it easy to use and most individuals are familiar with Google products. As has already been mentioned, the survey will be distributed via the A.L.L's Facebook page and during the sessions. As these are anonymous surveys, no consent form is necessary. The survey will contain the following statement of intent:

"Hello, my name is Jennifer Young I am Master's candidate at the University of Tennessee in the Information Science program. I am conducting a quick survey to find out what users think about the Bristol Public Library Adult Learning Lab. Your participation in the survey is voluntary and you are assured anonymity and confidentiality. Please respond to the survey as honestly as you can. Feel free to contact me with any questions at jyoung79@vols.utk.edu. Thank you for your time."

The structured question section of the survey will ask participants to rate the following using a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). This section of the survey is mandatory.

  • The material covered during my session(s) was relevant.
  • The instructor clearly presented the information.
  • The session(s) helped me to improve my skill set.
  • Indicate your overall satisfaction with the current offerings of the Adult Learning Lab.
  • I would recommend this service to a friend or family member.

The unstructured question section of the survey will ask participants to voluntarily answer the following questions in their own words.

  • Is there anything you would change about the Adult Learning Lab?
  • Are there any services that you would like the Adult Learning Lab to begin offering?
  • What days and times are most convenient for you to visit the Adult Learning Lab?

To assist in analyzing the satisfaction of each user group the survey will open by asking the user to identify their age. This is the only factual question asked in the study and cannot in any way be used by the staff to identify the participant. The question will be multiple choice and contain the following options: under 18, 18-35, 35-65, and 65+. The survey can be accessed at the following link http://goo.gl/iC1dYf.

5.7 Provide strategies to maintain anonymity and confidentiality of users participating in your study (if appropriate).

The participants can be assured that the information they provide in the survey will be anonymous and confidential by the structure of the survey tool. When viewing the results in Google Forms the only provided information besides the answers is the date and time the survey was taken. To further assure anonymity, the survey will only ask one demographic question relating to age group in order to assist with analysis. The researcher understands the absolute necessity for patron privacy and will only use the data collected for the stated purpose of evaluating the A.L.L. Any potentially identifying information provided by the participant in the open-ended questions will be kept confidential.

5.8 Create a budget in terms of investment of resources (e.g., money, time, efforts, human input, etc.) to orchestrate user-centered assessment/evaluation of various kinds. Identify stages for orchestrating a plan for the PL (e.g., previous efforts made, current efforts, future efforts) and describe what will be feasible as part of your efforts in this course during this semester.

The investment of resources will vary depending on the type of service being evaluated, the depth of the evaluation and the tools used to perform the evaluation. For most situations, low-cost options are available. BPL is already familiar with and have utilized user-centered assessment in the past and plan to continue its use as they add more programming and renovate their spaces. The main resource needed to orchestrate a variety of user-centered evaluations at BPL is time and effort.

BPL recently performed a user-centered evaluation of the A.L.L. in the form of focus groups. The primary cost associated with this method was the hired administrator of the focus groups. A lot of efforts went into advertising the focus groups through different routes to assure a good turnout. For the purposes of this study, a satisfaction survey will be administered during a three day period. This study will serve as what Mathews (2007) calls a "prototype of the study" (p. 12). Due to time constraints it is only feasible to collect a limited amount of data. It is the hopes of this project to demonstrate the importance of the ongoing collection of this data to the library administration. This study will hopefully be the beginning of the integration of user-centered evaluations into the A.L.L. sessions.

The costs associated with the assessment and evaluation proposed for this study are very low. Since the survey will be delivered via Google Forms there are no costs associated with its creation or distribution. The main advertising venue for the survey is Facebook, which is also a free service. The main cost associated with the assessment and evaluation is in the form of time and human input. The library staff time needed will be minimal though. A.L.L. staff will be asking users if they would like to participate in the evaluation after each session. If the user volunteers the staff member will direct them to a computer in a private area of the lab with the survey set as the browser homepage. The survey should take approximately 10 minutes to complete. Once the data has been collected it will take human input to analyze the data, identify trends, and create a report of the results. Google Form automatically compiles and tallies the results of the closed-ended questions, therefore, the only data that will require analysis is the open-ended answers.

5.9 Determine the kinds of analysis you will do with the data collected. How do you plan to report the findings based on the data collected?

This study will use "descriptive methods .to identify trends, perform comparisons, and make better-informed decisions" (Matthews, J.R., 2007, p. 79). Google Forms tallies the answers to the quantitative data and provides a bar chart displaying the results for each answer. The collected data will be exported to a Microsoft Excel spreadsheet to calculate the mean, median, and mode. This information will be used to illustrate the degree to which A.L.L. users are satisfied with the service provided. The qualitative data will be analyzed to discover trends. Pie charts will be created to clearly present any identified trends. A formal summary including the statistics will be reported to the A.L.L. Coordinator. This report will create a snapshot of user satisfaction, laying the foundation for future evaluations.

References

Matthews, J.R. (2007). The Evaluation and Measurement of Library Services. Westport, CT: Libraries Unlimited.

McCaffrey, A. (2015). Adult creativity lab focus group report.

Spring 2015

Contact K.C. Williams

451 Communications Building, 1345 Circle Park Drive, Knoxville, TN 37996-0341

Phone:865-974-2148
Fax:865-974-4967
Email