Skip to Main Content

The University of Tennessee

School of Information Sciences

Frequently Used Tools:



INSC 554


Main Navigation:

Brittany Metzger
5. Developing an Evaluative Action Plan

Abstract

This project is concerned with the Lawson McGhee library, part of the Knox County Public Library system and located in the city of Knoxville, Tennessee. The Children's Room of the Lawson McGhee library provides storytime kits for teachers and child care providers to check out and use. This service will be assessed as to its usefulness and visibility to its intended users

5.1 Develop a rationale to propose user-centered assessment/evaluation of the selected PL service. Assess the selected PL service carefully and explain your proposal in relation to participation of primary, secondary, and tertiary users identified in the earlier assignment.

The "Storytime to Go" kits offered by the Knox County Public Library are a valuable resource for preschool and daycare educators in the Knox county area. They allow educators to present high-quality and effective storytimes with minimal effort while providing all the materials necessary for such. They are also extremely helpful in creating and developing early literacy skills while teaching children that reading can be fun (American Library Association). Unfortunately, there is little structured assessment of the impact of these kits and very little knowledge of them outside of a select few members of the community. The proposed assessment will assess how useful current users find the kits as well as assess the likelihood of new users finding and using them.

As identified in Part 2, the primary users are adults ages 20-64, secondary users are children up to age 14, and the tertiary users are adults over 65. The use of this service is largely confined to a portion of the primary segment of users, adults who are also daycare or preschool educators. In such a case, only said portion of the primary users will participate in this assessment.

5.2 Identify the problem (related to your selected service) in terms of: production bottlenecks; tasks that are performed frequently; activities that require frequent movement; and declining budgets.

There are three major challenges with the Storytime to Go kits. The first is the total lack of visibility of the program. There is currently no advertisement for this service and the kits themselves are not visible in the library, as they are kept in a back room and not advertised even in the Children's Room itself. The second challenge is that the kits have not been updated in several years due to lack of fund allocation and low circulation. The third, related problem is that there has been no formal assessment of the impact of these kits. Understanding if the kits have had a positive impact for their users or not and what changes they would like to see to them is key to knowing the best ways to update and advertise them (E. Nguyen, personal communication, 2015). Production bottlenecks, tasks that are performed frequently, and activities that require frequent movement do not have an impact on this service.

5.3 Determine the scope of analysis in terms of what specific information/aspects about the selected PL service do you hope to gain by eliciting feedback from the PL users. In other words, what is the purpose for doing the evaluation? Is the library attempting to improve its operational efficiencies (internal focus), or is the study being done to better understand the effectiveness of a library service? Explain in detail.

The purpose of this evaluation will be two-fold. First, this evaluation will determine interest in the program by asking those who have not used the kits prior if they would be interested, given a short description of how they work. Second, this evaluation will determine if current users find the kits useful. Thus, this study will aim to understand and improve the effectiveness of the Storytimes to Go service by assessing how useful users find the kits, what they like about the kits, and the likelihood of new users finding them useful. By having this data, the library will better be able to decide on how the kits and the marketing efforts therefore should look in the future. As the kits are due for a planned overhaul and marketing push within the next year, this is a perfect time to conduct this assessment (E. Nguyen, personal communication, 2015).

5.4 Determine the kinds of data that you hope to gather (be specific). What evaluation methodology and design will be used (quantitative and/or qualitative methods: be very specific in the types of tools that will be employed)? How will data be collected?

This assessment is designed to collect data on two major aspects of the Storytimes to Go kits - one, user knowledge and interest, as assessed by asking if respondents have heard of the kits and, given a description, if they would be interested in using them with their students and why or why not; and two, previous user's perception of kits, including if they liked using them, what they did and did not like, and if their students enjoyed them. There will also be basic demographics questions related to age, gender, and the age of children the respondent works with. This data will be collected using both qualitative and quantitative methods in the form of a survey with both yes/no and open-ended questions.

5.5 Describe the user-centered assessment/evaluation strategies that you are proposing to conduct community analysis and gather feedback from users about the selected service at your PL. Identify and analyze what methods you will use to gather feedback from various kinds of users as a part of your efforts. How do you propose getting access/permissions to select users? Provide a timeline for implementing your proposed methods.

The assessment strategy that will be used to gather feedback will be a simple survey with both quantitative yes/no and qualitative open-ended questions, delivered both by paper and through the SurveyMonkey online platform. This survey will be delivered in two ways - paper copies handed out at the Children's Room in the library to daycare and preschool educators as well as a SurveyMonkey link that will be emailed out to daycares and preschools that may or may not elect to participate. The two surveys are identical aside from their delivery method. Paper copies will be retrieved from the library one week after dispersal, while the SurveyMonkey version has an online interface for immediate results. This is the most prudent course for this particular survey - as the target user population of the kits is small and specialized (daycare and preschool educators) and the number of such that come to the Children's Room cannot be certain, distributing the survey to randomly chosen daycares and preschools will make it more likely to get a useful number of responses from the target population. Logistical concerns about retrieving paper copies of the survey are the reason for these populations receiving the online version of the survey. The survey will also be distributed at the Children's Room in paper format for ease of use for patrons who are already physically in the building, as well as it being logistically simpler to retrieve these copies. Surveys distributed at the library will have a slightly higher chance of having respondents who have previously used the kits, a concern for this assessment as there are few of them. The survey design and implementation plan was approved by Erin Nguyen, the library partner for this assessment. Daycares and preschools will be contacted individually and link will likely be distributed by principals and managers of said facilities. Final approval on survey has been obtained, and the library's copies of the survey will be provided on April 1, with completed copies to be collected by April 8. Daycares and preschools will have survey links provided on April 2, and data will be collected through April 10. Data will be analyzed April 11-13. This timeline is subject to change as is necessary.

5.6 Share the data collecting instrument that you propose to use. Identify the qualitative and/or quantitative measures you propose to use (what aspects are you trying to measure). Provide strategies to gather consent (consent forms, assent forms as appropriate) and your data-collecting tool(s) with listed questions/prompts, etc. You can explore multiple research methods to gather user-centered assessment/evaluation of the selected service.

The data collection instrument that will be used is a short survey with some explanatory text regarding the program for use in assessing future interest in the program. The first questions are meant to assess current levels of knowledge about the program among its target user population and the likelihood of future use once they know of its existence, while the latter questions assess current user perceptions of the program and its usefulness. As discussed previously, yes/no questions function as quantitative measures to get solid numbers on how many potential and actual users exist, while the open-ended qualitative questions serve to find out why potential users are or are not current users and what both groups like or dislike about the program. Consent will be given by completing the survey - they are entirely voluntary and no one will be required to complete one, nor is there any reward for completing one. The text of the survey follows.

The Children's Room at Lawson McGhee Library currently offers a resource known as "Storytimes to Go." These kits are a complete story time program in a bag, complete with books, manipulatives, puppets, puzzles, felt board pieces, songs, rhymes, lists of additional possible materials for the story time available from the library, and instructions for how to put on the story time. They are designed to "offer professional preschool educators and caregivers the resources to help children build early literacy skills through enhanced cognition, social-emotional development, motor development, and language development." The library currently has kits on the following themes/subjects: All About Me, Colors, Community Helpers, Counting & Numeracy, Dinosaurs, Families, Food, Friendship, Gardening & Growing, Insects, Jungle Animals, Move It, Nursery Rhymes, Pets, School, Shapes, Transportation, and Weather.

Please answer the following questions to help the library assess this service, thanks!

Have you heard of the Library's storytime kits before?

Y / N

Have you used the Library's storytime kits before?

Y / N

If you have not used the kits before, would you want to in the future?

Y / N

Why or why not?

If you have used the kits before, did you find them useful to your curriculum?

Y / N

If you have used the kits before, did your students react well to the programming?

Y / N

If you have used the kits before, what would you add to them? What would you not use again?

Any additional comments?

Age:

Gender:

Ages of kids worked with:

5.7 Provide strategies to maintain anonymity and confidentiality of users participating in your study (if appropriate).

To maintain anonymity, all paper surveys will be distributed by Children's Room employees with no oversight by the researcher. All SurveyMonkey respondents are automatically anonymous due to the nature of the service. The surveys do not ask for names, contact information, where the respondents work, or what area of the county they live in. The demographics questions asked serve only to get an idea of who the target audience is as a group, and does not give enough information to be used as identifiers.

5.8 Create a budget in terms of investment of resources (e.g., money, time, efforts, human input, etc.) to orchestrate user-centered assessment/evaluation of various kinds. Identify stages for orchestrating a plan for the PL (e.g., previous efforts made, current efforts, future efforts) and describe what will be feasible as part of your efforts in this course during this semester.

The investment of money in this assessment is negligible. Paper copies of the survey are approximately $0.10 per copy and only a few copies are needed, while the SurveyMonkey version is free to create and use. Similar surveys could likewise be conducted in the same way, though a greater number of paper copies may be necessary depending on what is being assessed. However, in cases where there is little or no budget for the survey, a SurveyMonkey will cost the library nothing in terms of cost for the service, and the time required for an employee to create the survey on this platform is under one hour. In terms of time, the greatest amount of time needed is on the part of the researcher, to assess the community and what parts of the service need to be evaluated, then designing the actual assessment instrument and administering it. The library's time in this researcher's case is limited to helping with questions and distributing surveys. If a similar survey were being conducted by a library employee, the time needed to design the survey et all would become library time as well, and would thus require the employee to be paid for the time to conduct such a process, adding to the cost in terms of salary and time not spent doing other work. The time required by the respondents to answer all survey questions is approximately five-ten minutes, contingent on their answering the open-ended questions. The effort needed to collate the data afterwards would be contingent on if data was collected through paper surveys or electronic means, as electronic surveys will often total up the quantitative data for the user. The data would then need to be made meaningful for the consumption of others in the form of a graph or a report.

No previous efforts have been made to assess the Storytimes to Go kits on a formal basis, and the only current effort is this researcher's assessment. The library may consider doing additional research in the future as they get ready to re-launch the service, but there is no current set plan for this (E. Nguyen, personal communication, 2015). For the efforts put forth during this semester, it is feasible to distribute a fairly simple survey to a small number of participants who may or may not choose to answer. A more robust assessment method or a longer amount of the time would allow for the survey to be conducted for a greater length of time and involve more respondents. The library will use the assessment data to see how current users view the service and what could be changed, as well as who could become a user, in order to judge how to update the kits and in what ways to market the kits to said potential users.

5.9 Determine the kinds of analysis you will do with the data collected. How do you plan to report the findings based on the data collected?

Once all data has been collected, the answers to quantitative questions will be tallied and compared - how many people have heard of the kits vs. how many have actually used them vs. how many are interested now that they know of their existence, for example. This will be represented in chart and table format. The qualitative questions will be evaluated individually to assess if there are any common threads as to why or why not potential users are interested in the kits and what actual users do or do not like about them. Information will be collated into a single evaluation report and presented to Erin Nguyen, the Children's Room manager, along with a verbal explanation.

References

American Library Association. (2015). Early Literacy & Libraries. Retrieved from http://www.ala.org/advocacy/literacy/earlyliteracy
Matthews, J. (2007). The Evaluation and Measurement of Library Services. Westport, CT: Libraries Unlimited.

Spring 2015

Contact K.C. Williams

451 Communications Building, 1345 Circle Park Drive, Knoxville, TN 37996-0341

Phone:865-974-2148
Fax:865-974-4967
Email