INSC 554: Public Library Management and Services
5. Developing an Evaluation Action Plan (20 Points)
This assignment will help you prepare an evaluation by discussing various discrete activities in your proposed plan. For this assignment especially study Ch. 1 and Ch. 3-6 EMLS.
The evaluation plan is aimed to justify the proposed evaluation action on the Dolly Parton's Library service provided across the Knox County area. The Dolly Parton's Imagination Library has seen an increasing number of participants ever since it was offered to the local public. An evaluation of the service would help to assess its effectiveness in enhancing reading abilities of target audience. In addition to the analysis of possible problems related to the service, the plan also proposes a sample survey and data collection methods and strategies for the program director to review.
5.1 Develop a rationale to propose user-centered assessment/evaluation of the selected PL service. Assess the selected PL service carefully and explain your proposal in relation to participation of primary, secondary, and tertiary users identified in the earlier assignment.
In general, a user-centered assessment helps the library to be aware of the quality of their past service performance, to know about where they are, and to plan the their prospect (Class Slides, Week 4). Specifically, the evaluation allows the library to see the outcomes of certain services within a time period, such as a fiscal year. An analysis of the outcomes will show whether users are satisfied with the service, and if not, what their suggestions and expectations are. The qualitative statistics will help administration to determine future adjustment and enhancement. And the quantitative data such as how many target users are the actual users indicates whether the objectives have been achieved. The Dolly Parton's Imagination Library of Knox County has seen a big success since its existence in 2005. There are 68% of all Knox county children under the age of five have received books in the program (KCPL, 2014 Annual Report). And Knox County ranks in the first place of urban registration percentage in the United States (KCPL, 2014 Annual Report). However, there are 32% of target users that haven't registered the program. And the 2014 Annual Report doesn't give much information of the impact of the program in closing school readiness in Knox County. Since the final goal of the program is to close the readiness gap in school rather than solely ensuring book readings for the youngest children, it's necessary to gather and examine statistics about school readiness.
The sex and age distribution chart of 2013 Knox County (Burchett, 2013) indicates that there are 26,920 children under age five in 2013, which makes up 5.9% of the total population in the Knox County. The previous analysis of the user groups of Knox County Public Library (KCPL) suggests that the primary user group is children under age of 14. Hence the proposal is mainly targeted to a small part of KCPL's primary user group.
5.2 Identify the problem (related to your selected service) in terms of: production bottlenecks; tasks that are performed frequently; activities that require frequent movement; and declining budgets.
The service doesn't have any production bottlenecks at the moment. The major tasks involved in the service are to send age-appropriate books to registered users. The major challenge of the service might be raising funds to purchase books for target audience. The increasing number of audience demands for a stronger financial support in the future. The service now has sustainable funds from a volunteer board annually (KCPL, 2014 Annual Report), however, the library should also consider raising funds from other possible sources to make sure the service can be continuously offered to users.
5.3 Determine the scope of analysis in terms of what specific information/aspects about the selected PL service do you hope to gain by eliciting feedback from the PL users. In other words, what is the purpose for doing the evaluation? Is the library attempting to improve its operational efficiencies (internal focus), or is the study being done to better understand the effectiveness of a library service? Explain in detail.
The first information that's expected to be collected from the evaluation is whether the supporting parents regard the mailed books to be age-appropriate. Delivering age-appropriate books to registered users makes sure children get entertained and receive good education through reading. Moreover, the statistics of children's reading skills and study performance before and after children's participation in the program would be collected. Besides, information about whether the registered children receive a new book each month and whether each new book arrives in time would also contribute to the analysis of the service's effectiveness. As has been mentioned above, by asking parents' open-ended questions, the survey is expected to collect in-depth feedbacks from parents, which would help understand to what degree the goal of the service has been accomplished. Hence, the analysis is being done to better understand the effectiveness of the service.
5.4 Determine the kinds of data that you hope to gather (be specific). What evaluation methodology and design will be used (quantitative and/or qualitative methods: be very specific in the types of tools that will be employed)? How will data be collected?
Both quantitative and qualitative data would be collected and analyzed to evaluate the effectiveness of the identified service. The major method that would be applied is qualitative method. By using qualitative method, the director or manager of the service is able to find out what the users have experienced or observed in the program (Matthews, 2007, p. 47). A structured survey will be designed and distributed to target users to collect data. The survey will include both closed and open-ended questions. Descriptive information and statistics such as how much appropriate do the parents regard the mailed books to be will be collected. Parents would also be asked open-ended questions that would help to achieve a better understanding of the participation experiences.
Online survey tools such as Qualtrics and Survey Monkey could be used to gather statistics. The survey would start with an informed consent to explain the purpose of the survey and to declare that all the information filled in by the participants would be kept confidential. The types of the questions in the survey would be multi-choice questions and open-ended questions. In the end of the survey, participants would receive an appreciation note. Surveys that are completed by parents would be distributed through the survey system to the data analyzer. Since the Internet is very convenient and some parents may not have much time to regularly come to KCPL to deliver the survey in person, online-survey tools could be very helpful and time saving.
One-to-one interview or focus group could also be employed to obtain associated data. Matthew (2007) suggested that interviews offer a good opportunity to gain a in-depth understanding the situation of the library patron. In this evaluation, interviews and focus groups could be effective in providing more detailed information about children's school readiness than online surveys. The first possible explanation could be that some parents may prefer to talk than write long sentences. Another advantage of focus group is that parents are likely to be more open to talk when they are sitting in the environment of idea exchanging. In the one-to-one interview or focus group, interviewers could use recording instruments to collect data. Compared to note taking, a recording instrument is more advantaged at capturing detailed information that could be left over while taking notes.
5.5 Describe the user-centered assessment/evaluation strategies that you are proposing to conduct community analysis and gather feedback from users about the selected service at your PL. Identify and analyze what methods you will use to gather feedback from various kinds of users as a part of your efforts. How do you propose getting access/permissions to select users? Provide a timeline for implementing your proposed methods.
In the specific service that's going to be evaluated here, the user groups would be young children under the age of five. Since these children are not able to respond to research questions because these questions will not make much sense to them, their supporting parents will answer the survey questions for them. Online surveys could be distributed to supporting parents according to email records in the system. It is very likely that not each individual who receives the survey will respond to it. So it may also be necessary to get some feedbacks from parents who are physically present in the library. Workers at the reference desks could help deliver the survey copies to parents in the library. Besides, library workers who are working on the evaluation could ask parents to do a one-to-tone interview if parents happen to have spared time to be spent in the library or they could be invited to join a focus group on the other specific date.
5.6 Share the data collecting instrument that you propose to use. Identify the qualitative and/or quantitative measures you propose to use (what aspects are you trying to measure). Provide strategies to gather consent (consent forms, assent forms as appropriate) and your data-collecting tool(s) with listed questions/prompts, etc. You can explore multiple research methods to gather user-centered assessment/evaluation of the selected service.
Quantitative tools include counting, measuring, surveys, conjoint analysis, transaction log analysis and experiments (Matthews, 2007, p. 61). Qualitative tools include examining documents, diaries, observation, and interviewing (Matthews, 2007, p. 50). In the online survey, a consent form would be presented before the start of the survey questions. And participants would be only asked to provide their initials to start with the survey. If a person doesn't agree with the consent form, the survey will jump to the ending page of the survey. For people who finished and submitted the survey online, they will receive a "thank you for participating in the survey" note at the finish page. In the face-to-face interview, interviewers will not require parents to provide privacy information such as their names and addresses. A designed survey is presented as follows:
1. How did you hear about this program?
A. Through mass media (ads, posters, newspapers, etc)
B. Through a friend/colleague/family member
C. Other (please specify) ___________
2. How long have your child(ren) been in this program
A. <1 year
B. 1-2 years
C. 2-3 years
D. 3-4 years
E. >4 years
3. How often do you purchase new storybooks for your child(ren)?
D. Very often
4. How often do you read with your child(ren)?
A. Once or twice a week
B. Three or four times a week
C. Almost every day
5. How many minutes each time do you read with you read with your child(ren)
A. Less than 15 minutes
B. 15 minutes or so
C. 30 minutes or so
D. At least 1 hour
6. Does this program help you choose other books for your child(ren)? If so, how?
7. How satisfied are you with the program?
A. Barely satisfied
B. Somewhat satisfied
C. Relatively satisfied
D. Very satisfied
8. Will you introduce this program to your friends/colleagues/family members?
9. Do you think the program enhances your child(ren)'s reading skills? Why?
10. Do you have any specific suggestions for the program?
5.7 Provide strategies to maintain anonymity and confidentiality of users participating in your study (if appropriate).
Participants won't be asked to provide their privacy information and before the start of the survey and the interview, they would sign a consent form that says all the research results would be kept in confidential.
5.8 Create a budget in terms of investment of resources (e.g., money, time, efforts, human input, etc.) to orchestrate user-centered assessment/evaluation of various kinds. Identify stages for orchestrating a plan for the PL (e.g., previous efforts made, current efforts, future efforts) and describe what will be feasible as part of your efforts in this course during this semester.
Money: The research won't demand much extra money except in the circumstances that the library doesn't have any recording instruments and has to purchase new ones, which would cost 100 dollars. And there would be some cost in printing consent forms for the use in interviews, the cost would be only like 10 dollars.
Time: It usually takes a month for the results of online surveys to be collected. And researchers could use this period or another month to do personal interviews if needed. And researchers would need time to compile the data for further analysis. So in total, the research could take two months in maximum.
Effort & Human input: Before the start of the survey and interview, the library needs a team to discuss the question designs and issues such as how to select users. And it would be necessary to allocate several workers to work on email distributions and to get contact with potential participants. The budget is included in their salaries.
The possible efforts that could be made in this course are to design a feasible questionnaire for the future distribution to participants. They will also be printed out for the library workers to have them delivered to parents who physically come to the library.
5.9 Determine the kinds of analysis you will do with the data collected. How do you plan to report the findings based on the data collected?
The descriptive methods would be mainly used in the analysis of data. Some collected data would be converted into tables and charts as needed. Tables and charts "help reveal interesting structures present in the data" (Matthews, 2007, p. 81). For example, to analyze the age make-up of the registered children, the library could create a pie chart to visually see the percentage of each age. For open-ended questions such as the last one in the previous listed sample questions, more detailed written report are needed. The report is used to further identify the effectiveness of the program and to compare the difference in the children's before-and-after reading performance. Charts or graphs that represent factual information and analysis on parents' narratives would be integrated together into a final report as a complete evaluation.
Burchett, T. (2013). 2014-2015 Adopted Budget Detail. Retrieved from http://www.knoxcounty.org/finance/pdfs/2014_2015_budget/2014-2015adopted_budget_detail.pdf
Knox County Public Library. (2014). 2014 Annual Report. Retrieved from http://www.knoxlib.org/sites/default/files/kcpl-annual-report-2014.pdf
Matthews, J. R. (2007). The Evaluation and Measurement of Library Services. Westport, CT: Libraries Unlimited.
Contact K.C. Williams451 Communications Building, 1345 Circle Park Drive, Knoxville, TN 37996-0341