EvaluLand

I chat with Jannik Kaiser about regenerative evaluation, an approach that asks how evaluation itself can contribute to flourishing. We discussed Jannik's background in evaluation and what led him to regenerative evaluation, how and why the approach was developed, what regenerative evaluation is and an example of regenerative evaluation in practice, and more. "Too often, evaluation is experienced as a "tax" on doing good: an extractive exercise where we mine data from communities to feed upward reporting structures. It reinforces a power dynamic where the funder holds the yardstick and the community bears the burden of proof. We count the fruits while depleting the soil. But I believe - and have witnessed first hand - that evaluation can be a source of vitality and justice, not just accountability." - Regenerative Evaluation Living Paper: Influences, Pathways, & Practices Resources Introduction to the Regenerative Evaluation Living Paper Regenerative Evaluation Living Paper: Influences, Pathways, & Practices Impact Garden M&E in Complexity: Presenting a methodology for making credible claims Capacity Compass Regenerative Evaluation Community of Practice About Jannik Kaiser Jannik Kaiser is the Co-founder and CEO of Unity Effect, where he leads the area of Regenerative Measurement and Evaluation. As the main editor of the Regenerative Evaluation Living Paper, Jannik has dedicated his career to bridging complexity science with participatory practice. He advocates for transforming evaluation from a compliance-driven reporting requirement into a regenerative intervention in itself: one that gives more energy than it takes and actively shifts institutional power. His work encompasses a broad spectrum of contributions, ranging from advising global organizations such as the ILO and UN agencies, to developing open-source frameworks such as the Impact Garden, Capacity Compass, and Methodology for Credible Claims in Complexity, to empower purpose-driven organizations worldwide. Email: jannik@unityeffect.net LinkedIn: https://www.linkedin.com/in/jannik-kaiser/Support EvaluLand

In this episode, I chat with the authors of The Garden of Evaluation Approaches about their innovative and helpful framework for thinking about evaluation theory and approaches. They each share their personal journeys into evaluation as well as into this work they have been collaborating on for years. We also discuss evaluation theory more broadly, including how to make it more accessible to new and emerging evaluators. About guests: Bianca Montrosse-Moorhead is a Professor of Research Methods, Measurement, and Evaluation at the University of Connecticut, where she also directs the Partnership for Evaluation and Educational Research (PEER). As Co-Editor-in-Chief of New Directions for Evaluation and a fervent advocate for evaluation, Bianca has dedicated her career to bridging the space between evaluation theory and practice. Her work encompasses a broad spectrum of contributions, from evaluating various educational and social programs using diverse methodologies to enhancing the professional training of evaluators worldwide. Daniela Schroeter serves as a Presidential Innovation Professor and Associate Professor at Western Michigan University. With a Ph.D. in Interdisciplinary Evaluation and over twenty years of research, Daniela has contributed substantially to developing evaluation theories and methodologies. Her global work has involved diverse sectors, focusing on enhancing the capacity and effectiveness of evaluations. Daniela also co-edits the Teaching & Learning of Evaluation section of the American Journal of Evaluation. Lyssa Wilson Becho serves as a Principal Research Associate at The Evaluation Center of Western Michigan University. Garnering the 2024 Marcia Guttentag Promising New Evaluator Award, Lyssa has made a significant impact in advancing evaluation methodologies, focusing on culturally responsive practices and promoting equity within evaluation processes. Lyssa is a co-Executive Editor for the Journal of MultiDisciplinary Evaluation. Collectively, their innovative work in evaluation spans continents, cementing their reputation as leaders in advancing the practical, theoretical, and methodological facets of the discipline. Contact information: Bianca Montrosse-Moorhead Email: bianca@uconn.edu LinkedIn: https://www.linkedin.com/in/BMMoorhead/ ResearchGate: https://www.researchgate.net/profile/Bianca-Montrosse-Moorhead Daniela Schröter Email: daniela.schroeter@wmich.edu LinkedIn: https://www.linkedin.com/in/danielaschroeter/ ResearchGate: https://www.researchgate.net/profile/Daniela-Schroeter Lyssa Wilson Becho Email: lyssa.becho@wmich.edu LinkedIn: https://www.linkedin.com/in/lyssa-wilson-becho/ ResearchGate: https://www.researchgate.net/profile/Lyssa-Becho Resources mentioned: Project vita (biography or résumé) with links to all free and publicly available resources: https://tinyurl.com/EvalGardenVita Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches visualization. Journal of MultiDisciplinary Evaluation, 20(48), 49–58. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/1029 Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches. American Journal of Evaluation, 45(2), 166–185. https://doi.org/10.1177/10982140231216667 Bledsoe, K. L., & Graham, J. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26(3), 302-319. https://doi.org/10.1177/1098214005278749 King, J. (2024) Garden of evaluation approaches with SROI and CBA https://linkedin.com/posts/julian-king-87a015a_cba-sroi-vfi-activity-7201338499776618497-J3-z/ Miller, R. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(3), 390–399. https://doi.org/10/fmzjxpSupport EvaluLand

I share my reflections on teaching Foundations of Evaluation after a major course revision, including how it's been going for the first four weeks. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

This month I chat with Brittany Dernberger about her evaluation background and practice. Brittany Dernberger is a sociologist and gender inequality expert who has led research, evaluation, and organizational learning across academia, philanthropy, government, large international NGOs, and small nonprofits. Brittany currently leads global initiatives to measure systems-level change at CARE and co-chairs the American Evaluation Association Systems in Evaluation Topical Interest Group. Learn more about Brittany’s work at http://brittanydernberger.com/. Access CARE’s completed Systems-Level Impact evaluations at https://careevaluations.org/evaluation/keywords/systems-level-impact/. The AEA Systems in Evaluation TIG paper on Principles for Systems Thinking in is available at https://www.betterevaluation.org/sites/default/files/SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

In this episode, I chat with Corey Newhouse and Jessica Manta Meyer, both from Public Profit, about how to make evaluation capacity building sticky for the organizations we work with. Some resources mentioned: TIERS Framework Tipping Point Community Irvine Foundation Change Cadet NonProfit AF article on capacity building does not work Info on the Public Profit’s evaluation cohort Contact information: info@publicprofit.net About Corey: Corey got her start as a teacher at Summerbridge Cincinnati in the early 1990s and has been involved in educational equity and social justice movements ever since. She founded Public Profit to build a team that would seamlessly blend social science research methods, organizational change strategy, and a deep commitment to supporting changemakers. As the Founder and Principal of Public Profit, Corey leads the team’s strategic direction, external relationships, and business development. In addition, Corey serves as an internal thought partner to project teams, assisting with the design of Public Profit’s engagements in evaluation, capacity building, and strategic program design. She is co-author of Public Profit’s Creative Ways to Solicit Stakeholder Feedback and Dabbling in the Data, and a contributor to Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned. She is a co-editor of the volume, Measure, Use, Improve! Data Use in Out-of-School Time. About Jes: Jessica began her career as a peer volunteer on a national youth talkline providing referrals and support to youth in crisis. Challenged by what she heard on the talkline, and by some of her own experiences with the health care system, she developed an interest in health education, equity and advocacy, which launched a career in the social services sector. Eventually, Jessica combined that with a lifetime love of numbers and inquiry, which led her to program evaluation. Prior to joining Public Profit, Jessica had several years of experience creating, directing and evaluating a range of youth development, LGBT, and health programs. Her work has run the gamut of nonprofit and social service roles including nonprofit finance, human resources, development and agency-led evaluations. Jessica directs many of the projects at Public Profit. She designs evaluation studies, develops project strategy, and manages implementation including all aspects of data collection, analysis, and reporting. An expert facilitator, Jessica also facilitates large stakeholder meetings and evaluation capacity building trainings, and provides evaluation coaching to clients and staff alike.Support EvaluLand

In this episode, I chat with Dr. Tatiana Elisa Bustos on community-based participatory research (CBPR). We talked about what it is, how it compares to research and other similar forms of inquiry, and how to get started doing CBPR. Disclaimer: Views expressed here are personal and not reflective of the speaker's respective employers or agencies. Contact information Dr. Tatiana Elisa Bustos tbust002@gmail.com @TElisa72 https://www.linkedin.com/in/tebustos/ About Dr. Bustos Dr. Tatiana Elisa Bustos knows that community partner engagement is key to understanding social issues. She’ll share her experience applying community-based participatory research approaches. Dr. Bustos innovates outside the box ways to do research that invite community participation, improving programs through implementation with a social justice lens. As a 1st generation college student and the daughter of Nicaraguan immigrants, equity is deeply important to her. She is an author and award-winning researcher. She leads professional development workshops on implementation science and community based participatory research. She received her PhD in Community Psychology from Michigan State University, an MS in Psychology from Nova Southeastern University, and a BA in Psychology from Florida International University. Connect with her on LinkedIn. Dr. Bustos also appeared on The Sci-Files on Impact 89FM and Beyond the Manuscript, the podcast of Progress in Community Health Partnerships. Resources Professional Organizations Society for Community Research and Action American Evaluation Association Connect (CBPR search) Community Psychology TIG Training Institutes https://www.detroiturc.org/programs-expertise/cbpr-capacity-building https://www.detroiturc.org/about-cbpr/online-cbpr-course https://www.mitrainingcenter.org/courses/cbprs0218noce Toolkits https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/intervention-research/main Journals Global Journal of Community Psychology American Journal of Community Psychology Collaborations: A Journal of Community Based Research and Practice Support EvaluLand

I talked with Dr. Tamara Hamai of Hamai Consulting about contracting, subcontracting, and independent consulting in evaluation. We discussed the minor differences between contracting and subcontracting, cleared up a common misconception of what subcontracting relationships are like, how to get into subcontracting, and tips for setting up contracts. Contact information: Tamara Hamai Hamai Consulting assistant@hamaiconsulting.com https://sustainableimpact.co About Dr. Tamara Hamai: Tamara Hamai, Ph.D., has dedicated her career to empowering organizations and rebuilding our global systems to encourage children’s holistic growth and well-being, from prenatal through the completion of higher education – especially those who are most vulnerable and facing the greatest challenges. In 2008, she founded Hamai Consulting as a platform to help organizations increase their impact, stability, and strength to make a bigger impact in children’s lives. Dr. Hamai’s work spans most aspects of child development, such as early childhood education, higher education, child welfare, parenting and discipline, trauma, and adverse childhood experiences. She has previously been featured by KTLA, KPIX CBS San Francisco, ABC 7 News, NBC Radio, American Psychological Association, Western Psychological Association, Institute for Violence, Abuse, and Trauma, American Evaluation Association, National Head Start Association, and several blogs. She is also a reviewer and on the Editorial Board for publications such as the Journal of Aggression, Maltreatment & Trauma, Journal of Child & Adolescent Trauma, Journal of Sexual Abuse, and the Journal of Child Custody. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

This episode I chatted with Bradlie Nabours about how he got into the field of evaluation and his experience applying for evaluation jobs. He also talks about his evaluation experience and provides great tips on applying for jobs in the evaluation space. Job search & application recommendations: Don’t give up. Know your worth. The job process is difficult, but don’t settle and remember it doesn’t reflect necessarily on you. Keep applying! Don’t limit yourself in what positions you look for. Expand your horizons. Network like your life depends on it! This includes having an online presence and building and leveraging connections with people in your field such as through LinkedIn, EvalYouth (LinkedIn & Twitter @evalyouth_na), and your professors! Be a versatile applicant. Build your resume. Seek out learning opportunities to grow as a professional. Be willing to experiment and try new things. Contact information: Email: [bradlietnabours@gmail.com ](bradlietnabours@gmail.com) Twitter: bradn98 LinkedIn: www.linkedin.com/in/bnabours About Bradlie Nabours: Bradlie is a perinatal health program evaluator at Johns Hopkins All Children’s Hospital in St.Petersburg, Fl. He works for a federal Healthy Start program focused on reducing adverse perinatal outcomes for black women and their families. He holds a Bachelors of Science in Public Health from Georgia Southern University and completed his Master’s in Public Health with a graduate certificate in Maternal and Child Health from the University of South Florida. He started his evaluation journey while in grad school where he worked as a graduate assistance on the evaluation of a non-profit family violence prevention program. While in grad school his research focused racial disparities in birth outcomes which led him to start his career at the intersection of his research interest and evaluation. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

On this episode, I’m chatting with Dr. Ayesha Boyce about the evaluation job market. Ayesha provides a ton of great insights and information about how to search for jobs, what to consider when looking for jobs, and tips for being a strong job candidate. Ayesha was previously on the podcast discussing "teaching evaluation and supporting students and colleagues of color." Contact information: Ayesha Boyce Email: ayesha.boyce@asu.edu Twitter: @AyeshaBoyce Website: https://education.asu.edu/about/people/ayesha-boyce About GUEST: Ayesha Boyce is currently an associate professor in the Division of Educational Leadership and Innovation at Arizona State University. She also co-directs the STEM Program Evaluation Lab. Boyce’s scholarship focuses on attending to value stances and issues related to diversity, equity, inclusion, access, cultural responsiveness, and social justice within evaluation—especially multi-site, STEM, and contexts with historically marginalized populations. She also examines teaching, mentoring, and learning in evaluation. She has evaluated more than 55 programs funded by the National Science Foundation (NSF), US Department of Education, National Institutes of Health, and Spencer and Teagle foundations. Boyce is a 2019 American Evaluation Association Marcia Guttentag Promising New Evaluator Award recipient. In her teaching and mentorship, Boyce encourages students to develop a strong methodological foundation, conduct studies based on democratic principles, and promote equity, fairness, inclusivity, and diversity. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

This episode I chatted with Jan Noga about systemic design thinking. There’s a wealth of resources and information provided below! Contact information: Jan Noga Jan.Noga@pathfinderevaluation.com www.pathfinderevaluation.com About Jan Noga: Jan Noga is an independent evaluation consultant based in Cincinnati, Ohio. She holds a bachelor’s degree from Stanford in developmental and counseling psychology with specialization in early and middle childhood and a master’s degree from the University of Cincinnati in instructional design and technology. Jan has worked in the non-profit and public sectors in human services and education for more than 30 years in roles spanning teaching, research, policy, and program planning and evaluation. As a program evaluator, Jan has planned and conducted both large and small-scale evaluations and provided organizational consulting and capacity building support to clients. She has also taught courses and workshops on such topics as systems thinking, systemic design thinking, research methods and techniques, program planning and development, and survey design and analysis. Jan has been a member of AEA since 2000 and was one of the founding members of the Systems in Evaluation TIG, serving as program chair and then TIG chair from 2004-2012. She is particularly interested in the use of systems approaches as a foundation for design, planning, implementation, and evaluation of change efforts in the human service and education arenas. Systems Thinking Resources for Evaluators: Hands on resources: Williams, Bob. 2020. Systemic evaluation design: A workbook. Available for download from https://bobwilliams.gumroad.com/ Williams, Bob. 2021. Systems diagrams: A practical guide. Available for download from https://bobwilliams.gumroad.com/ Good for starting out Anderson, V. & Johnson, L. (1997). Systems thinking basics: From concepts to causal loops. Waltham, MA: Pegasus Communications. Meadows, D.H. (2008). Thinking in systems: A primer. White River Junction, VT: Chelsea Green Publishing. Ramage, M. & Shipp, K (2009). Systems Thinkers. New York: Springer. Sweeney, L.B. & Meadows, D. (2010). The systems thinking playbook. White River Junction, VT: Chelsea Green Publishing. Williams, B. & Hummelbrunner, R. (2011). Systems concepts in action: A practitioner’s toolkit. Stanford, CA: Stanford University Press. Williams, B. and Imam, I, eds. (2007). Systems concepts in evaluation: An expert anthology. Point Reyes, CA: EdgePress. Williams, B. and Van’t Hoft, S (2016). Wicked solutions: A systems approach to complex problems. Available at http://bit.ly/1SVoOH3 Good for more advanced reading: Bamberger, M, Vaessen, J., & Raimondo, E. (eds.) (2016) Dealing with complexity in development evaluation: A practical approach. Thousand Oaks, CA: Sage Publications. Cabrera, D., Colosi, L., & Lobdell, C. (2008) Systems thinking. Evaluation and Program Planning, 31(3), 299-310. Cabrera, D. & Cabrera, L (2015). Systems thinking made simple: New hope for solving wicked problems. Odyssean Publishing. Capra, F & Luisi, PL (2016). The systems view of life: A unifying vision (6th printing). New York: Cambridge University Press. Checkland, P. (1999). Systems thinking, systems practice. New York: John Wiley & Sons, Ltd. Cunliff, E., (2002) Connecting systems thinking to action, The Systems Thinker, 15(2), 6-7. Eoyang, G.H. & Holladay, R.J. (2013) Adaptive action: Leveraging uncertainty in your organization. Stanford: Stanford Business Books. Karach, R, (1997) How to see structure, The Systems Thinker, 8(4), 6-7. Patton M.Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press. Patton, M.Q., McKegg, K., & Wehipeihana, N., eds. (2015). Developmental evaluation exemplars: Principles in practice. New York: Guilford Press. Senge, P. (1990) The fifth discipline: The art and practice of the learning organization. New York: Doubleday. Stroh, DP (2015). Systems thinking for social change. White River Junction, VT: Chelsea Green Publishing. Ulrich, W & Reynolds, M (2010). Critical systems heuristics. In: Reynolds, Martin and Holwell, Sue eds. Systems approaches to managing change: A practical guide. London: Springer, pp. 243–292. von Bertalanffy, Ludwig. (1950). The theory of open systems in physics and biology. Science, 13, 23-29. von Bertalanffy, Ludwig. (1968). General systems theory. New York: George Braziller, Inc. Wolf-Branigin, M. (2013) Using complexity theory for research and evaluation. Oxford: Oxford University Press. Some other resources: International Society for Systems Sciences https://aea365.org/blog/systemic-design-thinking-for-evaluation-of-social-innovations-a-pd-for-intermediate-and-advanced-evaluators-by-jan-noga/ http://www.epreconsulting.com/SETIG%202018%20Principles.pdf https://systemic-design.org/ https://modus.medium.com/what-the-is-systems-design-e005c1e9fef8 https://rsdsymposium.org/ Martin Reynolds Open University Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

I chat with Libby Smith and Tiffany Tovey, two of the three hosts of the Radical Re(imagining) podcast, to chat about what felt radical about the 2021 AEA virtual conference. We'd love to hear from you! What felt radical to you about the conference? Feel free to add your reflections on LinkedIn or Twitter. Be sure to use the #Eval21 hashtag, and tag us in your reflections! Twitter accounts: @RadReImagining @work_with_libby @Tiffany7001 @EvaluLand @danawanzer Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand

Jennifer Bisgard and I talk about project management in evaluation, including how she manages evaluation projects from start to finish and useful tools in her company's practice. Jennifer has a wealth of information in her over 20 years of experience leading evaluation projects, and I learned a lot about how to better do this work from her. Khulisa has a wonderful library of resources including blog posts about QASP, analyzing WhatsApp data, and using rubrics. Every Tuesday they publish #EvalTuesdayTip. Ms Jennifer Bisgard co-founded Khulisa Management Services in 1993. An expert in monitoring and evaluation (M&E) and organisational development, she leads evaluations and capacity building assignments in the Education, Power, and Democracy and Governance sectors. Khulisa has about 100 staff, based in Johannesburg, Nairobi, Lusaka, Mbabane, and Bethesda. Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993. Ms Bisgard has served on boards for the: African Evaluation Association (AfrEA), International Organization for Cooperation in Evaluation (IOCE) and South African Monitoring and Evaluation Association (SAMEA). She is a current board member of BetterEvaluation. Jennifer co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned” published by Sage Publishers in August 2018. The book, edited by Kylie Hutchinson, includes a forward by Michael Quinn Paton. She has a Master’s Degree in Social Change and Development from Johns Hopkins University. She is an American but has been in South Africa for 33 years. She is married to Marc (a dual Dutch-South Africa citizen) and has three boys: Zuko (adopted, now 23); Dylan (18) and Thomas (14). The family is completed by Border Collie, Riley and Africanis, Milly. Music by Matt Ingelson, http://www.mattingelsonmusic.com/Support EvaluLand