More Than Data

Matt Kistler:
There is a reality that we are here to bear good fruit and not just to throw seeds on the ground. We’re hoping to use these tools not to take things away but to grow things and make things better.

Rich:
Welcome to the Essentially Translatable Podcast brought to you by Lutheran Bible Translators. I’m Rich Rudowske, the executive director here at LBT. Today we’re going to have a conversation with Matt Kistler, who’s joined our team recently as the senior measurement and evaluation analyst. And we’re going to talk about the importance of data metrics and evaluating whether your programs are having the impact and outcomes that you are looking for.

And if not What happens next? And so I’m just going to have that conversation with Matt before we dive into that, though, I [00:01:00] wanted to let you know that if you listen to essentially translatable on Google podcasts, that that platform will be going away after April 2nd, they’re going to migrate that content by via RSS feed to YouTube music.

So if you use Google podcasts. When you open it up, you should get a message that tells you you’ll no longer be able to use it after April 2nd and give you two options, learn more or export subscriptions. So if you like a lot of info you can read, but all you really have to do is push export subscriptions.

It’s going to ask you which ones make sure that among whatever else you listen to. Uh, Google podcasts that you choose essentially translatable. It’s going to migrate over there and it will be available to you. I think you want to download the YouTube music app to your Android or iPhone device first before you do all that.

So now if you don’t use Google podcasts and everything I just said doesn’t matter to you, you just keep listening the way you have been. And we hope that that migration works well for you. If you do use Google podcasts. Having said all that, with no further ado, we’re going to jump into the interview with Matt Kistler.[00:02:00] 

All right, we’re welcoming to the studio today Senior Measurement and Evaluation Analyst Matt Kistler. Welcome to the team and to the Essentially Translatable podcast. Hey, thanks for having me. So we always want to introduce our guests to our listeners, listeners to guests. So can you share a little bit about your background?

What is it that you first got involved in ministry and then Bible translation specifically? 

Matt Kistler:
Yeah. See, I have a wife, I have three children, and we live outside of Philadelphia, Pennsylvania. And how did I get into Bible translation? Well, I took a bit of a circuitous route. Growing up, I was thinking I would be an economist.

I really enjoyed watching the West Wing, and I thought I would be an economic advisor to the president. So I had big dreams. God had other plans. And I moved into the international community development space. So after getting a master’s in that, my wife and I moved to Kenya. where we worked with Mennonite Central Committee for three years.

Once we [00:03:00] came back, I had another few jobs, and really the Bible translation part of my story came in about five years ago when I started working at American Bible Society as part of their global scripture impact research team. So that’s where it kind of came through other Christian ministry, but now I’ve been in the Bible translation space for about five years.

you All 

Rich:
Right, and you’ve also served for several years as part of the Illuminations Collective Impact Alliance. Tell us a little bit about your roles there in the Alliance as well. 

Matt Kistler:
Yeah, so after ABS, I moved to Mission Mutual. So Mission Mutual refers to itself as the backbone organization of the Illuminations Alliance.

So the idea is, it’s hard work to make an alliance work, and I think the members of that alliance tried it for a little bit by themselves and then realized, we need somebody every day thinking about, how do we make this alliance work? How [00:04:00] do we, achieve the goals we have for ourselves and encourage everybody, facilitate all of the different day in, day out types of things.

So that’s where Mission Mutual came from, and I joined the team as their field data coordinator. So if it had to do with data, I Basically, I was looking at how can we connect the different partners with the data they need. So, one example. We often had people complaining about a situation where multiple organizations were working with the same people group and bumping into each other in not so friendly ways.

So it can be a real frustration if you’ve been working a number of years translating the gospel with people and then Only years down the road realizing someone else was doing the same work and you never learned about it So that was a real tension point So after discovering this and kind of interacting with this problem, we thought what kind [00:05:00] of systemic data solution Could we to address this.

And so Mission Mutual asked for people to provide solutions. We got proposals and that was kind of something I helped develop and shepherd into existence. And so that now we have a data system that really can support people so that that problem doesn’t exist so much. So that was the kind of work I did as well as program evaluations looking at what new things people were trying in Bible translation and saying, is this working?

Is this not working? Should we be doing more of this? So that was a lot of my role at Mission Mutual. 

Rich:
What’d you find challenging or energizing 

Matt Kistler:
About that work? Yeah, well, there’s plenty to be challenged by. I think one challenge is, The Bible translation, and it could be energizing as well, the whole space is in a state of change, I feel like, kind of from the vantage point I’ve [00:06:00] had.

There’s a lot of new ideas being tried out, a lot of things that people are considering, whether it’s AI or new approaches to translation. The positioning of the local community and their role within translation is drastically changing and being looked at. And that’s an opportunity, but. it’s very easy to see those changes as threats as well for organizations.

And so navigating that change season and, you know, being an advocate for change and learning and growth isn’t always welcome. But I think at the same time, what’s energizing is seeing new things, seeing God at work in new ways. So. I know one thing, you know, the LBT team was recently in Papua New Guinea working with the Evangelical Lutheran Church there.

I was in Papua New Guinea in May seeing work that’s kind of related to what’s being [00:07:00] considered and it’s interesting. It’s just awesome to see God at work, to see the local church take up translation and say, this is ours. We want to own it. You are welcome to come help us and we need your help, but we take responsibility for this and being there and seeing the way communities were responding to that approach and kind of method.

So, translation was, was really exciting. 

Rich:
So in your former role, you’re bringing some of that along with you to Lutheran Bible Translators. So some of the things that you’re going to be talking about with us, with our team members, with our partners, is about metrics. So can you share a little bit about what some of those conversations that have been happening already within Lutheran Bible Translators of metrics and evaluation of those?

Yeah. Well, 

Matt Kistler:
First of all, I’m going to give a shout out to you, Rich. I really appreciated your vision. I, I see that you have a clear idea of how data can help us grow as an organization. And so I’m excited to join the team in an environment where, you know, often evaluation and learning can be [00:08:00] something that’s like eating your vegetables.

You know, we all know we should probably eat broccoli and I know some people that my son loves broccoli. So good for him. But, yeah. A lot of organizations really aren’t excited about seeing, like, are we having the impact we say we are? Because if you find out you’re not, you probably should do something about it.

Right. And those are tough conversations. So, all that to say, What gets me passionate and excited about how I can serve this movement is helping organizations Take the information that is already kind of in the atmosphere around them and learning from it, growing from it, making wise decisions using it, and the easiest application where you can kind of imagine that is Taking a program and saying hey, this is really working.

What’s making that work? What’s making that tick so that we can try it somewhere else? And in the same way saying, we’ve always done it this way, [00:09:00] and maybe it’s time for an update. Like maybe we do need a way to be able to say this isn’t working and, and have those challenging conversations internally, not just based off of our hunch or intuition, which is valuable and completely relevant, but also off of some solid data.

So that’s one easy piece. I think another Kind of way I conceptualize the way I can serve at OBT is, you know, bringing that same kind of measurement Perspective to all of our systems. So how can we track growth in all of our areas in finance in development in marketing and use again the data that’s available To get better at what we do.

I think that’s the the banner mission statement over How I think about my vocation is we can be better as the church in serving God’s people and that’s [00:10:00] not just Pride that’s not to say it’s all us. You know, the Holy Spirit is absolutely a necessary component in that growth But we can’t just assume that we’re as good as we ever get we’re gonna close our eyes and leave it to God You know we can do better And I believe it’s part of stewardship to actively pursue that.

Yeah, I’d have to 

Rich:
Agree as well. I mean in non profit ministry in general and then in Bible translation ministry Yeah, as well. I think it holds true for both that in general we’re trying to solve the biggest problems that the world has We know and we do that out of a passion that we believe God’s called us to Make an impact in these different ways And if that’s the case then paying attention to how things are doing what the interventions are Accomplishing or not accomplishing.

This isn’t just a nice add on or a nice to have. This is, this is important. And there’s people who want to invest in making sure that the world is a better [00:11:00] place in whatever way is available to them. And, and we really have the opportunity to, to use data and metrics to one, ensure that’s happening and to be able to talk to people about.

How we can show them that that’s happening. So talk about how data informs best practices or changes and some of the terminology and concepts that you use as you kind of try to talk through that. Yeah. 

Matt Kistler:
Yeah. I mean, one, one term that is pretty hotly contested is best practice even, you know, like, is there such a thing?

I think it’s. Often what people end up getting into a habit of can just be what we’ve done before. Right. And so habit is just such a powerful, powerful force in how organizations work. And I think there’s Uh, benefit to that, but there can also be a downside, which is you get in a rut of just doing what we’ve always done without continuing to see how could this be better.

I think using data [00:12:00] again, I, I take a really wide, broad stance on data. You know, some people are thinking, well, what do you mean you didn’t have to put numbers on everything? No, to me, stories are really valuable data. And often to me are. I kind of have a special place in my heart for stories. It’s like the little stepchild in the data world, where, you know, often it’s like, Oh, stories, that’s just for like fundraising.

And, you know, we’re just trying to pull on people’s heartstrings. No, actually stories can tell us so much about what’s happening. The numbers often tell us what happened, but the stories tell us why and how. And so to me, that’s like a much richer data flow. So. All that to say, when we capture that kind of data, robust data that gives us a full picture of what’s happening, we can start to say, okay, here’s the context, here were the conditions, and here’s what happened.

And then we can start to [00:13:00] think, is this replicable? None of this is a, is a formula or a silver bullet or guaranteed, but with that information we can make informed decisions and say, okay, here’s Well, we know that this tweak in the system seemed to have resulted in this outcome or impact. And so that’s what we want.

So let’s try it again somewhere else. And so I think that’s the power of evaluation and learning and measurement to strengthen and grow. Yeah, 

Rich:
Ao just as you were talking, I was thinking about in Acts 2, if it wasn’t about story, of setting the context of Peter preaching to the crowds, and their number grew in the thousands that day, that wouldn’t have been at all 

Matt Kistler:
Helpful.

Like, the number of Christians increased by 273%. That’s right. Very unhelpful Bible. 

Rich:
And being able to say what the Bible says. What was the context surrounding it? What was it that their needs were? What was said there? And really, as you’re interpreting the data, [00:14:00] what is it that’s surrounding it? What maybe seems insignificant, but it’s going to inform a better, if not a practice, a better practice in the future.

And so that kind of moves in the direction of input and output versus outcomes. Can you break that down a little bit? The effects of outcomes? 

Matt Kistler:
So, I think, really, the question behind those different definitions is what are we trying to measure? There’s three different things that I think about, like general buckets of measurement.

One is, what did we do? A second is, what changed? And the third is, What transformed? Okay, so we need different words to talk about these different concept of almost like a ripple effect every intervention or program that we implement it’s like throwing a rock in the water and Some of those ripples you can anticipate some of them.

You have no idea what’s gonna happen and so the closest to where the rock lands is what you [00:15:00] can pretty well anticipate and That’s generally what we would call It’s what the project put out. So we’re thinking about the number of things we do. Number of trainings held. Number of attendees to a thing.

Number of chapters completed or probably like verses drafted. The basic work of what we have done is what we call an output. And it’s important to be able to distinguish that because that’s what You would put in your daily planner. This is what I did today. Five of these, two of those, seven of these, and that gives us a good sense of the progress we’ve made toward whatever plan.

But if you ask yourself, why are we doing this work? Outputs rarely answer that question. We’re not investing in a community so that five people can attend a training, right? There’s a deeper why. So the first why is points us to this next level, what changed, [00:16:00] and I call that an outcome. What came out of our output?

What change happened within a person, within a system, within a institution? And so outcomes are the next ring out. We still generally have some control over those outcomes. But not entire control. So, let’s take a literacy program, for example. If we have ten people who went through a literacy program, we can feel relatively confident that the literacy rate within that community will go up.

Right? That maybe even by ten or, let’s say, by seven people. You know, that’s something that we can measure toward that is a little bit deeper than we just attended a class. Now we’re measuring what changed somebody’s ability to read. Okay? So that’s your outcome. Now the next ring of the ripple of water kind of [00:17:00] spreading out is impact.

And just like before, as you get farther out, you have less control. Impact, usually, people think about a longer time frame, like three to five years. And it depends, but you’re thinking in that year’s time span, and often there are a lot of things that have to go right for the impact you hoped for to occur, right?

So in the literacy example, it might be the percentage of people within a community who are engaging with Scripture. In an example. We do the literacy classes. That’s the output. People become literate. That’s the outcome. And then the impact is they can engage with God’s Word. We can’t make them engage. We can’t control whether they take their literacy skills and actually utilize it in the way we’re hoping for.

We can encourage them and there’s [00:18:00] different, you know, things that we have in mind so that that is maximized. But, uh, That’s not something we can make happen and that’s where impact is and it’s often Not just kind of the immediate change but a change caused by a change that ripple effect that is looking more like a Transformation looking more like a long term change that we and the community desire to see so So that’s your kind of basic primer on three levels of measurement.

Like means and ends? Yeah. I think that helps. Like, that’s one way to think it. You, you know, your output generally will be measuring more means. Your outcomes is kind of like moving toward the end and your impact might be your final end. 

Rich:
Which you don’t have as much control over. And impacts also can be things that you didn’t know would happen or unintended.

So the literacy example, again, is. People engage with scripture, but people also fill out their [00:19:00] taxes or whatever because they they finally have the ability to to do it Or yeah, not the best example, but you know, it’s yeah. Yeah. No, I 

Matt Kistler:
Mean it’s one anyways, um We can always hope for increased tax revenue give to caesar with caesar.

There we 

Rich:
Go. All right, so ultimately then, you know evaluation As you’ve mentioned, it is a tool that can be used to say, okay, are we having the impact that we think we are having? And if not, how do we adjust inputs, outputs, you know, the outcomes even toward that. Talk about, you know, why that’s important and especially in the, in the type of work that we’re doing.

Yeah, I 

Matt Kistler:
Was just kind of thinking about writing a little thing up about this and I was looking at Luke 13, six through nine, which is a pretty grim example, but I think kind of makes the point where Jesus gives a little mini story about an owner of a fig tree. And for three years he says, I’ve looked here and there’s been no fruit, right?

It’s time to cut it down. [00:20:00] Yeah. And then the gardener’s like, well, just chill and give me a year. We’re going to treat it real nice, put some fertilizer down and then we’ll see. And I think to me. The value of evaluation is kind of in that same process where if something isn’t bearing fruit, it’s using up resources, right?

And you know, I think people can see that as like pretty harsh or like tough love maybe. But the reality is there are limited resources. There are many places that we could be investing in a lot of ways that we could invest in communities and in this work. And so given those restraints, we need to prioritize.

And what that teaches us is we have limited resources to invest. And even in the passage it talks about, it’s just eating up soil. And so if we have those limited resources, we need to be able to prioritize and dig in where the [00:21:00] harvest is ripe. So I think while that can feel daunting sometimes, there is.

a reality that we are here to bear good fruit and not just to throw seeds on the ground, right? And, you know, sometimes I think about the parable of the sower too. And, you know, like from an evaluation perspective, you’re like, maybe next time the sower could like, you know, focus on the good ground a little bit.

But I think We’re hoping to use these tools, not to take things away, but to grow things and make things better. There are ingredients to be considered that will grow the impact. And maybe that’s even where the fertilizer analogy, you know, do we need more phosphorus or nitrogen or potassium, lime? You know, what do we need to put on the soil to see more fruitfulness?

And I think that’s kind of the paradigm I have. Not so much like, hey, let’s cut down this tree, but You know, let’s [00:22:00] see this bear as much fruit as 

possible. 

Rich:
I love that analogy of stewardship in the garden and pruning. It’s a labor of love, but it’s also interpreting the elements. This is working. This is not working.

The other side of it too, though, is now that parable were different and the, the owner said to the manager, like, well, how much fruit are we getting out of here? And what kind of fruit is it? And the guy attending it was like, well, I mean, there’s a lot of fruit going on there and it’s really good. It’s like, okay.

But like, what kind is it and how much and, and what are we talking about specifically? Eventually that manager is going to say, well, I’m cutting this off because I can’t get good information about what’s actually happening there. So that’s the other side. Right. Also of evaluation. It’s not just the cutting, but also to.

Give reality and clear a picture to like here is the impact that’s actually happening, 

Matt Kistler:
Right? Right and describing so that we can invest more or Respond to what’s happening. We need to be able to quantify Observe document and honestly, that’s a lot of what I do and was [00:23:00] doing in my last position was just documenting What are you doing?

It’s very easy to just go about your day and not realize that you’re doing something new. Like, you are learning every day. And by writing down those things, you can really pass on a gift to the next person who is working in the same way. 

Rich:
So Dr. Henry Cloud has written many books, and one of those is a book called Necessary Endings, which we’ve kind of talked about just in passing in our conversations around the office.

But he also addressed the CEOs of the Bible translation organizations and the Impact Alliance of Illuminations. And he said, Most ministries miss the boat on evaluation. This lack of discipline means they can’t correct course and serve more effectively. Can you kind of share your reflections on that?

Matt Kistler:
Yeah, preach it. I mean, yeah, I was there. I was clapping, maybe just quietly because, you know, it was awkward. But I think a lot, and I think we can zoom out from Bible translation [00:24:00] specifically. This isn’t just like a Bible translation problem. This is, I’d say, a helping people problem. When organizations, especially, are built around a passion for helping people.

First of all, there’s limited resources again, and so taking the time and the energy to document and learn and respond feels like an extravagance. It feels like we don’t have that kind of time. We don’t kind of have that kind of resources, so it doesn’t happen. And then the second piece is people’s passion is what fuels these organizations.

And so you often will have people who are entirely convinced about the goodness of what they want to do. That’s how they got started. That’s how they took the entrepreneurial jump to get where they are. But the temptation there is to. Um, shut out or kind of isolate yourself from voices that are saying, is that really what’s happening?

And so that could be [00:25:00] just as likely in helping the homeless or drilling wells or, you know, in any context of helping others, that can be a real institutional temptation is to just say, we know what we’re doing is good because it’s clearly good. And so that can make it hard to. Adopt evaluation as a discipline, but I think there’s so much good fruit to bear there Like you’re just missing such an opportunity and I think that’s what Dr.

Cloud was kind of pointing at is there are hard decisions that need to be made there And there’s also like joyful decisions to be made when something works Well, you want to like shout it to the world But if you’re not just like you were saying if the manager doesn’t know that it was the best Harvest ever, you know, you lose the celebration of it and you can’t say, Hey, everybody, like, look at this.

This is awesome. Let’s do this again. And [00:26:00] so there’s just a, a real opportunity. And I think it’s, it’s a place where almost every nonprofit can grow to, to lean in and allow yourself to kind of open up and say, What’s under the hood? What’s happening? And are we having the effect that we were built for?

Rich:
It’s awesome. Yeah. I’m excited, obviously, also about, uh, you coming into the, the role here. And yeah, even in your first week here in the office, first couple of weeks on the job, already sitting on the podcast and trying to explain what you do. It won’t be the last time, I’m sure, but really excited about that.

How can we be praying for you as you start a new, uh, with our organization? 

Matt Kistler:
Yeah, I appreciate that. I think one thing is you can be praying that I model humble listening. That’s what I want. And I think it’s just challenging. There’s so much opportunity for fear to be part of the narrative around evaluation.

Like, you’re here to [00:27:00] tell us we’re wrong, to tell us we failed, and to take away our money. You know, and make us change. So, I think I want to, from the beginning, give off a spirit of humility, of listening, and a partnership. Right? I have no interest in being the auditor. I have no interest in being everyone’s, like, assistant.

Professor like slapping on the wrist or like giving out red marks, you know, that’s not the goal here So I think just being able to communicate that well to our partners and to our staff so that’s what they hear when I talk because even You know, sometimes people that you can hear, but you might not understand and, and doing that from a distance, you know, it’s a lot easier when you’re doing it over a meal.

And I think that would be maybe the other prayer thing is, you know, I’m not only not in the field, I’m, you know, I live in Philadelphia, so I’m not overseas where a lot of this work is happening, but I’m [00:28:00] not in Missouri, so I’m not in the office with people. And so if you could just be praying that. I can develop healthy and really, you know, fruitful relationships with the different staff that I have to interact with.

Remote work can be hard, and especially in something so relational and kind of vulnerable as evaluation, I want to kind of lean into that and, and build trust. Well, 

Rich:
We’re truly thankful that you’re on board the team, but also that you took time to be interviewed here today and we will definitely be praying for you.

So thank you, Matt. 

Matt Kistler:
Yeah, it’s my pleasure. Thanks for having me.

Rich:
All right. Just reflecting on that conversation with Matt and I’m really excited about the upcoming season here at Lutheran Bible Translators as we continue to become more of a learning organization so that we can serve the beneficiaries [00:29:00] of our work more effectively, our partners, the language communities, the churches, and really understand how it is that we’re working with them.

What we are accomplishing and whether that’s helping or not helping. So that’s, that’s so exciting to me to be entering that phase of our, our way of being as an organization and also then to, uh, be able to invite prayer and investment in the ministry and based on an improved understanding of how, what we’re doing.

is making an impact what those outcomes look like and, and inviting greater participation in God’s mission through that clarity of vision. So it’s really great to talk with Matt, looking forward to how this all unfolds. And thanks for spending time with us listening in today. Thank you for listening to the Essentially Translatable podcast brought to you by Lutheran Bible translators.

You can find past episodes of the podcast at lbt.org/podcast, or subscribe on audible apple podcast, Spotify, or wherever you listen to podcasts. Follow Lutheran [00:30:00] Bible Translators social media channels on Facebook, Instagram, or Twitter, or go to lbt.org to find out how you can get involved in the Bible translation movement and put God’s Word in their hands.

The Essentially Translatable Podcast is edited and produced by Audrey Seider. Artwork designed by Sarah Rudowske. Music written and performed by Rob Veith. I’m Rich Rudowske, so long for now.