Are You Leaving Evaluation Until The End?

I am not an expert on evaluation, but you don’t need to be one to understand that it is a key part of any project that has aims! I touch on it briefly in my post detailing seven steps for a successful science communication project, but I wanted to go into more detail about evaluation techniques and why they are anything but a hindrance to your project. Evaluation could even enhance the activities and provide some extra content for your audience to enjoy!

Why evaluate?

In school, we were constantly being graded on the work we produced and it’s the only way we knew if we’d hit all the important points in an essay, or missed the mark completely. Evaluation is essentially a way we can grade communication projects to find out if they were successful. They say ignorance is bliss, but no one sets out on a communication project without caring whether it will work or not!

Why do I love evaluation so much? Well, I enjoy the validation that it gives my projects and thrive on the opportunity to improve and come back with a bang! Evaluation is great because it doesn’t leave you guessing; it tells you exactly how you can rectify your mistakes without guessing or having to rummage too deep into statistics.

Unfortunately, you can’t jump straight into evaluation; you have to know what you’re evaluating and that means setting your aims and objectives.

Aims and objectives

I’ll admit that it took me a while to get my head around the differences between the two words. If you’re a researcher, you are probably familiar with how to set valuable aims and objectives for your projects, but I’ll try to explain it in a communication context. An aim could be found by asking “what do I want to achieve with this project in relation to the audience?”

Imagine you are a working for a charity that works to improve the lives of autistic people. An example aim for a project could be: To educate college staff on the challenges faced by autistic people in further education.

I happened to come across this brilliant definition of “Aim”:

The aim is about what you hope to do, your overall intention in the project. It signals what and/or where you aspire to be by the end. It’s what you want to know. It is the point of doing the research. An aim is therefore generally broad. It is ambitious, but not beyond possibility.

Patter, Pat Thomson

Objectives outline how you will achieve your aim and must be SMART: specific, measurable, achievable, realistic and time-bound. Following these guidelines is essential because you will need to able to define whether you have achieved your objectives in your evaluation. The easier you make that for yourself the more you will enjoy evaluation. Objectives could almost be a checklist that you can follow to complete your aim, using active language such as “I will measure/ construct/ select…”. This actively defines what you think will make your project successful instead of passively saying “I will appreciate/ understand/ consider…”.

From the example aim above, two objectives could be:

  • Create a webinar for college staff within 6 months.
  • Attract educators from 10 colleges across 3 UK counties.

Aims and objectives focus a project, preventing it from going off track and loosing sight of why and what you’re doing. I used to agonise about getting the right wording and sounding philosophical when setting aims and objectives, but the easier they are to understand, the easier it will be to plan how you will measure their success.

Techniques

Once you’ve solidified your aims and objectives, you can decide the best way to collect data for evaluation and it’s important to remember that words (qualitative data) are just as important as numbers (quantitative data).

My project supervisor, Dr Margarida Sardo, has written a fantastic and informative open-access paper about evaluation methods. The paper is called What Works in the Field? Evaluating Informal Science Events and will give you some great inspiration and practical advice for evaluating a science communication project.

Surveys

Surveys are the first thing that jumps in to most people’s heads when they hear the word evaluation and I’m not surprised! As consumers, we’re bombarded with surveys every day; supermarkets want to know our opinion every other week, there are TV shows based around them and we’re asked to rate our experience with a smiley face button when we leave a shop. Surveys can be a valuable way to collect evaluation data, but it’s hard to get people to fill them out and many participants won’t understand the importance of filling it out properly or completely.

I use them a lot in my job and they do work if you actively encourage people to fill them in and allocate time for the audience to take time with responses! Designing the perfect survey is about asking direct questions that will help you determine success and incorporate a mixture of different types of question. In other words; use your objectives to build your survey if this is an evaluation technique you want to use.

Following the example from above, if you want to attract people from ten colleges in three different UK counties, two of your questions will need to be:

  • Please specify the college you teach at.
  • What UK county are you from?

You will want to make your surveys anonymous to encourage people to share their thoughts and assure respondents that you value their feedback and it will only be used internally for future improvements.

Webinar/ Zoom Evaluation

The increase in online events doesn’t mean you just have to use online surveys for your evaluation. There are now more tools than ever. Old tools that I found out about 5 years ago are becoming common practice for online engagement. For online events your evaluation has a real chance to be part of the activities you do and play an important role in audience participation.

Have a look at Dr Adeayo Sotayo‘s list of apps for adding interactivity to an online project, these are a great way to get real-time feedback and survey responses for your event. You could run the same quiz at the beginning and the end to compare how much audience knowledge has improved. If you want to know where your participants are from you could ask your audience and use Mentimeter to create a word cloud of responses that you can save. Padlet could help you re-imagine a post-it note wall for feedback to specific questions that will help you evaluate your project.

Contribute to change

We all want to get better at science communication and the sci-comm community as a whole strives to continuously improve. You might not think that your small project can make a difference to sci-comm as a whole, but you’re wrong. You can influence how science is communicated in your organisation or wider world if you evaluate properly and make suggestions for best practice based on the evidence you collect.

Make sure that the mistakes made in your project aren’t repeated with another audience by writing a report and making the right people read it, or having a meeting with your managers to highlight your key findings and improve engagement in the future. Academic sci-comm researchers are important, but don’t underestimate the impact that you can have by physically doing projects and testing out sci-comm theories on real audiences.

Evaluation is a powerful tool and without it, we would still be using the deficit model to communicate with non-specific audiences.

What are your attitudes to evaluation? Have they changed over time like mine? Let me know in the comments below or tweet me @jenniefrench95

On the 31st March I’ll be hosting #SciCommChat on Twitter with @SciCommClub. My questions will be all about audience and evaluation so join us then for some healthy discussion and digital networking!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.