Let’s Get Real 10 Years On: How to evaluate online success

February 2021 to September 2021

In the decade since, the report from our first action research programme has been downloaded nearly 24,000 times, showing that this is still a big question that the sector is struggling to answer. Since January 2020 the cultural sector has been thrown in at the digital deep end and the struggle to effectively evaluate online success is more important now more than ever. LGR10YO aimed to help participants answer that question and meet the challenges of evaluating online success in the current climate. 

Across the 46 participating organisations, small scale interventions were conducted within their existing and planned digital activities with the purpose of interrogating how they measured their success and what success means to them. For example, some organisations experimented with optimising web content to drive more visits and increase user engagement, thinking about how data and analytics could help them to understand their success, while others explored challenges including the use of online surveys and how visitors interact with in-gallery digital content. This project brought together a community of people and organisations with a shared sense of purpose to foster open, honest and collaborative learning between participants as a cohort of peers, during one of the most turbulent times our sector has faced. 

We developed and ran LGR10YO in partnership with Chris Unitt, Founder of analytics and user research agency One Further. Chris supported our content design, webinar programme and the participants’ self-directed learning. Having Chris’ expertise, knowledge and experience of supporting the cultural sector with data and analytics was incredibly important for the project.

46 arts and heritage organisations participated in this collaboratively-funded project, each contributing in the region of £350 to £750 (depending on size). As part of our desire to encourage smaller organisations to attend, the subsidised places went to Black2Nature, Marlborough Productions and Creative Bexley.

The LGR10YO programme took place over eight months from February to September 2021, in the shadow of the ongoing Covid-19 pandemic, so was shaped to meet the challenge of online delivery. We adapted our usual four full-day, in-person workshops to become a series of whole-group live webinars where we shared theory and content. We combined this with discursive online workshops which allowed for participatory activities, reflections and discussions in smaller groups. We also scheduled regular expert-led ‘troubleshooting’ and mentoring sessions, alongside a range of informal social opportunities. This was all held together in a closed online community space hosted on the Mighty Networks platform, where we posted resources and content and encouraged collaboration and discussion. The price of participation was significantly lowered from previous years to reflect the online delivery. 

The number of participating organisations was a challenge – going from 20-30 in previous LGR cohorts, so 40-60 people, up to 46 organisations and around 90 participants in LGR10YO. Whilst this was a deliberate choice on our part to make the most of the potential virtual sessions provided (gathering people in rooms, in person, naturally restricts numbers) for creating a rich and rewarding community of practice, it led to difficulties in coordinating and maintaining the engagement of that many people. 

Unfortunately, a handful of organisations were unable to stick with the project all the way through and others, whilst they stayed the course, struggled to participate as much as they would have liked. For some, the challenges of online working – working from home and home life, combined with increased workloads and the stresses of the pandemic, became too much of a hurdle. Whilst Mighty Networks worked as a community hub for many of the LGR participants, it didn’t work for everyone. For the Culture24 team trying to build the personal, human connections online, as we would normally do in-person through LGR, was a challenge.

Having said all of that, the online delivery and lowered cost did enable some organisations to participate that wouldn’t have been able to otherwise and there were many connections made across the group. The online space also allowed more flexibility for participants to consume and revisit webinar and workshop content, with an added layer of further reading in their own time, which was never a part of our in-person workshops.

For more detail on our process, see the ‘Structure’ section below.

At the start of the project, we posed the question: how can we evaluate online success?

Unpacking this question, we needed to think about what success might look like for different organisations and what tools and techniques were available to measure that success. Every organisation has different motivations behind their activities but many lack a clear view of what really constitutes success. Combined with rigid reporting on data and the challenge of using analytics tools geared more towards measuring commercial success than social impact, it can be difficult for cultural organisations to truly understand and measure their success. This lead us to break the core question down: 

• What does success mean to you, your organisation and your audiences?
• What tools are available to collect the evidence of your success and how can we both interpret the data and act on it?
• How can we ensure that our digital activities map to our core values, missions, goals and objectives?
• How can we understand success beyond the numbers and how can we think more deeply about values-led digital practice?  

One of the most important aspects of our Let’s Get Real projects is for those taking part to choose a hunch to test or interrogate, something that has come up through their work which they would like to explore further and evidence. Among our own hunches – the areas that Culture24 wanted to better understand – was how we used Google Analytics to measure the success of our own activities, specifically focusing on our Museum Crush website. But at the core of the project, we were also testing and experimenting with how we could deliver a programme built on collaboration and action research entirely virtually. This was the first time that Culture24 had brought the Let’s Get Real model online, delivering the programme mostly through Zoom and on our community hub Mighty Networks. This project offered us an opportunity to test and iterate how we deliver an action-research programme virtually, whilst attempting to maintain the sense of community that we usually build through Let’s Get Real. As mentioned above, it was challenging!

As with all of our Let’s Get Real projects, LGR10YO looked outward for best practice ideas and approaches, and related these to the specific context of cultural organisations’ work. We took a ground-up approach to understanding this relationship, defined by the thinking and practice of our participating organisations. This ensured our understanding was always rooted in the specific contexts and nature of their work.

LGR10YO was supported by a range of fantastic guest speakers from within and beyond the cultural sector. This talented group of individuals and practitioners brought in and shared a variety of perspectives and experiences for the cohort. They were: 

Kati Price, Head of Digital at the V&A and Culture24 trustee. Kati is passionate about user-centred design, content and the digital realm and has worked on a number of award winning digital products and services. Kati joined Daffyd James (see below) for an in-conversation during Webinar 4, where they spoke about success, what it looks like for different organisations, their research into the structure of digital teams and how measurement can be used as a tool for tracking progress. 

Daffyd James, Head of Digital at Amgueddfa Cymru (National Museum Wales) and Chair of the Museums Computer Group. Dafydd’s focus is digital transformation at Amgueddfa Cymru and embedding user-focussed services across the museum as well as working on an internal digital skills framework and developing a more distributed model for digital content and audience development.

Cliff Manning, Research and Development Director at Parent Zone and instigator of More Than Robots. Cliff is interested in how tech impacts real people’s lives through education, government, art and science. He contributed to Webinar 7, alongside Katie Moffatt, which focussed on children, young people, families, creativity and qualitative evaluation. Cliff spoke to the latest thinking and evidence on the ways children, young people and families are using digital tools and channels.

Katie Moffat, Head of Sector Engagement at Substrakt (at the time of writing, when she joined us on LGR10YO, she was Head of Digital at The Audience Agency). Katie is an experienced digital strategist, consultant, and trainer with specialist expertise in audience development, digital marketing, social media engagement, digital analytics and evaluation, user behaviour and community building and management. In Webinar 7, Katie shared insights on the creative and qualitative side of evaluating digital engagement. 

Louise Cohen, Content Strategist at One Further. Louise’s digital content strategy, whilst working at the Royal Academy, was one of the first in the field, and won a GLAMi at Museums and the Web in Cleveland in 2017. In Webinar 8, Louise spoke about her time at the RA, discussing the realities of using dashboards for data and reporting to help an organisation make more informed decisions about their digital activity. 

Zak Mensah, Co-CEO of Birmingham Museums Trust and Culture24 trustee. Before his role at Birmingham Museums Trust, Zak was Head of Transformation for the Culture & Creative Industries Service at Bristol City Council. Zak’s background is in staff development and digital and he has been on the web since the late 90s. Zak joined us for Webinar 9, where we discussed the new normal of hybrid working and how the sector is slowly adapting.

LGR10YO encouraged practical action research. We supported participants to experiment in the context of their everyday work, testing out hunches developed through our collaborative discussions. The Culture24 team and partners supported participants to conceive, plan, track and analyse experiments using agile-based methodologies with a focus on clear objectives, audience involvement, a willingness to create and iterate and a culture of learning from failures. All of the experiments had the following characteristics: 

• They sought to answer a question 
• They involved a practical action 
• They were simple and small-scale 
• They used existing resources, content, channels and technologies 
• They were time-bound 
• They had an idea of what success would look like
• And, most importantly for the theme, they had feedback or tracking mechanisms built in

Where possible, each organisation had two members of staff participating to help form small working groups within their organisations. Many involved other members of staff within their experiments as well, nominating colleagues from other roles or departments to work and consult with. In this way the experiments sought to uncover organisational opportunities and challenges as well as personal ones. This was important in the context of the pandemic, as many teams were working entirely virtually, with some taking a hybrid approach.  

LGR10YO aimed to foster a shared sense of purpose amongst the project community. We wanted to encourage open, honest and collaborative learning between participants as a cohort of peers. This was more of a challenge in LGR10YO than in previous LGRs as this was the first time that the programme was delivered entirely virtually. We used the online community space as a cohort hub where they could share ideas and communicate with one-another. It also worked as a space for Culture24 to share information on events, recordings of webinars and interesting content related to the project theme. 

We encouraged the LGR10YO group to share perspectives, ways of working, opportunities and challenges – it is always important in our Let’s Get Real projects that participants feel able and supported to share failures and concerns as well as their successes. This was supported via structured and unstructured online discussions on platforms such as Zoom, Whereby, and Gather.Town as well as on Mighty Networks. 

This was the third LGR project to invite each participating organisation to nominate two individuals to represent them on the project. Not only did this ensure that participants had a wider peer group to draw support from, it also sought to promote more collaborative working within each participating organisation and to increase the likelihood of better effecting and embedding change. 

The LGR10YO project followed a structured process, delivered entirely online, involving: 

• Ten  1 hour webinars, with participatory activity and opportunity for discussion, expert presentations and individual reflection on a number of themes from framing digital to the basics of Google Analytics. These took place in February, early March, April, June, August and September
• Five 2 hour workshops, featuring group activities and discussion, supporting them to plan and deliver their experiments as well as round off and wrap up the programme. Due to the size of the cohort, the group was divided and each workshop was delivered two or three times over the period of a few days. These took place in March, April, May, July and September       
• Regular self-led research periods between workshops and webinars when participants were supported to run experiments to test out their thinking and ideas in their own contexts
• Three troubleshooting sessions on specific topics related to participant experiments.
• A series of four mentoring / support sessions for each participating organisation, run by Culture24 with LGR10YO partners, to help guide and support participants during research periods
• Ongoing online collaboration between all project participants via the Mighty Networks, an online community platform where we hosted course material, encourage discussion and conversation and gave a space for participants to have individual chats away from the group.
• Eight social drop-ins, where we experimented with a number of different platforms including Zoom, Gather.Town, Whereby and Wonder.

The project followed the fundamentals of design thinking which are embedded into all the workshops, webinars, mentoring and the actual ‘doing’ of the experiments. Each step is iteratively reflected upon so that learning is embedded throughout:

• Understanding – pinning down the key issues/ themes in the project
• Define – how project issues relate to each participant according to own personal and organisational context
• Ideation – generate project ideas in a creative and non-restrictive way
• Planning – shaping experiments based on the best fit between the define and ideation stage
• Review – where everyone is at and focus on approaches to unblock any issues/problems they are having to move forwards
• Test – everyone runs their experiments with possible iterations depending on time and scope
• Reflect – everyone reflects on their work and key learning 

This Let’s Get Real project was challenging and fascinating. As Chris describes in his reflective piece (see ‘Reflections’) framing the project before we began, we suspected that the findings from the first Let’s Get Real, ten years ago, still stand today and that despite a decade having passed, cultural organisations are still grappling with the challenge of evaluating success. 

The core difference now is that we understand the importance of measuring digital success across the whole organisation, not just in relation to our audience-facing outputs. That challenge is even more pertinent today than it was ten years ago, in the midst of a global pandemic when organisations were forced to engage their audiences solely online. Whilst this created an opportunity for greater audience engagement through digital platforms, it highlighted the inadequacies of remote access to our digital systems, the size of the digital skills gap in the sector and the fact that digital literacy is not spread evenly across organisations. 

For smaller organisations, where there is less capacity for everything, let alone digital activities, the pandemic gave the impetus to experiment and develop their digital literacies but for many this quickly became overwhelming. 

Those scenarios were all reflected in the people and organisations that took part in LGR10YO and led us to these overarching conclusions:

• Digital confidence and digital literacy are not as pervasive as they should be across organisations and are unequally spread amongst teams
• For those that do have digital confidence and literacy, the problems and challenges are always evolving – the job is never done
• As individuals in smaller organisations develop their digital literacies they can actually become more overwhelmed – the more they learn, the more skills they realise there are to learn and the more they can see they need to do
• Our concept of measuring digital success has changed – it is now more clearly about how digital impacts on every aspect of an organisation and its activities, not limited to audience engagement 
• The pandemic opened an opportunity for audience engagement, but highlighted the inadequacies of hardware provision, remote access to centralised digital systems and general capacity pressures.

Whilst each experiment was slightly different, general themes did begin to emerge. Challenges around capacity, understanding the platforms and understanding success were issues faced by all participating organisations. Morrigan Mason, Museums Development Manager at Amgueddfeydd Sir Gâr – Carmarthenshire Museums, puts it perfectly when she lists her top strategic insights from the project: 

• Defining digital leadership and understanding where responsibility for this sits within a team
• The cross-cutting nature of digital activity across a service and how this is reflected in future structures and job roles
• Recognising digital needs to be given appropriate attention in business planning – it takes time to develop, resources to support development, and commitment from everyone in the team.

Our LGR10YO participants’ journeys, tracked through their individual organisational experiments, represent the wider sector in microcosm. Their findings and the challenges they encountered in their experiments will have resonance for many within the sector. They were not only struggling with the challenges of measuring and understanding success, but also with their shift in working practice, their reliance on online audiences and the general uncertainty of the times. We asked the participants for their reflections and top tips from the 8 months they spent over the course of the project. We have shared a selection of those below:

Analytics and data can be challenging even for the people that spend their careers focusing on it. For the majority of us, evaluating the success of our digital activities makes up a very small part of our jobs, so we need to take the pressure off a little, to recognise that this is a learning curve. If Let’s Get Real as a whole tells us anything, it’s that learning together, learning from others and learning by doing is one of the best ways to make meaningful progress. As you read through a selection of our participants’ and partners’ reflections, responses and case studies we hope that you appreciate, as we do, their openness and willingness to honestly reflect on their experiences. As with every other Let’s Get Real, it was a pleasure to gather and work with such a wonderful bunch of people at a time of such momentous change, pressure and uncertainty for the sector. Thank you all. 

Michael Hardy, Digital Engagement Curator, Barnsley Museums: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-barnsley-museums/

Alison Bowyer, Executive Director, and Jocelyn Murdoch, Content and Communications Manager, Kids in Museums: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-kids-in-museums/

Sian Woodward, Collections Manager, and Zoe Hendon, Museum of Domestic Design & Architecture: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-museum-of-domestic-design-and-architecture/

Jack Stanley, Marketing Campaigns Manager, Museum of London: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-museum-of-london/

Russell Dornan, Digital Media Content, and David Weinczok, Digital Media Content Producer Manager, National Museums Scotland: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-national-museums-scotland/

Sarah-Jane Harknett, Visitor Engagement Project Coordinator, and Richard White, Marketing & Communications Coordinator, University of Cambridge Museums: https://digipathways.co.uk/lets-get-real-10-years-on-case-study-university-of-cambridge-museums/