What more can we do to protect the dignity and identity of the children we work with, given the increased connectedness of the world we now live in? How can we do this while ensuring that we continue to spend as efficiently as possible and that as much of your donation goes towards providing school meals? How do we balance achieving this with the safety and well-being of our team who sacrifice their time, relationships and health to travel far and beyond to capture the essence of our work through photography and videography?
These are some of the questions that frequently cause robust debates within our team here at Charity Right. So, this Ramadan, we decided to conduct a series of tests to see if we can get closer to answering these questions.
First, a little about us...
We are Charity Right – a British charity dedicated to providing school meals to children living in some of the most vulnerable communities around the world. Our aim is simple, keep kids in school. A child will stay in school if they can receive a meal there. The longer the child stays in school, the more chance they have of obtaining an education and breaking themselves out of the generational cycle of poverty they’re born into.
What We Did: The A/B Test
We ran a display advertising campaign on Reddit, promoting each of the countries we work in. For each of the countries, we used key messaging and creatives related to the challenges faced by the communities living there. The A/B test was between two photographs used for each location while using the same key messaging. One group of photographs was captured by our team who travelled to each location; let’s call these the team photos. The other photograph group was generated using Midjourney, an AI tool that can be used to generate images via text prompts; we'll call these the AI photos.
Why We Carried Out the Test
Our aim was to assess the differences in engagement, traffic and lead generation between the two types of photos. We wanted to see which of the two types of photographs would be noticed and clicked on the most by Reddit users.
What Factors We Took into Consideration for the Test
Reddit is the best: We considered the different advertising platforms for visual content such as Google, Facebook and Instagram, and decided to go with Reddit. We felt the audience there was right for the test as we would get the most learning from it.
Faking it to making it: We considered the potential for there to be a perception amongst some that this was somehow fraudulent or fakery as the team photos contained the actual children we work with.
We decided to press ahead with the test on the basis that we do sometimes use illustrations or stock photographs of children to reflect the nature of the kind of communities we work with, and wanted to see how the AI photos would compare.
To watermark or not to watermark: We decided if we were to yield the clearest results, we should not make it too obvious that the test image was an AI photo. Therefore, we did not use a watermark or any identifier on the AI photo type.
Given Reddit is a more discerning audience we deliberately left some mistakes in the AI photos to see if this would affect the results.
Consistency is key: We wanted to ensure there was a consistent visual tone and language between all of the AI photos. This is particularly challenging to achieve with our team photos as it requires consistent environmental conditions and lighting in each location, leaving much to be done with the photographs post-production.
Don’t smile please: We wanted to convey emotions in the AI photos to align with the key messaging used in the advertising
Again, this is challenging to achieve with our team photos as children cannot switch between positive and negative emotions at the request of a photographer!
Our Discovery
Like any new technology, the adoption and attitude towards AI, in general, is polarised between those that embrace it and those that perceive it negatively. We’re in the former camp and see a future filled with increased usage of AI across a number of cases.
What we discovered during this test is that the public perception is more nuanced.
Our AI photos triggered a fascinating discussion on Reddit which was an interesting discussion on the merits and risks of charities using AI-generated images to promote their work. We decided not to take part in the discussion as we didn’t want to detract from the debate, but we learned much from it.
You can see the discussion on Reddit by going to the following Reddit threads:
Thread 1 – Click here
Thread 2 – Click here
Thread 3 – Click here
The Results
Now on to the techy bit...
After completing the campaign, we analysed the data and discovered some intriguing findings:
In total, Test A (the AI-generated ad creative) received 486,901 impressions and 723 clicks, resulting in a CTR of 0.21%. On the other hand, Test B (the original photo ad creative) received 342,247 impressions and 815 clicks, resulting in a CTR of 0.24%.
After analysing the results, it appears that Test B, with the Charity Right Photo Ad Creative, was the most successful. This campaign received more clicks, resulting in a higher CTR than Test A, with the AI Ad Creative. It should be noted, however, that the eCPM and CPC of Test A were slightly higher than those of Test B.
Key Takeaways
Visual content is king: The results of our A/B test underscore the importance of visual content in advertising, as well as the potential benefits of incorporating AI-generated images in marketing campaigns. Especially when compared to other forms of visual content such as stock photographs and illustrations.
Emotionally connected: Many people commented in the debate about the emotions conveyed in the AI photos. The reaction was mixed in that some felt negatively about the use of AI photos while others shared views that were more open to its use. However, both groups of commenters implied feelings of connection with the emotion conveyed in the AI photos. It was only their reaction that was different.
Not obvious enough: A minority of people did spot the mistakes we deliberately left in the AI photos. That said, many commenters did not and as a result suspected the ads as fraudulent because they were expecting to see a disclaimer or watermark included in them.
A/B testing helps: The experiment demonstrates that A/B testing is a valuable method for charities to optimise our advertising strategies and ensure we’re getting the best value from our investment. That said, although AI-generated visuals can be eye-catching, in this specific test, the team photos outperformed the AI photos in terms of engagement and clicks.
The Future
Back to the questions that underlie this test. Do we have answers? Yes and no.
We know AI is not currently the panacea as some would have us believe. It will take time for the technology to mature and for people to trust it. Having said that, it’s important for us at Charity Right to be at the forefront of its adoption so long as it continues to help us further our work. We have built a solid foundation on digital to ensure we are positioned to make the most of new and emerging technologies such as AI, and we will continue to leverage this where possible.
There are already some use cases in that AI can help us achieve our goals while protecting our beneficiaries and staff, and we will continue to test its use in other areas where beneficial. In those future tests, we will incorporate the factors we considered this time around but configured in slightly different ways while incorporating the learning from our key takeaways to see how that affects the results.
Conclusion
The A/B test we carried out on Reddit provided valuable insights into the effectiveness of our AI vs team photographs. The results demonstrated that while AI-generated images attracted attention, our team photographs still had a significant impact on audience engagement and clicks.
We thoroughly enjoyed the debate this has generated both inside and outside of Charity Right. We will continue to test, learn and hope that the wider charity sector considers doing so too.
Share
Comments (1)
Please Sign In to leave a comment
-
Doc Pop
May 26, 2023
Thank you for answering those questions. I didn't know there was an A/B test going on here. It's interesting to know the Team Photos slightly outperformed the AI Photos in the test in terms of click-throughs. Was there also any difference in terms of donations received? Or did the two campaigns generate about the same much in terms of donations made?
