An experiment demonstrates that people can no longer reliably tell the difference between real and AI B&W landscape photos
Like many folks with an interest in photography and technology, I have been playing with generative AI tools. A key question that I have been exploring is when images from these tools will be able to consistently fool the average online viewer. I don’t believe that point has come for many types of photographs but I do believe that we have now passed that point in the narrow category of black and white landscape photography. Please read on to see a simple experiment that I performed and consider subscribing if you are interested in seeing similar content in the future.
The experiment was executed as follows:
I chose six black and white images from my personal photography collection. I took all of these in British Columbia or Alberta.
I then created a number of black and white images in Midjourney using V6 and relatively simple prompts. From that collection, I chose six that I felt were the most convincing. An example of a prompt used for one of the images is:
“Black and White photograph, detailed texture, cloudy day, snow covered trees in the pacific northwest --ar 16:9 --c 10 --s 225 --v 6.0”
Although the selected Midjourney images appeared to all be greyscale, I processed them in Photoshop to save them as greyscale images to be sure. Sometimes when you ask Midjourney for a black and white photo, it will create an image that has a sepia tone or even has hints of richer colors in parts of the image.
I then uploaded all twelve photos to a local photo group on Facebook with a message that included these instructions:
”This is a collection of 12 photos. Six of these are real photos that I have taken in our corner of the world (I can disclose the exact locations after voting) and I used AI to create the other six. They are presented in a random order. Can you tell which are which? If you care to guess, vote with a thumbs up for each picture you think is real and a laughing face for each picture you think is AI generated…”This was the first and only time I have run this type of experiment.
Over a period of slightly longer than eight hours, there were approximately 350 reactions for each photo and I made a record of those reactions at that point. The post was removed before additional data could be collected as it was deemed to not be consistent with the group guidelines.
There are a few obvious limitations of this experiment. The most significant limitation is that the votes were not independent since each respondent could view the current tally of thumbs up and laughing face reactions before placing their own vote. The second limitation is that some people may not have read the instructions and may have just been clicking a reaction (in particular the thumbs up,) after they saw the photo come across their feed. The third limitation is that one of the images (image 11) was recognized by a number of respondents as a fairly notable waterfall in a provincial park and they shared that observation in the comments. Obviously that skewed the results for that image significantly.
Results:
Overall, the crowd correctly identified whether six of the twelve images were AI or real. Mistakes were made on three of the six AI images (where the crowd thought they were real,) and three of the six real images (where the crowd thought they were AI.) Given that outcome, a crude statistical interpretation may be that it would have been just as effective to flip a coin.
I was struck at the balance of votes on each individual image. I had expected that the first few votes on each image would carry significant weight with following respondents going along with the crowd. Instead, the the results were mixed for every image. If we ignore image 11 (where the crowd was able to “dox” the image as a location that many respondents were familiar with,) the image for which the crowd had the strongest conviction was the fourth image with 77% of the respondents believing it was AI. Interestingly, this was incorrect and the photo was actually real.
See the full results below. For fun, I suggest scrolling slowly and guessing along the way.
Thanks for reading,
Cheers,
Steve