Book a Call
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
ARTICLE #32
Table of contents
Evaluating creative work with AI: What are the possible risks?
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
Written by:
7 min read
Updated on: May 27, 2024
Toni Hukkanen
Head of Design
Creative Direction, Brand Direction
Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.
AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking.
Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI.
AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.
Susceptibility to user bias
AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.
For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.
Inaccurate and unreliable algorithms
Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors.
Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones.
Absence of real-world context and insight
AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.
One example of this limitation is evident in the interpretation of humour and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works.
Deficiency in Creativity and Original Thought
According to research, creativity has two components, novelty and usefulness. Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality.
Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.
Lack of emotional insight and empathy
AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.
Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.
Undermining human expertise and intuition
Excessive use of AI for creative critique could devalue the importance of human expertise, intuition, and judgment in the creative process.
Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.
Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.
Ethical bias and considerations
AI chatbots are trained on large datasets that may contain inherent biases, such as gender, racial, and cultural biases. AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.
Creative work has the potential to influence societal perceptions, attitudes, and behaviours. AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.
Failure to understand strategic objectives
The creative work of an agency for a brand is typically informed by a set of strategic objectives, target audiences, and market positioning. These factors play a crucial role in shaping the direction and content of creative campaigns.
But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.
Challenges in grasping brand voice
Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.
AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.
Final Thoughts
While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.
ARTICLE #32
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
ARTICLE #32
Table of contents
Evaluating creative work with AI: What are the possible risks?
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
Written by:
7 min read
Updated on: May 27, 2024
Toni Hukkanen
Head of Design
Creative Direction, Brand Direction
Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.
AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking.
Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI.
AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.
Susceptibility to user bias
AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.
For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.
Inaccurate and unreliable algorithms
Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors.
Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones.
Absence of real-world context and insight
AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.
One example of this limitation is evident in the interpretation of humour and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works.
Deficiency in Creativity and Original Thought
According to research, creativity has two components, novelty and usefulness. Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality.
Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.
Lack of emotional insight and empathy
AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.
Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.
Undermining human expertise and intuition
Excessive use of AI for creative critique could devalue the importance of human expertise, intuition, and judgment in the creative process.
Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.
Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.
Ethical bias and considerations
AI chatbots are trained on large datasets that may contain inherent biases, such as gender, racial, and cultural biases. AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.
Creative work has the potential to influence societal perceptions, attitudes, and behaviours. AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.
Failure to understand strategic objectives
The creative work of an agency for a brand is typically informed by a set of strategic objectives, target audiences, and market positioning. These factors play a crucial role in shaping the direction and content of creative campaigns.
But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.
Challenges in grasping brand voice
Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.
AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.
Final Thoughts
While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.
ARTICLE #32
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
ARTICLE #32
Table of contents
Evaluating creative work with AI: What are the possible risks?
AI and Creativity
AI Evaluation
AI Limitations
AI Chatbot
AI for Design
Written by:
7 min read
Updated on: May 27, 2024
Toni Hukkanen
Head of Design
Creative Direction, Brand Direction
Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.
AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking.
Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI.
AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.
Susceptibility to user bias
AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.
For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.
Inaccurate and unreliable algorithms
Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors.
Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones.
Absence of real-world context and insight
AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.
One example of this limitation is evident in the interpretation of humour and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works.
Deficiency in Creativity and Original Thought
According to research, creativity has two components, novelty and usefulness. Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality.
Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.
Lack of emotional insight and empathy
AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.
Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.
Undermining human expertise and intuition
Excessive use of AI for creative critique could devalue the importance of human expertise, intuition, and judgment in the creative process.
Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.
Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.
Ethical bias and considerations
AI chatbots are trained on large datasets that may contain inherent biases, such as gender, racial, and cultural biases. AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.
Creative work has the potential to influence societal perceptions, attitudes, and behaviours. AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.
Failure to understand strategic objectives
The creative work of an agency for a brand is typically informed by a set of strategic objectives, target audiences, and market positioning. These factors play a crucial role in shaping the direction and content of creative campaigns.
But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.
Challenges in grasping brand voice
Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.
AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.
Final Thoughts
While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.
ARTICLE #32
More news
Work with us
Click to copy
work@for.co
- FOR® Brand. FOR® Future.
We’re remote-first — with strategic global hubs
Click to copy
Helsinki, FIN
hel@for.co
Click to copy
Tallinn,EST
ee@for.co
Click to copy
New York, NY
ny@for.co
Click to copy
Miami, FL
mia@for.co
Click to copy
Dubai, UAE
uae@for.co
Click to copy
Kyiv, UA
kyiv@for.co
Copyright © 2024 FOR®