AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

Evaluating Creative Work with AI: What Are The Possible Risks

AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

WRITTEN BY

Updated on: May 27nd, 2024

Toni Hukkanen

Head of Design

Creative Direction, Brand Direction

7 Minute read

Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.

AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking. 

Keep reading to find out the possible risks if you ask ChatGPT or other LLMs for an expert opinion.

What Are The Risks of Evaluating Creative Work with AI?

Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete,  when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI. 

AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.

Here are some possible reasons for why you should not use AI to evaluate creative work:

Susceptibility To User Bias

AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.

For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.

Inaccurate And Unreliable Algorithms

Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors. 

Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones. 

Absence of Real-World Context And Insight

AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.

One example of this limitation is evident in the interpretation of humor and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works

Deficiency In Creativity And Original Thought

According to research, creativity has two components:

  • Novelty

  • Usefulness

Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality. 

Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.

Lack of Emotional Insight And Empathy

AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.

Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.

Undermining Human Expertise And Intuition

The excessive use of AI for creative critique could devalue the importance of: 

  • Human expertise

  • Intuition

  • Judgment in the creative process

Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.

Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.

Ethical Bias And Considerations

AI chatbots are trained on large datasets that may contain biases inherent in the data, such as:

  • Gender

  • Racial

  • Cultural biases

AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.

Creative work has the potential to influence: 

  • Societal perceptions

  • Attitudes

  • Behaviors

AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.

Failure To Understand Strategic Objectives

The creative work of an agency for a brand is typically informed by: 

  • A set of strategic objectives

  • Target audiences

  • Market positioning

These factors play a crucial role in shaping the direction and content of creative campaigns. 

But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.

Challenges In Grasping Brand Voice

Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.

AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.

Read More: How to Detect AI Content

Final Thoughts on Evaluating Creative Work with AI

While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.

AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

Evaluating Creative Work with AI: What Are The Possible Risks

AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

WRITTEN BY

Updated on: May 27nd, 2024

Toni Hukkanen

Head of Design

Creative Direction, Brand Direction

7 Minute read

Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.

AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking. 

Keep reading to find out the possible risks if you ask ChatGPT or other LLMs for an expert opinion.

What Are The Risks of Evaluating Creative Work with AI?

Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete,  when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI. 

AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.

Here are some possible reasons for why you should not use AI to evaluate creative work:

Susceptibility To User Bias

AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.

For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.

Inaccurate And Unreliable Algorithms

Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors. 

Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones. 

Absence of Real-World Context And Insight

AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.

One example of this limitation is evident in the interpretation of humor and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works

Deficiency In Creativity And Original Thought

According to research, creativity has two components:

  • Novelty

  • Usefulness

Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality. 

Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.

Lack of Emotional Insight And Empathy

AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.

Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.

Undermining Human Expertise And Intuition

The excessive use of AI for creative critique could devalue the importance of: 

  • Human expertise

  • Intuition

  • Judgment in the creative process

Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.

Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.

Ethical Bias And Considerations

AI chatbots are trained on large datasets that may contain biases inherent in the data, such as:

  • Gender

  • Racial

  • Cultural biases

AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.

Creative work has the potential to influence: 

  • Societal perceptions

  • Attitudes

  • Behaviors

AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.

Failure To Understand Strategic Objectives

The creative work of an agency for a brand is typically informed by: 

  • A set of strategic objectives

  • Target audiences

  • Market positioning

These factors play a crucial role in shaping the direction and content of creative campaigns. 

But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.

Challenges In Grasping Brand Voice

Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.

AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.

Read More: How to Detect AI Content

Final Thoughts on Evaluating Creative Work with AI

While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.

AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

Evaluating Creative Work with AI: What Are The Possible Risks

AI and Creativity

AI Evaluation

AI Limitations

AI Chatbot

LLM

WRITTEN BY

Updated on: May 27nd, 2024

Toni Hukkanen

Head of Design

Creative Direction, Brand Direction

7 Minute read

Artificial Intelligence (AI), such as ChatGPT or other LLMs, is entertaining and sounds human-like. But, it is also unreliable and could pose risks when it comes to evaluating creative work.

AI is already being used to write books and articles, assist healthcare practitioners, support legal professionals, and provide mental health support. As its capabilities grow, so will its impact on our lives. But, the use of AI to evaluate creative work is not ideal. It is not well-suited for tasks that require genuine creativity and original thinking. 

Keep reading to find out the possible risks if you ask ChatGPT or other LLMs for an expert opinion.

What Are The Risks of Evaluating Creative Work with AI?

Clients sometimes use ChatGPT to get an 'expert opinion' to evaluate the creative work of an agency. The worst cases happen after the project is complete,  when adding a new decision-maker too late causes chaos. This not only disrupts the process but also highlights the risks associated with evaluating creative work with AI. 

AI can be unreliable, biased, and lack the nuanced understanding required for creative judgment. These things make it a risky choice for such evaluations.

Here are some possible reasons for why you should not use AI to evaluate creative work:

Susceptibility To User Bias

AI is easily influenced by the prompts it receives. If you ask it to critique a piece of creative work, it will generate a critical assessment, pointing out flaws, real or imagined. But if you prompt it to praise the same work, it will gladly comply, producing a glowing review, emphasizing strengths, whether they exist or not. This susceptibility to direction highlights a significant limitation of AI: it simply reflects the instructions given to it rather than independently assessing the quality of creative work.

For example, research studies have demonstrated the influence of prompts on AI-generated responses. Researchers have found that GPT-3 was highly responsive to prompt manipulation. It often generated biased or inaccurate outputs based on the provided instructions.

Inaccurate And Unreliable Algorithms

Most AI tools use data and models to evaluate or provide feedback that can be outdated or incomplete. For example, an AI tool used for evaluating writing skills might not recognize different writing styles, contexts, or genres. It might even penalize non-native speakers for grammatical errors. 

Similarly, if you use an AI tool to provide feedback on performance, it might not account for individual goals, preferences, and differences. It might also overemphasize quantitative metrics instead of qualitative ones. 

Absence of Real-World Context And Insight

AI indeed has access to vast amounts of data, but it lacks the real-world experiences and contextual understanding humans possess. As a result, it may struggle to grasp the nuances, cultural references, and creative intent behind the verbal and visual expressions of a brand. This limitation can lead to superficial or misguided critiques or evaluations.

One example of this limitation is evident in the interpretation of humor and cultural references. AI may struggle to understand sarcasm, irony, or wordplay, which are prevalent in creative works

Deficiency In Creativity And Original Thought

According to research, creativity has two components:

  • Novelty

  • Usefulness

Novelty involves originality and uniqueness, which highlights what is new and different. This aspect is mainly emphasized in creativity evaluations. But novelty alone isn't sufficient to be considered valuable. Creative work also needs to be useful, which generally includes functionality. 

Large Language Models (LLMs) or AI can produce text that resembles human writing; they primarily replicate patterns from their training data rather than possessing genuine insight or creativity. Consequently, AI may encounter difficulties recognizing the originality, innovation, and artistic value of groundbreaking creative works that challenge conventional norms. This limitation could potentially result in AI’s critique discouraging bold and visionary brand expressions.

Lack of Emotional Insight And Empathy

AI algorithms can analyze data and generate content based on patterns. But, they lack the ability to comprehend emotions or empathize with human experiences.

Emotional insight and empathy play important roles in the interpretation and evaluation of creative work. Humans bring their emotional responses and subjective experiences to their assessments. AI operates solely based on predefined algorithms and objective criteria, which may not always align with human emotions or preferences.

Undermining Human Expertise And Intuition

The excessive use of AI for creative critique could devalue the importance of: 

  • Human expertise

  • Intuition

  • Judgment in the creative process

Creative directors, strategists, and clients have valuable industry knowledge and an intuitive understanding of what connects with audiences and communicates brand messages effectively. Relying solely on AI for evaluation could overlook these unique human perspectives and insights.

Depending too much on AI might standardize evaluations, making creative work less unique. Human creativity adds authenticity to brand expressions, helping brands stand out in the market.

Ethical Bias And Considerations

AI chatbots are trained on large datasets that may contain biases inherent in the data, such as:

  • Gender

  • Racial

  • Cultural biases

AI may inadvertently reinforce or amplify these biases when used to evaluate creative work. This leads to unfair or discriminatory evaluation.

Creative work has the potential to influence: 

  • Societal perceptions

  • Attitudes

  • Behaviors

AI-generated evaluations that lack cultural sensitivity or perpetuate stereotypes can have negative societal impacts, such as reinforcing bias or spreading misinformation.

Failure To Understand Strategic Objectives

The creative work of an agency for a brand is typically informed by: 

  • A set of strategic objectives

  • Target audiences

  • Market positioning

These factors play a crucial role in shaping the direction and content of creative campaigns. 

But AI often lacks the inherent understanding of this strategic context. This often leads to critiques that overlook the alignment of the work with the brand's goals and resonance with its intended audience.

Challenges In Grasping Brand Voice

Brand voice consists of a unique tone and personality. It is a style of communication that distinguishes a brand from others and resonates with its target audience. But, AI may struggle to capture and understand the nuances of brand voice accurately. It leads to misinterpretations or misrepresentations in their evaluations.

AI is trained on vast amounts of text data, which may include diverse styles of communication from various sources. This breadth of training data allows AI to generate human-like text. It also means that AI lacks a deep contextual understanding of individual brands and their specific brand voices. As a result, it may produce evaluations that do not align with the intended brand identity or messaging strategy. This potentially undermines the effectiveness of creative work.

Read More: How to Detect AI Content

Final Thoughts on Evaluating Creative Work with AI

While AI tools offer impressive capabilities, their application in evaluating creative work is not without risks. Ethical considerations, understanding of brand voice, appreciation for subtlety, and awareness of cultural trends are among the many aspects where AI falls short. Creative evaluations need human insight, empathy, and a profound understanding of context that AI cannot replicate. By relying on human expertise and taking a balanced approach, agencies and brands can protect their creative work. They can also keep it fresh, powerful, and meaningful, even as AI becomes more tempting.

Work with us

Click to copy

work@for.co

Copyright © 2024 FOR®

Cookie Settings

  • FOR® Growth

  • FOR® Digital

  • FOR® Brand

  • FOR® Studio

Work with us

Click to copy

work@for.co

Copyright © 2024 FOR®

Cookie Settings

  • FOR® Growth

  • FOR® Digital

  • FOR® Brand

  • FOR® Studio

Work with us

Click to copy

work@for.co

Copyright © 2024 FOR®

Cookie Settings