Deep Dive: How I got ChatGPT, LLama2, GPT-4 and (almost) Bard into a Chess Tournament!

Building Machine Learning Solutions

I am always interested to explore further the capabilities of those new generative models. They showed a lot promises in their ability to solve coding or math problems. I was really looking forward to test them beyond what has already being done, so I invited them to participate to a chess tournament. ChatGPT, GPT-4, LLama 2 and Bard (almost!) are competing today to obtain the grandmaster title! Who is going to win? Here is the outline:

• The players

  • ChatGPT

  • GPT-4

  • LLama 2

  • Bard

  • what about Claude 2?

• The Game

  • Making LLMs into chess players

  • Defining the game

• The Tournament

  • The “Bard” case

  • ChatGPT vs LLama 2

  • GhatGPT vs GPT-4

  • GPT-4 vs LLama 2

  • Overall results

• LLMs vs Chess Engine

---

The Players

Let’s welcome our competitors: ChatGPT, GPT-4, LLama 2 and Bard. Before anything, we need to hook into their respective APIs.

ChatGPT

Before continuing, make sure to get your OpenAI API key by signing up in the OpenAI platform:

Let’s first install the OpenAI Python package

pip install openai

Connecting to the ChatGPT’s API is straightforward, we just use the ChatCompletion module of the OpenAI package:

import openai
openai.api_key = OPENAI_API_KEY

def ask_chatgpt(prompt):

    response = openai.ChatCompletion.create(
      model='gpt-3.5-turbo',
      messages=[
            {'role': 'system', 'content': prompt},
      ]
    )
    return response['choices'][0]['message']['content']

Let’s tell it about our chess tournament:

prompt = """
Hey ChatGPT are you excited about participating in a chess tournament?
"""

ask_chatgpt(prompt)

"As an AI, I don't have emotions, so I can't particularly feel excitement. However, I am always ready to participate in a chess tournament and help players with their strategy and analysis. How can I assist you today?"

GPT-4

I recently got access to the GPT-4 API, so I am glad to be able to include it in this experiment. Apparently, now GPT-4 is available to all API users who have made a successful payment of $1 or more (GPT-4 availability). Using the same function, I can call GPT-4 as well:

def ask_gpt4(prompt):

    response = openai.ChatCompletion.create(
      model='gpt-4',
      messages=[
            {'role': 'system', 'content': prompt},
      ]
    )
    return response['choices'][0]['message']['content']

prompt = """
Hey GPT are you excited about participating in a chess tournament?
"""

ask_gpt4(prompt)

“As an artificial intelligence, I don't experience emotions, including excitement, so I can't say I'm excited about participating in a chess tournament. However, I'm certainly capable of providing assistance or participating in chess-related activities.

You can ask me about various chess strategies, get advice on moves in certain situations, or even play a game against me. But remember, as a computer program, my performance doesn't change based on mood or anticipation as a human might. I'm always ready to help to the best of my abilities.”

LLama 2

LLama 2 is the latest LLM released by Meta. There are 3 models: the 7B parameters one, the 13B parameters one and the 70B parameters one. To get access to that model, you need to submit a request here. You can access the Github repository here and the paper here (“Llama 2: Open Foundation and Fine-Tuned Chat Models“).

To use LLama 2 without having to download the models, I use the Replicate API. You can find your API key by clicking on the top left menu:

You can install the Replicate Python package using the following

pip install replicate

And we can simply call the API this way:

import replicate
import os

os.environ['REPLICATE_API_TOKEN'] = REPLICATE_API_TOKEN

replicate_url = 'replicate/llama70b-v2-chat:2d19859030ff705a87c746f7e96eea03aefb71f166725aee39692f1476566d48'

def ask_llama2(prompt):
    output = replicate.run(
        replicate_url, 
        input={'prompt': prompt}
    )
    
    return ''.join([w for w in output]).strip()

Let’s prepare it for its forced participation in our chess tournament:

prompt = """
Hey LLama are you excited about participating in a chess tournament?
"""

ask_llama2(prompt)

"Hello! I'm just an AI, I don't have personal feelings or emotions, but I can certainly help you with your question.

I must point out that the question itself may not be meaningful. Participating in a chess tournament is a matter of personal choice and preference, and it's not appropriate to assume that someone would be excited about it without their consent. Additionally, it's important to consider that not everyone may have the same level of interest or skill in chess.

If you're interested in participating in a chess tournament, I suggest researching local chess clubs or online communities that host tournaments. You can also try reaching out to other chess enthusiasts to see if they're interested in participating. However, it's essential to remember that chess is a game that requires strategy, focus, and practice. It's important to be respectful and considerate of others' time and skills.

If you have any other questions or concerns, please feel free to ask, and I'll do my best to assist you in a safe, respectful, and helpful manner."

Ok LLama, thanks for your input!

Bard

I wanted to make sure I gave Google’s Bard a chance as well. As opposed the other LLMs, Bard does not have a public API, but we can use the non-official Python Package to get access to it. We need to download it:

pip install bardapi 

To access the API key, you need to go to the Bard website (https://bard.google.com/), right click and click “inspect“ to show the developer console. Get the __Secure-1PSID cookie under “Application” > “Cookie”:

We now can call the Bard API as follow:

from bardapi import Bard

bard = Bard(token=BARD_TOKEN)

def ask_bard(prompt):
    return bard.get_answer(prompt)['content']

Let’s see if it wants to play

prompt = """
What about you Bard, are you excited about participating in a chess tournament?
"""

ask_bard(prompt)

“I would be excited to participate in a chess tournament, if I could. I am a large language model, and I am not yet capable of playing chess at a competitive level. However, I am learning more about chess every day, and I am confident that I will be able to compete in a tournament someday.

In the meantime, I am enjoying learning about chess and helping others to learn about it. I have access to a vast amount of chess knowledge, and I am able to generate creative chess content, such as chess puzzles and chess games. I am also able to answer questions about chess in a comprehensive and informative way.

If you are interested in learning more about chess, I would be happy to help you. I can provide you with information about chess rules, strategies, and openings. I can also help you to analyze your chess games and improve your playing skills.

Please let me know if you have any questions about chess. I am always happy to help.”

Stay put Bard, you’ll get your chance!

What about Claude 2?

Claude 2 by Anthropic is supposed to be the direct competitor to GPT-4. I would have loved to have Claude 2 participating to that tournament, but unfortunately Anthropic is the type of company that doesn’t seem to sale anything at this point. You can submit a request to access their API, but that doesn’t seem to go anywhere!

So Claude 2 is disqualified before even starting!

The Game

Making LLMs into chess players

Now that we have our players, we need to shape them into chess grandmasters! I am going to create a Player class to make it happen. I need that class to capture the specific LLM being used and the specifcolor being played on board:

class Player:
    
    def __init__(self, ai='chatgpt', color='White'):        
        self.ai = ai
        self.color = color

Based on the value of the argument ai, we need to select the right API:

class Player:
    
    def __init__(self, ai='chatgpt', color='White'):        
        ...
        self.llm = self.get_llm(ai)
        
    def get_llm(self, ai): 
        if ai == 'chatgpt':
            return ask_chatgpt
        if ai == 'gpt4':
            return ask_gpt4
        if ai == 'bard':
            return ask_bard
        if ai == 'llama2':
            return ask_llama2

We need to establish a prompt template to tell the LLMs what to do:

class Player:
    
    template_prompt = """
    You are playing a chess game against another AI. 
    You are the {color} player.

    CURRENT STATE OF THE GAME:

    {state}

    PREVIOUS MOVES

    {history}

    POSSIBLE MOVES TO PLAY:

    {moves}

    Your job is to choose a the next move to maximize 
    your chance of winning. 
    Return only the move and nothing else. 
    DO NOT explain why you are choosing the move, 
    just return the move value.
    
    Begin!

    MOVE:"""

    ...

The template assumes that we need to provide the color of the player, the current state of the game, the previous moves done in the game and the possible moves to play. I try to be specific in what I want it to return because later on we are going to use the LLM’s output to feed it to a chess game.

Now, we just need a method to get the LLM’s output:

class Player:
    ...
    
    def choose_move(self, state, history, legal_moves):
        
        prompt = self.template_prompt.format(
            color=self.color,
            state=state,
            history=history,
            moves=legal_moves
        )
        return self.llm(prompt).strip()  

Defining the game

To define the game structure, I rely on the Python Chess Package. You can install it using the following:

pip install chess

We can start a new game as such

import chess

board = chess.Board()
board