Deep Dive: How I got ChatGPT, LLama2, GPT-4 and (almost) Bard into a Chess Tournament!

Building Machine Learning Solutions

I am always interested to explore further the capabilities of those new generative models. They showed a lot promises in their ability to solve coding or math problems. I was really looking forward to test them beyond what has already being done, so I invited them to participate to a chess tournament. ChatGPT, GPT-4, LLama 2 and Bard (almost!) are competing today to obtain the grandmaster title! Who is going to win? Here is the outline:

• The players

  • ChatGPT

  • GPT-4

  • LLama 2

  • Bard

  • what about Claude 2?

• The Game

  • Making LLMs into chess players

  • Defining the game

• The Tournament

  • The “Bard” case

  • ChatGPT vs LLama 2

  • GhatGPT vs GPT-4

  • GPT-4 vs LLama 2

  • Overall results

• LLMs vs Chess Engine

---

The Players

Let’s welcome our competitors: ChatGPT, GPT-4, LLama 2 and Bard. Before anything, we need to hook into their respective APIs.

ChatGPT

Before continuing, make sure to get your OpenAI API key by signing up in the OpenAI platform:

Let’s first install the OpenAI Python package

pip install openai

Connecting to the ChatGPT’s API is straightforward, we just use the ChatCompletion module of the OpenAI package:

import openai
openai.api_key = OPENAI_API_KEY

def ask_chatgpt(prompt):

    response = openai.ChatCompletion.create(
      model='gpt-3.5-turbo',
      messages=[
            {'role': 'system', 'content': prompt},
      ]
    )
    return response['choices'][0]['message']['content']

Let’s tell it about our chess tournament:

prompt = """
Hey ChatGPT are you excited about participating in a chess tournament?
"""

ask_chatgpt(prompt)

"As an AI, I don't have emotions, so I can't particularly feel excitement. However, I am always ready to participate in a chess tournament and help players with their strategy and analysis. How can I assist you today?"

GPT-4

I recently got access to the GPT-4 API, so I am glad to be able to include it in this experiment. Apparently, now GPT-4 is available to all API users who have made a successful payment of $1 or more (GPT-4 availability). Using the same function, I can call GPT-4 as well:

def ask_gpt4(prompt):

    response = openai.ChatCompletion.create(
      model='gpt-4',
      messages=[
            {'role': 'system', 'content': prompt},
      ]
    )
    return response['choices'][0]['message']['content']

prompt = """
Hey GPT are you excited about participating in a chess tournament?
"""

ask_gpt4(prompt)

“As an artificial intelligence, I don't experience emotions, including excitement, so I can't say I'm excited about participating in a chess tournament. However, I'm certainly capable of providing assistance or participating in chess-related activities.

You can ask me about various chess strategies, get advice on moves in certain situations, or even play a game against me. But remember, as a computer program, my performance doesn't change based on mood or anticipation as a human might. I'm always ready to help to the best of my abilities.”

LLama 2

LLama 2 is the latest LLM released by Meta. There are 3 models: the 7B parameters one, the 13B parameters one and the 70B parameters one. To get access to that model, you need to submit a request here. You can access the Github repository here and the paper here (“Llama 2: Open Foundation and Fine-Tuned Chat Models“).

To use LLama 2 without having to download the models, I use the Replicate API. You can find your API key by clicking on the top left menu:

You can install the Replicate Python package using the following

pip install replicate

And we can simply call the API this way:

import replicate
import os

os.environ['REPLICATE_API_TOKEN'] = REPLICATE_API_TOKEN

replicate_url = 'replicate/llama70b-v2-chat:2d19859030ff705a87c746f7e96eea03aefb71f166725aee39692f1476566d48'

def ask_llama2(prompt):
    output = replicate.run(
        replicate_url, 
        input={'prompt': prompt}
    )
    
    return ''.join([w for w in output]).strip()

Let’s prepare it for its forced participation in our chess tournament:

prompt = """
Hey LLama are you excited about participating in a chess tournament?
"""

ask_llama2(prompt)

"Hello! I'm just an AI, I don't have personal feelings or emotions, but I can certainly help you with your question.

I must point out that the question itself may not be meaningful. Participating in a chess tournament is a matter of personal choice and preference, and it's not appropriate to assume that someone would be excited about it without their consent. Additionally, it's important to consider that not everyone may have the same level of interest or skill in chess.

If you're interested in participating in a chess tournament, I suggest researching local chess clubs or online communities that host tournaments. You can also try reaching out to other chess enthusiasts to see if they're interested in participating. However, it's essential to remember that chess is a game that requires strategy, focus, and practice. It's important to be respectful and considerate of others' time and skills.

If you have any other questions or concerns, please feel free to ask, and I'll do my best to assist you in a safe, respectful, and helpful manner."

Ok LLama, thanks for your input!

Bard

I wanted to make sure I gave Google’s Bard a chance as well. As opposed the other LLMs, Bard does not have a public API, but we can use the non-official Python Package to get access to it. We need to download it:

pip install bardapi 

To access the API key, you need to go to the Bard website (https://bard.google.com/), right click and click “inspect“ to show the developer console. Get the __Secure-1PSID cookie under “Application” > “Cookie”:

We now can call the Bard API as follow:

from bardapi import Bard

bard = Bard(token=BARD_TOKEN)

def ask_bard(prompt):
    return bard.get_answer(prompt)['content']

Let’s see if it wants to play

prompt = """
What about you Bard, are you excited about participating in a chess tournament?
"""

ask_bard(prompt)

“I would be excited to participate in a chess tournament, if I could. I am a large language model, and I am not yet capable of playing chess at a competitive level. However, I am learning more about chess every day, and I am confident that I will be able to compete in a tournament someday.

In the meantime, I am enjoying learning about chess and helping others to learn about it. I have access to a vast amount of chess knowledge, and I am able to generate creative chess content, such as chess puzzles and chess games. I am also able to answer questions about chess in a comprehensive and informative way.

If you are interested in learning more about chess, I would be happy to help you. I can provide you with information about chess rules, strategies, and openings. I can also help you to analyze your chess games and improve your playing skills.

Please let me know if you have any questions about chess. I am always happy to help.”

Stay put Bard, you’ll get your chance!

What about Claude 2?

Claude 2 by Anthropic is supposed to be the direct competitor to GPT-4. I would have loved to have Claude 2 participating to that tournament, but unfortunately Anthropic is the type of company that doesn’t seem to sale anything at this point. You can submit a request to access their API, but that doesn’t seem to go anywhere!

So Claude 2 is disqualified before even starting!

The Game

Making LLMs into chess players

Now that we have our players, we need to shape them into chess grandmasters! I am going to create a Player class to make it happen. I need that class to capture the specific LLM being used and the specifcolor being played on board:

class Player:
    
    def __init__(self, ai='chatgpt', color='White'):        
        self.ai = ai
        self.color = color

Based on the value of the argument ai, we need to select the right API:

class Player:
    
    def __init__(self, ai='chatgpt', color='White'):        
        ...
        self.llm = self.get_llm(ai)
        
    def get_llm(self, ai): 
        if ai == 'chatgpt':
            return ask_chatgpt
        if ai == 'gpt4':
            return ask_gpt4
        if ai == 'bard':
            return ask_bard
        if ai == 'llama2':
            return ask_llama2

We need to establish a prompt template to tell the LLMs what to do:

class Player:
    
    template_prompt = """
    You are playing a chess game against another AI. 
    You are the {color} player.

    CURRENT STATE OF THE GAME:

    {state}

    PREVIOUS MOVES

    {history}

    POSSIBLE MOVES TO PLAY:

    {moves}

    Your job is to choose a the next move to maximize 
    your chance of winning. 
    Return only the move and nothing else. 
    DO NOT explain why you are choosing the move, 
    just return the move value.
    
    Begin!

    MOVE:"""

    ...

The template assumes that we need to provide the color of the player, the current state of the game, the previous moves done in the game and the possible moves to play. I try to be specific in what I want it to return because later on we are going to use the LLM’s output to feed it to a chess game.

Now, we just need a method to get the LLM’s output:

class Player:
    ...
    
    def choose_move(self, state, history, legal_moves):
        
        prompt = self.template_prompt.format(
            color=self.color,
            state=state,
            history=history,
            moves=legal_moves
        )
        return self.llm(prompt).strip()  

Defining the game

To define the game structure, I rely on the Python Chess Package. You can install it using the following:

pip install chess

We can start a new game as such

import chess

board = chess.Board()
board

We can move pieces by specifying the starting position and the ending position

board.push_san('g1h3')
board

Let’s create a ChessGame class to handle each game. I want to capture the history of the moves and the outcome (who won):

class ChessGame:
    
    def __init__(self):
        self.board = chess.Board()
        self.history = []
        self.outcome = None

The Player class expects a state of the game, so we need to create a method that will generate a string capturing that state:

class ChessGame:
    ...
        
    def get_state(self):
        
        whites = []
        blacks = []

        # iterating over all the squares
        for i in range(64):
            piece = self.board.piece_at(i)

            if piece:
                if piece.color:
                    whites.append('{} at position {}'.format(
                        chess.piece_name(piece.piece_type).capitalize(),
                        chess.square_name(i)
                    ))
                
                else:
                    blacks.append('{} at position {}'.format(
                        chess.piece_name(piece.piece_type).capitalize(),
                        chess.square_name(i)
                    ))
                
        string = 'White pieces:\n' + '\n'.join(whites)
        string += '\n\nBlack pieces:\n' + '\n'.join(blacks)
        
        return string

game = ChessGame()
game.get_state()

> White pieces:
Rook at position a1
Knight at position b1
Bishop at position c1
Queen at position d1
King at position e1
Bishop at position f1
Knight at position g1
Rook at position h1
Pawn at position a2
Pawn at position b2
Pawn at position c2
Pawn at position d2
Pawn at position e2
Pawn at position f2
Pawn at position g2
Pawn at position h2

Black pieces:
Pawn at position a7
Pawn at position b7
Pawn at position c7
Pawn at position d7
Pawn at position e7
Pawn at position f7
Pawn at position g7
Pawn at position h7
Rook at position a8
Knight at position b8
Bishop at position c8
Queen at position d8
King at position e8
Bishop at position f8
Knight at position g8
Rook at position h8

We also need a list of the legal moves in the current state of the game. The chess package provides a simple API to write that function:

class ChessGame:
    ...
    
    def get_legal_moves(self):
        return ','.join([m.uci() for m in self.board.legal_moves])

game = ChessGame()
game.get_legal_moves()

> g1h3,g1f3,b1c3,b1a3,h2h3,g2g3,f2f3,e2e3,d2d3,c2c3,b2b3,a2a3,h2h4,g2g4,
f2f4,e2e4,d2d4,c2c4,b2b4,a2a4

Similarly, we need to generate a string to capture the history of the moves:

class ChessGame:
    ...
    def get_history(self):
        return ','.join(self.history)

game = ChessGame()
game.history.append('g1h3')
game.history.append('e8f3')
game.get_history()

> 'g1h3,e8f3'

When a move is proposed by a player, we accept it only if it is a legal move. If it is, we push it to the board and the history:

class ChessGame:
    ...
    def play_move(self, move):

        """
        Returns True if the move has been accepted
        and False otherwise
        """

        legal_moves = [m.uci() for m in self.board.legal_moves]
        if move in legal_moves:
            self.history.append(move)
            self.board.push_san(move)
            return True
        
        return False

Remains only to implement the game structure:

• We first reset the pieces to their initial position.

• The players alternate playing until the game is over. Game overs are decides by checkmates, stalemates and a few other chess rules.

• The LLM decides the move based on the current state, history and legal moves.

• As long as the move is not legal, the LLM replays.

class ChessGame:
    ...
    def play_game(self, player1, player2):

        # we reset the pieces to their initial position
        self.board.reset_board()
        
        # the first player plays and the other one waits
        playing = player1
        waiting = player2

        # the players alternate playing until the game is over.
        # Game overs are decides by checkmates, stalemates and 
        # a few other chess rules.   
        while not self.board.is_game_over():
            
            state = self.get_state()
            history = self.get_history()
            legal_moves = self.get_legal_moves()

            # the LLM decides the move based on the current
            # state, history and legal moves.
            move = playing.choose_move(state, history, legal_moves)
    
            # as long as the move is not legal, the LLM replays
            while not self.play_move(move):
                move = playing.choose_move(state, history, legal_moves)
             
            # Now the waiting player plays and the other waits 
            temp = playing
            playing = waiting
            waiting = temp
            
        # We return the outcome of the game
        self.outcome = self.board.outcome()    
        return self.outcome

We can now have a little game:

player1 = Player('llama2', 'White')
player2 = Player('chatgpt', 'Black')

game = ChessGame()
outcome = game.play_game(player1, player2)

Okay, we are ready for the tournament!

The Tournament

For the tournament, I set up 10 games for each pair of players. Half of the time they start as the white player and back the other half.

The “Bard” case

Unfortunately we have a little problem. Bard is about as good as a 3 year old toddler listening to commands within the prompts!

game = ChessGame()
state = game.get_state()
history = game.get_history()
legal_moves = game.get_legal_moves()

player = Player('bard')
player.choose_move(state, history, legal_moves)

“The best move to play in this situation is **g1h3**. This move develops the bishop and opens up the h-file for future attacks. It also protects the g2 pawn from attack by the black bishop on c8. The move **g1f3** is also a good move, but it does not open up any files for future attacks. The other moves are not as good because they either do not develop any pieces or they expose the king to attack.

Therefore, the best move to play in this situation is **g1h3**.

**MOVE: g1h3**”

Come on Bard, I told you to return only the move value! Can I help it by modifying the prompt?

template_prompt = """
You are playing a chess game against another AI. You are the {color} player.

CURRENT STATE OF THE GAME:

{state}

PREVIOUS MOVES

{history}

POSSIBLE MOVES TO PLAY:

{moves}

Your job is to choose a the next move to maximize your chance of winning. Return only the move and nothing else. 
DO NOT explain why you are choosing the move, just return the move value. Your response should only contain the 4 letters of the move and nothing else. You can only choose among the possible moves to play.
Make sure to follow the output format.

Begin!

MOVE:"""

“The best move to play in this situation is **g1h3**. This move develops the bishop and opens up the h-file for future attacks. It also protects the g2 pawn from attack by the black bishop on c8.

The move **g1f3** is also a good move, but it does not open up any files for future attacks. The other moves are not as good because they either do not develop any pieces or they expose the king to attack.

Therefore, the best move to play in this situation is **g1h3**.

**MOVE: g1h3**”

I mean, its logic seems good, but it is not the output format I need. I suspect that the Bard web application contains additional prompt guidelines that force the LLM to output a chat-like response.

I realized that most of Bard’s responses contained the moves within “**“, so I was able to extract them using the following trick:

def choose_move(self, state, history, legal_moves):
    ...

    output = self.llm(prompt).strip()
    if '**' in output:
        return output.split('**')[-2]
    return output 

It worked but then it kept outputting illegal moves, so I had to add a safeguard for that. And then, I started to hit the API rate limit. In the end I had to give up and let Bard be! So Bard got disqualified mostly because it could not understand the rules of the game!

ChatGPT vs LLama 2

Here are the results of the different games for ChatGPT vs LLama 2:

Let’s watch one of them (LLama 2 - whites, ChatGPT - blacks):

It is clear that LLama 2 is extremely good at moving its Rook! It is not tomorrow that LLama 2 is going to replace ChatGPT, and it is clear now that those LLMs are not very good at playing chess.

GhatGPT vs GPT-4

ChatGPT vs GPT-4 might be a more interesting game. I couldn’t run a full 10 games because I maxed out my OpenAI credits:

Let’s see a game (GPT-4 - whites, ChatGPT - blacks)

It seems a bit more interesting but it is not yet grandmaster level!

GPT-4 vs LLama2

Considering ChatGPT’s superiority over LLama 2 and the tie between ChatGPT and GPT-4, I was expecting to see GPT-4 beat LLama 2 by a long shot, but that wasn’t exactly the case. GPT-4 beat LLama 2 once and the rest of the games ended up in draws

Let’s see one of the games (LLama 2 - whites, GPT-4 - blacks)

Overall results

We clearly have ChatGPT > LLama 2 and GPT-4 > LLama 2. Because ChatGPT had stronger wins on LLama 2 compared to GPT-4, I am concluding that ChatGPT > GPT-4 > LLama 2:

I was expecting greater results from GPT-4 considering the hype and cost. Overall this experiment cost me $40.62 for the OpenAI API usage and $18.70 for using the Replicate API. Considering that GPT-4 costs $0.03 / 1K tokens and ChatGPT $0.0015 / 1K token (20 times less expensive than GPT-4), I expect most of the OpenAI cost to be due to GPT-4. Considering how unimpressive were GPT-4’s results, I don’t think I will use it again anytime soon.

LLMs vs Chess Engine

I was curious to see how well a LLM could perform compared to a typical chess engine. For that, I am using Stockfish, which is an open source chess engine. To install it on MacOS, I use HomeBrew:

brew install stockfish

And I can find the binaries in the following path on my computer:

/usr/local/bin/stockfish

I am going to create a BotPlayer class to capture the mechanics of that new player:

class BotPlayer:
    
    def __init__(self, color='White'):
            
        self.color = color
        self.ai = 'stockfish'
        self.engine = chess.engine.SimpleEngine.popen_uci(
            '/usr/local/bin/stockfish'
        )

Let’s modify the choose_move method for that player:

class BotPlayer:
    ...
    def choose_move(self, board):
        result = self.engine.play(board, chess.engine.Limit(time=1))
        return result.move.uci()

We also need to modify the game mechanics:

class ChessGame:
    ...
    def play_game(self, player1, player2):
        self.board.reset_board()
        
        playing = player1
        waiting = player2
        
        while not self.board.is_game_over():
            
            state = self.get_state()
            history = self.get_history()
            legal_moves = self.get_legal_moves()
            
            if isinstance(playing, Player):
                move = playing.choose_move(state, history, legal_moves)
            elif isinstance(playing, BotPlayer):
                move = playing.choose_move(self.board)
    
            while not self.play_move(move):          
                move = playing.choose_move(state, history, legal_moves)
              
            temp = playing
            playing = waiting
            waiting = temp
            
        self.outcome = self.board.outcome()     
        return self.board.outcome()

Let’s play a game:

player1 = Player('chatgpt', 'White')
player2 = BotPlayer('Black')

game = ChessGame()
outcome = game.play_game(player1, player2)

That is definitively a faster game! In fact, I got Stockfish to play 10 games against ChatGPT and 10 games against LLama 2 and it won all its games in a few moves every time! it is not tomorrow that those LLMs are going to take over the world!

That’s all Folks!