---------------------------------------------------------------------------- Summary ---------------------------------------------------------------------------- The content of this file is mirrored on the Fuego wiki at: https://sourceforge.net/apps/trac/fuego/wiki/AdditiveKnowledgeTypeComparison This file contains the results of comparing the different types of additive knowledge used by Fuego's UCT search. At the time of writing, the trunk version of Fuego is at revision r1718 and implements these types of additive knowledge - KNOWLEDGE_NONE - KNOWLEDGE_GREENPEEP - KNOWLEDGE_RULEBASED ---------------------------------------------------------------------------- Builds ---------------------------------------------------------------------------- The following builds were used for the tests. Build 1 - Fuego trunk r1711 - This build was used before KNOWLEDGE_NONE was implemented in r1715 - Available knowledge types: KNOWLEDGE_GREENPEEP (0) and KNOWLEDGE_RULEBASED (1) Build 2 - Fuego trunk r1718 - This build was used after KNOWLEDGE_NONE was implemented in r1715 - Available knowledge types: KNOWLEDGE_NONE (0), KNOWLEDGE_GREENPEEP (1) and KNOWLEDGE_RULEBASED (2) Build 3 - Fuego 1.1 Build 4 - Fuego trunk with customization - The customization consists of a set of Greenpeep patterns that are reduced to 1 byte per pattern (down from the original 2 bytes per pattern) in order to reduce the amount of memory ---------------------------------------------------------------------------- Test environment ---------------------------------------------------------------------------- Build configuration - Mac OS X 10.9 - boost 1.55 (installed via Homebrew) - autoconf 2.69 (installed via Homebrew) - Xcode 5.0.2, Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Runtime configuration - MacBook Pro 15-inch, Late 2013 - 2.6 GHz Intel Core i7 - 16 GB RAM ---------------------------------------------------------------------------- Test runs ---------------------------------------------------------------------------- The following configuration settings apply for all test runs: - The default opening book is not used (--nobook) - 1000 games are played to have a good sample - Komi 6.5 - No handicap - To make the tests somewhat reproducible on different test systems with different CPU speeds, the limiting factor for playing strength is the number of simulations calculated per move ("uct_param_player max_games"). To eliminate the time factor, all tests set "uct_param_player ignore_clock 1": This guarantees that the full number of "max_games" is played out regardless of how fast the machine that runs the tests is. If "ignore_clock" were not set, a slower machine might hit the time limit (default is 10 seconds) before all games are played out. - All tests are conducted by gogui-twogtp from GoGui release 1.4.9 - gogui-twogtp is run with the command line option "-alternate" - GNU Go 3.8 is used as the referee program, with this command line: gnugo --mode gtp --never-resign --chinese-rules --capture-all-dead Run 1: Fast, 19x19 - boardsize 19 - uct_param_player max_games 1000 - uct_param_player ignore_clock 1 Run 2: Fast, 9x9 - boardsize 9 - uct_param_player max_games 1000 - uct_param_player ignore_clock 1 Run 3: Slow, 19x19 - boardsize 19 - uct_param_player max_games 25000 - uct_param_player ignore_clock 1 - uct_max_memory 3221225472 Run 4: Slow, 9x9 - boardsize 9 - uct_param_player max_games 25000 - uct_param_player ignore_clock 1 - uct_max_memory 3221225472 Run 5: Medium, 19x19 - boardsize 19 - uct_param_player max_games 10000 - uct_param_player ignore_clock 1 - uct_max_memory 3221225472 Run 6: Medium, 19x19 - boardsize 9 - uct_param_player max_games 10000 - uct_param_player ignore_clock 1 - uct_max_memory 3221225472 ---------------------------------------------------------------------------- Test results ---------------------------------------------------------------------------- This matrix tells which result set to look at for a specific combination of configurations/builds. KNOWLEDGE_NONE KNOWLEDGE_GREENPEEP KNOWLEDGE_RULEBASED Fuego 1.1 ----------------------------------------------------------------------------------------- KNOWLEDGE_NONE - B C F KNOWLEDGE_GREENPEEP B - A D KNOWLEDGE_RULEBASED C A - E Fuego 1.1 F D E - Full disclosure result details are available here: http://www.herzbube.ch/fuego/additive-knowledge-type-comparison ---------------------------------------------------------------------------- Result set A ---------------------------------------------------------------------------- - Player 1: Build 1, KNOWLEDGE_GREENPEEP - Player 2: Build 1, KNOWLEDGE_RULEBASED 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown greenpeep rulebased results greenpeep rulebased results -------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 73.1% 26.9% - 66.9% 33.1% - - Greenpeep bot 73.1% 26.9% - 66.2% 33.8% 5 - Rulebased bot 73.1% 26.9% - 66.3% 33.7% 5 Note: No wins by resignation. Medium (max_games 10000) - Referee (GNU Go 3.8) 66.0% 34.0% - 66.3% 33.7% - - Greenpeep bot 66.0% 34.0% - 66.3% 33.7% - - Rulebased bot 66.0% 34.0% - 66.3% 33.7% - Note: Almost all games are won by resignation. Slow (max_games 25000) - Referee (GNU Go 3.8) 63.6% 36.4% - 62.7% 37.3% - - Greenpeep bot 63.6% 36.4% - 62.4% 37.6% 2 - Rulebased bot 63.6% 36.4% - 62.5% 37.5% 2 Note: Almost all games are won by resignation. ---------------------------------------------------------------------------- Result set B ---------------------------------------------------------------------------- - Player 1: Build 2, KNOWLEDGE_GREENPEEP - Player 2: Build 2, KNOWLEDGE_NONE 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown greenpeep no-knowledge results greenpeep no-knowledge results -------------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 73.6% 26.4% - 69.5% 30.5% - - Greenpeep bot 73.6% 26.4% 1 68.5% 31.5% 1 - No-knowledge bot 73.6% 26.4% 1 68.7% 31.3% 1 Medium (max_games 10000) - Referee (GNU Go 3.8) 71.9% 28.1% - 70.2% 29.8% - - Greenpeep bot 71.9% 28.1% 1 69.9% 30.1% - - No-knowledge bot 71.9% 28.1% - 69.9% 30.1% - Note: Almost all games are won by resignation. Slow (max_games 25000) - No tests run ---------------------------------------------------------------------------- Result set C ---------------------------------------------------------------------------- - Player 1: Build 2, KNOWLEDGE_RULEBASED - Player 2: Build 2, KNOWLEDGE_NONE 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown rulebased no-knowledge results rulebased no-knowledge results -------------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 45.0% 55.0% - 53.7% 46.3% - - Rulebased bot 45.0% 55.0% - 53.7% 46.3% 6 - No-knowledge bot 45.0% 55.0% - 53.9% 46.1% 6 Medium (max_games 10000) - Referee (GNU Go 3.8) 62.3% 37.7% - 52.4% 47.6% - - Rulebased bot 62.3% 37.7% - 52.4% 47.6% 2 - No-knowledge bot 62.3% 37.7% - 52.5% 47.5% 1 Note: Almost all games are won by resignation. Slow (max_games 25000) - No tests run ---------------------------------------------------------------------------- Result set D ---------------------------------------------------------------------------- - Player 1: Build 1, KNOWLEDGE_GREENPEEP - Player 2: Build 3 (fuego-1.1) 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown greenpeep fuego-1.1 results greenpeep fuego-1.1 results -------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 93.4% 6.6% - 81.8% 18.2% - - Greenpeep bot 93.4% 6.6% 5 80.8% 19.2% 7 - Fuego 1.1 bot 93.4% 6.6% 5 80.9% 19.1% 7 Medium (max_games 10000) - Referee (GNU Go 3.8) 83.0% 17.0% - 79.5% 20.5% - - Greenpeep bot 83.0% 17.0% - 79.6% 20.4% - - Fuego 1.1 bot 82.9% 17.1% - 79.4% 20.6% 1 Note: Almost all games are won by resignation. Slow (max_games 25000) - No tests run ---------------------------------------------------------------------------- Result set E ---------------------------------------------------------------------------- - Player 1: Build 1, KNOWLEDGE_RULEBASED - Player 2: Build 3 (fuego-1.1) 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown rulebased fuego-1.1 results rulebased fuego-1.1 results -------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 82.3% 17.7% - 65.1% 34.9% - - Rulebased bot 82.3% 17.7% 1 64.2% 35.8% 4 - Fuego 1.1 bot 82.3% 17.7% 1 64.0% 36.0% 5 Medium (max_games 10000) - Referee (GNU Go 3.8) 83.6% 16.4% - 68.3% 31.7% - - Rulebased bot 83.6% 16.4% - 68.3% 31.7% - - Fuego 1.1 bot 83.4% 16.6% - 68.1% 31.9% 2 Note: Almost all games are won by resignation. Slow (max_games 25000) - No tests run ---------------------------------------------------------------------------- Result set F ---------------------------------------------------------------------------- - Player 1: Build 2, KNOWLEDGE_NONE - Player 2: Build 3 (fuego-1.1) 19x19 board 9x9 board wins by wins by unknown wins by wins by unknown no-knowledge fuego-1.1 results no-knowledge fuego-1.1 results -------------------------------------------------------------------------------------------------- Fast (max_games 1000) - Referee (GNU Go 3.8) 83.8% 16.2% - 65.8% 34.2% - - No-knowledge bot 83.7% 16.3% 4 64.5% 35.5% 11 - Fuego 1.1 bot 83.7% 16.3% 4 64.6% 35.4% 12 Medium (max_games 10000) - Referee (GNU Go 3.8) 69.0% 31.0% - 62.1% 37.9% - - No-knowledge bot 69.0% 31.0% - 61.7% 38.3% - - Fuego 1.1 bot 68.3% 31.7% - 61.8% 38.2% 2 Note: Almost all games are won by resignation. Slow (max_games 25000) - No tests run ---------------------------------------------------------------------------- Result set X ---------------------------------------------------------------------------- - Player 1: Build 1, KNOWLEDGE_GREENPEEP - Player 2: Build 4 (custom build), KNOWLEDGE_GREENPEEP reduced 19x19 board 9x9 board greenpeep greenpeep-reduced unknown greenpeep greenpeep-reduced unknown ------------------------------------------------------------------------------------------------------------ Fast (max_games 1000) - Referee (GNU Go 3.8) ? ? - ? ? - - Greenpeep bot ? ? - ? ? - - Rulebased bot ? ? - ? ? - Medium (max_games 10000) - Referee (GNU Go 3.8) ? ? - ? ? - - Greenpeep bot ? ? - ? ? - - Rulebased bot ? ? - ? ? - Slow (max_games 25000) - Referee (GNU Go 3.8) ? ? - ? ? - - Greenpeep bot ? ? - ? ? - - Rulebased bot ? ? - ? ? -