Adaptive AI primer/database tool

Discussion in 'Community Workshop' started by pixeljetstream, Jun 4, 2016.

  1. hotak

    hotak New Member

    Joined:
    Jun 3, 2016
    Ratings:
    +4 / 0 / -0
    I mostly race against AI, so i should have some usefull data, but for some car/track combo i may have very bad times because i was just getting settled with them or was letting some friend play. Tell me if my file can be usefull anyways, i can upload it this evening
     
  2. Mich Angel

    Mich Angel Well-Known Member

    Joined:
    Feb 16, 2015
    Ratings:
    +216 / 0 / -0
    Well my file is 467kb at the moment and growing..
     
  3. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    @hotak, your time won't matter mostly interested in the AI time :)
     
    • Agree Agree x 1
    • Funny Funny x 1
  4. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    updated html once more, and generated some statistics
    http://htmlpreview.github.io/?https...aptive-ai-primer/blob/master/docs/sample.html

    We now have reached almost 1% of all possible combinations (40 ai levels * 65 tracks * 28 carclasses)
    Total (track, car, ai) Entries: 746 (0.97%) Times: 786
    Only have very little entries with more than one time, just shows how much variety in the 12 people who contributed so far
    In terms of having at least one value per track & car we are at 16%
    Track (track, car) Entries: 316(16.36%)
     
  5. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    I think we may get good values for laptimes for 80 and 120 % AI level and just linearly interpolate between, maybe have 100% as another value. But when I look at some races the times scale quite linearly.
    The more dense data we have, the more outliers/noisy our data gets.
     
  6. kingtin

    kingtin New Member

    Joined:
    Jun 27, 2015
    Ratings:
    +3 / 0 / -0
    Here's mine. Hope it doesn't make the data too noisy :D
     

    Attached Files:

  7. Paul Darke

    Paul Darke Moderator Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +198 / 0 / -0
    Here's mine. I hope it is of some use.:cool:
     

    Attached Files:

  8. hotak

    hotak New Member

    Joined:
    Jun 3, 2016
    Ratings:
    +4 / 0 / -0
    here's mine
     

    Attached Files:

  9. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    thanks for those contributing, we got 20% track/car coverage, although not all will be useful. We need at least two distinct ai - levels per track to have better data for the autogenerated values.

    worked on that portion today, and the attached html report gives AI times per track/car that passed some "plausibility" criteria that I made up. Right now only a quarter of the content we have passes that criteria, maybe it's too harsh now we will see.

    if you are brave you could use the attached table and make edits to your aiadaptation file as @CheerfullyInsane described. Otherwise wait until I have that part of the tool done which lets you modify the file via some UI (though warning this may be couple weeks away).

    edit removed file, didn't check plausibility as well
     
    Last edited: Jun 16, 2016
  10. Gareth Smith

    Gareth Smith Well-Known Member

    Joined:
    Oct 29, 2015
    Ratings:
    +53 / 0 / -0
    Here's mine, I used to race in single player only but moved to MP only now. Anyway hope it helps.
     

    Attached Files:

  11. CheerfullyInsane

    CheerfullyInsane Well-Known Member

    Joined:
    Oct 24, 2015
    Ratings:
    +334 / 0 / -0
    Sure, I'll give it a whack or two during the week. :)
    Btw, I think I may know why there are such large differences in the AI lap-times.
    If you 'train' it like me using only one AI, the driver selected for each race differs.
    And seeing as it is always the same drivers in the top ten during 'real' SP races, I suspect there might be something like the GTR2 talent files hidden away somewhere.
    Not that it matters all that much. Since we have a span of six AI levels, we are mainly looking for a ball-park figure.
     
  12. CheerfullyInsane

    CheerfullyInsane Well-Known Member

    Joined:
    Oct 24, 2015
    Ratings:
    +334 / 0 / -0
    First impressions:
    This actually works rather beautifully. Exceedingly well done @pixeljetstream :)
    There is however a bug in the algorithm you use to generate lap-times.
    Take a look at Aquilas@Raceway.
    Starting from 120, it the laptimes slowly increase until around level 108.....
    Then they start to decrease again.
    Level 80 is in fact 25 seconds faster than level 120. ;):p
    What I suspect is happening is that you've taught the algorithm to decrease the lap-times by X amount from a baseline, which is great as long as you want it to go faster.
    But it should be reversed when going the other way.
    Speaking of which, is there a way you can highlight (e.g. through text-colour) which times are player-generated, and which are algorithm generated?
    In order to test the algorithm, it kinda helps if you know which lap-times you're comparing against. :)

    First I tried to see if it works (which it does), then I tried to break it. :D

    Couple of things to note.
    If you haven't been following any of the other threads, one major thing to remember:
    R3E cannot be running while you edit the XML file!
    The XML file gets loaded into memory when R3E starts up, and after it shuts down, it dumps the memory into a new XML file that overwrites the old one.
    So if you edit it while R3E is running, not only will the game not load any of the settings you've typed in, it'll also overwrite them afterwards, making the whole thing a colossal waste of time.

    Next, in order to get full enjoyment out of the adaptive AI, you have to be fairly consistent.
    The shorter the races you run, the more consistent you have to be.
    The idea here is that if you spin in a 30 minute race, you have maybe 14 other laps to pull the average up.
    If however you spin in an 8m race, you only have 3-4 laps to make up the difference.
    One example was when I tried the generated lap-times in NSUs@Zandvoort. I plugged in the lap-times and got an index running:
    110-111-112-113-114-115
    Then I did an 8 minute race, and spun on one lap. After that the index ran:
    107-111-112-113-114-115
    So the AI threw in a 107 to deal with my slower lap.
    Now, it's not the end of the world since the lap-times are weighted.
    R3E takes note of how often each AI level is used in each race, and adds a value to them. So just because I had one spin, doesn't mean the AI will start at 107 in my next race.
    However, if your lap-times are all over the place, the AI can't find a baseline to adjust to.

    Next, I tried to see if you needed your own lap-times in there as well in order for it to work.
    I simply plugged in 5 identical lap-times at the level i wanted and tried racing.
    Then I tried it again with only the AI times.
    Either works fine. As I kinda figured from my earlier testing, your own lap-times only comes into play AFTER the AI is trained.

    Last thing I tried was to see if the number of decimals matter.
    The XML file has 8 decimals, and the table supplies us with 4.
    As it turns out, the AI doesn't care either way. You can plug in 4 decimal lap-times, and that'll work just fine.

    EDIT: Updated version of my XML.
    Lot more tracks for both the FRJ and Aquilas. (I seem to be the only one running them :D )
     

    Attached Files:

    Last edited: Jun 14, 2016
  13. Riverside

    Riverside Well-Known Member

    Joined:
    Aug 30, 2015
    Ratings:
    +85 / 0 / -0
    Sorry i just saw this Lars, if it is not too late i can contribute because i do a fair amount of AI and also because you guys are going to struggle populating data at levels lower that 120... :D:D:D I can give lots of stuff at 100, 105... ;)

    Will send my file later this week...
     
    Last edited: Jun 16, 2016
  14. CheerfullyInsane

    CheerfullyInsane Well-Known Member

    Joined:
    Oct 24, 2015
    Ratings:
    +334 / 0 / -0
    Great. :)
    Everyone are volunteers, so no pressure.
    Just as and when you get to it. :)
     
  15. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    Thanks @CheerfullyInsane indeed there were some bugs. The tool will mostly never use values directly reported by users but will fit a curve to the data. Reason is that the data is kinda noisy anyway... I may change the fit logic over time, but the good news is that we can always change it.

    I actually use the fact that my numbers will generate 0005 or say 000? or last 4 digits missing to detect values the tool generated, as it's unlikely real data has those properties. To avoid the problem of enriching database by values we generated that @mr_belowski mentioned before.

    The tool is ready and should work for the brave once more. Just command-line so far (sorry no time for UI yet).

    But in theory it works.
    • Run the exe file (it comes with the database of times we gathered so far)
    • Look at the results/processed.html file, find the ai levels you want to race against
    • edit the "edits.lua" file (see below)
    • BACKUP your old aiadaptation.xml !!
    • Drag drop the edits.lua file onto exe or use commandline to pass it (preferred as you can read error messages)
    Here is what the edits.lua file looks like, ideally self explanatory.

    You can see the "safety" is on, as the target points to a file that won't exist for you, so you have to switch the if (true) to if (false) as well.

    The sample does just one edit operation, but you could throw in multiple of course.

    Code:
    -- for now no UI
    -- change what you want to edit here
    -- lookup classid, trackid and your preferred ai level via results/processed.html
    
    -- modifyAdaptive (filename, database to use, trackid, classid,
    --                 ai start, ai end, ai spacing (1: e.g 90,91,92,93... 2: 90,92,94...))
    
    print "executing edits"
    
    local target
    
    if (true) then
      -- absolute path
      target = [[D:\projects\r3e-adaptive-ai-primer\aiadaptation.xml]]
    else
      -- danger zone
      -- make backups!
      -- this will look at the default location of the aiapdatiation.xml
      target = specialFilename([[$USER_DOCUMENTS$\My Games\SimBin\RaceRoom Racing Experience\UserData\Player1\aiadaptation.xml]])
    end
    
    -- this changes "RaceRoom Raceway - Classic Sprint" (265) for "Aquila CR1 Cup" (255)
    -- it will generate ai for 97,99,101,103
    -- use tighter spacing 1 instead of 2, if you are more consistent
    modifyAdaptive(target, processed, "265", "255",  97, 103, 2)
    
     
    Last edited: Jun 17, 2016
  16. CheerfullyInsane

    CheerfullyInsane Well-Known Member

    Joined:
    Oct 24, 2015
    Ratings:
    +334 / 0 / -0
    Fair point.

    Will do.
    Going to have to wait until tomorrow though.
    It would presumably be a good idea to be at least semi-awake for this. :D
     
  17. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    ui version is done :)

    [​IMG]
     
    • Love it! Love it! x 2
    • Like Like x 1
  18. Riverside

    Riverside Well-Known Member

    Joined:
    Aug 30, 2015
    Ratings:
    +85 / 0 / -0
    Very impressive and advanced already, thanks @pixeljetstream @CheerfullyInsane; will give a try today; in the meantime and in case you are still collecting udata here is my current file. Thanks!
     

    Attached Files:

  19. mr_belowski

    mr_belowski Well-Known Member Beta tester

    Joined:
    Apr 23, 2015
    Ratings:
    +1,287 / 0 / -0
    Woah, that's a bit awesome :D

    Are the data points you've collected going to invalidated if / when the ai gets a kick up the arse? I believe Robert is working on their performance at the moment, which might make the current time->ai level data inaccurate
     
  20. pixeljetstream

    pixeljetstream Well-Known Member Beta tester

    Joined:
    Jan 29, 2015
    Ratings:
    +378 / 0 / -0
    @Riverside yes keep it coming :) we have tons of gaps.

    @mr_belowski yes good point. We don't have a "versioning" of sorts, although maybe I should add file-dates where the timestamps came from. R3E itself probably wipes the adaptation file clean whenever bigger changes occurred (or at least it should).

    what I hope is that some day we get a dedicated server with ai support and then we could just brute-force fill the database with tons of AI values running a server in "accelerated simulation time". I'd hope the developers have such a mode, cause then you can tweak AI much easier by simulating long races in a minute or so depending on how fast the PC is.