Skip to content

Commit

Permalink
Integrate boxcars (#225)
Browse files Browse the repository at this point in the history
* decompile replay with boxcars

* refactor header parsing to boxcars format

* refactor frame parsing to boxcars format 1

* update boxcars-py

* readd flagged attribute handling

* workaround for checking invalid actor ids

* temp json file no longer needed

* dropshot fixes

* fix rotation on old replays

* fix party leader parsing

* fix error test

* fix more tests

* fix rest of the tests

* clean up rattletrap

* Added benchmarking to this pr

* Add boxcars-py==0.1.1 to setup.py

* Update benchmarking.yml

* Update unsigned check for Engine.PlayerReplicationInfo:Team

* Add safety check to GameEventHandler

* add safety check for player team coming in at 4294967295

* update boxcars-py to 0.1.2

* handle new boxcars actor id format

* update boxcars-py to 0.1.3

* fix camera settings

* Update version number

Co-authored-by: dtracers <[email protected]>
Co-authored-by: Paul Seelman <[email protected]>
Co-authored-by: Sciguymjm <[email protected]>
  • Loading branch information
4 people authored Jun 18, 2020
1 parent 7f188c8 commit 5d4385d
Show file tree
Hide file tree
Showing 37 changed files with 284 additions and 593 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/benchmarking.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.7'
python-version: '3.6.10'
architecture: 'x64'

- name: Install/Update pip and wheel.
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ venv/
output
.env
.coverage
rattletrap-*
*.iml
build/
carball.egg*
Expand Down
2 changes: 1 addition & 1 deletion CARBALL_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
36
0
2 changes: 0 additions & 2 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
include requirements.txt
include CARBALL_VERSION
include init.py
include carball/rattletrap/*
include carball/analysis/*
exclude carball/generated/*
exclude carball/rattletrap/rattletrap-*
recursive-include carball/generated/api *.py
recursive-include carball *.xlsx
12 changes: 1 addition & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,13 +127,9 @@ optional arguments:
--proto PROTO The result of the analysis will be saved to this file
in protocol buffers format.
--json JSON The result of the analysis will be saved to this file
in json file format. This is not the decompiled replay
json from rattletrap.
in json file format.
--gzip GZIP The pandas dataframe will be saved to this file in a
compressed gzip format.
-sd, --skip-decompile
If set, carball will treat the input file as a json
file that Rattletrap outputs.
-v, --verbose Set the logging level to INFO. To set the logging
level to DEBUG use -vv.
-s, --silent Disable logging altogether.
Expand Down Expand Up @@ -166,12 +162,6 @@ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1
```
This assumes you already have 3.6 installed.

Linux Error (Potential):
`PermissionError: [Errno 13] Permission denied: 'carball/rattletrap/rattletrap-6.2.2-linux'`
Fix:
`chmod +x "carball/rattletrap/rattletrap-6.2.2-linux"`


## Developing
Everyone is welcome to join the carball (and calculated.gg) project! Even if you are a beginner, this can be used as an opportunity to learn more - you just need to be willing to learn and contribute.

Expand Down
2 changes: 1 addition & 1 deletion carball/analysis/PROTOBUF_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
6
7
15 changes: 2 additions & 13 deletions carball/command_line.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
import carball
import logging
import gzip
from carball.json_parser.game import Game
from carball.analysis.analysis_manager import AnalysisManager


def main(program_args=None):
Expand All @@ -14,13 +12,10 @@ def main(program_args=None):
parser.add_argument('--proto', type=str, required=False,
help='The result of the analysis will be saved to this file in protocol buffers format.')
parser.add_argument('--json', type=str, required=False,
help='The result of the analysis will be saved to this file in json file format. This is not '
'the decompiled replay json from rattletrap.')
help='The result of the analysis will be saved to this file in json file format.')
parser.add_argument('--gzip', type=str, required=False,
help='The pandas data frame containing the replay frames will be saved to this file in a '
'compressed gzip format.')
parser.add_argument('-sd', '--skip-decompile', action='store_true', default=False,
help='If set, carball will treat the input file as a json file that Rattletrap outputs.')
parser.add_argument('-v', '--verbose', action='count', default=0,
help='Set the logging level to INFO. To set the logging level to DEBUG use -vv.')
parser.add_argument('-s', '--silent', action='store_true', default=False,
Expand All @@ -47,13 +42,7 @@ def main(program_args=None):
else:
logging.basicConfig(handlers=[logging.StreamHandler()], level=log_level)

if args.skip_decompile:
game = Game()
game.initialize(loaded_json=args.input)
manager = AnalysisManager(game)
manager.create_analysis()
else:
manager = carball.analyze_replay_file(args.input)
manager = carball.analyze_replay_file(args.input)

if args.proto:
with open(args.proto, 'wb') as f:
Expand Down
22 changes: 9 additions & 13 deletions carball/decompile_replays.py
Original file line number Diff line number Diff line change
@@ -1,44 +1,40 @@
import logging
import os
from boxcars_py import parse_replay

from carball.analysis.analysis_manager import AnalysisManager
from carball.controls.controls import ControlsCreator
from carball.extras.per_goal_analysis import PerGoalAnalysis
from carball.json_parser.game import Game
from carball.json_parser.sanity_check.sanity_check import SanityChecker
from carball.rattletrap import run_rattletrap

BASE_DIR = os.path.dirname(__file__)


def decompile_replay(replay_path, output_path: str = None, overwrite: bool = True, rattletrap_path: str = None):
def decompile_replay(replay_path):
"""
Takes a path to the replay and outputs the json of that replay.
:param replay_path: Path to a specific replay.
:param output_path: The output path of rattletrap.
:param overwrite: True if we should recreate the json even if it already exists.
:param rattletrap_path: Custom location for rattletrap executable. Path to folder.
:return: The json created from rattle trap.
:return: The object created from boxcars.
"""
return run_rattletrap.decompile_replay(replay_path, output_path, overwrite, rattletrap_path)
with open(replay_path, 'rb') as f:
buf = f.read()
return parse_replay(buf)


def analyze_replay_file(replay_path: str, output_path: str = None, overwrite=True, controls: ControlsCreator = None,
sanity_check: SanityChecker = None, analysis_per_goal=False, rattletrap_path: str = None,
def analyze_replay_file(replay_path: str, controls: ControlsCreator = None,
sanity_check: SanityChecker = None, analysis_per_goal=False,
logging_level=logging.NOTSET,
calculate_intensive_events: bool = False,
clean: bool = True):
"""
Decompile and analyze a replay file.
:param replay_path: Path to replay file
:param output_path: Path to write JSON
:param overwrite: If to overwrite JSON (suggest True if speed is not an issue)
:param controls: Generate controls from the replay using our best guesses (ALPHA)
:param sanity_check: Run sanity check to make sure we analyzed correctly (BETA)
:param analysis_per_goal: Runs the analysis per a goal instead of the replay as a whole
:param rattletrap_path: Custom location for rattletrap executable. Path to folder.
:param force_full_analysis: If True full analysis will be performed even if checks say it should not.
:param logging_level: Sets the logging level globally across carball
:param calculate_intensive_events: Indicates if expensive calculations should run to include additional stats.
Expand All @@ -49,7 +45,7 @@ def analyze_replay_file(replay_path: str, output_path: str = None, overwrite=Tru
if logging_level != logging.NOTSET:
logging.getLogger('carball').setLevel(logging_level)

_json = decompile_replay(replay_path, output_path=output_path, overwrite=overwrite, rattletrap_path=rattletrap_path)
_json = decompile_replay(replay_path)
game = Game()
game.initialize(loaded_json=_json)
# get_controls(game) # TODO: enable and optimise.
Expand Down
2 changes: 1 addition & 1 deletion carball/json_parser/actor/ball.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
if self.parser.game.ball_type is None:
self.parser.game.ball_type = BALL_TYPES.get(actor['TypeName'], mutators.DEFAULT)

ball_data = BallActor.get_data_dict(actor, self.parser.replay_version)
ball_data = BallActor.get_data_dict(actor)
self.parser.ball_data[frame_number] = ball_data

if self.parser.game.ball_type == mutators.BREAKOUT:
Expand Down
2 changes: 1 addition & 1 deletion carball/json_parser/actor/boost.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class BoostHandler(BaseActorHandler):
type_name = 'Archetypes.CarComponents.CarComponent_Boost'

def update(self, actor: dict, frame_number: int, time: float, delta: float) -> None:
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', None)
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', {}).get('actor', None)

if car_actor_id is None or car_actor_id not in self.parser.current_car_ids_to_collect:
return
Expand Down
2 changes: 1 addition & 1 deletion carball/json_parser/actor/camera.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
if 'TAGame.CameraSettingsActor_TA:PRI' not in actor:
return

player_actor_id = actor['TAGame.CameraSettingsActor_TA:PRI'] # may need to try another key
player_actor_id = actor['TAGame.CameraSettingsActor_TA:PRI']['actor'] # may need to try another key
# add camera settings
if player_actor_id not in self.parser.cameras_data and \
'TAGame.CameraSettingsActor_TA:ProfileSettings' in actor:
Expand Down
17 changes: 8 additions & 9 deletions carball/json_parser/actor/car.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
if 'Engine.Pawn:PlayerReplicationInfo' not in actor:
return

player_actor_id = actor['Engine.Pawn:PlayerReplicationInfo']
player_actor_id = actor['Engine.Pawn:PlayerReplicationInfo']['actor']
if player_actor_id == -1:
self.add_demo(actor, frame_number)
return
Expand All @@ -23,12 +23,12 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
RBState = actor.get(REPLICATED_RB_STATE_KEY, {})
# bDriving is missing?! TODO: Investigate bDriving in RBState
# car_is_driving = RBState.get("rigid_body_state", {}).get("TAGame.Vehicle_TA:bDriving", False)
car_is_sleeping = RBState.get("rigid_body_state", {}).get('sleeping', True)
car_is_sleeping = RBState.get('sleeping', True)
# only collect data if car is driving and not sleeping
if not car_is_sleeping:
self.parser.current_car_ids_to_collect.append(actor['Id'])

data_dict = CarActor.get_data_dict(actor, version=self.parser.replay_version)
data_dict = CarActor.get_data_dict(actor)
# save data from here
self.parser.player_data[player_actor_id][frame_number].update(data_dict)

Expand All @@ -38,7 +38,7 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
if player_actor_id not in self.parser.car_dicts:
self.parser.car_dicts[player_actor_id] = {'team_paint': {}}

team_paint = actor['TAGame.Car_TA:TeamPaint']['team_paint']
team_paint = actor['TAGame.Car_TA:TeamPaint']

self.parser.car_dicts[player_actor_id]['team_paint'][team_paint['team']] = {
'primary_color': team_paint['primary_color'],
Expand All @@ -49,12 +49,11 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N

def add_demo(self, actor, frame_number):
if 'TAGame.Car_TA:ReplicatedDemolish' in actor:
demo_data = actor['TAGame.Car_TA:ReplicatedDemolish']['demolish']
demo_data = actor['TAGame.Car_TA:ReplicatedDemolish']
# add attacker and victim player ids
attacker_car_id = demo_data['attacker_actor_id']
victim_car_id = demo_data['victim_actor_id']
if attacker_car_id != -1 and victim_car_id != -1 and \
attacker_car_id < 1e9 and victim_car_id < 1e9:
attacker_car_id = demo_data['attacker']
victim_car_id = demo_data['victim']
if attacker_car_id != -1 and victim_car_id != -1 and attacker_car_id < 1e9 and victim_car_id < 1e9:
# Filter out weird stuff where it's not a demo
# frame 1 of 0732D41D4AF83D610AE2A988ACBC977A (rlcs season 4 eu)
attacker_player_id = self.parser.car_player_ids[attacker_car_id]
Expand Down
21 changes: 4 additions & 17 deletions carball/json_parser/actor/dropshot.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,11 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
if tile_id not in self.parser.dropshot['tile_states']:
self.parser.dropshot['tile_states'][tile_id] = 0

damage_state = actor['TAGame.BreakOutActor_Platform_TA:DamageState']['damage_state']
damage_state = actor['TAGame.BreakOutActor_Platform_TA:DamageState']

# unknown1: 0 - undamaged, 1 - damaged, 2 - destroyed
state = damage_state['unknown1']

# unknown2: False when undamaged, True otherwise?

# unknown3: damaging player actor id
player_actor_id = damage_state['unknown3']

# unknown4: (size, bias, x, y, z) properties that have a value of (0, 2, 0, 0, 0) when the tile is undamaged
# probably the position of the ball when the tile was damaged as tiles in the same event have the same value

# unknown5: In a single damage event only one tile has a value of True, the others are False
# probably the center of the damage aka the tile that was hit
tile_hit = damage_state['unknown5']

# unknown6: seems to be always False
state = damage_state['tile_state']
player_actor_id = damage_state['offender']
tile_hit = damage_state['direct_hit']

if state > self.parser.dropshot['tile_states'][tile_id]:

Expand Down
5 changes: 4 additions & 1 deletion carball/json_parser/actor/game_event.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ class GameEventHandler(BaseActorHandler):

@classmethod
def can_handle(cls, actor: dict) -> bool:
return actor['ClassName'].startswith('TAGame.GameEvent_Soccar_TA') or actor['ClassName'].startswith('TAGame.GameEvent_GodBall_TA')
if actor['ClassName'] is not None:
return actor['ClassName'].startswith('TAGame.GameEvent_Soccar_TA') or actor['ClassName'].startswith('TAGame.GameEvent_GodBall_TA')
else:
return False

def update(self, actor: dict, frame_number: int, time: float, delta: float) -> None:
self.parser.soccar_game_event_actor = actor
Expand Down
2 changes: 1 addition & 1 deletion carball/json_parser/actor/jump.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ def __init__(self, parser: object, data_key: str):
self.data_key = data_key

def update(self, actor: dict, frame_number: int, time: float, delta: float) -> None:
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', None)
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', {}).get('actor', None)
if car_actor_id is not None:
if car_actor_id in self.parser.current_car_ids_to_collect:
player_actor_id = self.parser.car_player_ids[car_actor_id]
Expand Down
45 changes: 16 additions & 29 deletions carball/json_parser/actor/player.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,50 +16,37 @@ def update(self, actor: dict, frame_number: int, time: float, delta: float) -> N
'name': actor["Engine.PlayerReplicationInfo:PlayerName"],
}
# Conditionally add ['team'] key to player_dict
player_team = actor.get("Engine.PlayerReplicationInfo:Team", None)
player_team = actor.get("Engine.PlayerReplicationInfo:Team", {}).get('actor', None)
if player_team is not None and player_team != -1:
player_dict['team'] = player_team

if "TAGame.PRI_TA:PartyLeader" in actor:
if "TAGame.PRI_TA:PartyLeader" in actor and actor["TAGame.PRI_TA:PartyLeader"] is not None:
try:
actor_type = \
list(actor["Engine.PlayerReplicationInfo:UniqueId"]['unique_id'][
'remote_id'].keys())[
0]
actor_type = list(actor["Engine.PlayerReplicationInfo:UniqueId"]['remote_id'].keys())[0]

# handle UniqueID for plays_station and switch
unique_id = None
if actor_type == "play_station" or actor_type == "psy_net":
if actor_type == "PlayStation" or actor_type == "PsyNet":
actor_name = actor["Engine.PlayerReplicationInfo:PlayerName"]
for player_stat in self.parser.game.properties['PlayerStats']['value']["array"]:
if actor_name == player_stat['value']['Name']['value']['str']:
unique_id = str(player_stat['value']['OnlineID']['value']['q_word'])
for player_stat in self.parser.game.properties['PlayerStats']:
if actor_name == player_stat['Name']:
unique_id = str(player_stat['OnlineID'])
if unique_id is None:
unique_id = str(
actor['Engine.PlayerReplicationInfo:UniqueId']['unique_id']['remote_id'][actor_type])
unique_id = str(actor['Engine.PlayerReplicationInfo:UniqueId']['remote_id'][actor_type])

# only process if party_leader id exists
if "party_leader" in actor["TAGame.PRI_TA:PartyLeader"] and \
"id" in actor["TAGame.PRI_TA:PartyLeader"]["party_leader"]:
leader_actor_type = list(
actor["TAGame.PRI_TA:PartyLeader"]["party_leader"]["id"][0].keys()
)[0]
if "remote_id" in actor["TAGame.PRI_TA:PartyLeader"]:
leader_actor_type = list(actor["TAGame.PRI_TA:PartyLeader"]["remote_id"].keys())[0]
leader = None
if leader_actor_type == "play_station" or leader_actor_type == "psy_net":
leader_name = actor[
"TAGame.PRI_TA:PartyLeader"
]["party_leader"]["id"][0][leader_actor_type][0]
if leader_actor_type == "PlayStation" or leader_actor_type == "PsyNet":
leader_name = actor["TAGame.PRI_TA:PartyLeader"]["remote_id"][leader_actor_type]['name']

for player_stat in self.parser.game.properties['PlayerStats']['value']["array"]:
if leader_name == player_stat['value']['Name']['value']['str']:
leader = str(player_stat['value']['OnlineID']['value']['q_word'])
for player_stat in self.parser.game.properties['PlayerStats']:
if leader_name == player_stat['Name']:
leader = str(player_stat['OnlineID'])

if leader is None: # leader is not using play_station nor switch (ie. xbox or steam)
leader = str(
actor[
"TAGame.PRI_TA:PartyLeader"
]["party_leader"]["id"][0][leader_actor_type]
)
leader = str(actor["TAGame.PRI_TA:PartyLeader"]["remote_id"][leader_actor_type])

if leader in self.parser.parties:
if unique_id not in self.parser.parties[leader]:
Expand Down
2 changes: 1 addition & 1 deletion carball/json_parser/actor/rumble.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ def can_handle(cls, actor: dict) -> bool:
return actor['TypeName'].startswith('Archetypes.SpecialPickups.SpecialPickup_')

def update(self, actor: dict, frame_number: int, time: float, delta: float) -> None:
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', None)
car_actor_id = actor.get('TAGame.CarComponent_TA:Vehicle', {}).get('actor', None)
if car_actor_id is not None and car_actor_id in self.parser.current_car_ids_to_collect:
player_actor_id = self.parser.car_player_ids[car_actor_id]
item_name = actor['TypeName'] \
Expand Down
Loading

5 comments on commit 5d4385d

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carball Benchmarks short_dropshot

Benchmark suite Current: 5d4385d Previous: 7f188c8 Ratio
carball/tests/benchmarking/benchmarking.py::test_short_dropshot 0.7242620648921544 iter/sec (stddev: 0.013851141960900187) 0.6077960552972108 iter/sec (stddev: 0.14301398941268187) 0.84

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carball Benchmarks intensive_oce_rlcs

Benchmark suite Current: 5d4385d Previous: 7f188c8 Ratio
carball/tests/benchmarking/benchmarking.py::test_intensive_oce_rlcs 0.06628853108761427 iter/sec (stddev: 0.16453871651497817) 0.06158342967316056 iter/sec (stddev: 0.19138208928114414) 0.93

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carball Benchmarks short_sample

Benchmark suite Current: 5d4385d Previous: 7f188c8 Ratio
carball/tests/benchmarking/benchmarking.py::test_short_sample 0.8343127981107309 iter/sec (stddev: 0.006526974199851653) 0.8930344604838997 iter/sec (stddev: 0.010219842775693095) 1.07

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carball Benchmarks full_rumble

Benchmark suite Current: 5d4385d Previous: 7f188c8 Ratio
carball/tests/benchmarking/benchmarking.py::test_full_rumble 0.06857996977966617 iter/sec (stddev: 0.3222570123747885) 0.05764721815835153 iter/sec (stddev: 0.6732893019363476) 0.84

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carball Benchmarks oce_rlcs

Benchmark suite Current: 5d4385d Previous: 7f188c8 Ratio
carball/tests/benchmarking/benchmarking.py::test_oce_rlcs 0.059391772044466225 iter/sec (stddev: 0.20357461942764168) 0.06239892238499711 iter/sec (stddev: 0.5863510863501828) 1.05

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.