JSON file read/write errors / corruption causing crash

[Lntnam]

Expected Behavior

Bot can start normally

Actual Behavior

Bot could not start with ValueError("No JSON object could be decoded")

Your FULL config.json (remove your username, password, gmapkey and any other private info)

{
    "auth_service": "google",
    "username": "",
    "password": "",
    "location": "taodanin",
    "favorite_locations":[
        {"name": "home", "coords": "10.761636, 106.654087"},
        {"name": "bachdang", "coords": "10.775286, 106.706436"},
        {"name": "taodanout", "coords": "10.774686, 106.693133"},
        {"name": "taodanin", "coords": "10.774779, 106.691739"}
    ],
    "gmapkey": "",
    "encrypt_location": "",
    "websocket_server": false,
    "heartbeat_threshold": 10,
    "enable_social": false,
    "live_config_update": {
      "enabled": true,
      "tasks_only": true
    },
    "tasks": [
      {
        "//NOTE: This task MUST be placed on the top of task list": {},
        "type": "RandomAlivePause",
        "config": {
          "enabled": false,
          "min_duration": "00:00:10",
          "max_duration": "00:10:00",
          "min_interval": "00:05:00",
          "max_interval": "01:30:00"
        }
      },
      {
        "type": "HandleSoftBan"
      },
      {
        "type": "CompleteTutorial",
        "config": {
          "enabled": false,
          "// set a name": "",
          "nickname": "DurianKing"
        }
      },
      {
        "type": "CollectLevelUpReward",
        "config": {
          "collect_reward": true,
          "level_limit": -1
        }
      },
      {
        "type": "IncubateEggs",
        "config": {
          "enabled": true,
          "longer_eggs_first": true,
          "min_interval": 120,
         "infinite": [2,5,10],
         "breakable": [2,5,10]
        }
      },
      {
        "type": "UpdateLiveStats",
        "config": {
          "enabled": true,
          "min_interval": 600,
          "stats": ["uptime", "km_walked", "stops_visited", "xp_earned", "pokemon_stats", "most_perfect_pokemon"],
          "terminal_log": true,
          "terminal_title": true
        }
      },
      {
        "type": "TransferPokemon",
        "config": {
          "enabled": true,
          "min_free_slot": 9900,
          "transfer_wait_min": 5,
          "transfer_wait_max": 10
        }
      },
      {
        "type": "EvolvePokemon",
        "config": {
            "enabled": false,
            "evolve_all": "none",
            "evolve_iv_min": 0.8,

            "first_evolve_by": "iv",
            "evolve_above_cp": 0,
            "evolve_above_iv": 0.8,
            "logic": "and",
            "min_evolve_speed": 20,
            "max_evolve_speed": 30,
            "use_lucky_egg": false
        }
      },
      {
        "type": "NicknamePokemon",
        "config": {
          "enabled": true,
          "nickname_above_iv": 0,
          "dont_nickname_favorite": true,
          "nickname_template": "{name:.10s}{iv_pct2}"
        }
      },
      {
        "type": "RecycleItems",
        "config": {
          "min_empty_space": 15,
          "max_balls_keep": 160,
          "max_potions_keep": 40,
          "max_berries_keep": 60,
          "max_revives_keep": 40,
          "item_filter": {
            "Pokeball":       { "keep" : 60 },
            "Greatball":      { "keep" : 50 },
            "Ultraball":      { "keep" : 50 },
            "Potion":         { "keep" : 10 },
            "Super Potion":   { "keep" : 10 },
            "Hyper Potion":   { "keep" : 20 },
            "Revive":         { "keep" : 40 },
            "Razz Berry":     { "keep" : 60 }
          },
          "recycle_wait_min": 1,
          "recycle_wait_max": 4,
          "recycle_force": false,
          "recycle_force_min": "00:01:00",
          "recycle_force_max": "00:05:00"
        }
      },
      {
        "type": "CatchPokemon",
        "config": {
          "daily_catch_limit": 800,
          "treat_unseen_as_vip": true,
          "catch_visible_pokemon": true,
          "catch_lured_pokemon": true,
          "min_ultraball_to_keep": 5,
          "berry_threshold": 0.35,
          "vip_berry_threshold": 0.9,
          "catch_throw_parameters": {
            "excellent_rate": 0.1,
            "great_rate": 0.5,
            "nice_rate": 0.3,
            "normal_rate": 0.1,
            "spin_success_rate" : 0.6,
            "hit_rate": 0.9
          },
          "catch_simulation": {
            "flee_count": 2,
            "flee_duration": 2,
            "catch_wait_min": 2,
            "catch_wait_max": 6,
            "berry_wait_min": 2,
            "berry_wait_max": 3,
            "changeball_wait_min": 2,
            "changeball_wait_max": 3,
            "newtodex_wait_min": 20,
            "newtodex_wait_max": 30
          }
        }
      },
      {
        "type": "SpinFort",
        "config": {
          "enabled": true,
          "spin_wait_min": 2,
          "spin_wait_max": 5
        }
      },
      { "type": "UpdateWebInventory",
        "config": {
          "enabled": true
        }
      },
      {
        "type": "FollowPath",
        "config": {
          "enabled": true,
          "walker": "PolylineWalker",
          "path_mode": "linear",
          "path_start_mode": "closest",
          "path_file": "configs/path.q1.json",
          "number_lap": 10,
          "timer_restart_min": "00:10:00",
          "timer_restart_max": "00:20:00",
          "log_interval": 5
        }
      },
      {
        "type": "MoveToFort",
        "config": {
          "enabled": true,
          "lure_attraction": true,
          "lure_max_distance": 2000,
          "walker": "PolylineWalker"
        }
      }
    ],

    "map_object_cache_time": 5,
    "pokemon_bag": {
      "show_at_start": true,
      "show_count": true,
      "show_candies": true,
      "pokemon_info": ["cp", "ivcp", "level"]
    },
    "forts": {
      "avoid_circles": true,
      "max_circle_size": 40,
      "cache_recent_forts": true
    },
    "walk_max": 3.5,
    "walk_min": 1.8,
    "alt_min": 500,
    "alt_max": 1000,
    "gps_default_altitude": 8.0,
    "replicate_gps_xy_noise": false,
    "replicate_gps_z_noise": false,
    "gps_xy_noise_range": 0.000125,
    "gps_z_noise_range": 12.5,
    "walker_limit_output": true,
    "action_wait_min": 1,
    "action_wait_max": 4,
    "debug": false,
    "test": false,
    "health_record": false,
    "location_cache": true,
    "distance_unit": "km",
    "reconnecting_timeout": 20,
    "logging_color": true,

    "catch": {
      "any": {"catch_above_cp": 0, "catch_above_iv": 0, "logic": "or"}
    },

    "release": {
      "Mewtwo": {"release_below_iv": 0},
      "Dragonite": {"release_below_iv": 0},
      "Mew": {"release_below_iv": 0},
      "Moltres": {"release_below_iv": 0},
      "Snorlax": {"release_below_iv": 0},
      "Zapdos": {"release_below_iv": 0},
      "Lapras": {"release_below_iv": 0},
      "Articuno": {"release_below_iv": 0},
      "Arcanine": {"release_below_iv": 0},
      "Vaporeon": {"release_below_iv": 0},
      "Exeggutor": {"release_below_iv": 0},
      "Gyarados": {"release_below_iv": 0.6},
      "Flareon": {"release_below_iv": 0.6},
      "Muk": {"release_below_iv": 0.6},
      "Slowbro": {"release_below_iv": 0.6},
      "Machamp": {"release_below_iv": 0.6},
      "Venusaur": {"release_below_iv": 0.6},
      "Charizard": {"release_below_iv": 0.6},
      "Victreebel": {"release_below_iv": 0.6},
      "Pollwrath": {"release_below_iv": 0.7},
      "Blastoise": {"release_below_iv": 0.7},
      "Nidoking": {"release_below_iv": 0.7},
      "Nidoqueen": {"release_below_iv": 0.7},
      "Vileplume": {"release_below_iv": 0.7},

      "any": {"release_below_cp": 0, "release_below_iv": 0.9, "logic": "or", "keep_best_iv":1}
    },

    "vips" : {
         "Any pokemon put here directly force to use Berry & Best Ball to capture, to secure the capture rate!": {},
        "any": {"catch_above_cp": 1500, "catch_above_iv": 0.9, "logic": "or" },

        "Mewtwo": {},
        "Dragonite": {},
        "Mew": {},
        "Moltres": {},
        "Snorlax": {},
        "Zapdos": {},
        "Lapras": {},
        "Articuno": {},
        "Arcanine": {},
        "Vaporeon": {},
        "Exeggutor": {},
        "Gyarados": {},
        "Flareon": {},
        "Muk": {},
        "Slowbro": {},
        "Machamp": {},
        "Venusaur": {},
        "Charizard": {},
        "Victreebel": {},
        "Pollwrath": {},
        "Blastoise": {},
        "Nidoking": {},
        "Nidoqueen": {},
        "Vileplume": {}
    }
}

Output when issue occurred

Nam-MBP:pokemongo-bot namle$ ./run.sh
2016-08-28 11:45:36,378 [       cli] [INFO] PokemonGO Bot v1.0
2016-08-28 11:45:36,389 [       cli] [INFO] commit: ed095936e8fb2bdb7442c9966630664f0cb739ea
2016-08-28 11:45:36,392 [       cli] [INFO] Configuration initialized
2016-08-28 11:45:36,527 [PokemonGoBot] [INFO] [set_start_location] Setting start location.
2016-08-28 11:45:36,529 [PokemonGoBot] [INFO] Favorite location found: taodanin ([u'10.774779', u'106.691739'])
2016-08-28 11:45:36,529 [PokemonGoBot] [INFO] [location_found] Location found: taodanin (10.774779, 106.691739, 8.0)
2016-08-28 11:45:36,530 [PokemonGoBot] [INFO] [login_started] Login procedure started.
2016-08-28 11:45:39,175 [PokemonGoBot] [INFO] [login_successful] Login successful.
2016-08-28 11:45:39,175 [PokemonGoBot] [INFO] Found encrypt.so! Platform: darwin encrypt.so directory: /Users/namle/Downloads/PokemonGo-Bot
2016-08-28 11:45:39,176 [PokemonGoBot] [INFO] 
Traceback (most recent call last):
  File "pokecli.py", line 746, in <module>
    main()
  File "pokecli.py", line 115, in main
    bot = initialize(config)
  File "pokecli.py", line 83, in initialize
    bot.start()
  File "/Users/namle/Downloads/PokemonGo-Bot/pokemongo_bot/__init__.py", line 123, in start
    init_inventory(self)
  File "/Users/namle/Downloads/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1234, in init_inventory
    _inventory = Inventory(bot)
  File "/Users/namle/Downloads/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1086, in __init__
    self.refresh()
  File "/Users/namle/Downloads/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1096, in refresh
    self.update_web_inventory()
  File "/Users/namle/Downloads/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1116, in update_web_inventory
    json_inventory = json.load(infile)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Sun Aug 28 11:45:41 ICT 2016 Pokebot  Stopped.
Press any button or wait 20 seconds to continue.

Steps to Reproduce

Yesterday it was still running fine. Tried to start it for today and got this error.

Other Information

OS: OSx
Branch: master
Git Commit: ed09593
Python Version: 2.7.10
Any other relevant files/configs (eg: path files)
Path file:

[
  {"location": "10.774779, 106.691739", "loiter": 7200, "// tao dan": 0},
  {"location": "10.779082, 106.697320", "loiter": 1800, "// cong vien 30/4": 0},
  {"location": "10.787485, 106.706190", "loiter": 3600, "// so thu": 0},
  {"location": "10.782313, 106.705991", "loiter": 1200, "// ton duc thang": 0},
  {"location": "10.775163, 106.706552", "loiter": 7200, "// bach dang": 0},
  {"location": "10.771474, 106.698284", "loiter": 1800, "// tuong dai tran nguyen han": 0},
  {"location": "10.771254, 106.693135", "loiter": 1200, "// nga 6 phu dong": 0}
]

[leadboots5]

does your config file run through jsonlint fine?

[Lntnam]

@leadboots5 yup, valid JSON

[Lntnam]

I don't know why but after deleting everything and run installation again, the bot is back to work with exactly the same config files...

The only difference is I'm using dev branch now.

So you guys may keep this open for further debugging if necessary.

[rawgni]

it's something to do with inventory-USERNAME.json file under web/

[Gobberwart]

Yes, it is. The file can get corrupted if the bot is stopped while this file is open for write. Next time, just delete web/inventory-USERNAME.json and restart.

[Gobberwart]

I'll leave this open for debugging. Sorry, can't do it now. Will be tomorrow morning (AEST) then I'll see what I can do.

[rawgni]

@Gobberwart we have the inventory data in the database and probably can get it from the server. can't we just use that?

[Jasperrr91]

@Gobberwart have you been able to replicate this?

[mjmadsen]

@Jasperrr91 I personally have not, but we have had a decent number of folks who end up needing to delete one or more of the web json files to stop such errors.

[Jasperrr91]

Ah yes, I wrongfully assumed this was about the config.json but now I see it's about the web_inventory.

If anyone experiences these errors. Please post a dump of the web jsons. (Or are there already any posted @mjmadsen?)

[mjmadsen]

I don't think so @Jasperrr91

But as @rawgni said, we should be using our db instead (same with the live_update).

[Gobberwart]

@Jasperrr91 I have seen it happen. Not sure exactly what causes it tbh. Best guess is an IO conflict. It's a very rare occurrence which makes it tricky to debug.

@mjmadsen Agreed re the database. For now, I'll put some exception catching around the file reads/writes to hopefully suppress this crash, then we can look into using the db better.

[Gobberwart]

@rawgni Annoyingly, the inventory file contains both inventory and player data. Not sure who made that choice, but that's what the web interface wants. Overwriting the whole .json file with inventory data is simple, but the player data is (currently) not that simple and needs an api call. Therefore the method opens the .json file, removes all the "inventory" stuff, then writes it back from what's kept in the bot's internal memory, and keeps the player data in tact.

If there were separate outfiles for each (player data, inventory (items) and pokemons), that would be a lot simpler to work with.

[Jasperrr91]

@Gobberwart I think that choice was made because the inventory and player data is also grouped together in the API call. We're using an old commit of the Web interface so branching that and splitting the file shouldn't be too much of a hassle. But you still have to work with a JSON file that can give read/write errors.

So although it may simplify things, it will not solve this particular bug. What will solve this, is adding a try/catch. I'll put out a PR request in the morning.

Edit: Sorry, didn't read the rest of your first message. You already came up with that haha. Nvm me then.

[Gobberwart]

@Jasperrr91 I'm already working on try/catch. PR should be up in a minute.

[Gobberwart]

PR #4877 up for review.

[mjmadsen]

Closing #4841