This package evaluates the Average Path Length Similarity (APLS) metric to measure the difference between ground truth and proposal graphs. The metric sums the differences in optimal path lengths between all nodes in the ground truth graph G and the proposal graph G’. This metric was used to score the SpaceNet 3 challenge. For further details, see Blog1 and Blog2.
pip install apls
All dependencies can be found in environment.yml
See LICENSE.
apls.py compares ground truth and proposal graphs. The actual metric takes up a relatively small portion of this code (functions: single_path_metric, path_sim_metric_, and compute_metric). Much of the code (e.g. cut_linestring, get_closest_edge, insert_point_into_G, insert_control_points, create_graph_midpoints) is concerned with injecting nodes into graphs. We inject nodes (i.e. control points) into the graph at a predetermined distance along edges, and at the location nearest proposal nodes. Injecting nodes is essential to properly compare graphs, though it unfortunately requires quite a bit of code. graphTools.py parses geojson labels into networkx graphs for analysis. Examples for running apls.py are shown below with the attached sample data
# for further details: python apls.py --help
# 1. Compare a ground truth SpaceNet geojson with a submission csv
python apls.py --test_method=gt_json_prop_wkt --output_name=gt_json_prop_wkt \
--truth_dir=data/gt_json_prop_wkt/ground_truth_randomized \
--wkt_file=data/gt_json_prop_wkt/proposal/sn3_sample_submission_albu.csv \
--im_dir=data/images
# 2. Compare a ground truth geojson with a proposal json
python apls.py --test_method=gt_json_prop_json --output_name=gt_json_prop_json \
--truth_dir=data/gt_json_prop_json/AOI_2_Vegas_Train/spacenetroads \
--prop_dir=data/gt_json_prop_json/AOI_2_Vegas_Train/osm
# 3. Compare a ground truth geojson with a pickled proposal graph
python apls.py --test_method=gt_json_prop_pkl --output_name=gt_json_prop_pkl \
--truth_dir=data/gt_json_prop_pkl/ground_truth_randomized \
--prop_dir=data/gt_json_prop_pkl/proposal \
--im_dir=data/images
# 4. Compare a pickled ground truth graph with a pickled proposal graph
python apls.py --test_method=gt_pkl_prop_pkl --output_name=gt_pkl_prop_pkl \
--truth_dir=data/gt_pkl_prop_pkl/ground_truth_randomized \
--prop_dir=data/gt_pkl_prop_pkl/proposal \
--im_dir=data/images
Running apls.py yields a number of plots in the outputs directory, along with the APLS score
Run the create_spacenet_masks.py script to create training masks with spacenet geojsons