Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bands input in load_collection causes error #28

Closed
lforesta opened this issue Oct 16, 2019 · 4 comments
Closed

bands input in load_collection causes error #28

lforesta opened this issue Oct 16, 2019 · 4 comments
Assignees

Comments

@lforesta
Copy link

In a process graph, when adding the bands input, e.g.

"dc": {
    "process_id": "load_collection",
    "arguments": {
      "id": "CGS_SENTINEL2_RADIOMETRY_V102_001",
      "spatial_extent": {
        "west": 2.052030657924054,
        "east": 4.063236553549667,
        "north": 51.00726308446294,
        "south": 50.99458367677388,
        "crs": "EPSG:4326"
        },
      "temporal_extent": [
        "2018-06-04T00:00:00.000Z",
        "2018-06-23T00:00:00.000Z"
      ],
      "bands": ["8", "4"]
    },
    "result": false
  }

the back-end returns the following error:

{"message":"An error occurred while calling z:org.openeo.geotrellis.geotiff.package.saveStitched.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 23 in stage 266.0 failed 4 times, most recent failure: Lost task 23.3 in stage 266.0 (TID 35790, epod53.vgt.vito.be, executor 147): java.lang.ClassCastException\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)\n\tat org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:363)\n\tat org.apache.spark.rdd.RDD.collect(RDD.scala:944)\n\tat geotrellis.spark.stitch.SpatialTileLayoutRDDStitchMethods.stitch(StitchRDDMethods.scala:81)\n\tat org.openeo.geotrellis.geotiff.package$.saveStitched(package.scala:39)\n\tat org.openeo.geotrellis.geotiff.package$.saveStitched(package.scala:28)\n\tat org.openeo.geotrellis.geotiff.package.saveStitched(package.scala)\n\tat sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.ClassCastException\n"}

If the bands input is not given, everything works fine. If needed I can provide full examples for this.

@soxofaan
Copy link
Member

which backend url are you using?
a full code example that triggers the issue would also help indeed

@lforesta
Copy link
Author

I am using this url: http://openeo.vgt.vito.be/openeo/0.4.0

The following process graph works (no bands specified anywhere):

{
  "dc": {
    "process_id": "load_collection",
    "arguments": {
      "id": "CGS_SENTINEL2_RADIOMETRY_V102_001",
      "spatial_extent": {
        "west": 2.052030657924054,
        "east": 4.063236553549667,
        "north": 51.00726308446294,
        "south": 50.99458367677388,
        "crs": "EPSG:4326"
        },
      "temporal_extent": [
        "2018-06-04T00:00:00.000Z",
        "2018-06-23T00:00:00.000Z"
      ]
    },
    "result": false
  },
  "ndvi": {
    "process_id": "ndvi",
    "arguments": {
      "data": {"from_node": "dc"}
    }
  },
  "min_time": {
    "process_id": "reduce",
    "arguments": {
      "data": {
        "from_node": "ndvi"
      },
      "reducer": {
        "callback": {
          "min": {
            "arguments": {
              "data": {
                "from_argument": "data"
              }
            },
            "process_id": "min",
            "result": true
          }
        }
      },
      "dimension": "temporal"
    }
  },
  "save_output": {
    "process_id": "save_result",
    "arguments": {
      "data": {
        "from_node": "min_time"
      },
      "format": "Gtiff"
    },
    "result": true
  }
}

Adding the input bands with band names "nir" and "red" in the first node "dc" causes the error above

"dc": {
    "process_id": "load_collection",
    "arguments": {
      "id": "CGS_SENTINEL2_RADIOMETRY_V102_001",
      "spatial_extent": {
        "west": 2.052030657924054,
        "east": 4.063236553549667,
        "north": 51.00726308446294,
        "south": 50.99458367677388,
        "crs": "EPSG:4326"
        },
      "temporal_extent": [
        "2018-06-04T00:00:00.000Z",
        "2018-06-23T00:00:00.000Z"
      ],
      "bands": ["nir", "red"]
    },
    "result": false
  }

If the bands are specified by their band_id ["8", "4"] (collection metadata), there is an integer/string error:
{"message":"java.lang.String cannot be cast to java.lang.Integer"}

If the bands are specified as integers [8, 4], they don't exist:
{"message":"Band 8 does not exist"}

@soxofaan
Copy link
Member

thanks for the info
is probably also related to Open-EO/openeo-python-client#76

@soxofaan soxofaan transferred this issue from Open-EO/openeo-python-driver Nov 4, 2019
soxofaan added a commit to Open-EO/openeo-python-client that referenced this issue Nov 4, 2019
soxofaan added a commit that referenced this issue Nov 5, 2019
…on (#28, #29)

related to issue #28
needs a more in-depth solution: see #29
@soxofaan
Copy link
Member

soxofaan commented Nov 5, 2019

should be fixed with commits above
(still to be deployed at the moment)

@soxofaan soxofaan closed this as completed Nov 5, 2019
soxofaan added a commit that referenced this issue Nov 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants