Skip to content

Commit

Permalink
Add error code for Spark Fabric test connection error (#472)
Browse files Browse the repository at this point in the history 
* Update diagnostics.md

* Update diagnostics.md
  • Loading branch information
@alexanderahn
alexanderahn authored Dec 16, 2024
1 parent eca548f commit 7850800
Showing 1 changed file with 19 additions and 16 deletions.
35 changes: 19 additions & 16 deletions docs/Spark/fabrics/diagnostics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,22 @@ Troubleshooting Prophecy Fabrics is much easier with built-in diagnostics. The d

## Diagnostics error codes

| Error Code | Symptom | Provider | Cause | Resolution |
| ---------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `10000` | `Is missing from the classpath` | [Databricks](/docs/Spark/fabrics/databricks.md#databricks) | Prophecy Library(Scala) is incorrect. You're probably using thin jar. | Use assembly `jar(${scalaFatJarName})` in the library section of the Fabric settings |
| `10001` | `DRIVER_LIBRARY_INSTALLATION_FAILURE` | [Databricks](/docs/Spark/fabrics/databricks.md#databricks) | Prophecy Library(Scala/Python) is incorrect. Databricks could not install it | Please provide the valid library path in the Fabric |
| `10002` | `object prophecy is not a member of package` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incorrect | Please ensure that the library path exists and you’re using the assembly `jar(${scalaFatJarName})` |
| `10003` | `cannot be added to user sessions` and `prophecy_libs` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) is incorrect | Please ensure that the library path exists and you’re using correct `file(${pythonPLibName})` |
| `10004` | `for method` and `too many arguments` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incompatible | Please use the correct `version(${Globals.prophecyLibsVersion})` in the library section of Fabric settings |
| `10005` | `No module named` and `prophecy` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) is incorrect | Please provide the valid library path in the Fabric |
| `10006` | `illegal start of simple expression` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Python version in livy/hadoop is incorrect | Please make sure you have python3 there |
| `10007` | `IncompatibleClassChangeError` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incompatible with your Spark version | Please use the correct assembly `jar(${scalaFatJarName})` in the library section of the Fabric settings. |
| `10008` | `"FileNotFoundException` and `prophecy_libs"` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) path does not exist | Please ensure that the file exists as per the path in the library section of the Fabric settings |
| `10009` | `503 Service Temporarily Unavailable` and `LivyRestClient` | [Livy](https://livy.apache.org/docs/latest/rest-api.html) | Livy service is down | Please make sure the livy service is up before executing this command |
| `10010` | `SQLNonTransientConnectionException, rds.amazonaws.com` or `Unable to instantiate, HiveMetaStoreClient` | [Unity Catalog](https://docs.databricks.com/en/resources/supported-regions.html#rds) | Databricks cluster can't access RDS service | Please ensure that the cluster can access to the same region's RDS endpoint as documented [here](https://docs.databricks.com/en/resources/supported-regions.html#rds) |
| `10011` | `UnauthorizedCommandException` and `This execution contained at leas` and `disallowed language` | [Unity Catalog](/docs/concepts/project/project.md#1-create-new-project) | Shared cluster in unity catalog does not allow Scala commands | Please use this cluster with Python Pipeline |
| `10012` | `UnauthorizedCommandException` and `This execution contained at leas` and `disallowed language` | [Databricks](https://docs.databricks.com/en/administration-guide/users-groups/index.html) | This cluster does not allow `${pipeline's language}` command | Please check with the Databricks workspace administrator to provide the execution access to `${pipeline's language}` language |
| `10013` | `javax.net.ssl.SSLHandshakeException` and `PKIX path building failed` | Livy / [EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-security.html) | Certificates provided in EMR cluster's security configuration are wrong | Please ensure that EMR cluster's security configuration is using correct certificates |
| Error Code | Symptom | Provider | Cause | Resolution |
| ---------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `10000` | `Is missing from the classpath` | [Databricks](/docs/Spark/fabrics/databricks.md#databricks) | Prophecy Library(Scala) is incorrect. You're probably using thin jar. | Use assembly `jar(${scalaFatJarName})` in the library section of the Fabric settings. |
| `10001` | `DRIVER_LIBRARY_INSTALLATION_FAILURE` | [Databricks](/docs/Spark/fabrics/databricks.md#databricks) | Prophecy Library(Scala/Python) is incorrect. Databricks could not install it. | Please provide the valid library path in the Fabric. |
| `10002` | `object prophecy is not a member of package` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incorrect. | Please ensure that the library path exists and you’re using the assembly `jar(${scalaFatJarName})`. |
| `10003` | `cannot be added to user sessions` and `prophecy_libs` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) is incorrect. | Please ensure that the library path exists and you’re using correct `file(${pythonPLibName})`. |
| `10004` | `for method` and `too many arguments` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incompatible. | Please use the correct `version(${Globals.prophecyLibsVersion})` in the library section of Fabric settings. |
| `10005` | `No module named` and `prophecy` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) is incorrect. | Please provide the valid library path in the Fabric. |
| `10006` | `illegal start of simple expression` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Python version in livy/hadoop is incorrect. | Please make sure you have python3 there. |
| `10007` | `IncompatibleClassChangeError` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Scala) is incompatible with your Spark version. | Please use the correct assembly `jar(${scalaFatJarName})` in the library section of the Fabric settings. |
| `10008` | `"FileNotFoundException` and `prophecy_libs"` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Prophecy Library(Python) path does not exist. | Please ensure that the file exists as per the path in the library section of the Fabric settings. |
| `10009` | `503 Service Temporarily Unavailable` and `LivyRestClient` | [Livy](https://livy.apache.org/docs/latest/rest-api.html) | Livy service is down. | Please make sure the livy service is up before executing this command. |
| `10010` | `SQLNonTransientConnectionException, rds.amazonaws.com` or `Unable to instantiate, HiveMetaStoreClient` | [Unity Catalog](https://docs.databricks.com/en/resources/supported-regions.html#rds) | Databricks cluster can't access RDS service. | Please ensure that the cluster can access to the same region's RDS endpoint as documented [here](https://docs.databricks.com/en/resources/supported-regions.html#rds). |
| `10011` | `UnauthorizedCommandException` and `This execution contained at leas` and `disallowed language` | [Unity Catalog](/docs/concepts/project/project.md#1-create-new-project) | Shared cluster in unity catalog does not allow Scala commands. | Please use this cluster with Python Pipeline. |
| `10012` | `UnauthorizedCommandException` and `This execution contained at leas` and `disallowed language` | [Databricks](https://docs.databricks.com/en/administration-guide/users-groups/index.html) | This cluster does not allow `${pipeline's language}` command. | Please check with the Databricks workspace administrator to provide the execution access to `${pipeline's language}` language. |
| `10013` | `javax.net.ssl.SSLHandshakeException` and `PKIX path building failed` | Livy / [EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-security.html) | Certificates provided in EMR cluster's security configuration are wrong. | Please ensure that EMR cluster's security configuration is using correct certificates. |
| `10014` | `HostUnreachableErrorCode_10014` | [Databricks](https://docs.databricks.com/en/administration-guide/users-groups/index.html) | Unable to reach Databricks endpoint. | Make sure the workspace is active and reachable. |
| `10015` | `HiveMetastoreNotEnabledErrorCode_10015` | Hive Metastore | We were unable to write execution metrics because Hive Metastore is not enabled on your Spark. | Please enable Hive Metastore on Spark, or disable execution metrics in Prophecy. |
| `10016` | `AuthenticationFAiled_10016` | [Livy](/docs/Spark/fabrics/fabrics.md#fabrics-using-apache-livy) | Authentication failed. Wrong or no auth credentials were provided. | Make sure correct auth credentials are provided. |

0 comments on commit 7850800

Please sign in to comment.