Not able to debug UDF

[guruvonline]

HI ,
I am trying to debug my UDF, for testing i am limiting the dataframe to single row, but still when my UDF hits, i keep getting the debug window and it breaks into TaskRunner.Run, but doing F5 from there doesn't break into my UDF code. this is happening in continuous loop and every time it hit TaskRunner, but never reaches my UDF code.

any suggestions on what could be wrong.

Thanks

[Niharikadutta]

Kindly refer to this documentation to understand how to debug UDFs. You will need to add breakpoints in the new debugger instance of Visual Studio that pops up for each task.

[guruvonline]

thanks for reply. I am following the above mentioned article, but not able to hit breakpoint inside my UDF. i tried with another UDF and was able to hit break point, so not sure what I am doing wrong in this UDF, because of which it is not hitting break point.
Few additional details for scenario, where UDF debug is not working

• my UDF is taking Row and returning string

• I am starting debug session with --master local[1]

• DataFrame has only 1 row and i can do Show() to see the data before passing to UDF.
  |MyModel|
  +---------+
  | [USD]|
  +---------+

• Debugger breaks in TaskRunner.cs

• when i continue debugging and Step Into TaskRunner, in PicklingSqlCommandExecutor.ExecuteCore(), messageLength is coming -1.
  messageLength = SerDe.ReadInt32(inputStream).

In my another UDF scenario, where i am able to hit debug point in my UDF, i can see the messageLength > 0

Can you kindly suggest, what might i am doing wrong, due to which messageLenght is coming -1.

[Niharikadutta]

Could you please paste your code snippet of the failing UDF as well as the working UDF here?

[guruvonline]

to troubleshoot, i have removed all code from UDF and just returning dummy string. the difference in working and non-working UDF is the input DataFrame.

• working UDF, dataframe is created dummy as spark.CreateDataFrame()
• non working UDF- dataframe is created from my application code, by reading a CSV file. It's difficult to get a trimmed version of my application to repro.

Rest of the code in both cases is same. If i disable UDF debug, my UDF returns response.

In non workig UDF, Dataframe.show, display a single row. but still when debugger step into PicklingSqlCommandExecutor.ExecuteCore(), it's not reading any rows/data.

Can you kindly suggest what to check for DataFrame.

Thanks

[Niharikadutta]

Could you share the error response you get when you disable debug and submit your failing UDF as part of an application? Could you also share the output of DataFrame.PrintSchema() for the dataframe you are passing to your UDF (the one created by reading the CSV file)?