pb2nc sporadic seg faults using -index

[georgemccabe]

Describe the Problem

Running pb2nc with the -index option succeeds sometimes but other times it causes a segmentation fault.

Expected Behavior

The index of input fields output every time instead of occasionally causing a segfault.

Environment

Describe your runtime environment:
1. Machine: kiowa (NCAR)
2. OS: Linux
3. Software version number(s): Bug was experienced using 8.1.1 and 9.1

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to kiowa
2. Change directory to:

 cd /d1/projects/MET/MET_Help/98219

3. Run this command multiple times:

/usr/local/met-9.1/bin/pb2nc gdas.adpupa.t00z.20160620.bufr out.nc PB2NCConfig_test -index

Error message should look like this:

DEBUG 1: Default Config File: /usr/local/met-9.1/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File: PB2NCConfig_local
DEBUG 1: Pre-processing BUFR File for variable names from gdas.adpupa.t00z.20160620.bufr
Segmentation fault

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to relevant bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
• Close this issue.

[JohnHalleyGotway]

Debugging this issue on kiowa in /d1/personal/johnhg/MET/MET_development/MET/met/bugfix_1618.
Running pb2nc from the main_v9.1 branch.
When running through the gdb debugger, the segfault never appears.
Switch to adding print statements and traced the segfault to a call to the UFBTAB function from BUFRLIB. The segfault occurs in there!

Here's output when it doesn't fail:

DEBUG 1: JHG: calling numpbmsg(unit, npbmsg)... unit = 22 and npbmsg = 21877
 JHG calling UFBTAB with FID2=         -22  and R8ARR=   6.9526598702468682E-310
 JHG back from UFBTAB.

Here's output when it does fail:

DEBUG 1: JHG: calling numpbmsg(unit, npbmsg)... unit = 22 and npbmsg = 21905
 JHG calling UFBTAB with FID2=         -22  and R8ARR=   6.9528904731617930E-310
Segmentation fault

[JohnHalleyGotway]

I confirmed that this problem still persists when linking against the latest version of BUFRLIB, version 11.3.0 in:
/d1/projects/MET/MET_releases/external_libs/gnu_6.3.0/bufrlib/BUFRLIB_v11.3.0

When I remove the -index option and reconfigure to process all variables and message type, I do get a successful run:

obs_bufr_var = [ ];
message_type = [ ];

But it does produce a warning:

WARNING: 
WARNING: process_pbfile() -> The observation time should remain the same for all Bufr messages
WARNING: process_pbfile()   251 messages with different reference time (20160620_000000):
WARNING: process_pbfile()	20160619_230000
WARNING: process_pbfile()	20160620_010000
WARNING: process_pbfile()	20160619_220000
WARNING: process_pbfile()	20160620_020000
WARNING: 

Re-assigning to Howard to investigate this more.
Howard, can you find any good reason for this sporadic segfault from the UFBTAB Fortran function?

[hsoh-u]

The same code works without -index options (MET calls openpb_ and numpbmsg_).
I will focus workarounds (different way to call APIs) instead of digging into the Fortran code.