Simple working IDoc receiver using pyRFC...

[duanmark]

I’m curios to find out if anyone has managed to build an “IDoc Receiver” using the pyRFC Server, and if so, how was the tRFC/qRFC transactions handled within pyRFC for the received IDocs?

As stated previously in SAP/pyRFC#340 (comment) and SAP/node-rfc#279 (comment) receiving the IDocs is only half the battle, however I’m still curious to see if anyone has managed to get something working?

[Ulrich-Schmidt]

Not in Python, but here is a detailed example in C. If all required NW RFC functions have been exported to Python, you may be able to convert the C sample to Python?!

https://help.sap.com/docs/SUPPORT_CONTENT/abapconn/3354079405.html

See the PDF document for some instructions and the samples.zip for the C coding. (Can be compiled into a complete executable that receives and manages simple IDocs of type TXTRAW. There is also a client program, that can send TXTRAW IDocs.)

[duanmark]

Thank you @Ulrich-Schmidt!

The iDocServer.c example from the samples download on the SAP Help article contains the following C function call which is needed to register the associated functions (onCheckTID, onCommit etc.) which will be used to handle the tRFC/qRFC calls from the ABAP backend when the IDoc is received by the RFC Server:

rc = RfcInstallTransactionHandlers(NULL, onCheckTID, onCommit, onRollback, onConfirm, &errorInfo); if (rc != RFC_OK){ printfU(cU("Unable to install TransactionHandlers: %s: %s\n"), RfcGetRcAsString(errorInfo.code), errorInfo.message); return 1; }

I don't think this C function has been exposed in the pyRFC implementation. I can see that provision has been made for the handling of bgRFC and its associated handler functions to handle bgRFC units (onCheck, onCommit etc.) however, as far as I understand, IDocs are only sent using tRFC/qRFC and as such the current pyRFC Server implementation won't be able to handle iDocs it receives.

(Within the ABAP backend bgRFC can be used to process iDocs, but this is only after they have already been received).

[bsrdjan]

Experimental support is implemented in transaction_handlers branch, to be used as follows

1. Implement transactional rfc handlers in Python application, similar to bgRFC handlers example
2. Register application transaction handlers using Server transaction_rfc_init
3. Start server and receive IDocs

Two properties are provided, just like for bgRFC handlers, transaction_handlers and transaction_handlers_count

Can you build from source and test?

[duanmark]

Wow @bsrdjan, thank you so much for adding this so quickly!

I've managed to build the transaction_handlers branch from source, I will create a simple IDoc receiver and test the newly added functions and will let you know my results.

[duanmark]

@bsrdjan I wrote a very simple script to test if the pyRFC Server is able to process IDocs after you added the transaction_handlers.

I keep getting the following error in the dev_rfc.log file after sending the IDoc from the ABAP backend server:
RfcSystemException: Key: RFC_ABAP_RUNTIME_FAILURE, Code: RFC_ABAP_RUNTIME_FAILURE, Message: Application's transaction state database currently unavailable.

None of the registered transaction handler functions are executed. It looks like it is failing before it executes any of them.

Am I missing something?

[Ulrich-Schmidt]

The error message “Application's transaction state database currently unavailable” means that the onCheck-function was executed, but it returned an error code. So it looks like the registered C-function was called by the NW RFC lib, but it couldn’t find/execute the corresponding Python function?

[bsrdjan]

Is server logging activated and what is it telling? To activate, set server_log configuration option to True
https://github.com/SAP/PyRFC/blob/transaction_handlers/src/pyrfc/_cyrfc.pyx#L1831

The check handler should provide more info then:

    cdef RFC_RC __onCheckTransaction(RFC_CONNECTION_HANDLE rfcHandle, const SAP_UC *tid) with gil:
        handler = Server.__transactionHandler["check"]
        if not callable(handler):
            _server_log("Transaction check handler is not registered for server connection handle '{<uintptr_t>rfcHandle}'")
            return RCStatus.OK.value
        try:
            transaction_id = wrapString(tid)
            return handler(<uintptr_t>rfcHandle, transaction_id).value
        except Exception as ex:
            _server_log("Error in transaction check handler:", ex)
            return RCStatus.RFC_EXTERNAL_FAILURE.value

[duanmark]

Thanks @Ulrich-Schmidt, I fixed a syntax error in the Python script which resolved that problem. Both the C function was well as the onCheck Python function is now being executed successfully.

However, when it gets to actually executing the Python function that should handle the IDoc, the connection between the Python RFC Server and the ABAP backend gets broken. I get the the following message on the ABAP backend in transaction SM58: connection to partner '172.17.0.1:35960' broken /

@bsrdjan I added the server_log option, below my output for the simple Python IDoc receiver:

[2024-03-06 12:48:31.890969 UTC] Server '1962248167024 created'
[2024-03-06 12:48:32.011059 UTC] Server function IDOC_INBOUND_ASYNCHRONOUS 'installed'
[2024-03-06 12:48:32.011059 UTC] Server function IDOC_INBOUND_ASYNCHRONOUS '{'func_desc_handle': 1962247323776, 'callback': <function handle_idoc at 0x000001C8DEBC8B80>, 'server': <pyrfc._cyrfc.Server object at 0x000001C8DF32F400>}'
[2024-03-06 12:48:32.011565 UTC] Server 1962248167024 'Transaction handlers installed: 4'
[2024-03-06 12:48:32.011565 UTC] Transaction handler check '<function onCheckTransaction at 0x000001C8DEBC9BC0>'
[2024-03-06 12:48:32.011945 UTC] Transaction handler commit '<function onCommitTransaction at 0x000001C8DEBC9D00>'
[2024-03-06 12:48:32.012534 UTC] Transaction handler rollback '<function onRollbackTransaction at 0x000001C8DF371DA0>'
[2024-03-06 12:48:32.012534 UTC] Transaction handler confirm '<function onConfirmTransaction at 0x000001C8DF371D00>'
[2024-03-06 12:48:32.019468 UTC] Server '1962248167024 launched'
[2024-03-06 12:48:32.019468 UTC] Server '1962248167024 started'

Press Enter to stop server...
Executed onCheckTransaction Python function 1962247047024 AC110002852065E8662C0000
[2024-03-06 12:48:45.950294 UTC] metadataLookup 'Function 'IDOC_INBOUND_ASYNCHRONOUS' handle 1962247323776.'

The IDoc is sent from the backend server, the onCheckTransaction Python function is successfully executed and thereafter the connection gets broken.

[duanmark]

Based on the rfc trace files (.trc) it looks like the IDOC_INBOUND_ASYNCHRONOUS function is being called successfully:

Dispatching function IDOC_INBOUND_ASYNCHRONOUS on handle 1962247047024
Looking up ServerFunction for IDOC_INBOUND_ASYNCHRONOUS from A4H.
Looking up ServerFunction for IDOC_INBOUND_ASYNCHRONOUS from A4H.
Found user function in cache for A4H.
2024-03-06 14:48:45.951843 [32072] >> RfcFunction::rfcDeserialize
2024-03-06 14:48:45.951894 [32072] >> RfcConnection::readUpTo
...
2024-03-06 14:48:45.953507 [32072] >> Invoking server function implementation (user function)
2024-03-06 14:48:45.953544 [32072] >> RfcGetServerContext
2024-03-06 14:48:45.953567 [32072] << RfcGetServerContext returned RFC_OK

Looks like the problem is when it gets to executing the Python user function.

[bsrdjan]

After metadataLookup log entry, the genericHandler log entry shall follow, which is apparently not happening because connection is broken. We need the detailed RFC trace for further analysis. It is activated by adding RFC_TRACE=3 in DEFAULT section of sapnwrfc.ini file, used by Python application. Check sapnwrfc.ini in RFC SDK demo folder for further details:

# top of file
DEFAULT 
RFC_TRACE=3

Please repeat the test with detailed RFC trace and send us full RFC trace and Python server trace. If any sensitive data in traces, create customer incident for BC-MID-RFC-SDK component, attach the traces there, with reference to this thread.

[duanmark]

@bsrdjan attached the ABAP backend RFC trace files as well as the pyRFC trace files. (The dev_rfc.log file contains no entries).
Please let me know if you need any additional detail?

dev_rfc.log
dev_rfc5 ABAP Backend.txt
dev_rfc6 ABAP Backend.txt
rfc13688_07252 pyRFC.txt
rfc13688_15956 pyRFC.txt

[Ulrich-Schmidt]

The trace simply ends. Well, if you ask me, I would say that the “user function” crashes at that point. Do you see a core dump file being generated? How does the Python runtime behave, when a segmentation fault is caused in native code outside the virtual machine? I know that the Java VM writes a special dump file called “hs_err_<pid>.log” with detailed stack information about where the crash happened.

[bsrdjan]

How does the Python runtime behave, when a segmentation fault is caused in native code outside the virtual machine?

Python has faulthandler, let try it if following does not help.

@duanmark, can you send your python server test script, including idoc_inbound_asynchronous python handler function, so that I can try to reproduce? Also describe how the IDOC is created and sent.

I extended exceptions catching and pushed changes to transaction_handlers branch.

You can pull the changes, repeat the test and send also the PyRFC server log (redirect Python server console to log file). It was not included in already sent logs and could show more error info now

[duanmark]

Hello @bsrdjan, I built the transaction_handlers branch again and repeated the same test, still the same problem.
Here are the two Python server trace files as well as the console output:

console log.txt
rfc26440_15752.txt
rfc26440_26820.txt

The problem is not only for "IDOC Server" scenario, I created another simple test, where I try and handle STFC_WRITE_TO_TCPIC and then use the SAP standard RSARFCT0 report to generate tRFC function module calls which are to be handled by the pyRFC server. (Exactly the same as the IDoc scenario, with the same result).

Step 1: Set up pyRFC server to handle STFC_WRITE_TO_TCPIC:

Step 2: Run RSARFCT0 report on ABAP backend to generate STFC_WRITE_TO_TCPIC tRFC calls:

Step 3: Transaction SM58:

I hope this helps?

[bsrdjan]

I could reproduce the issue.

There was one PyRFC bug in transactional handler logic, now fixed.

There was also an error in your Python server function handle_rfc. Python server function should behave like corresponding ABAP function and return a dict instance, not RCStatus. That error was causing another exception, which is now exposed in server log, in case it occurs again.

You can now rebuild PyRFC from transaction_handlers branch and test with fixed test_trfc.py

[duanmark]

Excellent, thank you very much @bsrdjan, test_trfc.py is working as expected! 😁

I'm going to do some more tests using IDOC_INBOUND_ASYNCHRONOUS to receive actual IDocs on the pyRFC Server. Will let you know if I encounter any further problems.

[duanmark]

I initially received the following error while testing actual IDocs from an ABAP backend with IDOC_INBOUND_ASYNCHRONOUS:

[2024-03-11 14:35:38.410106 UTC] genericHandler 'Request for 'IDOC_INBOUND_ASYNCHRONOUS' raises ExternalRuntimeError 22 (rc=22): key=RFC_CONVERSION_FAILURE, message=PSGNUM of type RFCTYPE_NUM value=       cannot be converted to type RFCTYPE_NUM [MSG: class=, type=, number=, v1-4:=;;;] - code set to RFC_EXTERNAL_FAILURE.'

After changing the SM59 TCP/IP ABAP backend connection to Unicode, everything now works as expected:

I see the transaction_handlers branch has already been merged with main, is tRFC/qRFC now officially supported in the pyRFC server scenario?

[bsrdjan]

The support is still experimental and backward incompatible changes might happen.

Could you please describe how IDocs are sent in your test scenario?

[duanmark]

For testing the IDocs received in the pyRFC Server the following was done:

• On the ABAP backend, using transaction SM59, created a TCP/IP RFC destination (Type 'T') using the same name as the 'PROGRAM_ID' value used to start the pyRFC Server for the 'Registered Server Program' Program ID.
• Created an IDoc distribution model between the ABAP backend system and the PyRFC server (created as a new logical system), using transaction BD64.
• To this distribution model simple IDoc message types (ALEAUD, TXTRAW etc.) were added.
• Generated the partner profiles from the distribution model. (Transaction BD82).
• Validated partner profiles in transaction WE20, specifically the 'receiver port' for each of the Outbound IDoc message types for the pyRFC Server logical system.
• Used transaction WE19 to send 'test IDocs' from the ABAP backend, to the pyRFC Server where the server is registered to handle function IDOC_INBOUND_ASYNCHRONOUS, and all of the required tRFC/qRFC transaction handlers.

Using the above, the pyRFC server successfully received the IDocs sent from the ABAP backend.