FF Importer - Cloudflare error 524 if proxy enable

[rubbo898]

Support guidelines

• I've read the support guidelines

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem
• ... it's definitely a Firefly III issue, not me

Description

Hi,

first of all thanks for the GREAT software, it's very solid with a huge amount of flexibility and available customizations.
I'm not sure this is a issue, it could be a "gray area".
My configuration is exposed via a domain hosted and handled in Cloudflare, it points to my public interface and forwarded to a reverse proxy (Traefik) via https. This is both for Firefly and the importer, they have differents subdomains ma follow the very same path.
Everything works fine except an error 524 for long-running import jobs: "AxiosError: Request failed with status code 524"

It's not a bug but an icompatibility issue with a specific Cloudflare configuration (proxyed target + free tier plan). Without any code change there are 2 options:

1. Disable proxy for a specific endpoint
2. Upgrade to Enterprise plan

More details in [https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-524-a-timeout-occurred](Error 524: a timeout occurred)

Google returns a Github repo with a possible solution --> https://github.com/marcialpaulg/Fixing-Cloudflare-Error-524
I'm not a developer, if it's not an option please just ignore.

Again this is not a blocking issue and workarounds do exist, my goal is to highlight this specific scenario and let someone more skilled to evaluate the solutions, beside the ones posted above.

If you need any further details just let me know.

Last but not the least --> The conversion and import jobs complete without any error in background!

Debug information

[2025-02-09 11:26:36] local.DEBUG: Found 339 transactions on the drive.  
[2025-02-09 11:26:36] local.DEBUG: Now have 339 transaction(s) in RoutineManager  
[2025-02-09 11:26:36] local.DEBUG: Now in setSubmissionStatus(submission_running)  
[2025-02-09 11:26:36] local.DEBUG: Found "submission-eADFVl8f2OLn" in the session  
[2025-02-09 11:26:36] local.DEBUG: Now in startOrFindJob(submission-eADFVl8f2OLn)  
[2025-02-09 11:26:36] local.DEBUG: Try to see if file exists for job submission-eADFVl8f2OLn.  
[2025-02-09 11:26:36] local.DEBUG: Status file exists for job submission-eADFVl8f2OLn.  
[2025-02-09 11:26:36] local.DEBUG: Status: waiting_to_start  
[2025-02-09 11:26:36] local.DEBUG: Messages: 0, warnings: 0, errors: 0  
[2025-02-09 11:26:36] local.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(submission-eADFVl8f2OLn): submission_running  
[2025-02-09 11:26:36] local.DEBUG: Messages: 0, warnings: 0, errors: 0  
[2025-02-09 11:26:36] local.DEBUG: Created APISubmitter in RoutineManager  
[2025-02-09 11:26:36] local.DEBUG: Start of shared import routine.  
[2025-02-09 11:26:36] local.DEBUG: First collect account information from Firefly III.  
[2025-02-09 11:26:36] local.DEBUG: Now in collectAccountTypes()  
[2025-02-09 11:26:36] local.DEBUG: has key in cache?  
[2025-02-09 11:26:36] local.DEBUG: generateKey("http://firefly:8080-collectAccountTypes", "eyJ0eXAiOi...") results in "0337945ed0..."  
[2025-02-09 11:26:36] local.DEBUG: has key "0337945ed0" in cache? no  
[2025-02-09 11:26:36] local.DEBUG: Get response fresh!  
[2025-02-09 11:26:36] local.DEBUG: isReadyForFileStep("submit")  
[2025-02-09 11:26:36] local.DEBUG: isReadyForFileStep: Return true  
[2025-02-09 11:26:36] local.DEBUG: Now at App\Http\Controllers\Import\SubmitController::status(submission-eADFVl8f2OLn)  
[2025-02-09 11:26:36] local.DEBUG: Now in startOrFindJob(submission-eADFVl8f2OLn)  
[2025-02-09 11:26:36] local.DEBUG: Try to see if file exists for job submission-eADFVl8f2OLn.  
[2025-02-09 11:26:36] local.DEBUG: Status file exists for job submission-eADFVl8f2OLn.  
[2025-02-09 11:26:36] local.DEBUG: Status: submission_running  
[2025-02-09 11:26:36] local.DEBUG: Messages: 0, warnings: 0, errors: 0  
xxx.xxx.xxx.xxx- - [09/Feb/2025:11:26:36 +0100] "GET /import/submit/status?identifier=submission-eADFVl8f2OLn HTTP/1.1" 200 90 "https://ff-importer.#############/import/submit"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36" "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx"
[2025-02-09 11:26:36] local.DEBUG: Collected 3 account(s) in collectAccountTypes()  
[2025-02-09 11:26:36] local.DEBUG: set key forever!  
[2025-02-09 11:26:36] local.DEBUG: generateKey("http://firefly:8080-collectAccountTypes", "eyJ0eXAiOi...") results in "0337945ed0..."  
[2025-02-09 11:26:36] local.DEBUG: Now starting submission by calling API Submitter  
[2025-02-09 11:26:36] local.DEBUG: Custom tag is "importcsv", parsed into "importcsv"  
[2025-02-09 11:26:36] local.INFO: Going to submit 339 transactions to your Firefly III instance.  
[2025-02-09 11:26:36] local.DEBUG: Vanity URL : "https://firefly.#############";  
[2025-02-09 11:26:36] local.DEBUG: Now submitting transaction 1/339  
[2025-02-09 11:26:36] local.DEBUG: Duplicate detection method is "none", so this method is skipped (return true).  
[2025-02-09 11:26:36] local.DEBUG: Transaction #1 is not checked beforehand on uniqueness.  
[2025-02-09 11:26:36] local.DEBUG: Going to map data for this line.  
[2025-02-09 11:26:36] local.DEBUG: Submitting to Firefly III: {#########################################}  
[2025-02-09 11:26:37] local.INFO: [s] Add message to index #1 ###########################  
[2025-02-09 11:26:37] local.DEBUG: Add message on index #1 #############################
[2025-02-09 11:26:37] local.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(submission-eADFVl8f2OLn): submission_running  
[2025-02-09 11:26:37] local.DEBUG: Messages: 1, warnings: 0, errors: 0  
[2025-02-09 11:26:37] local.INFO: Created withdrawal <a target="_blank" href="https://firefly.#############/transactions/show/455";>############################## 

Expected behaviour

Complete the jobs without the error "AxiosError: Request failed with status code 524"

Steps to reproduce

1. Use a public URL exposed via Cloudflare proxied endpoint
2. Start a new import job
3. Configure all the parameters
4. Use a csv with many entries (mine was 360+ transactions)
5. Wait 100 seconds

Additional info

No response

[JC5]

https://docs.firefly-iii.org/references/faq/data-importer/import/#i-get-504-gateway-timeout-errors

Hey, thanks for opening an issue, and thanks for the compliments!

This is something I can't fix. Long-running imports will time out eventually. The front-end will stop checking after about 600 seconds, but like the FAQ says, reverse proxies may be more strict about it.

Ironically, like you say, the php process in the back-end will usually not be killed and will continue until it's actually finished. But the front-end won't be able to report it.

If you import large batches, if you have a lot of rules or something else is slow, use the CLI or smaller sets of files.

[rubbo898]

I completely understand your points, I'd do the same. Also, is something very specific related to Cloudflare and its WAF free plan limitations.

Again, great job!

[JC5]

Thanks!