Amplify v6, Storage uploadData throws ECONNABORTED error consistently in React Native Testflight Production App #12912
Comments
nadetastic
added
Storage
|
Hi @ChristopherGabba thank you for opening this issue. Taking a look at this, it appears that this issue is intermittent, only affecting some users, and is not easily reproducible. Im working to test this on my end and one thought I had was possibly the size of the upload is causing an error where S3 aborts the connection. Are you testing with files that are larger than 100MB with no error? Im also working to verify the same, but it also seems that this issue may be specific to production builds. |
|
@nadetastic async function sendResponse() {
if (!response.sender) return
setIsLoading(true)
const client = generateClient()
try {
// Upload main phone media to S3
const compressedVideo = await compressVideo(uri)
const videoUri = await uploadToS3({ // caught the exception here
filePath: compressedVideo,
key: `${response.id}/videoUrl.mp4`,
options: {
accessLevel: "guest",
contentType: "video/mp4",
},
})
// Get the new result Uri and update dynamoDB
const apiResponse = await client.graphql({
query: updateResponse,
variables: {
input: {
id: response.id,
caption: response.caption,
videoUri: videoUri
},
},
})
console.log("successfully updated response", apiResponse)
await sendResponseNotification(response.sender?.pushToken, true, response?.caption)
await showConfirmation("common.sent")
} catch (error) {
reportError({
error,
})
}Error: ECONNABORTED , Description: Network Error It is a failure to upload to S3 |
|
@nadetastic If you would like me to modify my error output in some way to gather more information during the ECONNABORTED I can to help try and solve it. Feel free to modify my uploadToS3 function to try and see what step its occurring at. Also if there is a way to program this to try again if this occurs maybe two more times or something that would be awesome. Maybe modify to try two more times max to get a successful upload? |
|
@ChristopherGabba thanks for the additional information - one question I have - is there a way for you to tell the size of the file being uploaded when this occurs? I'd like to try eliminate that if it is not the case. My thinking is something may be happening where s3 aborts the connection in the middle of a large file upload. And to confirm, you do not see the video in s3 when this error occurs? Additionally, you can do some retry logic that loops up to 2 times or breaks if the call is succesful. That is one way to mitigate this, but I would still like to understand what the root cause is here. |
nadetastic
added
investigating
|
@nadetastic Sorry I should have answered you when you asked that the first time! My uploads are typically from my photo library or recording from a camera with a max recorded length of 30s and on relatively low quality. But we always perform a video compression before the upload to save S3 memory. I just ran a test and it was 2.5MB after compression. All of ours should be < 10MB if I had to guess. Here was my log output prior to the S3 upload: And to answer your last question, the video does not upload successfully, we do not see it in S3. |
|
@nadetastic What are the chances this is related to this miserable ExpiredToken error I was getting intermittently in V5? That error would come after the app sat in the background for a few hours then you immediately open it and try to upload a file to S3. I'm not sure on the backend side of things what's different between V5 and V6, but we are seeing a similar pattern with this ECONNABORTED: Network Error where the app sits in the background for a while, you open it up and upload, and then it fires off ECONNABORTED. I upload things throughout the day for testing and the error only seems to come when the app is in the background for a while and then you try to upload something. I changed my authentication patterns every which way but hell to try and fix that ExpiredToken error, to the point where I upgraded from V5 amplify to V6 and when I didn't get that error again I was so ecstatic. Then shortly into testing V6, we started getting this ECONNABORTED errors as a replacement... TBH these two errors have kept me from releasing the app for over 3 weeks now. I can't go into production with this happening. Maybe it's a combination of our S3 bucket policy and our authentication? Reading about this error elsewhere on other platforms, here are a few helpful links:
It seems based off the various articles that I've read, this error is related to a few different causes:
Really just trying to help, sorry for the long message. |
ChristopherGabba
changed the title
|
HI @ChristopherGabba I think that is a great point - especially considering that the app sits in the background for a while. One thing you can try to do is to call |
|
@nadetastic I am still experiencing this every now. I am now using the fetchAuthSession with a force refresh. When I try the upload again, most of the time it goes through on the second attempt. The issue is definitely still present so I don't want to close. The kicker is that it occurs infrequently and usually after the app has been static for a while. |
|
@ChristopherGabba thanks for the additional info, Is it possible for you to share a minimal sample app for me to reproduce and test with? |
|
@nadetastic We have been using our app in TestFlight now for about 2 months. During this time, it's pretty random. Every day or every other day one of the users gets the ECONNABORTED error when they attempt the S3 upload. We have done a decent job at trying to make a graceful error that allows the user to try and re-upload, and usually when they try again, it works. However, I have been unable to get this to fire on command, but it is happening for several different users on several different types of iPhones. It would seem that the best way to replicate this would be:
I think the reason why it doesn't happen on development builds is that I'm always closing and refreshing the development builds, so its not getting the time to disconnect or lose connection to the aws server / web socket. |
|
@ChristopherGabba, this appears to be an error tied to This comment from the |
|
@cwomack The place where this error comes the most often is this function: async function sendResponse() {
if (!response.sender) return
setIsLoading(true)
const client = generateClient()
try {
// Upload main phone media to S3
const [reactionVideoUrl, reactionThumbnailUrl] = await Promise.all([
await uploadToS3({
filePath: reactionVideo,
key: `${response.sharedMedia.id}/rf-${response.id}/reactionVideoUrl.mp4`,
options: {
accessLevel: "guest",
contentType: "video/mp4",
},
}),
await uploadToS3({
filePath: thumbnailUrl,
key: `${response.sharedMedia.id}/rf-${response.id}/reactionThumbnailUrl.jpg`,
options: {
contentType: "image/jpg",
},
}),
])
const response = await client.graphql({
query: updateResponse,
variables: {
input: {
id: response.id,
// ....
},
},
})
console.log("successfully updated response", response)
await sendResponseNotification(
response.sender?.pushToken,
true,
)
setConfirmation({ tx: "common.sent", durationMillis: 1000 })
await new Promise((resolve) => setTimeout(resolve, 1000))
setResponseSent(true)
setIsLoading(false)
} catch (error) {
setIsLoading(false)
setConfirmation({ tx: "errors.sendingResponseError" })
reportCrash({
error,
method: "sendResponse",
component: "RecordingScreen",
})
}
}The only method in here that is not an amplify related method is the try {
await fetch("https://exp.host/--/api/v2/push/send", {
method: "POST",
headers: {
Accept: "application/json",
"Accept-encoding": "gzip, deflate",
"Content-Type": "application/json",
},
body: JSON.stringify(message),
})
} catch (error) {
reportCrash({
error,
method: "sendReelFeelNotification",
})
}I know it doesn't get to this point though because the other user does not get the notification, and it is the uploadToS3 that actually fails. Do any of these amplify functions use |
|
@cwomack @nadetastic Okay, I finally spent time hunting down the exact source (line of code) that is triggering this error in production. It's happening at least 5 times a day among my test users and it's been driving me up the wall. I put export async function uploadToS3(
input: Omit<UploadDataInput, "data"> & { filePath: string },
): Promise<string> {
let response: Response
let data: Blob
let output: UploadDataOutput
let result: any
let key: any
let urlResponse: any
try {
response = await fetch(input.filePath)
} catch (error) {
reportCrash({ error, component: "uploadToS3-CatchBlock1" })
throw error
}
try {
data = await response.blob()
} catch (error) {
reportCrash({ error, component: "uploadToS3-CatchBlock2" })
throw error
}
try {
output = uploadData({ key: input.key, data, options: input?.options })
} catch (error) {
reportCrash({ error, component: "uploadToS3-CatchBlock3" })
throw error
}
try {
result = await output.result // <------------------------- It's this line right here
key = result.key
} catch (error) {
reportCrash({ error, component: "uploadToS3-CatchBlock4" }). // <-------- see error screenshot below
throw error
}
try {
urlResponse = await getUrl({
key,
options: {
accessLevel: input.options?.accessLevel,
},
})
} catch (error) {
reportCrash({ error, component: "uploadToS3-CatchBlock5" })
throw error
}
const fullUrl = urlResponse.url.toString()
return stripUrlParams(fullUrl)
}
Here is what I can tell you so far:
This &$*@ has plagued us like a menace for the last 3 months and I will Venmo whoever solves this 50$ for beer. If there is anything I can do, like you guys provide me with a patch file that has better logging for you using If you send me the background code of this uploadData function and I can use it in place of the uploadData function and break it down with Here are screenshots of the native alert I have displayed for the user:
|
|
@cwomack a few more users got it today. Originating from the same function every time. The screenshots below show that it is coming from
All the error messages show the user is still authenticated with cognito, neither token is expired (access or identity) and the user has a good internet connection. |
|
Hi @ChristopherGabba. I looked into the origin point of this error a bit and found it here A couple issues with what we did and I'm talking to other engineers on what we should do to address:
I will add items to our task queue to address/evaluate these 2 issues. But, in the meantime, if it's possible to patch this line in your testing, you can try to get more information from the XHR event itself directly. |
|
Following up my comment above. Ultimately, the error is coming from |
|
@cshfang First of all, thank you so much for digging deep and trying to find the source of this, I am very grateful. I patched the line below: - xhr.addEventListener('error', () => {
- const networkError = buildHandlerError(constants_1.NETWORK_ERROR_MESSAGE, constants_1.NETWORK_ERROR_CODE);
+ xhr.addEventListener('error', (event) => {
+ const networkError = buildHandlerError(JSON.stringify(event, null, 4), constants_1.NETWORK_ERROR_CODE);to capture the full error event. I do not know how this will end up being displayed, but I'll work a few different iterations if it comes out poorly. If you have any suggestions on the exact patch, I'll jump right on it. Will get the error data here as soon as we get the error. |
|
@ChristopherGabba, appreciate you taking the time to help us get the context of what's happening here. It looks like the underlying issue being experienced isn't quite something that we'll be able to immediately fix from a bug standpoint, but wanted to let you know we're evaluating how to expose the underlying errors when this happens in a better format. Given that the underlying |
cwomack
added
question
|
@cwomack Before closing, just so I have a better understanding, where exactly is this "bug" happening? Understanding I'm still trying to gather the raw error event message. Is it some sort of server rejection to the users upload? I just want to make sure that if this issue gets Closed, it's not forgotten. I really would like this bug to not happen again 😂. It definitely still needs a solution, so should I open another issue somewhere else? |
|
Okay guys, I just got the following error message for the raw event data:
Looks like the problem is |
Hey @ChristopherGabba. The error message is coming from the The isTrusted property coming back from the XHR error message unfortunately tells us very little because successful events are going to contain that same property and value as well. At this time, I would try to inspect the actual network logs to see if I can glean further insight as to what the issue is but I suspect it's a low level network connectivity resolution that is at fault here. Either searching through Expo issues or SO for similar issues and continuing to narrow down poor network conditions would be my approach. But, at this time, I'm afraid there's very little that we can do to the library code to narrow it down further. |
|
@cshfang @cwomack Thanks a bunch for digging deep. I'm okay with closing this issue, please keep me updated on any updates you guys do make in this area. Because I'm wading in some unknown waters here, how could I inspect the actual network logs? How is this different than what I've already done? I did open the above issues with |
Hmm typically I would do this via Flipper, React Native Debugger or Reactotron. But it may require being able to replicate the issue locally somehow. I'm not sure if there is a way to configure those tools to log to some remote logging tool like CloudWatch. Ultimately, I suspect the root issue is poor network conditions, maybe it would be possible to replicate locally using network link conditioner? This seems a tricky one to track down for sure. |
Before opening, please confirm:
JavaScript Framework
React Native
Amplify APIs
Storage
Amplify Version
v6
Amplify Categories
storage
Backend
Amplify CLI
Environment information
Describe the bug
Uploading media to S3 via the
uploadDataapi occasionally fails upload and throwsECONNABORTED Network Error. Unable to reproduce exact replication scenario, but it is happening ~30% of the time in production - beta.Expected behavior
The video should complete the upload successfully, which it does most of the time. Connected to good wifi connection, no bad signal, still receiving this error.
Reproduction steps
Code Snippet
Most code is provided above in the reproduction steps.
Log output
Occuring in production copy of app in testflight Beta, so can't get logs. Refer to Sentry screenshot below.
aws-exports.js
Here is my aws-exports.js:
/* eslint-disable */
// WARNING: DO NOT EDIT. This file is automatically generated by AWS Amplify. It will be overwritten.
Manual configuration
Here is my amplifyconfiguration.json:
{ "aws_project_region": "us-east-1", "aws_appsync_graphqlEndpoint": "https://5tvnvesrgjfjdclupp5rzrh5gi.appsync-api.us-east-1.amazonaws.com/graphql", "aws_appsync_region": "us-east-1", "aws_appsync_authenticationType": "API_KEY", "aws_appsync_apiKey": *REDACTED*, "aws_cognito_identity_pool_id": *REDACTED*, "aws_cognito_region": "us-east-1", "aws_user_pools_id": "us-east-1_eDAImGHL9", "aws_user_pools_web_client_id": " *REDACTED*, "oauth": {}, "aws_cognito_username_attributes": [], "aws_cognito_social_providers": [], "aws_cognito_signup_attributes": [ "GIVEN_NAME", "FAMILY_NAME", "BIRTHDATE", "PHONE_NUMBER" ], "aws_cognito_mfa_configuration": "OFF", "aws_cognito_mfa_types": [], "aws_cognito_password_protection_settings": { "passwordPolicyMinLength": 8, "passwordPolicyCharacters": [ "REQUIRES_LOWERCASE", "REQUIRES_UPPERCASE", "REQUIRES_NUMBERS", "REQUIRES_SYMBOLS" ] }, "aws_cognito_verification_mechanisms": [ "PHONE_NUMBER" ], "aws_user_files_s3_bucket": *REDACTED*, "aws_user_files_s3_bucket_region": "us-east-1" }Additional configuration
Here is my configuration in App.tsx:
Mobile Device
iPhone 13 Pro (Physical)
Mobile Operating System
iOS 17.2.1
Mobile Browser
Safari
Mobile Browser Version
Unsure
Additional information and screenshots
Sentry Error with a little more error details:
iOS error alert:
The text was updated successfully, but these errors were encountered: