-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS-SDK: Fixing broken exports #2410
Merged
Merged
Changes from 13 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
dac4adf
new means of creating a bucket
engelhartrueben 98b8563
remove now unused S3 import
engelhartrueben 8338ac1
remove double await that was doing nothing
engelhartrueben 8cc52b8
remove expires in parameter in getSignedURL that is taken care of in …
engelhartrueben 43fd02d
change var name s3bucket to client + comment
engelhartrueben 5e31c04
Merge branch 'main' into re/fix-eports
engelhartrueben 3fcab79
call region from env
engelhartrueben af2bddd
move bucket name to func call. add location var found in example docs
engelhartrueben 6818d2a
implement PutObjectCommand w/ supporting parameters.
engelhartrueben 0a52da7
make pretty
engelhartrueben 78c2aaa
remove logging
engelhartrueben 2c101dc
add front end language for when export will end up in local Spoke dir…
engelhartrueben 2e623b6
add hard check on email set-up. Next step is to (re-add) fix front e…
engelhartrueben 87538cf
Add error handling when exporting to an S3 bucket
mau11 fb0f56d
misspointed check on emailEnabled
engelhartrueben 48f095e
typos
engelhartrueben 243b610
change Snackbar logic to better fit outcome
engelhartrueben 052685b
change exportCacheKey expiration to match AWS expiration of 1 day
engelhartrueben 7417b63
cache error
engelhartrueben 0d1055a
reduce timout to 15 sec
engelhartrueben ee6a29c
adjust export UI logic to only show each respective methods output, a…
engelhartrueben be0bd30
revert back to emailEnabled check
engelhartrueben File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,7 +28,7 @@ import { rawIngestMethod } from "../extensions/contact-loaders"; | |
|
||
import { Lambda } from "@aws-sdk/client-lambda"; | ||
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; | ||
import { GetObjectCommand, S3 } from "@aws-sdk/client-s3"; | ||
import { CreateBucketCommand, GetObjectCommand, waitUntilBucketExists, S3Client, PutObjectCommand } from "@aws-sdk/client-s3"; | ||
import { SQS } from "@aws-sdk/client-sqs"; | ||
import Papa from "papaparse"; | ||
import moment from "moment"; | ||
|
@@ -861,46 +861,74 @@ export async function exportCampaign(job) { | |
(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) | ||
) { | ||
try { | ||
const s3bucket = new S3({ | ||
// The transformation for params is not implemented. | ||
// Refer to UPGRADING.md on aws-sdk-js-v3 for changes needed. | ||
// Please create/upvote feature request on aws-sdk-js-codemod for params. | ||
params: { Bucket: process.env.AWS_S3_BUCKET_NAME } | ||
const client = new S3Client({ | ||
region: process.env.AWS_REGION | ||
}); | ||
const bucketName = process.env.AWS_S3_BUCKET_NAME; | ||
const command = new CreateBucketCommand({ Bucket : bucketName }); | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we want to create a new bucket each time we export data, it's likely someone will already have bucket created for the data and want to upload directly to it we can add a check, if the bucket exists, save the exported data, if it doesn't, create a new one and save |
||
// this can return Location and $metadata, we just don't need that info | ||
await client.send(command); | ||
|
||
// verifies that the bucket exists before moving forward | ||
// if for some reason this fails, Spoke defensively deletes the job | ||
await waitUntilBucketExists({ client, maxWaitTime: 60 }, { Bucket : bucketName }); | ||
|
||
const campaignTitle = campaign.title | ||
.replace(/ /g, "_") | ||
.replace(/\//g, "_"); | ||
const key = `${campaignTitle}-${moment().format( | ||
"YYYY-MM-DD-HH-mm-ss" | ||
)}.csv`; | ||
const messageKey = `${key}-messages.csv`; | ||
let params = { Key: key, Body: campaignCsv }; | ||
await s3bucket.putObject(params); | ||
params = { Key: key, Expires: 86400 }; | ||
const campaignExportUrl = await await getSignedUrl(s3bucket, new GetObjectCommand(params), { | ||
expiresIn: "/* add value from 'Expires' from v2 call if present, else remove */" | ||
}); | ||
params = { Key: messageKey, Body: messageCsv }; | ||
await s3bucket.putObject(params); | ||
params = { Key: messageKey, Expires: 86400 }; | ||
const campaignMessagesExportUrl = await await getSignedUrl(s3bucket, new GetObjectCommand(params), { | ||
expiresIn: "/* add value from 'Expires' from v2 call if present, else remove */" | ||
}); | ||
let params = { Key: key, | ||
Body: campaignCsv, | ||
Bucket: bucketName }; | ||
await client.send(new PutObjectCommand(params)); | ||
params = { Key: key, | ||
Expires: 86400, | ||
Bucket: bucketName }; | ||
const campaignExportUrl = await getSignedUrl(client, new GetObjectCommand(params)); | ||
params = { Key: messageKey, | ||
Body: messageCsv, | ||
Bucket: bucketName }; | ||
await client.send(new PutObjectCommand(params)); | ||
params = { Key: messageKey, | ||
Expires: 86400, | ||
Bucket: bucketName }; | ||
const campaignMessagesExportUrl = await getSignedUrl(client, new GetObjectCommand(params)); | ||
exportResults.campaignExportUrl = campaignExportUrl; | ||
exportResults.campaignMessagesExportUrl = campaignMessagesExportUrl; | ||
|
||
await sendEmail({ | ||
to: user.email, | ||
subject: `Export ready for ${campaign.title}`, | ||
text: `Your Spoke exports are ready! These URLs will be valid for 24 hours. | ||
Campaign export: ${campaignExportUrl} | ||
Message export: ${campaignMessagesExportUrl}` | ||
}).catch(err => { | ||
log.error(err); | ||
log.info(`Campaign Export URL - ${campaignExportUrl}`); | ||
log.info(`Campaign Messages Export URL - ${campaignMessagesExportUrl}`); | ||
}); | ||
log.info(`Successfully exported ${id}`); | ||
// extreme check on email set-up | ||
if (( | ||
process.env.EMAIL_FROM && | ||
process.env.EMAIL_HOST && | ||
process.env.EMAIL_HOST_PASSOWRD && | ||
process.env.EMAIL_HOST_PORT && | ||
process.env.EMAIL_HOST_USER) || | ||
( | ||
process.env.MAILGUN_DOMIAN && | ||
process.env.MAILGUN_SMTP_LOGN && | ||
process.env.MAILGUN_SMTP_PASSWORD && | ||
process.env.MAILGUN_SMTP_PORT && | ||
process.env.MAILGUN_SMTP_SERVER && | ||
process.env.MAILGUN_PUBLIC_KEY | ||
) | ||
) { | ||
await sendEmail({ | ||
to: user.email, | ||
subject: `Export ready for ${campaign.title}`, | ||
text: `Your Spoke exports are ready! These URLs will be valid for 24 hours. | ||
Campaign export: ${campaignExportUrl} | ||
Message export: ${campaignMessagesExportUrl}` | ||
}).catch(err => { | ||
log.error(err); | ||
log.info(`Campaign Export URL - ${campaignExportUrl}`); | ||
log.info(`Campaign Messages Export URL - ${campaignMessagesExportUrl}`); | ||
}); | ||
log.info(`Successfully exported ${id}`); | ||
} | ||
} catch (err) { | ||
log.error(err); | ||
exportResults.error = err.message; | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just tried this locally, if we have AWS S3 set up, even if email isn't enabled, the data will save to S3, not in the spoke directory
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
disregard my previous comment, the last change to this check fixed the issue I was seeing