Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS-SDK: Fixing broken exports #2410

Merged
merged 22 commits into from
Aug 19, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
dac4adf
new means of creating a bucket
engelhartrueben Jul 30, 2024
98b8563
remove now unused S3 import
engelhartrueben Jul 30, 2024
8338ac1
remove double await that was doing nothing
engelhartrueben Jul 30, 2024
8cc52b8
remove expires in parameter in getSignedURL that is taken care of in …
engelhartrueben Jul 30, 2024
43fd02d
change var name s3bucket to client + comment
engelhartrueben Jul 30, 2024
5e31c04
Merge branch 'main' into re/fix-eports
engelhartrueben Jul 30, 2024
3fcab79
call region from env
engelhartrueben Jul 30, 2024
af2bddd
move bucket name to func call. add location var found in example docs
engelhartrueben Jul 30, 2024
6818d2a
implement PutObjectCommand w/ supporting parameters.
engelhartrueben Jul 30, 2024
0a52da7
make pretty
engelhartrueben Jul 30, 2024
78c2aaa
remove logging
engelhartrueben Jul 30, 2024
2c101dc
add front end language for when export will end up in local Spoke dir…
engelhartrueben Jul 31, 2024
2e623b6
add hard check on email set-up. Next step is to (re-add) fix front e…
engelhartrueben Aug 2, 2024
87538cf
Add error handling when exporting to an S3 bucket
mau11 Aug 14, 2024
fb0f56d
misspointed check on emailEnabled
engelhartrueben Aug 14, 2024
48f095e
typos
engelhartrueben Aug 14, 2024
243b610
change Snackbar logic to better fit outcome
engelhartrueben Aug 15, 2024
052685b
change exportCacheKey expiration to match AWS expiration of 1 day
engelhartrueben Aug 15, 2024
7417b63
cache error
engelhartrueben Aug 15, 2024
0d1055a
reduce timout to 15 sec
engelhartrueben Aug 15, 2024
ee6a29c
adjust export UI logic to only show each respective methods output, a…
engelhartrueben Aug 15, 2024
be0bd30
revert back to emailEnabled check
engelhartrueben Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions src/containers/AdminCampaignStats.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -424,9 +424,10 @@ class AdminCampaignStats extends React.Component {
message={
<span>
Export started -
{this.props.organizationData &&
this.props.organizationData.emailEnabled &&
" we'll e-mail you when it's done. "}
{(this.props.organizationData &&
this.props.organizationData.emailEnabled) ?
" we'll e-mail you when it's done. " :
" check your Spoke directory. "}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried this locally, if we have AWS S3 set up, even if email isn't enabled, the data will save to S3, not in the spoke directory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disregard my previous comment, the last change to this check fixed the issue I was seeing

{campaign.cacheable && (
<span>
<Link
Expand Down
88 changes: 58 additions & 30 deletions src/workers/jobs.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { rawIngestMethod } from "../extensions/contact-loaders";

import { Lambda } from "@aws-sdk/client-lambda";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { GetObjectCommand, S3 } from "@aws-sdk/client-s3";
import { CreateBucketCommand, GetObjectCommand, waitUntilBucketExists, S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { SQS } from "@aws-sdk/client-sqs";
import Papa from "papaparse";
import moment from "moment";
Expand Down Expand Up @@ -861,46 +861,74 @@ export async function exportCampaign(job) {
(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY)
) {
try {
const s3bucket = new S3({
// The transformation for params is not implemented.
// Refer to UPGRADING.md on aws-sdk-js-v3 for changes needed.
// Please create/upvote feature request on aws-sdk-js-codemod for params.
params: { Bucket: process.env.AWS_S3_BUCKET_NAME }
const client = new S3Client({
region: process.env.AWS_REGION
});
const bucketName = process.env.AWS_S3_BUCKET_NAME;
const command = new CreateBucketCommand({ Bucket : bucketName });

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to create a new bucket each time we export data, it's likely someone will already have bucket created for the data and want to upload directly to it

we can add a check, if the bucket exists, save the exported data, if it doesn't, create a new one and save

// this can return Location and $metadata, we just don't need that info
await client.send(command);

// verifies that the bucket exists before moving forward
// if for some reason this fails, Spoke defensively deletes the job
await waitUntilBucketExists({ client, maxWaitTime: 60 }, { Bucket : bucketName });

const campaignTitle = campaign.title
.replace(/ /g, "_")
.replace(/\//g, "_");
const key = `${campaignTitle}-${moment().format(
"YYYY-MM-DD-HH-mm-ss"
)}.csv`;
const messageKey = `${key}-messages.csv`;
let params = { Key: key, Body: campaignCsv };
await s3bucket.putObject(params);
params = { Key: key, Expires: 86400 };
const campaignExportUrl = await await getSignedUrl(s3bucket, new GetObjectCommand(params), {
expiresIn: "/* add value from 'Expires' from v2 call if present, else remove */"
});
params = { Key: messageKey, Body: messageCsv };
await s3bucket.putObject(params);
params = { Key: messageKey, Expires: 86400 };
const campaignMessagesExportUrl = await await getSignedUrl(s3bucket, new GetObjectCommand(params), {
expiresIn: "/* add value from 'Expires' from v2 call if present, else remove */"
});
let params = { Key: key,
Body: campaignCsv,
Bucket: bucketName };
await client.send(new PutObjectCommand(params));
params = { Key: key,
Expires: 86400,
Bucket: bucketName };
const campaignExportUrl = await getSignedUrl(client, new GetObjectCommand(params));
params = { Key: messageKey,
Body: messageCsv,
Bucket: bucketName };
await client.send(new PutObjectCommand(params));
params = { Key: messageKey,
Expires: 86400,
Bucket: bucketName };
const campaignMessagesExportUrl = await getSignedUrl(client, new GetObjectCommand(params));
exportResults.campaignExportUrl = campaignExportUrl;
exportResults.campaignMessagesExportUrl = campaignMessagesExportUrl;

await sendEmail({
to: user.email,
subject: `Export ready for ${campaign.title}`,
text: `Your Spoke exports are ready! These URLs will be valid for 24 hours.
Campaign export: ${campaignExportUrl}
Message export: ${campaignMessagesExportUrl}`
}).catch(err => {
log.error(err);
log.info(`Campaign Export URL - ${campaignExportUrl}`);
log.info(`Campaign Messages Export URL - ${campaignMessagesExportUrl}`);
});
log.info(`Successfully exported ${id}`);
// extreme check on email set-up
if ((
process.env.EMAIL_FROM &&
process.env.EMAIL_HOST &&
process.env.EMAIL_HOST_PASSOWRD &&
process.env.EMAIL_HOST_PORT &&
process.env.EMAIL_HOST_USER) ||
(
process.env.MAILGUN_DOMIAN &&
process.env.MAILGUN_SMTP_LOGN &&
process.env.MAILGUN_SMTP_PASSWORD &&
process.env.MAILGUN_SMTP_PORT &&
process.env.MAILGUN_SMTP_SERVER &&
process.env.MAILGUN_PUBLIC_KEY
)
) {
await sendEmail({
to: user.email,
subject: `Export ready for ${campaign.title}`,
text: `Your Spoke exports are ready! These URLs will be valid for 24 hours.
Campaign export: ${campaignExportUrl}
Message export: ${campaignMessagesExportUrl}`
}).catch(err => {
log.error(err);
log.info(`Campaign Export URL - ${campaignExportUrl}`);
log.info(`Campaign Messages Export URL - ${campaignMessagesExportUrl}`);
});
log.info(`Successfully exported ${id}`);
}
} catch (err) {
log.error(err);
exportResults.error = err.message;
Expand Down
Loading