Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Refactor notifications and make them injectable #2034

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import {getSleepTime} from './util';
import {logger} from './logger';
import {storeList} from './store/model';
import {tryLookupAndLoop} from './store';
import {sendNotification} from './notification';

let browser: Browser | undefined;

Expand Down Expand Up @@ -76,7 +77,13 @@ async function main() {
store.setupAction(browser);
}

setTimeout(tryLookupAndLoop, getSleepTime(store), browser, store);
setTimeout(
tryLookupAndLoop,
getSleepTime(store),
browser,
store,
sendNotification
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the main function we inject the existing sendNotification function

);
}

await startAPIServer();
Expand Down
23 changes: 23 additions & 0 deletions src/notification/link_poll_event.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import {Link, Store} from '../store/model';

type RequestFailed = {
failureReason: 'request_failed';
statusCode: number;
};
type Captcha = {failureReason: 'captcha'};
type MaxPriceExceeded = {failureReason: 'max_price'; maxPrice: number};
type OutOfStock = {failureReason: 'out_of_stock'};
type BannedSeller = {failureReason: 'banned_seller'};
type LinkPollError = {result: 'failure'} & (
| RequestFailed
| Captcha
| MaxPriceExceeded
| OutOfStock
| BannedSeller
);
type LinkPollSuccess = {result: 'in_stock'; url: string};

export type LinkPollEvent = (LinkPollError | LinkPollSuccess) & {
link: Link;
store: Store;
};
Copy link
Contributor Author

@gmalette gmalette Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LinkPollEvent could be made shorter but I liked to express it in a way that prevents representing illegal states.

I wasn't sure if events should be provided for cloudflare, tryLookupAndLoop caught errors, inStockWaiting, lookupCard caught errors, NoResponse, RateLimit, etc.

32 changes: 32 additions & 0 deletions src/notification/logstream.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import {Print, logger} from '../logger';
import {LinkPollEvent} from './link_poll_event';

function assertNever(x: never): never {
throw new Error('Unexpected object: ' + x);
}

export function sendLogstream(pollEvent: LinkPollEvent) {
const {link, store} = pollEvent;
if (pollEvent.result === 'in_stock') {
logger.info(`${Print.inStock(link, store, true)}\n${pollEvent.url}`);
} else {
switch (pollEvent.failureReason) {
case 'captcha':
logger.warn(Print.captcha(link, store, true));
return;
case 'banned_seller':
logger.warn(Print.bannedSeller(link, store, true));
return;
case 'out_of_stock':
logger.info(Print.outOfStock(link, store, true));
return;
case 'max_price':
logger.info(Print.maxPrice(link, store, pollEvent.maxPrice, true));
return;
case 'request_failed':
return;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this case logs too but I haven't found where it happens.

default:
assertNever(pollEvent);
}
}
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the logging w.r.t polling had been moved here. Some logging still happends in lookup.ts but looked ancillary, except for what's mentioned in the comments on notifications.

11 changes: 9 additions & 2 deletions src/notification/notification.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import {Link, Store} from '../store/model';
import {adjustPhilipsHueLights} from './philips-hue';
import {playSound} from './sound';
import {sendDesktopNotification} from './desktop';
Expand All @@ -17,8 +16,16 @@ import {sendTwitchMessage} from './twitch';
import {updateRedis} from './redis';
import {activateSmartthingsSwitch} from './smartthings';
import {sendStreamLabsAlert} from './streamlabs';
import {sendLogstream} from './logstream';
import {LinkPollEvent} from './link_poll_event';

export function sendNotification(link: Link, store: Store) {
export function sendNotification(pollEvent: LinkPollEvent) {
sendLogstream(pollEvent);
if (pollEvent.result === 'failure') {
return;
}

const {link, store} = pollEvent;
// Priority
playSound();
sendDiscordMessage(link, store);
Expand Down
98 changes: 81 additions & 17 deletions src/store/lookup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,15 @@ import {fetchLinks} from './fetch-links';
import {filterStoreLink} from './filter';
import open from 'open';
import {processBackoffDelay} from './model/helpers/backoff';
import {sendNotification} from '../notification';
import useProxy from '@doridian/puppeteer-page-proxy';
import {LinkPollEvent} from '../notification/link_poll_event';

const inStock: Record<string, boolean> = {};

const linkBuilderLastRunTimes: Record<string, number> = {};

export type SendNotification = (pollEvent: LinkPollEvent) => void;

function nextProxy(store: Store) {
if (!store.proxyList) {
return;
Expand Down Expand Up @@ -151,7 +153,11 @@ async function handleAdBlock(request: HTTPRequest, adBlockRequestHandler: any) {
* @param browser Puppeteer browser.
* @param store Vendor of graphics cards.
*/
async function lookup(browser: Browser, store: Store) {
async function lookup(
browser: Browser,
store: Store,
sendNotification: SendNotification
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the added code in this file is passing the sendNotification to the functions that need it.

The rest is sending notifications in lieu of logging

) {
if (!getStores().has(store.name)) {
return;
}
Expand Down Expand Up @@ -252,7 +258,13 @@ async function lookup(browser: Browser, store: Store) {
let statusCode = 0;

try {
statusCode = await lookupCard(browser, store, page, link);
statusCode = await lookupCard(
browser,
store,
page,
link,
sendNotification
);
} catch (error: unknown) {
if (store.currentProxyIndex !== undefined && store.proxyList) {
const proxy = `${store.currentProxyIndex + 1}/${
Expand Down Expand Up @@ -294,7 +306,8 @@ async function lookupCard(
browser: Browser,
store: Store,
page: Page,
link: Link
link: Link,
sendNotification: SendNotification
): Promise<number> {
const givenWaitFor = store.waitUntil ? store.waitUntil : 'networkidle0';
const response: HTTPResponse | null = await page.goto(link.url, {
Expand All @@ -305,21 +318,32 @@ async function lookupCard(
const statusCode = await handleResponse(browser, store, page, link, response);

if (!isStatusCodeInRange(statusCode, successStatusCodes)) {
sendNotification({
result: 'failure',
failureReason: 'request_failed',
statusCode,
link,
store,
});
return statusCode;
}

if (await lookupCardInStock(store, page, link)) {
if (await lookupCardInStock(store, page, link, sendNotification)) {
const givenUrl =
link.cartUrl && config.store.autoAddToCart ? link.cartUrl : link.url;
logger.info(`${Print.inStock(link, store, true)}\n${givenUrl}`);

if (config.browser.open) {
await (link.openCartAction === undefined
? open(givenUrl)
: link.openCartAction(browser));
}

sendNotification(link, store);
sendNotification({
result: 'in_stock',
url: givenUrl,
link,
store,
});

if (config.page.inStockWaitTime) {
inStock[link.url] = true;
Expand Down Expand Up @@ -407,7 +431,12 @@ async function checkIsCloudflare(store: Store, page: Page, link: Link) {
return false;
}

async function lookupCardInStock(store: Store, page: Page, link: Link) {
async function lookupCardInStock(
store: Store,
page: Page,
link: Link,
sendNotification: SendNotification
) {
const baseOptions: Selector = {
requireVisible: false,
selector: store.labels.container ?? 'body',
Expand All @@ -416,7 +445,12 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {

if (store.labels.captcha) {
if (await pageIncludesLabels(page, store.labels.captcha, baseOptions)) {
logger.warn(Print.captcha(link, store, true));
sendNotification({
result: 'failure',
failureReason: 'captcha',
link,
store,
});
await delay(getSleepTime(store));
return false;
}
Expand All @@ -426,14 +460,24 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {
if (
await pageIncludesLabels(page, store.labels.bannedSeller, baseOptions)
) {
logger.warn(Print.bannedSeller(link, store, true));
sendNotification({
result: 'failure',
failureReason: 'banned_seller',
link,
store,
});
return false;
}
}

if (store.labels.outOfStock) {
if (await pageIncludesLabels(page, store.labels.outOfStock, baseOptions)) {
logger.info(Print.outOfStock(link, store, true));
sendNotification({
result: 'failure',
failureReason: 'out_of_stock',
link,
store,
});
return false;
}
}
Expand All @@ -444,7 +488,13 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {
link.price = await getPrice(page, store.labels.maxPrice, baseOptions);

if (link.price && link.price > maxPrice && maxPrice > 0) {
logger.info(Print.maxPrice(link, store, maxPrice, true));
sendNotification({
result: 'failure',
failureReason: 'max_price',
maxPrice,
link,
store,
});
return false;
}
}
Expand All @@ -466,7 +516,12 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {
};

if (!(await pageIncludesLabels(page, store.labels.inStock, options))) {
logger.info(Print.outOfStock(link, store, true));
sendNotification({
result: 'failure',
failureReason: 'out_of_stock',
link,
store,
});
return false;
}
}
Expand All @@ -479,7 +534,12 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {
};

if (!(await pageIncludesLabels(page, link.labels.inStock, options))) {
logger.info(Print.outOfStock(link, store, true));
sendNotification({
result: 'failure',
failureReason: 'out_of_stock',
link,
store,
});
return false;
}
}
Expand Down Expand Up @@ -540,20 +600,24 @@ async function runCaptchaDeterrent(browser: Browser, store: Store, page: Page) {
}
}

export async function tryLookupAndLoop(browser: Browser, store: Store) {
export async function tryLookupAndLoop(
browser: Browser,
store: Store,
sendNotification: SendNotification
) {
if (!browser.isConnected()) {
logger.debug(`[${store.name}] Ending this loop as browser is disposed...`);
return;
}

logger.debug(`[${store.name}] Starting lookup...`);
try {
await lookup(browser, store);
await lookup(browser, store, sendNotification);
} catch (error: unknown) {
logger.error(error);
}

const sleepTime = getSleepTime(store);
logger.debug(`[${store.name}] Lookup done, next one in ${sleepTime} ms`);
setTimeout(tryLookupAndLoop, sleepTime, browser, store);
setTimeout(tryLookupAndLoop, sleepTime, browser, store, sendNotification);
}
2 changes: 1 addition & 1 deletion test/functional/test-notification.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ const store: Store = {
/**
* Send test email.
*/
sendNotification(link, store);
sendNotification({result: 'in_stock', url: 'test:url', link, store});

/**
* Open browser.
Expand Down