[HOLD for payment 2024-07-22] [HOLD for payment 2024-07-17] [$250] [Performance] [MEDIUM] Add "Export Onyx state" option to Troubleshoot menu and four finger tap

[muttmuure]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

What performance issue do we need to solve?

e.g. memory consumption, storage read/write times, React native bridge concerns, inefficient React component rendering, etc.

When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.

What is the impact of this on end-users?

List specific user experiences that will be improved by solving this problem e.g. app boot time, time to for some interaction to complete, etc.

This is more of a debugging tool, since Onyx impacts on pretty much everything you see in app, it can impact on a multitude of end user functionality.

The most important end user impact of a debugging tool which exports the Onyx state is around privacy. We need to sanitize the authToken, emails and messages of the data since those are private and shouldn't be shared with anyone.

List any benchmarks that show the severity of the issue

Please also provide exact steps taken to collect metrics above if any so we can independently verify the results.

This is more of a debugging tool, so this isn't something that will show benchmarks.

Proposed solution (if any)

Please list out the steps you think we should take to solve this issue.

Add a "Export Onyx State" option to the Troubleshoot menu.

This downloads the values in the Onyx keyvaluepairs table to a txt file.
The authTokens, emails and messages in the file are redacted

List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)

Note: These should be the same as the benchmarks collected before any changes.

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Version Number:
Reproducible in staging?:
Reproducible in production?:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
Expensify/Expensify Issue URL:
Issue reported by:
Slack conversation:

View all open jobs on Upwork

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~014fd561267f73f88e
• Upwork Job ID: 1803048668106098615
• Last Price Increase: 2024-06-18

Issue Owner

Current Issue Owner: @muttmuure

[melvin-bot]

Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.

[melvin-bot]

Huh... This is 4 days overdue. Who can take care of this?

[burczu]

Hi, I am Bartek from Callstack - expert contributor group. I would like to work on this issue.

[burczu]

Daily update: I’ve been working on the proposal for this issue today, mostly on finding the way to collect the whole onyx snapshot as a string (to be able to save it as a txt file later), I hope I’ll be able to post something by the end of tomorrow - I’ll keep you posted. I’ve also created a draft PR in GH.

[burczu]

Daily update: I think I’ve found the way to collect all the Onyx data for web environment and already tested it; for native I think I’ve also find the way but it still needs testing. I believe the proposal is behind the corner and I’ll be able to post it tomorrow.

[burczu]

Daily update: the solution for collecting the Onyx data and saving as a text file is done and can be checked in the draft PR. I've forgotten that we also need to mask data such as authToken, emails etc, so this is the last thing I need to sort out.

[melvin-bot]

@burczu Whoops! This issue is 2 days overdue. Let's get this updated quick!

[burczu]

not overdue

[burczu]

Proposal

Please re-state the problem that we are trying to solve in this issue.

When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.

What is the root cause of that problem?

There is no specific problem - this is more like a new feature. We want to add a "Export Onyx State" option in the Troubleshoot menu in the App. Pressing it should download the values in the Onyx keyvaluepairs table as a *.txt file.

What changes do you think we should make in order to solve the problem?

Add new button to the Troubleshoot menu

First we need to add the new button in the Troubleshoot section. To do so, we need to add new translation (initialSettingsPage.troubleshoot.exportOnyxState) in en.ts and es.ts files

exportOnyxState: 'Export Onyx state',

exportOnyxState: 'Exportar estado Onyx',

Next, we need to add the button in the /src/pages/settings/Troubleshoot/TroubleshootPage.tsx, here:

App/src/pages/settings/Troubleshoot/TroubleshootPage.tsx

Lines 63 to 69 in b278b8f

	const baseMenuItems: BaseMenuItem[] = [
	{
	translationKey: 'initialSettingsPage.troubleshoot.clearCacheAndRestart',
	icon: Expensicons.RotateLeft,
	action: () => setIsConfirmationModalVisible(true),
	},
	];

The code after addition:

const baseMenuItems: BaseMenuItem[] = [
    {
        translationKey: 'initialSettingsPage.troubleshoot.clearCacheAndRestart',
        icon: Expensicons.RotateLeft,
        action: () => setIsConfirmationModalVisible(true),
    },
    {
        translationKey: 'initialSettingsPage.troubleshoot.exportOnyxState',
        icon: Expensicons.Download,
        action: exportOnyxState,
        },
];

The exportOnyxState action method implementation will be described in the last step.

Reading data from Onyx

Next, we need to add the code that will be responsible for reading data from Onyx's keyvaluepairs table. We need to maintain two different implementations for web and mobile, because on web we use the IndexedDB database, and on mobile we use the SQLite database.

To do it, we need to create the ExportOnyxState folder inside /src/libs, and create two files there: index.ts that will be responsible for web implementation and index.native.ts for mobile implementation.

IndexedDB on web

Here's the implementation of reading data from Onyx stored in the IndexedDB on web (using IDBDatabase available in the browser) in the index.ts file:

const readFromIndexedDB = () =>
new Promise<Record<string, unknown>>((resolve) => {
    let db: IDBDatabase;
    const openRequest = indexedDB.open('OnyxDB', 1);
    openRequest.onsuccess = () => {
        db = openRequest.result;
        const transaction = db.transaction('keyvaluepairs');
        const objectStore = transaction.objectStore('keyvaluepairs');
        const cursor = objectStore.openCursor();

        const queryResult: Record<string, unknown> = {};

        cursor.onerror = () => {
            console.error('Error reading cursor');
        };

        cursor.onsuccess = (event) => {
            const {result} = event.target as IDBRequest<IDBCursorWithValue>;
            if (result) {
                queryResult[result.primaryKey as string] = result.value;
                result.continue();
            } else {
                resolve(queryResult);
            }
        };
    };
});

SQLite on mobile

Here's the implementation of reading data from Onyx stored in the SQLite on native devices (using react-native-quick-sqlite we already use in the App) in the index.native.ts file:

const readFromIndexedDB = () => new Promise((resolve) => {
    const db = open({name: 'OnyxDB'});
    const query = 'SELECT * FROM keyvaluepairs';

    db.executeAsync(query, []).then(({rows}) => {
        // eslint-disable-next-line no-underscore-dangle
        const result = rows?._array.map((row) => ({[row.record_key]: JSON.parse(row.valueJSON as string)}));

        resolve(result);
        db.close();
    });
});

Masking fragile data

Now that we have date retrieved from the Onyx database we need to mask some fragile data such as authToken or emails. Do to it, we can simply iterate over the keys of retrieved object, check if its the fragile data and replace it with the *** string. Let's add another method to the ExportOnyxState lib (index.ts):

const maskFragileData = (data: Record<string, unknown>): Record<string, unknown> => {
    const maskedData: Record<string, unknown> = {};

    for (const key in data) {
        if (Object.prototype.hasOwnProperty.call(data, key)) {
            const value = data[key];
            if (typeof value === 'string' && (Str.isValidEmail(value) || key === 'authToken' || key === 'encryptedAuthToken')) {
                maskedData[key] = '***';
            } else if (typeof value === 'object') {
                maskedData[key] = maskFragileData(value as Record<string, unknown>);
            } else {
                maskedData[key] = value;
            }
        }
    }

    return maskedData;
};

As we will use the same code for both web and native in this case, can simply import the above method in the index.native.ts and export it with no modifications.

Sharing result as *.txt file

The last thing is to share the retrieved data as a text file. We also need to do it in a different way for web and for mobile.

Web

For web implementation we can simply use the download attribute of the a element that can be temporarily added to the DOM:

// eslint-disable-next-line @lwc/lwc/no-async-await,@typescript-eslint/require-await
const shareAsFile = async (value: string) => {
    const element = document.createElement('a');
    element.setAttribute('href', `data:text/plain;charset=utf-8,${encodeURIComponent(value)}`);
    element.setAttribute('download', 'onyx-state.txt');

    element.style.display = 'none';
    document.body.appendChild(element);

    element.click();

    document.body.removeChild(element);
};

Mobile

For mobile implementation we can use react-native-fs and react-native-share libraries that we already use in the App:

// eslint-disable-next-line @lwc/lwc/no-async-await
const shareAsFile = async (value: string) => {
    try {
        // Define new filename and path for the app info file
        const infoFileName = `onyx-state.txt`;
        const infoFilePath = `${RNFS.DocumentDirectoryPath}/${infoFileName}`;
        const actualInfoFile = `file://${infoFilePath}`;

        await RNFS.writeFile(infoFilePath, value, 'utf8');

        const shareOptions = {
            urls: [actualInfoFile],
        };

        Share.open(shareOptions);
    } catch (error) {
        console.error('Error renaming and sharing file:', error);
    }
}

Wrapping things up in the exportOnyxState method

At last, we need to utilize all the methods created above in the exportOnyxState method in the TroubleshootPage.tsx:

const exportOnyxState = () => {
    ExportOnyxState.readFromOnyxDatabase().then((value: Record<string, unknown>) => {
        const maskedData = ExportOnyxState.maskFragileData(value);

        ExportOnyxState.shareAsFile(JSON.stringify(maskedData));
    });
};

What alternative solutions did you explore? (Optional)

n/a

Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~014fd561267f73f88e

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ZhenjaHorbach (External)

[ZhenjaHorbach]

@burczu 's proposal looks cool 💪
🎀👀🎀 C+ reviewed

[melvin-bot]

Triggered auto assignment to @hayata-suenaga, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.5-13 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Feature/43097 - "Export onyx state" button in the troubleshoot section #43552

If no regressions arise, payment will be issued on 2024-07-17. 🎊

For reference, here are some details about the assignees on this issue:

• @burczu does not require payment (Contractor)
• @ZhenjaHorbach requires payment (Needs manual offer from BZ)

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.6-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Feature/43097 - "Export onyx state" button in the troubleshoot section #43552

If no regressions arise, payment will be issued on 2024-07-22. 🎊

For reference, here are some details about the assignees on this issue:

• @burczu does not require payment (Contractor)
• @ZhenjaHorbach requires payment (Needs manual offer from BZ)

[ZhenjaHorbach]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@ZhenjaHorbach] The PR that introduced the bug has been identified. Link to the PR:

Not a regression. This is a new feature.

• [@ZhenjaHorbach] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:

Not a regression. This is a new feature.

• [@ZhenjaHorbach] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:

NA

• [@ZhenjaHorbach] Determine if we should create a regression test for this bug.

• [@ZhenjaHorbach] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Regression Test Steps

1. Log in to the App
2. Navigate to Accout Settings -> Troubleshoot section
3. Press the "Export Onyx state" button
4. Wait for the file to be generated and saved (you may be asked to decide where to save the file)
5. Check the content of the saved file (it should contain the Onyx state as a string with emails and authTokens masked by "***" string)

Do we agree 👍 or 👎?

[ZhenjaHorbach]

Not overdue
Waiting for payment !

[melvin-bot]

Payment Summary

Upwork Job

• Contributor: @burczu is from an agency-contributor and not due payment
• ROLE: @ZhenjaHorbach paid $(AMOUNT) via Upwork (LINK)

BugZero Checklist (@muttmuure)

• I have verified the correct assignees and roles are listed above and updated the neccesary manual offers
• I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants/1803048668106098615/hired)
• I have paid out the Upwork contracts or cancelled the ones that are incorrect
• I have verified the payment summary above is correct

[melvin-bot]

@burczu, @muttmuure, @ZhenjaHorbach, @hayata-suenaga 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

[muttmuure]

Handling

[muttmuure]

Invited @ZhenjaHorbach

[ZhenjaHorbach]

Invited @ZhenjaHorbach

Done !

[Christinadobrzyn]

hey I have this GH #44841 that's also ready for payment.

The PRs are different but is this the same job? I've already paid out @etCoderDysto in this GH for this PR.

Should we pay @ZhenjaHorbach once for both jobs or are these separate jobs?

@burczu @hayata-suenaga

[hayata-suenaga]

@burczu ?

[muttmuure]

It should be one payment since this was the original issue and the linked issue is a follow-up/polish issue caught in testing

[muttmuure]

Paid