Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrectly formatted json in results data column #310

Closed
8 tasks
alexjustesen opened this issue Jan 11, 2023 · 1 comment · Fixed by #812
Closed
8 tasks

Incorrectly formatted json in results data column #310

alexjustesen opened this issue Jan 11, 2023 · 1 comment · Fixed by #812
Assignees
Labels
🐛 bug Something isn't working

Comments

@alexjustesen
Copy link
Owner

alexjustesen commented Jan 11, 2023

Describe the bug
The output of a speedtest is being store in results.data in the incorrect format. The field contains escaping characters \ and new lines \n which breaks Laravel's model casting from working correctly.

To Reproduce
Steps to reproduce the behavior:

  1. Go to the dashboard
  2. Click on "Queue Speedtest" and wait for it to run
  3. Open a database IDE and browse the table results
  4. See incorrectly formatted json

Expected behavior
Modify app/Jobs/ExecSpeedtest.php to correctly store the results received by the speedtest-cli.

Result::create([
    //...
   'data' => $output, // <- change this
   'data' => $results, // <- to this and let Laravel convert the array automatically.
]);

Screenshots
The last row of this screenshot has the correct data format when using the decoded $output value from $results.
image

Additional Context
This has been the root cause of a lot of the other issues and the model attribute casting not working. The issue keeps cropping up when passing data to InfluxDB.

Things that will break

  • InfluxDb integration
  • Dashboard charts
  • Results table
  • Legacy APIs
  • tbd...

Databases to test

  • sqlite
  • mysql/mariadb
  • postgresql

Steps to fix the issue

Option 1 - iterate through each record and re-encode it.

DB::table('results')->chunkById(100, function ($result) {
    foreach ($results as $result) {
        $data = json_decode($result->data);

        DB::table('results')
            ->where('id', $result->id)
            ->update(['data' => $data]);
    }
});

Option x - tbd...

@alexjustesen alexjustesen added the 🐛 bug Something isn't working label Jan 11, 2023
@alexjustesen alexjustesen added this to the v0.10.0 milestone Jan 11, 2023
@alexjustesen alexjustesen self-assigned this Jan 11, 2023
@alexjustesen alexjustesen modified the milestones: v0.11.0, v0.13.0 Jan 11, 2023
@alexdelprete
Copy link
Contributor

If that's the only main data issue we have, an automatic routine that runs at startup, that sanity checks the results table and if it finds wrong characters updates all records would be a good solution.

Otherwise, we could introduce a new Maintenance menu, and let users run the task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants