RangeError: offset is not uint

[kimfucious]

Hi there,

I'm loving music-metadata-browser. Nice work!

Every so often, though, when processing a file, it throws an error like the below for certain songs:

RangeError: offset is not uint
    at checkOffset (index.js:1165)
    at Uint8Array.readUInt32LE (index.js:1218)
    at ID3v24TagMapper.postMap (ID3v24TagMapper.js:173)
    at ID3v24TagMapper.mapGenericTag (GenericTagMapper.js:78)
    at CombinedTagMapper.mapTag (CombinedTagMapper.js:42)
    at MetadataCollector.toCommon (MetadataCollector.js:239)
    at MetadataCollector.addTag (MetadataCollector.js:102)
    at ID3v2Parser.addTag (ID3v2Parser.js:195)
    at ID3v2Parser.parseId3Data (ID3v2Parser.js:189)

When I say "certain songs" I mean mp3 files, and that the error is consistent, in that it will always happen for some files and it will never happen on others.

What are some things to look at with regards to the file to determine why a problematic file is causing the error?

I've read through issue #15 but couldn't quite figure out how to troubleshoot the issue.

I dropped some of the files in question on the Audio Tag Analyzer, and compared those to "known good" files, but I'm not seeing thing that sticks out.

Any advice would be greatly appreciated.

P.S. I'm using "music-metadata-browser": "^1.1.1" in a create-react-app

[Borewit]

I am happy to hear you like it.

Under the assumption you suffering from the same issue as described in #15, there is nothing wrong with these 'problematic files'. These files just trigger some functionality in your buffer module which is missing.

Which is caused by an older version of buffer getting compiled / packed in your react web app.

Note that music-metadata is designed to operate node.js API. The buffer module is part of node.js. In order to get this working in the browser you need to 'shim' this functionality with a buffer replacing the standard node.js module.

If you manage to get the right version of buffer packed in, it will solve your issue.

[kimfucious]

Thanks for the tips @Borewit

My first thought was to move this process out to a serverless function and use Node version of music-metadata there, but then I realized I'd have to send the entire file to the function before it got processed.

So I took a look at your implementation of the Audio Tag Analyzer, as it seems to do this all in the browser, to see how you implemented the buffer shim.

From what I can tell, you import buffer from npm and then attach it to the window object as shown here on lines 42-43, like so:

(window as any).global = window;
import * as _buffer from 'buffer';
(window as any).Buffer = _buffer.Buffer; // note: the trailing slash is important!

I'm not using TypeScript, so my block looks like this:

window.global = window;
import * as _buffer from 'buffer';
window.Buffer = _buffer.Buffer; // note: the trailing slash is important!

Side note: I don't see a trailing slash there, but I think that's a left-over comment from ferros's docs where he's using require instead of import.

Anyhow, my attempts a "shimming" have failed, as the above doesn't seem to fix the offset is not uint error on certain "problematic" files.

I'm admittedly a toddler at the keyboard, but this stuff interests me.

If you have the time and patience, kindly let me know if I'm missing a step somewhere or if I've totally lost the plot.

Danke je,

Kim

[kimfucious]

One thing I've just noticed is that, when viewing the data on one "problematic file" in the Audio Tag Analyzer is that the file the following Common tag, where I do not see that in files that don't throw and error:

Average gain level. | 10107 | averageLevel

[Borewit]

it's not a coding issue, it's more tweaking the packager in such a way that it includes the right buffer module.

Average gain level. | 10107 | averageLevel

To decode that tag a function of the buffer module is called, which is has an in issue in the buffer module you are using. It don't think this is the only decoding part where this error can occure.

To trick the packager, you could:

• add buffer v5.2.1 as dependency in your project (npm install buffer)

Add something like this at the beginning of your code:

const BrowserBuffer= require('buffer/').Buffer  // note: the trailing slash is important!

// Trigger the buffer to ensure the packeger doesn't kick it out because it is not used
const buffer = BrowserBuffer.from([0x01, 0x02, 0x03, 0x04,]);

// Ensure the readUInt32LE is working
if (buffer.readUInt32LE() !== 0x04030201)
  throw new Error(`Expected ${buffer.readUInt32LE()} == ${0x04030201}`);

Note that I have not tested this code snippet.

Maybe you can publish your code so I can have a look.

[kimfucious]

@Borewit you're a ⭐️ !

Here's my measly attempt at coding:

import { cleanThing, uncleanThing } from "./helpers";
import { withAuthenticator } from "aws-amplify-react";
import * as _buffer from "buffer";
import * as mm from "music-metadata-browser";
import $ from "jquery";
import Amplify, { API, Storage, Auth } from "aws-amplify";
import awsconfig from "./aws-exports";
import Dropzone from "react-dropzone";
import InfoModal from "./InfoModal";
import NavBar from "./NavBar";
import React, { useEffect, useState } from "react";
import slug from "./images/slug.png";
import slugfeeding from "./images/slug_feeding.gif";
import SlugModal from "./SlugModal";

window.global = window;
window.Buffer = _buffer.Buffer;
Amplify.configure(awsconfig);
Storage.configure({ level: "public" });

const App = () => {
  const [modalAction, setModalAction] = useState("");
  const [deleteStatus, setDeleteStatus] = useState("");
  const [downloadLink, setDownloadLink] = useState("");
  const [dropzoneDisabled, setDropzoneDisabled] = useState(false);
  const [error, setError] = useState("");
  const [fileName, setFileName] = useState("");
  const [likeToggle, setLikeToggle] = useState(false);
  const [songs, setSongs] = useState([]);
  const [s3FileList, setS3FileList] = useState([]);
  const [uploadProgress, setUploadProgress] = useState(0);
  const [uploadStatus, setUploadStatus] = useState(0);

  // TODO
  // const massFeedtheSlug = async acceptedFiles => {
  //   acceptedFiles.forEach(file => {
  //     prepareToFeedTheSlug([file]);
  //   });
  // };

  const openStreamingWindow = async (fileName, index) => {
    console.log(fileName);
    const url = await Storage.get(fileName, {
      contentDisposition: "inline",
      contentType: "audio/mpeg"
    });
    window.open(url, "_blank");
  };

  let suffix;
  const prepareToFeedTheSlug = async acceptedFiles => {
    if (acceptedFiles.length > 1) {
      alert("There can be only one Highlander!");
      return;
    }
    const file = acceptedFiles[0];
    const fileName = cleanThing(file.name);
    let metadata;
    try {
      metadata = await mm.parseBlob(file);
      console.log("Metadata extracted!");
      console.log(metadata);
      let { album, title, artist, year: releaseYear } = metadata.common;
      if (title) {
        title = cleanThing(title);
      } else {
        title = cleanThing(fileName.slice(0, -4));
      }
      if (artist) {
        artist = cleanThing(artist);
      } else {
        artist = "n/a";
      }
      if (album) {
        album = cleanThing(album);
      } else {
        album = "n/a";
      }
      suffix = fileName.slice(-3).toLowerCase();
      if (suffix === "mp3" || suffix === "m4a") {
        console.log(`Ooh, a lovely ${suffix} to slurp!`);
      } else {
        alert("The slug only eats mp3s and m4as!");
        return;
      }

      console.log("Calling the Slug...");

      // check if song exists in dynamoDB, if not feed the Slug
      try {
        const response = await API.get(
          "SlugBucketApi",
          `/songs/object/${artist}/${title}`
        );
        if (
          Object.entries(response).length === 0 &&
          response.constructor === Object
        ) {
          feedTheSlug(album, artist, file, fileName, releaseYear, title);
        } else {
          const message = `The Slug has previously slurped ${response.title
            .replace(/</g, "(")
            .replace(/>/g, ")")} by ${response.artist}.`;
          console.log(message);
          alert(message);
        }
      } catch (e) {
        console.log(e);
      }
    } catch (e) {
      setError(e.message);
      console.log(e);
    }
  };

  const feedTheSlug = async (
    album,
    artist,
    file,
    fileName,
    releaseYear,
    title
  ) => {
    console.log(`Feeding the slug... ${artist} / ${title}`);
    setUploadProgress(1);
    setUploadStatus("Warming up...");
    setDropzoneDisabled(true);
    try {
      await Storage.put(`${artist} ||| ${title}.${suffix}`, file, {
        contentType: "audio/mpeg",
        // contentDisposition: "attachment",
        progressCallback(progress) {
          setUploadProgress(
            `${parseInt(100 * (progress.loaded / progress.total))}`
          );
          setUploadStatus(
            `${parseInt(100 * (progress.loaded / progress.total))}%`
          );
        }
      });
      console.log("The Slug has been fed");
      UpdateSlugBucketMetaData(artist, title, album, releaseYear, fileName);
      setTimeout(() => {
        setUploadStatus("Upload complete!");
        setDropzoneDisabled(false);
        setUploadStatus(0);
        setUploadProgress(0);
      }, 2000);
    } catch (e) {
      console.log(e);
    }
  };

  const UpdateSlugBucketMetaData = async (
    artist,
    title,
    album,
    releaseYear,
    fileName
  ) => {
    try {
      const response = await API.post("SlugBucketApi", `/songs/`, {
        body: {
          artist,
          title,
          album,
          releaseYear,
          fileName
        }
      });
      console.log("Slug Bucket Metadata Updated!");
      console.log(response);
    } catch (e) {
      console.log(e);
    }
  };
  const getUrl = async (fileName, action) => {
    console.log("geturl filename", fileName);
    try {
      const url = await Storage.get(fileName);
      setFileName(fileName);
      setDownloadLink(url);
      setModalAction(action);
    } catch (e) {
      console.log(e);
    }
  };

  const handleDelete = file => {
    console.log("file name to delete", file);
    Storage.remove(file)
      .then(result => {
        console.log(result);
        setDeleteStatus("deleted");
        setFileName("");
        $("#slug-modal").modal("hide");
      })
      .catch(err => console.log(err));
  };

  const handleLike = async (artist, title, fileName) => {
    const who = await Auth.currentAuthenticatedUser();
    console.log(who.username);
    const songsTemp = songs;
    songsTemp.map(song => {
      if (song.fileName === fileName) {
        console.log("Match!");
        if (song.liked) {
          delete song.liked;
        } else {
          song.liked = true;
        }
        console.log(`${song.liked ? "Liking" : "Unliking"} song on DynamoDB`);
        try {
          API.put("SlugBucketApi", "/songs", {
            body: { ...song }
          });
          console.log(
            `${title} by ${artist} has been ${song.liked ? "liked" : "unliked"}`
          );
        } catch (e) {
          console.log(e);
        }
      }
      return song;
    });
    await setSongs(songsTemp);
    setLikeToggle(!likeToggle);
  };

  useEffect(() => {
    console.log("Event: Like/Unlike");
  }, [likeToggle]);

  const renderFileList = () => {
    if (s3FileList.length === 0) {
      return (
        <div className="lead text-center">
          The slug bucket is empty (or loading)...
        </div>
      );
    } else {
      console.log("Slurping...");
      return songs.map(file => {
        if (!file.album) file.album = "n/a";
        const index = songs.indexOf(file);
        const { album, artist, title, fileName, liked } = file;
        const slugBucketFileName = `${artist} ||| ${title}.${fileName.slice(
          -3
        )}`;
        return (
          <li
            key={index}
            className="list-group-item d-flex flex-column justify-contents-center"
          >
            <div className="d-flex justify-content-between w-100">
              <div className="d-flex flex-column">
                <div className="d-flex align-items-center">
                  <i
                    className="material-icons mr-2 text-secondary"
                    id={`playSong-${index}`}
                    onClick={() =>
                      openStreamingWindow(slugBucketFileName, index)
                    }
                    style={{ cursor: "pointer" }}
                  >
                    play_circle_filled
                  </i>
                  <div className="lead mr-auto">{uncleanThing(title)}</div>
                </div>
                <div>
                  <small>
                    <strong>Artist: </strong>
                    {uncleanThing(artist)}&nbsp;&nbsp;
                    <strong>Album: </strong>
                    {uncleanThing(album)}
                  </small>
                </div>
              </div>
              <div className="d-flex align-items-center">
                <i
                  onClick={() => handleLike(artist, title, fileName)}
                  className={` material-icons mx-1 ${
                    liked ? "text-danger" : "text-muted"
                  }`}
                  style={{ cursor: "pointer" }}
                >
                  {liked ? "favorite" : "favorite_border"}
                </i>
                <i
                  className="material-icons mx-1 text-primary"
                  onClick={() => getUrl(slugBucketFileName, "download")}
                  style={{ cursor: "pointer" }}
                  data-toggle="modal"
                  data-target="#slug-modal"
                >
                  cloud_download
                </i>
                <i
                  onClick={() => {
                    setFileName(slugBucketFileName);
                    setModalAction("delete");
                  }}
                  className="material-icons mx-1 text-danger"
                  style={{ cursor: "pointer" }}
                  data-toggle="modal"
                  data-target="#slug-modal"
                >
                  delete_forever
                </i>
              </div>
            </div>
          </li>
        );
      });
    }
  };

  const getS3Files = async () => {
    try {
      const response = await Storage.list("", { level: "public" });
      setS3FileList(response);
    } catch (e) {
      console.log(e);
    }
  };

  useEffect(() => {
    console.log("Effect! Getting...");
    getS3Files();
  }, []);

  useEffect(() => {
    console.log("Effect! Getting metadata...");
    const getSong = async song => {
      const artist = song.key.split(" ||| ")[0];
      const title = song.key.split(" ||| ")[1].slice(0, -4);
      let metadata = await API.get(
        "SlugBucketApi",
        `/songs/object/${artist}/${title}`
      );
      if (
        Object.entries(metadata).length === 0 &&
        metadata.constructor === Object
      ) {
        metadata = { artist, title, fileName: song.key };
        console.log("Dynamo metadata is missing ! Trying to add...");
        try {
          await API.post("SlugBucketApi", "/songs", {
            body: {
              artist,
              title,
              fileName: song.key
            }
          });
          console.log("Song metadata updated!");
        } catch (e) {
          console.log(e);
        }
      }
      console.log("Here's your metadata", metadata);
      return metadata;
    };
    const pending = s3FileList.map(song => getSong(song));
    Promise.all(pending).then(songs => setSongs(songs));
  }, [s3FileList]);

  useEffect(() => {
    console.log(
      `Effect | Upload Status: ${uploadStatus}, DeleteStatus: ${deleteStatus}`
    );
    if (uploadStatus === "Upload complete!" || deleteStatus === "deleted") {
      getS3Files();
      setDeleteStatus("");
      setUploadStatus(0);
    }
  }, [deleteStatus, uploadProgress, uploadStatus]);

  return (
    <>
      <NavBar />
      <div
        className="d-flex flex-column justify-content-center align-items-center container mt-5"
        // style={{ position: "relative" }}
      >
        {error ? (
          <div
            className="alert alert-dismissible alert-danger"
            style={{
              position: "absolute",
              bottom: "0px",
              zIndex: "999"
            }}
          >
            <button
              type="button"
              className="close"
              onClick={() => setError("")}
              style={{ "&:hover": { opacity: "1" } }}
            >
              &times;
            </button>
            <strong>Oh, nos! The Slug has indigestion.</strong>
            <div>{`Message for Slug:  "${error}"`}</div>
          </div>
        ) : null}
        <div className="d-block d-sm-none display-4 text-center">
          Slug Bucket
        </div>
        <div className="d-none d-sm-block display-2 text-center">
          Slug Bucket
        </div>
        <div className="d-flex justify-content-center input-group mb-3 text-center">
          <Dropzone
            onDrop={acceptedFiles => {
              // massFeedTheSlug(acceptedFiles);
              prepareToFeedTheSlug(acceptedFiles);
            }}
            disabled={dropzoneDisabled}
          >
            {({ getRootProps, getInputProps }) => (
              <section>
                <div style={{ outline: "none" }} {...getRootProps()}>
                  <input {...getInputProps()} />
                  <img
                    className="m-3"
                    src={dropzoneDisabled ? slugfeeding : slug}
                    alt="slug bucket logo"
                    height="201"
                    width="200px"
                    title="Feed me!"
                    style={{ cursor: "pointer" }}
                  />
                  <p className="lead">
                    {uploadProgress > 0 && uploadProgress < 100
                      ? "The Slug is Feeding!"
                      : "Feed the Slug!"}
                  </p>
                </div>
              </section>
            )}
          </Dropzone>
        </div>
        {uploadProgress > 0 ? (
          <div className="d-flex flex-column align-items-center justify-content-center w-100">
            <div
              className="progress"
              style={{ maxWidth: "768px", minWidth: "300px" }}
            >
              <div
                className={`progress-bar progress-bar-striped ${
                  uploadProgress < 100
                    ? "progress-bar-animated bg-primary"
                    : "bg-success"
                }`}
                role="progressbar"
                aria-valuenow={uploadProgress}
                aria-valuemin="0"
                aria-valuemax="100"
                style={{ width: `${uploadProgress}%` }}
              />
            </div>
            <div className="text-center lead text-muted mb-3">
              <small>{uploadStatus ? uploadStatus : null}</small>
            </div>
          </div>
        ) : null}
        {songs.length > 0 ? (
          <>
            <div
              className="align-items-center d-flex d-sm-none display-4 mb-3"
              style={{ fontSize: "24px" }}
            >
              <span>Songs in the Key of Slug</span>
              <i
                className="material-icons text-muted ml-2"
                data-toggle="modal"
                data-target="#info-modal"
                title="How to Slug..."
                style={{ cursor: "pointer" }}
              >
                info_outline
              </i>
            </div>
            <div
              className="align-items-center d-none d-sm-flex display-4 mb-3"
              style={{ fontSize: "32px" }}
            >
              <span>Songs in the Key of Slug</span>
              <i
                className="material-icons text-muted ml-2"
                data-toggle="modal"
                data-target="#info-modal"
                title="How to Slug..."
                style={{ cursor: "pointer" }}
              >
                info_outline
              </i>
            </div>
          </>
        ) : null}
        <ul
          className="d-flex justify-content-center list-group w-100 mb-5"
          style={{ maxWidth: "800px", minWidth: "300px" }}
        >
          {renderFileList()}
        </ul>
        <SlugModal
          action={modalAction}
          artist={fileName.split(" ||| ")[0]}
          title={fileName.slice(0, -4).split(" ||| ")[1]}
          fileName={fileName}
          url={downloadLink}
          fn={handleDelete}
        />
        <InfoModal />
      </div>
    </>
  );
};

export default withAuthenticator(App, true);

[kimfucious]

More than you wanted, or needed, I'm sure 😸

[Borewit]

Can you please share the entire project in a repo?

Ideally it is project setting to get the right buffer in.

[kimfucious]

Sure, thanks @Borewit

Let me check that repo before I make it public, as it's part of a CI deploy via Amplify, and I'm not sure how much private info regarding my AWS resources is exposed, though I'm pretty sure those resources are locked down regardless.

[Borewit]

Make sure nothing sensitive is in your history neither.

You can consider granting me access to the private repo if you fear going public. Or maybe copy all files required and push a to new repo.

[kimfucious]

Cool; thanks. Almost there.

[Borewit]

You make me curious.

[Borewit]

Any luck Kim?

[kimfucious]

Ha ha! Sorry for the delay. And thanks for the follow up.

I'm battling with getting a "clean" repo up. Amplify CLI uses a key file (with all my goodies in there) for the build. If I remove that file, then the build fails.

I've just learned of another way to do this, and will let you know if/when I publicize my repo.

[dansharisan]

Any updates yet please? I'm also having this issue and is struggling to solve it.

[Borewit]

How can I reproduce the issue @dansharisan? Would a react app be sufficient?

[Borewit]

I have been able to reproduce the issue in a React application, I will look into it.

[dansharisan]

@Borewit I've just created a really simple vuejs app (default vuejs project + music-metadata-browser package) to reproduce the issue, in case you still need another reference, here is the link to the repo: https://github.com/dansharisan/vuesamples.git

(haven't seen your latest comment above before creating this)

[Borewit]

@dansharisan thanks a lot for your effort, let's see if we can get all of the test setups to work.

My first React app demonstrating this issue: https://github.com/Borewit/music-metadata-react

My react app uses webpack v4, which is using node-libs-browser v2.2.0, which depends on buffer v4.9.1.

It looks like buffer v4.9.1 missing some features music-metadata relies on. This is the source of evil 😉. In buffer v5 this issues are solved.

A PR has been submitted to update to buffer v5. In that PR they claim this issue should be solved in webpack v5, because webpack v5 (in alpha state) allows to use any buffer. Which implies a bit, that we cannot freely check another buffer version for webpack v4.

These are the options I can think of to address this issue:

1. Trick webpack to use buffer version 5 (update: see Borewit/music-metadata-react#2)
2. Put a workarnounds for buffer v4 shortcomings in music-metadata (respectviely the underlying libraries, like strtok3). Very ugly in my opinion. (update: minor change, has been done)
3. Maybe a buffer version 4 release fixing these issue (not sure of that is even possible) (update: created issue: Calling readUInt32LE with no arguments throws an exception in buffer v4 feross/buffer#237)
4. Ugrade to webpack 6?

To be continued...

Help with figuring out why it exactly breaks on buffer v4, link to existing issues, hooks to hack buffer v4 in are ofcourse welcome.

@feross: can you please give some advice?

[kimfucious]

Hi @Borewit sorry for the delay, I got distracted.

Please find a (hopefully) clean repo here.

My flailing can be found at /src/App.js on line 28

I look forward to your insight on how to do the shim right.

[Borewit]

I managed to fix my React app, using yarn package manager, in PR: Borewit/music-metadata-react#2

[dansharisan]

↑ that saved my life. Thanks @Borewit .

[Borewit]

Glad to here it is working for you!

If you can create a PR on your repo's addressing the issue that may help other users. Preferably without merging it. I may ask your help to test an alternative solution. One of the alternatives I want to investigate, if buffer v4 can be fixed and adapted in webpack v4. That would prevent other users running into the issue.

[Borewit]

One of the incompatibility issues I found (there maybe more) is buffer.readUInt32LE() is called without any argument, which does not seem to translate to buffer.readUInt32LE(offset = 0) in buffer v4.

This is a dead simple workaround can both include in music-metadata, and be proposed in a PR to buffer v4.
This seem to only occure once in music-metadata: id3v2/ID3v24TagMapper.ts#L172.

[Borewit]

Workaround released in music-metadata-browser v1.2.2.

Does this work for you guys, without the yarn buffer resolution?

[dansharisan]

Excellent. Yes, I did a fresh install to my dumb project with the update of music-metadata-browser to 1.2.2, without the yarn buffer resolution and confirm that it works.
Now I can continue using npm without having to install yarn. Big thanks.

[kimfucious]

Hi @Borewit , thanks for your continued efforts on this cool project.

I just removed node_modules, removed buffer from packages.json, removed yarn.lock, and bumped mm-b to v1.2.2, then did a clean npm install.

I also removed the shim bits in the app.

And... 🎉 !

[kimfucious]

Feel free to close this if you're satisfied with the results. I am. Or let me know if you need me to test anything.

[Borewit]

Thanks a lot for letting me know gents!

[kimfucious]

One last note from me: I just ran ~6500 music files through this beast, and it performed with 100% success! MM FTW! 🚀

[Borewit]

That's great to hear Kim!

If you use it to parse that many files, maybe music-metadata, which has node-js file support, fits your purpose better (it will parse audio files much faster). If you run the music-metadata parsing part on the node-js / server side, you can ofcourse still put all your GUI controls on the browser / client side.

[kimfucious]

Thanks for the tip, @Borewit

This bulk job was a one-off after a db change. The normal usage in my current app is rather low volume, so it's all good. In future applications of your awesome, music-metadata, I'll keep this in mind.