Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JS class for FSStream, FSNode and LazyUint8Array. NFC #21406

Merged
merged 3 commits into from
Feb 23, 2024
Merged

Use JS class for FSStream, FSNode and LazyUint8Array. NFC #21406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f250f64
Use JS class for FSStream. NFC
sbc100 Feb 23, 2024
e09d383
FSNode
sbc100 Feb 23, 2024
8f466ae
LazyUint8Array
sbc100 Feb 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 169 additions & 180 deletions src/library_fs.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,55 +39,7 @@ if (!Module['noFSInit'] && !FS.init.initialized)
FS.ignorePermissions = false;
`)
addAtExit('FS.quit();');
// We must statically create FS.FSNode here so that it is created in a manner
// that is visible to Closure compiler. That lets us use type annotations for
// Closure to the "this" pointer in various node creation functions.
return `
var FSNode = /** @constructor */ function(parent, name, mode, rdev) {
if (!parent) {
parent = this; // root node sets parent to itself
}
this.parent = parent;
this.mount = parent.mount;
this.mounted = null;
this.id = FS.nextInode++;
this.name = name;
this.mode = mode;
this.node_ops = {};
this.stream_ops = {};
this.rdev = rdev;
};
var readMode = 292/*{{{ cDefs.S_IRUGO }}}*/ | 73/*{{{ cDefs.S_IXUGO }}}*/;
var writeMode = 146/*{{{ cDefs.S_IWUGO }}}*/;
Object.defineProperties(FSNode.prototype, {
read: {
get: /** @this{FSNode} */function() {
return (this.mode & readMode) === readMode;
},
set: /** @this{FSNode} */function(val) {
val ? this.mode |= readMode : this.mode &= ~readMode;
}
},
write: {
get: /** @this{FSNode} */function() {
return (this.mode & writeMode) === writeMode;
},
set: /** @this{FSNode} */function(val) {
val ? this.mode |= writeMode : this.mode &= ~writeMode;
}
},
isFolder: {
get: /** @this{FSNode} */function() {
return FS.isDir(this.mode);
}
},
isDevice: {
get: /** @this{FSNode} */function() {
return FS.isChrdev(this.mode);
}
}
});
FS.FSNode = FSNode;
FS.createPreloadedFile = FS_createPreloadedFile;
FS.staticInit();` +
// Get module methods from settings
Expand Down Expand Up @@ -145,6 +97,79 @@ FS.staticInit();` +
}
},

FSStream: class {
constructor() {
this.shared = {};
#if USE_CLOSURE_COMPILER
this.node = null;
this.flags = 0;
#endif
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this ifdefing on closure compiler in the old code - why is it needed?

Also the old code seems to store the flags on .shared and this does not - is that an optimization?

Copy link
Collaborator Author

@sbc100 sbc100 Feb 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With real ES6 classes closure requires that all members be initialized in the construtor. If you try to add new properties later it generates a warning. Quite a nice feature but it does mean we need to predeclare these things.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this ifdefing on closure compiler in the old code - why is it needed?

Also the old code seems to store the flags on .shared and this does not - is that an optimization?

The new code also stores the flags on .shared. see the getter and setter for flags below. I'm not trying to change anything with this PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you try to add new properties later it generates a warning.

Hm but in this case properties are declared as getters and setters - I'd expect Closure to recognise those as "statically defined" properties and not require initialisation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah true, I tried and was able to remove that one line and it seems to work. I'm not sure what happened there because I was adding those specifically on demand to fix warnings.

}
get object() {
return this.node;
}
set object(val) {
this.node = val;
}
get isRead() {
return (this.flags & {{{ cDefs.O_ACCMODE }}}) !== {{{ cDefs.O_WRONLY }}};
}
get isWrite() {
return (this.flags & {{{ cDefs.O_ACCMODE }}}) !== {{{ cDefs.O_RDONLY }}};
}
get isAppend() {
return (this.flags & {{{ cDefs.O_APPEND }}});
}
get flags() {
return this.shared.flags;
}
set flags(val) {
this.shared.flags = val;
}
get position() {
return this.shared.position;
}
set position(val) {
this.shared.position = val;
}
},
FSNode: class {
constructor(parent, name, mode, rdev) {
if (!parent) {
parent = this; // root node sets parent to itself
}
this.parent = parent;
this.mount = parent.mount;
this.mounted = null;
this.id = FS.nextInode++;
this.name = name;
this.mode = mode;
this.node_ops = {};
this.stream_ops = {};
this.rdev = rdev;
this.readMode = 292/*{{{ cDefs.S_IRUGO }}}*/ | 73/*{{{ cDefs.S_IXUGO }}}*/;
this.writeMode = 146/*{{{ cDefs.S_IWUGO }}}*/;
}
get read() {
return (this.mode & this.readMode) === this.readMode;
}
set read(val) {
val ? this.mode |= this.readMode : this.mode &= ~this.readMode;
}
get write() {
return (this.mode & this.writeMode) === this.writeMode;
}
set write(val) {
val ? this.mode |= this.writeMode : this.mode &= ~this.writeMode;
}
get isFolder() {
return FS.isDir(this.mode);
}
get isDevice() {
return FS.isChrdev(this.mode);
}
},

//
// paths
//
Expand Down Expand Up @@ -421,44 +446,7 @@ FS.staticInit();` +
// object isn't directly passed in. not possible until
// SOCKFS is completed.
createStream(stream, fd = -1) {
if (!FS.FSStream) {
FS.FSStream = /** @constructor */ function() {
this.shared = { };
};
FS.FSStream.prototype = {};
Object.defineProperties(FS.FSStream.prototype, {
object: {
/** @this {FS.FSStream} */
get() { return this.node; },
/** @this {FS.FSStream} */
set(val) { this.node = val; }
},
isRead: {
/** @this {FS.FSStream} */
get() { return (this.flags & {{{ cDefs.O_ACCMODE }}}) !== {{{ cDefs.O_WRONLY }}}; }
},
isWrite: {
/** @this {FS.FSStream} */
get() { return (this.flags & {{{ cDefs.O_ACCMODE }}}) !== {{{ cDefs.O_RDONLY }}}; }
},
isAppend: {
/** @this {FS.FSStream} */
get() { return (this.flags & {{{ cDefs.O_APPEND }}}); }
},
flags: {
/** @this {FS.FSStream} */
get() { return this.shared.flags; },
/** @this {FS.FSStream} */
set(val) { this.shared.flags = val; },
},
position : {
/** @this {FS.FSStream} */
get() { return this.shared.position; },
/** @this {FS.FSStream} */
set(val) { this.shared.position = val; },
},
});
}

// clone it, so we can return an instance of FSStream
stream = Object.assign(new FS.FSStream(), stream);
if (fd == -1) {
Expand Down Expand Up @@ -1652,111 +1640,112 @@ FS.staticInit();` +
// XHR, which is not possible in browsers except in a web worker! Use preloading,
// either --preload-file in emcc or FS.createPreloadedFile
createLazyFile(parent, name, url, canRead, canWrite) {
// Lazy chunked Uint8Array (implements get and length from Uint8Array). Actual getting is abstracted away for eventual reuse.
/** @constructor */
function LazyUint8Array() {
this.lengthKnown = false;
this.chunks = []; // Loaded chunks. Index is the chunk number
}
LazyUint8Array.prototype.get = /** @this{Object} */ function LazyUint8Array_get(idx) {
if (idx > this.length-1 || idx < 0) {
return undefined;
}
var chunkOffset = idx % this.chunkSize;
var chunkNum = (idx / this.chunkSize)|0;
return this.getter(chunkNum)[chunkOffset];
};
LazyUint8Array.prototype.setDataGetter = function LazyUint8Array_setDataGetter(getter) {
this.getter = getter;
};
LazyUint8Array.prototype.cacheLength = function LazyUint8Array_cacheLength() {
// Find length
var xhr = new XMLHttpRequest();
xhr.open('HEAD', url, false);
xhr.send(null);
if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status);
var datalength = Number(xhr.getResponseHeader("Content-length"));
var header;
var hasByteServing = (header = xhr.getResponseHeader("Accept-Ranges")) && header === "bytes";
var usesGzip = (header = xhr.getResponseHeader("Content-Encoding")) && header === "gzip";

#if SMALL_XHR_CHUNKS
var chunkSize = 1024; // Chunk size in bytes
#else
var chunkSize = 1024*1024; // Chunk size in bytes
// Lazy chunked Uint8Array (implements get and length from Uint8Array).
// Actual getting is abstracted away for eventual reuse.
class LazyUint8Array {
constructor() {
this.lengthKnown = false;
this.chunks = []; // Loaded chunks. Index is the chunk number
#if USE_CLOSURE_COMPILER
this.getter = undefined;
this._length = 0;
this._chunkSize = 0;
#endif

if (!hasByteServing) chunkSize = datalength;

// Function to get a range from the remote URL.
var doXHR = (from, to) => {
if (from > to) throw new Error("invalid range (" + from + ", " + to + ") or no bytes requested!");
if (to > datalength-1) throw new Error("only " + datalength + " bytes available! programmer error!");

// TODO: Use mozResponseArrayBuffer, responseStream, etc. if available.
}
get(idx) {
if (idx > this.length-1 || idx < 0) {
return undefined;
}
var chunkOffset = idx % this.chunkSize;
var chunkNum = (idx / this.chunkSize)|0;
return this.getter(chunkNum)[chunkOffset];
}
setDataGetter(getter) {
this.getter = getter;
}
cacheLength() {
// Find length
var xhr = new XMLHttpRequest();
xhr.open('GET', url, false);
if (datalength !== chunkSize) xhr.setRequestHeader("Range", "bytes=" + from + "-" + to);
xhr.open('HEAD', url, false);
xhr.send(null);
if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status);
var datalength = Number(xhr.getResponseHeader("Content-length"));
var header;
var hasByteServing = (header = xhr.getResponseHeader("Accept-Ranges")) && header === "bytes";
var usesGzip = (header = xhr.getResponseHeader("Content-Encoding")) && header === "gzip";

#if SMALL_XHR_CHUNKS
var chunkSize = 1024; // Chunk size in bytes
#else
var chunkSize = 1024*1024; // Chunk size in bytes
#endif

if (!hasByteServing) chunkSize = datalength;

// Function to get a range from the remote URL.
var doXHR = (from, to) => {
if (from > to) throw new Error("invalid range (" + from + ", " + to + ") or no bytes requested!");
if (to > datalength-1) throw new Error("only " + datalength + " bytes available! programmer error!");

// TODO: Use mozResponseArrayBuffer, responseStream, etc. if available.
var xhr = new XMLHttpRequest();
xhr.open('GET', url, false);
if (datalength !== chunkSize) xhr.setRequestHeader("Range", "bytes=" + from + "-" + to);

// Some hints to the browser that we want binary data.
xhr.responseType = 'arraybuffer';
if (xhr.overrideMimeType) {
xhr.overrideMimeType('text/plain; charset=x-user-defined');
}

// Some hints to the browser that we want binary data.
xhr.responseType = 'arraybuffer';
if (xhr.overrideMimeType) {
xhr.overrideMimeType('text/plain; charset=x-user-defined');
xhr.send(null);
if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status);
if (xhr.response !== undefined) {
return new Uint8Array(/** @type{Array<number>} */(xhr.response || []));
}
return intArrayFromString(xhr.responseText || '', true);
};
var lazyArray = this;
lazyArray.setDataGetter((chunkNum) => {
var start = chunkNum * chunkSize;
var end = (chunkNum+1) * chunkSize - 1; // including this byte
end = Math.min(end, datalength-1); // if datalength-1 is selected, this is the last block
if (typeof lazyArray.chunks[chunkNum] == 'undefined') {
lazyArray.chunks[chunkNum] = doXHR(start, end);
}
if (typeof lazyArray.chunks[chunkNum] == 'undefined') throw new Error('doXHR failed!');
return lazyArray.chunks[chunkNum];
});

if (usesGzip || !datalength) {
// if the server uses gzip or doesn't supply the length, we have to download the whole file to get the (uncompressed) length
chunkSize = datalength = 1; // this will force getter(0)/doXHR do download the whole file
datalength = this.getter(0).length;
chunkSize = datalength;
out("LazyFiles on gzip forces download of the whole file when length is accessed");
}

xhr.send(null);
if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status);
if (xhr.response !== undefined) {
return new Uint8Array(/** @type{Array<number>} */(xhr.response || []));
this._length = datalength;
this._chunkSize = chunkSize;
this.lengthKnown = true;
}
get length() {
if (!this.lengthKnown) {
this.cacheLength();
}
return intArrayFromString(xhr.responseText || '', true);
};
var lazyArray = this;
lazyArray.setDataGetter((chunkNum) => {
var start = chunkNum * chunkSize;
var end = (chunkNum+1) * chunkSize - 1; // including this byte
end = Math.min(end, datalength-1); // if datalength-1 is selected, this is the last block
if (typeof lazyArray.chunks[chunkNum] == 'undefined') {
lazyArray.chunks[chunkNum] = doXHR(start, end);
return this._length;
}
get chunkSize() {
if (!this.lengthKnown) {
this.cacheLength();
}
if (typeof lazyArray.chunks[chunkNum] == 'undefined') throw new Error('doXHR failed!');
return lazyArray.chunks[chunkNum];
});

if (usesGzip || !datalength) {
// if the server uses gzip or doesn't supply the length, we have to download the whole file to get the (uncompressed) length
chunkSize = datalength = 1; // this will force getter(0)/doXHR do download the whole file
datalength = this.getter(0).length;
chunkSize = datalength;
out("LazyFiles on gzip forces download of the whole file when length is accessed");
return this._chunkSize;
}
}

this._length = datalength;
this._chunkSize = chunkSize;
this.lengthKnown = true;
};
if (typeof XMLHttpRequest != 'undefined') {
if (!ENVIRONMENT_IS_WORKER) throw 'Cannot do synchronous binary XHRs outside webworkers in modern browsers. Use --embed-file or --preload-file in emcc';
var lazyArray = new LazyUint8Array();
Object.defineProperties(lazyArray, {
length: {
get: /** @this{Object} */ function() {
if (!this.lengthKnown) {
this.cacheLength();
}
return this._length;
}
},
chunkSize: {
get: /** @this{Object} */ function() {
if (!this.lengthKnown) {
this.cacheLength();
}
return this._chunkSize;
}
}
});

var properties = { isDevice: false, contents: lazyArray };
} else {
var properties = { isDevice: false, url: url };
Expand All @@ -1775,7 +1764,7 @@ FS.staticInit();` +
// Add a function that defers querying the file size until it is asked the first time.
Object.defineProperties(node, {
usedBytes: {
get: /** @this {FSNode} */ function() { return this.contents.length; }
get: function() { return this.contents.length; }
}
});
// override each stream op with one that tries to force load the lazy file first
Expand Down
2 changes: 1 addition & 1 deletion test/other/metadce/test_metadce_cxx_ctors1.gzsize
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
9953
9887
Loading
Loading