api documentation updates - devel branch

[ghost]

While implementing v2 in an app, I'm coming across some irregularities and parts that I seem to be missing in the documentation or needs clarification, so I thought starting a thread to gather them here is useful, feel free to chip in, to help new adopters.

@raix @aldeed Can we keep this issue/thread open for now, to have a single go to place for api docs updates for the devel branch? If so, then I'll simply add issues here one by one, as I stumble upon them.

Missing

• After having uploaded files serverside, eg. an image file for use as a graphics element in an app, a natural/intuitive/naive next step seems to be to show it (the single image) in a template. Either via a direct like as if it was uploaded to /public, or via a helper, eg. Template.[template].image = function () { return [fsCollectionName].findOne({_id: id}).url(); }; Maybe there is a better implementation already? What is the way to do this?

Irregularities

• cfs-handlebars: is it still recommended or is it deprecated/obsolete due to integrated UI helpers in the new v2 API? (the current doc recommends it)
• Insert One File From a Remote URL mentions FS.File.fromUrl, however this no longer works.

[raix]

@casperanderseneu I thinks its a good idea - The issue will prop. grow long (we had a similar thread called "rewrite" - a google group could perhaps solve it better but what ever works)

So the id in the helper comes from some other document? I would say that the preferred way would be:

  {{#with foo}}
    My name is {{name}} and my image <img src="{{image.url store='thumbnail'}}">
  {{/with}}

Near full example (not tested but should show the intended use)

  // We depend on gm - can this be done from app level yet?
  var gm = Npm.require('gm');

  // We have a regular meteor collection
  var fooList = new Meteor.Collection('fooList');

  // Our images
  var images = new FS.Collection('images', { stores: [
    new FS.FileSystem('thumbnail', {
      path: '/path/to/images',
      transform: function(fileObj /* or readStream, writeStream? */) {
        // This api is WIP not implemented yet - 
        gm(fileObj.createReadStream())
        .resize(10)
        .stream()
        .pipe(fileObj.createWriteStream('thumbnail'));
      }
    })
  ]});

 // Add some data - this would normally be done via events

// Insert + upload a file
 var fileObj = images.insert(file);

 // Insert the mounted image into the normal collection
 fooList.insert({ name: 'bar', image: fileObj });

  // Template helper
  Template.hello.foo = function() {
    return fooList.findOne();
  };

• cfs-handlebars - is deprecating should be obsolete at some point - The current api is tightly integrated with the way Blaze works - We have added a package for dropped events - ui-dropped-event there is a very short code example on the package read me (only thing there is really)
• Correct fromUrl no longer works - we use the attachData api now

  var fileObj = new FS.File({ name: 'myfile.ext' });
  fileObj.attachData(/* file/blob/Buffer/readStream/url */);
  images.insert(fileObj);

  // I think we will also support this directly or already does?
  var fileObj = images.insert(/* file/url */);
  var fileObj = images.insert('filename.ext', /* file/blob/Buffer/readStream/url */);

One point we need to add / stress in the docs - don't write your own Meteor.methods for insert/remove etc. its a common mistake to think thats any safer than letting the client-side api do the works. Its more secure to actually use the client-side api since we do check allow/deny rules on the server - and its basically Meteor.methods behind the scenes any way.

[raix]

We are also talking about deprecating cfs-graphicsmagic since the streaming api in the transform function is basically pure gm api.

It would be possible to create transform helper functions eg. in separate packages:

// Our simple transformation function for resizing images
FS.Transform.resizeImage = function(size) {
  // Maybe it makes sense to serve the read and write stream
  return function(fileObj, readStream, writeStream) {
        // This api is WIP not implemented yet - 
        gm(readStream)
        .resize(size)
        .stream()
        .pipe(writeStream);
      }
};

Would simplify api:

  // Our images
  var images = new FS.Collection('images', { stores: [
    new FS.FileSystem('thumbnail', {
      path: '/path/to/images',
      transform: FS.Transform.resizeImage(10)
    })
  ]});

[aldeed]

@casperanderseneu, just to add one thing, we would accept PR to update the README. At the moment, the API has been changing almost daily so we have not yet bothered to update everything, but it will smooth out within 2 weeks. Often it seems that someone trying to understand from outside the development group is the best qualified to write documentation because they know what questions people will have.

But that said, we plan to release around the end of the month, and the docs will be updated at that point. We'll use this issue as a point of reference to see what we need to explain better.

[ghost]

@aldeed, perfect, that was the intention. Focus is on identifying best practices using the existing api design, as inspiration for the readme.

[ghost]

@raix Indeed a google group, seems better for discussing these issues one by one in each their thread. An alternative is to create an issue here for each individual things we stumble upon. And a google group can also motivate to more user-user discussions, which will be quite useful when the v2 rocket launches for release, to support adoption.

@aldeed @raix would that be useful to you? to have a google group eg. meteor-collectionFS, and perhaps meteor-collectionfs-docs (which can also cover QA and testing since it should be in harmony) ? and then we can close this issue, and refer to the group ?

[raix]

I would say ok for meteor-collectionFS or meteor-cfs?

Docs would be ok too - agree that QA/tests/docs are tightly connected - I haven't tested https://hackpad.com but it may be useful in documentation case? - since (as I get it) we could discuss while forming a spec/project/task and it got syntax highlighting? thats the nice thing about having the discussion in git issues

Note: we have been discussing strategy for version numbers etc. target is 1.0.0 - this way we can follow the major version of Meteor (same goes for packages) - we have been using v1/v2 we should really have been saying "old" / "new" api I guess. My bad - so v1.0.0 release.

@aldeed could you confirm? I'm unsure if we actually agreed on this in reality or just in my head, I may have continued our discussion on my own :)

[aldeed]

Truth is I really don't like google groups, but whatever works best for the most people is fine with me. IMO, github issues (which we can tag with discussion or question tag) serve the same purpose and give better code markdown, etc.

Yes, @raix, that sounds like what we decided about versions. :)

[raix]

So you vote for keeping it simple? I'm fine with all - we could use the meteor talk for community discussions?

MDG uses google groups for feature requests - that pattern doesn't work well, its a black hole for new ideas - so we should avoid gg when discussing something we want to keep a track of.

We can tag in gg but it doesn't have syntax highlighting - why I like github - but discussion threads can become very long - why I'm thinking about hack pad for arc discussions etc. but haven't had the time to check it out.

[ghost]

ok, either works for me. going with github issues. I also prefer having code highlighting.

[lleonard188]

For the section "Securing Based on User Information" , has that code changed? I'm trying to attach Meteor userIDs to images.

Also, is there a way to limit how many files a user can upload?

[aldeed]

@lleonard188, that example should still work. For limiting uploads, security for insert is done through allow/deny as if it's a normal collection, so you can run a search in a deny function to see how many files already belong to that user and return true if they already have the maximum. Also, for the example in the readme, if you use that code on the client, you would want to verify the owner ID is the current userId in a deny function as well, since client could set it to any user.

You client code could also do a redundant check for max uploads if all the user's files are published to the client, which would save you the server trip and processing.

[lleonard188]

@aldeed I'm still a bit confused. From what I've read the code below is what I've come up with for an upload-image-and-display-it app, but I don't know how I would find the images belonging to a particular user. Thanks in advance for any help.

HTML

<template name="imageUploader">
    <h2>Picture</h2>
    <p>
    {{#each images}}
            <img src="{{url}}" alt="">
            {{cfsDeleteButton}}
        {{else}}
            No files uploaded.
    {{/each}}
    </p>
    <p>Upload profile pic:</p>
    <p>{{cfsFileInput myImageFiles metadata}}</p>
</template>

client

var imageStore = new FS.Store.S3("images");

var Images = new FS.Collection("images", {
    stores: [imageStore],
    filter: {
        allow: {
            contentTypes: ['image/*']
        }
    }
});

Meteor.subscribe("images");

Template.imageUploader.myImageFiles = function() {
    return Images;
};

Template.imageUploader.images = function () {
    return Images.find();
};

Template.imageUploader.metadata = {
    userId:Meteor.userId()
};

server

var imageStore = new FS.Store.S3("images", {
    region: "us-east-1",
    accessKeyId: "***********", 
    secretAccessKey: "*********", 
    bucket: "*******"
});

var Images = new FS.Collection("images", {
    stores: [imageStore],
    filter: {
        allow: {
            contentTypes: ['image/*']
        }
    }
});

Images.allow({
    insert: function(userId) {
        return !!userId;
    },
    update: function(userId) {
        return !!userId;
    },
    remove: function(userId) {
        return !!userId;
    },
    download: function(userId) {
        return true;
    },
    fetch: []
});

Meteor.publish("images", function() {
    return Images.find();
});

[raix]

@lleonard188: image-uploader example gist

[aldeed]

I just went through the main package readme and attempted to get all the errors corrected. Still needs work but in general the API examples should be correct as of this moment.

[raix]

Cool 👍 The api is settling, feels nice - Auto tests should be up and running, cfs-file may need some love, We need to work a bit on the worker etc. I'll try to update milestones etc. To get some overview of status.

[lleonard188]

I have couple of suggestions for the documentation:

1. Give a full working simple example e.g. HTML, Subscribe/Publish, etc. along with the javascript, maybe in the Getting Started?
2. In the Example code section under the heading "Insert One or More Files From File Input", I think it's same code used in the getting started section. Is this redundant? Maybe have simple example in getting started then have more complex code in examples section?
3. For Allow/Deny rules show code example?
4. Url helper section should show function that returns files to template

Let me know what you think and if it's OK I'll work on a PR

[aldeed]

Sounds good to me. I don't know if a complete example should be in readme or just some links to example app repos that show the complete code. We all have our various testing apps, but it would be nice to have a simple showcase app that shows simple file management with ownership, drag/drop, metadata, virtual folders, sorting, filtering, etc.

[raix]

Documentation and examples roundup
Theres a couple of contributors that have created examples and contributed with code:
@nooitaf did the cfs-public-folder and create a small repo example on the new api #97 (comment)
@sanjo just fixed the s3 SA and created a small test on transformWrites #210 (comment) Maybe collect some of these use cases?
@copleykj Did also speak about having some examples to get people started and would be able to contribute with code documentation? :) #95 (comment)
@casperanderseneu Is doing some QA this week and will hopefully chip in with docs, api notes and related stuff
@mxab Did feature some nice examples on the old and new api?
@cramhead Did play cfs in his collectionFSPlay (new api?)
@marcoscuevas Did some groundwork integrating cfs and native objective-C
@vsivsi Did help make the gridfs use the official / real gridfs spec, he created an example #182 (comment) and he also did some really nice benchmarks (I can't remember where they are) - And made us aware that cfs is really in many ways a FMS - File Managing System

I think its a good approach - the small cloneable repos makes it easy for starters - the docs should have small principle examples explaining the api and the system. We could have this in the wiki but thinking about it better yet organized in a /docs folder using md files, it follows the repo code / version / api?

1. The CollectionFS repo should document the whole system and how to work with it, its a bit difficult going into depth here and have it be consistent with the module documentation
2. We should reference the module repos for more in depth api docs? (modules docs would be for advanced features and further development - where the collectionFS docs is for people to get started with the most common problems)
3. We already auto generate code documentation, in a jsdoc similar way - using docmeteor allows us to use Meteor specific annotations

I think we should agree on some basic documentation structure, and thinking about it I think it would be best with a docs folder structured by topics:

• Getting started
  • Installation
  • Basic upload
  • etc.
• Concepts
  • FS.File
  • FS.Collection
  • Storeage Adapters
  • Transform Streams
  • Access Points
  • etc.
• Architecture
  • The mechanics
  • Data flows
  • The overview diagram
  • etc.
• Modules
  • Short explanation of the cfs modules
  • ui-dropped-event
  • etc.
• Usecases / examples
  • Small walkthroughs / screen casts etc.
• Contributing
  • Code conventions
  • Tools (mbr + docmeteor etc.)
  • etc.

The outline is just from the tip of my head - @lleonard188 It would be really great if you want to contribute any way you can, starting up the docs folder, maybe @casperanderseneu and others will chip in :) (I hope, its much welcome)

I have to use @aldeed for the last internal code + test writing - we are focused on writing tests and make the code solid - Eric we got one week left...

The api should not change - and we are here nearly 24 hours a day, most days - if you got questions etc. :)

[aldeed]

Other than docs and tests, I think we have to rewrite the worker and address some "TODO" comments throughout. Shouldn't be too much else.

I've been busy with other projects, but I think I'll be able to finish up tests this week (as well as fixing any issues they find). If others could work on writing docs and creating examples, that would be helpful. I don't think those need to be 100% finished by April, but we should try to have the doc outline in place. @raix, I'm assuming you can do the worker changes?

[raix]

Yep, the rewrite of the worker is not too bad - I'm considering the pull arc mentioned by @vsivsi - Would be awesome if you work on some tests - I'm pretty busy too (got 2 projects with milestone deadlines friday) - I tend to keep my deadlines, but hope to do the same on cfs :) EDIT: so I may cross your timezone :)

[ghost]

is there an api change in progress?

• after running 'mrt update' to get the latest, I get an error:

W20140325-20:01:54.857(1)? (STDERR) packages/meteor.js:964
W20140325-20:01:54.857(1)? (STDERR)     Fiber(runWithEnvironment).run();
W20140325-20:01:54.858(1)? (STDERR)                               ^
W20140325-20:01:54.871(1)? (STDERR) Error: SA images type storage.filesystem did not return a fileKey
W20140325-20:01:54.872(1)? (STDERR)     at packages/cfs-storage-adapter/storageAdapter.server.js:110
W20140325-20:01:54.872(1)? (STDERR)     at runWithEnvironment (packages/meteor/dynamics_nodejs.js:89)

[raix]

just published it to mrt - when released we'll use the devel branches to avoid updating mrt by accident

[lingz]

I know the API is rapidly changing right now but I agree with @casperanderseneu in terms of needing some information with regards to all the ways one one can access the files. I see that the URL is available, but is it possible to get the datastream of images directly from a collection cursor that's already been subscribed to client-side for example?

[aldeed]

@lingz, what do you mean by datastream? There aren't streams on the client. If you explain what you're trying to do, we can tell you the best way.

[lingz]

@aldeed sorry, data steam was the wrong terminology. I meant have all the data for the images ready via subscriptions so they can be used via data uris without further load times.

[raix]

@lingz thats a very large subject and we have been discussing this a lot - And Im not 100% of what you are asking, but:

Publish
You can use regular Meteor publish for publishing collectionFS

Download
You can basically download data in tree ways from cfs but you have to know the id and collection - maybe a store name (optional defaults to the primary store, the first in store array) and the http baseUrl
the user have to be granted access in allow/deny rules download section

1. via http
2. via ddp
3. directly via http from services like s3

Joins
So how to publish the collectionFS that is joined in other collections?

1. We did play with a no schema model its the experimental package join but it lacks in security and tests - could be implemented - but we are not sure its the right pattern.
2. There are a couple of join packages these could perhaps prove valid too
3. Meteor got joins on their roadmap - stuff to come

Client-side Streams
I've actually been playing with the idea of having streams on the client - that could pipe file data via sockets/xhr or via ddp - the file reader and xhr + power-queue could be contained in these client-side streams (this is not for 1.0.0 - its future discussion its in the issues somewhere)

[raix]

Or are you talking about a "pre" loader for files?

[lingz]

Essentially the behavior I'm thinking is to be able to throw up a "loading" animation until I know that all the images for the objects I'm displaying have been downloaded already, and then have some kind of reactive data source let me know that the images are ready.

For example if I have the data-uri of all the images, I can display them all straight away without having to let the browser deal with fetching all the images from their own URLs.

[aldeed]

Basically what you want to do is download the binary and attach it to the file object so that you can display it from a data URI. So:

Download the data:

Use fileObj.url({store: yourStoreName}) to get the URL. Then use an XMLHttpRequest to get the arraybuffer data from that URL.

Attach the data:

Pass the arraybuffer to attachData. (fileObj.attachData(arrayBuffer))

Get data URI:

fileObj.getDataUri(function (uri) {
  //save URI to a reactive variable or set into img src
});

For those first two steps, I think we can add an attachDataFromStore method on the client to make that simple, or maybe attachData({store: storeName}).

[raix]

Update:
FS.TempStore now uses the Storage Adapter api and defaults to FS.Store.FileSystem or FS.Store.GridFS depending on installed packages. Filesystem is preferred unless cfs-worker is in the project, then FS.Store.GridFS is preferred due to scalability over multiple instances of Meteor.

Added code documentation FS.TempStore https://github.com/CollectionFS/Meteor-cfs-tempstore

[ghost]

For inserting a file, server side, via file path, I read the most simple methods in the docs to be:

Images.insert( '/path/to/file/image.png'); // where Images is a FS.Collection
Images.insert( '/path/to/file/image.png', function (error, filObj) { } ); // with a callback

Is this correct?
Is this working for anyone?

Experience:

• The file gets created on both the file system and in the cfs mongodb collection
• Using cfs-filesystem as Storage Adapter
• from the latest code as of today (collectionFS and cfs-filesystem)

Problem:

• The file is empty, it is a 0 byte file on both the file system and in the 'cfs.images.filerecord' mongodb collection document.

[raix]

I think we have to rework some of the stuff to take better advantage of the simpler streaming api #239
I havent tested that part of the api, but I see a node fs dependency and thats potentially a problem when deploying.

But yes, the docs seems to be ok, we just need to align the api :)

[ghost]

neat.

Just for reference
until api alignment, a quick resolution is to use url instead of filepath eg.

Images.insert( 'http:/example.org:port/path/to/file/image.png' );
Images.insert( 'http:/example.org:port/path/to/file/image.png', function (error, fileObj) { } );

Both works.

[aaronjudd]

Two comments, in the above example:

 fileObj.url({store: yourStoreName}) 

would be great if examples of accessing data from code, not just templates were provided wherever possible . Took me some time to find this.

Also at the moment:

FS.HTTP.setBaseUrl('/files')

causes ReferenceError: mountUrls is not defined

Wasn't sure if it was doc error, as I saw some comments on the update api docs on this. (but not working for me yet)

Last note, @raix and @aldeed - this is becoming an incredibly useful project - right on par with Collections2 - my kudos for an amazing job. Hope we all have stable projects out for meteor 1.0

[aldeed]

@aaronjudd, I fixed the setBaseUrl issue now.

[aldeed]

@casperanderseneu, currently I'm getting a "RangeError: Maximum call stack size exceeded" error when doing server-side insert of filepath or URL, like your example. I don't know if this is what you're getting, too. I couldn't figure out the cause at the moment, but maybe @raix will have an idea. Further discussion can move to #246.

[calculon0]

The current README uses

var newFile = new FS.File();
newFile.attachData(file);

in a few locations. But that throws an error of "DataMan constructor received data that it doesn't support". Changing it to

var newFile = new FS.File(file);

makes it work.

[aldeed]

@calculon0, what type is file? Is it a browser File object? DataMan should support it, especially because if you pass it to the FS.File constructor, it passes it along to attachData for you anyway.

[calculon0]

@aldeed I think the issue is the error is being thrown when the constructor is empty. I don't think DataMan allows an empty constructor.

[aldeed]

OK, I guess that's correct behavior then, but we could add a more specific error.

[abuddenb]

An example of how to insert a Buffer would be great. I wrestled with it all afternoon:

request.get({url: result.downloadUrl, encoding: null}, Meteor.bindEnvironment(function(e ,r, body){
    var newFile = new FS.File();
    newFile.attachData(body, {type: 'image/png'}, function(error){
        if(error) throw error;
        newFile.name('mypicture.png');
    });
    Images.insert(newFile);

The options object on attachData() could really use a writeup.

[aldeed]

@abuddenb, if you want to submit a PR with readme updates, we would accept that. There are also lots of examples in the unit tests for the cfs-file package, which could be linked to.

https://github.com/CollectionFS/Meteor-cfs-file/blob/master/tests/file-tests.js#L123