Remove unnecessary code

[jcnelson]

blockstack-profiles should not be responsible for constructing zonefiles or resolving them to data.
This should be the exclusive responsibility of the blockstack-client package, since

• blockstack-client is already responsible for constructing and manipulating zonefiles for
  storing things like immutable data and data public keys;
• the Blockstack-specific uses of the zonefile are implementation details that are subject to change, so we don't want external programs to assume anything about their contents;
• the protocol-specific URL construction, formatting, and resolution logic is already provided by
  the blockstack-storage-drivers package;
• blockstack-client already provides a unified interface for locating data via the blockstack:// URL scheme.

[shea256]

OK this is a good thing to discuss although I'm unsure about the best way to organize this functionality.

I'd point out that the Blockstack client isn't the only package using Blockstack Profiles. Currently, both onename.com and the blockstack desktop app use these functions:

• make_zone_file_for_hosted_data
• resolve_zone_file_to_profile

Are you suggesting that both apps start importing the Blockstack client or the Blockstack Storage Drivers library?

Also, I'm confused by what you mean by resolving via the URL scheme. Can you explain? We designed zone files such that traditional DNS resolvers will already know how to resolve the information. New URI protocols can be useful in other contexts, of course. Is this what you mean or do you mean something else?

[jcnelson]

I'd point out that the Blockstack client isn't the only package using Blockstack Profiles.

Ah, I didn't know that.

Are you suggesting that both apps start importing the Blockstack client or the Blockstack Storage Drivers library?

Hmmm...what if we implemented make_zone_file_for_hosted_data and resolve_zone_file_to_profile as top-level methods in the blockstack_client package? They'd work exactly the same, but would live in the blockstack_client package so they could use its underlying machinery. Module import notwithstanding, would that break compatibility?

Also, I'm confused by what you mean by resolving via the URL scheme. Can you explain? We designed zone files such that traditional DNS resolvers will already know how to resolve the information. New URI protocols can be useful in other contexts, of course. Is this what you mean or do you mean something else?

The DNS zonefile would continue to contain the URI record for external consumption, but the URI record is primarily consumed by the storage drivers (since they're the entities that put them there in the first place). The URI can contain driver-specific information that's not obvious to client applications--for example, a URI can contain post fields for the service that the client does not know about. Also, not every URI can be handled by existing Python packages like requests, and drivers for particularly new services like IPFS might not even have standard URIs. By going through blockstack_client, we spare client applications from having to know about the driver and service implementations.

[shea256]

Hmmm...what if we implemented make_zone_file_for_hosted_data and resolve_zone_file_to_profile as top-level methods in the blockstack_client package? They'd work exactly the same, but would live in the blockstack_client package so they could use its underlying machinery. Module import notwithstanding, would that break compatibility?

OK so then onename.com would import the Blockstack Client package? That could work too but then it just has to import everything.

Side note: we currently call it Blockstack CLI on GitHub but maybe that name should change to Blockstack Client to make it clear that people are importing the Blockstack client library and not importing a CLI, which doesn't really make sense.

Also, keep in mind that we also have a JS version of this library which has the same functions and it would be nice to continue to keep them somewhat in sync. Would we start a Blockstack Client library in JS?

[jcnelson]

OK so then onename.com would import the Blockstack Client package? That could work too but then it just has to import everything.

Would have to confirm with @muneeb-ali, but I'm pretty sure the backend of Onename.com already imports blockstack_client somewhere. blockstack-server depends on it, for example.

Side note: we currently call it Blockstack CLI on GitHub but maybe that name should change to Blockstack Client to make it clear that people are importing the Blockstack client library and not importing a CLI, which doesn't really make sense.

Hmm, we could split the blockstack-cli repo into blockstack-client (which holds the library), blockstack-cli (which holds the tool), and blockstack-runtime (which holds the API endpoint and local RPC daemon). Would that make it cleaner?

Also, keep in mind that we also have a JS version of this library which has the same functions and it would be nice to continue to keep them somewhat in sync. Would we start a Blockstack Client library in JS?

It would be nice to keep them in sync. What currently uses blockstack-profiles-js? Could we direct these applications to either blockstack-runtime or some server-side blockstack-client process that handles resolving URIs to data via the storage drivers?

Alternatively, we can hammer out a standard for what goes into URIs, so we can ensure that both vanilla Javascript and Python libraries will be able to handle them correctly (without storage drivers).

[shea256]

Would have to confirm with @muneeb-ali, but I'm pretty sure the backend of Onename.com already imports blockstack_client somewhere. blockstack-server depends on it, for example.

No, just confirmed - onename.com does not import the blockstack server or client.

Hmm, we could split the blockstack-cli repo into blockstack-client (which holds the library), blockstack-cli (which holds the tool), and blockstack-runtime (which holds the API endpoint and local RPC daemon). Would that make it cleaner?

That seems to make sense. The next question I would have would be "which repo do we direct users to?" to install the Blockstack CLI but then my instinctive answer is github.com/blockstack/blockstack, with the following installation pattern:

brew install blockstack
blockstack install server
blockstack install api
blockstack install atlas

The initial installation of blockstack would come with the client and the CLI and then everything else would have to be installed on top, including the full node server, the HTTP(S) API, and the Atlas plugin.

It would be nice to keep them in sync. What currently uses blockstack-profiles-js? Could we direct these applications to either blockstack-runtime or some server-side blockstack-client process that handles resolving URIs to data via the storage drivers?

I'm still of the view that we should allow software developers to do all of this stuff without a server. Consider that all frontend developers will be using Javascript, so we would really only need to maintain a library in a single language. Also consider that we'll need to be able to support name wallets in many different environments and on many different browsers. This includes mobile apps compiled from Javascript as well as client-only javascript environments where we might not be able to have a server running and where we wouldn't want to force the user to outsource functionality to a server not on their device (e.g. an app loaded through IPFS may use Blockstack libraries and shouldn't require an external server outside of connecting to a simple blockstack resolver that just returns the zone file).

Here's an alternative idea: what if we port the blockstack drivers over to Javascript?

If we only want to implement and maintain this functionality in a single language, I would go even futher and suggest that we use Javascript as our main language for working with the drivers and have the CLI run a node server process. Seem a bit radical, I know, but we've ported libraries to javascript fairly quickly in the past and if we consider our audience of developers, we'll really need to cater to a large frontend crowd.

Alternatively, we can hammer out a standard for what goes into URIs, so we can ensure that both vanilla Javascript and Python libraries will be able to handle them correctly (without storage drivers).

Yeah, I do like this idea as well and it shouldn't be too difficult.

[jcnelson]

I'm still of the view that we should allow software developers to do all of this stuff without a server. Consider that all frontend developers will be using Javascript, so we would really only need to maintain a library in a single language. Also consider that we'll need to be able to support name wallets in many different environments and on many different browsers. This includes mobile apps compiled from Javascript as well as client-only javascript environments where we might not be able to have a server running and where we wouldn't want to force the user to outsource functionality to a server not on their device (e.g. an app loaded through IPFS may use Blockstack libraries and shouldn't require an external server outside of connecting to a simple blockstack resolver that just returns the zone file).

I'm confused...the proposed blockstack-runtime functionality (currently part of blockstack-cli, in the blockstack_client package) is completely client-side. It runs locally on the device already as a daemon, just like IPFS. It doesn't depend on blockstack-server, and does not need to talk to blockstack-server unless you call a method that requires it.

It doesn't run in the browser, however.

Here's an alternative idea: what if we port the blockstack drivers over to Javascript?

That could be done :)

If we only want to implement and maintain this functionality in a single language, I would go even futher and suggest that we use Javascript as our main language for working with the drivers and have the CLI run a node server process. Seem a bit radical, I know, but we've ported libraries to javascript fairly quickly in the past and if we consider our audience of developers, we'll really need to cater to a large frontend crowd.

That would be a pretty big job--blockstack_client is ~6,700 lines of executable Python, and depends on non-trivial functionality in virtualchain, protocoin, keylib, pybitcoin, and blockstack-utxo (which would also need to be re-written).

How about this? I port blockstack-storage-drivers to Javascript, I make blockstack_client use the Javascript version (we drop the Python version), and I port enough of blockstack_client's driver-agnostic storage functionality to Javascript as well--enough to manage zonefiles and provide the top-level storage interface to Javascript clients. I'll update blockstack_client to call into the Javascript implementation, so we can drop the Python implementation there as well.

[shea256]

I'm confused...the proposed blockstack-runtime functionality (currently part of blockstack-cli, in the blockstack_client package) is completely client-side. It runs locally on the device already as a daemon, just like IPFS. It doesn't depend on blockstack-server, and does not need to talk to blockstack-server unless you call a method that requires it.

It doesn't run in the browser, however.

Sorry, what I meant by completely client-side is "running in the browser, without the need for a local server."

That would be a pretty big job--blockstack_client is ~6,700 lines of executable Python, and depends on non-trivial functionality in virtualchain, protocoin, keylib, pybitcoin, and blockstack-utxo (which would also need to be re-written).

Sorry, to clarify, I don't mean that we should implement all of the CLI's functionality in JS. I'm more referring to the drivers and related functionality. I'd also point out that we already have JS libraries for key encoding, keychains, and token management. In addition, there are actually some really well done libraries in JS like all the ones on the BitcoinJS GitHub. Contrast this with Python, where there wasn't any quality library available and we had to make pybitcoin a couple years back. Bitcore Lib also has a wonderfully designed interface but I prefer not to use them because they don't properly modularize and import way to many things and sometimes have import errors. I'm still not suggesting we port everything from the CLI over but it's good to know that there's a lot available and we wouldn't have to port over keylib, pybitcoin, or protocoin (or potentially Blockstack UTXO).

How about this? I port blockstack-storage-drivers to Javascript, I make blockstack_client use the Javascript version (we drop the Python version), and I port enough of blockstack_client's driver-agnostic storage functionality to Javascript as well--enough to manage zonefiles and provide the top-level storage interface to Javascript clients. I'll update blockstack_client to call into the Javascript implementation, so we can drop the Python implementation there as well.

Yes, I believe we're on the same page and this is exactly what I was suggesting. To clarify, though, when you say you'll make the Blockstack Client use the Javascript version, do you mean you'll have it start up a NodeJS process and then make calls to it?

[muneeb-ali]

I feel like there are multiple things being discussed in this thread. Here are some comments:

@jcnelson wrote:

blockstack-profiles should not be responsible for constructing zonefiles or resolving them to data.
This should be the exclusive responsibility of the blockstack-client package

I think of the blockstack-profiles package as the repo where "blockstack ID" specific stuff goes i.e., specifications for what type of data goes to the .id namespace. Obviously, other namespaces might have some overlap e.g., how zone files are used etc. But ignoring the overlap, I think of blockstack-profiles as a package that belong in the "identity layer" of the stack.

Should the identity layer be responsible for checking statements and constructing profile data? That's an easy yes.

Should the identity layer be responsible for constructing zone_files? That's unclear because zone_files is probably a more generic concept and tied more to the "data storage infrastructure". Correct me if I'm wrong, but I think part of the discussion stems from the fact that Jude is viewing this as data storage infrastructure and Ryan is taking an app-developer perspective.

@jcnelson wrote:

That would be a pretty big job--blockstack_client is ~6,700 lines of executable Python, and depends on non-trivial functionality in virtualchain, protocoin, keylib, pybitcoin, and blockstack-utxo (which would also need to be re-written).

I agree. I don't think we should go down that path. It's going to be a large engineering task and we can't do this for each new language. Having a common "sys call" layer might be the right thing to do here.

@shea256 wrote:

Side note: we currently call it Blockstack CLI on GitHub but maybe that name should change to Blockstack Client to make it clear that people are importing the Blockstack client library and not importing a CLI, which doesn't really make sense.

I do think that we need to re-factor some components. The downside of re-factoring is that it slows us down and we get into weird version issues. If we are refactoring, I'd say we need:

1. The "sys call package/daemon/background-process" that I think Jude is calling blockstack-runtime
2. The python client, which can be light-weight as most of the work is done by the background process
3. The CLI

@jcnelson wrote:

blockstack-client already provides a unified interface for locating data via the blockstack:// URL scheme.

I feel that this hasn't been discussed enough. I'd say that use of blockstack:// is not standard right now. It's in a proposal stage and needs more discussion. I think it's perfect for a RFC and we can discuss it there.

@shea256 wrote:

I'm still of the view that we should allow software developers to do all of this stuff without a server.

Can you explain that a bit? You mean without a blockstack-server as well?

@jcnelson wrote:

How about this? I port blockstack-storage-drivers to Javascript, I make blockstack_client use the Javascript version (we drop the Python version)

I think the Python drivers are important because (a) they'll go into blockstack-server which is written in Python and (b) they'll go into the common syscall daemon thing (we really need a name for it) which is also written in Python. If anything we can have separate JS drivers. Ryan is already sort of doing that with S3 data.

[shea256]

@muneeb-ali wrote:

Should the identity layer be responsible for constructing zone_files? That's unclear because zone_files is probably a more generic concept and tied more to the "data storage infrastructure". Correct me if I'm wrong, but I think part of the discussion stems from the fact that Jude is viewing this as data storage infrastructure and Ryan is taking an app-developer perspective.

To clarify, this library does not construct zone files. It calls this function:

from zone_file import make_zone_file

Then it makes a zone file by passing in a URI.

That said, this function is not specific to profile-centric namespaces and could be applied to any situation where you want a name to resolve to a specific hosted file. As an example, let's say you want blockstack.app to resolve to app.blockstack.org/index.html. You could pass in the URI to the make_zone_file_for_hosted_data function. Thus, I do buy the argument that this function does not belong in a library that is explicitly for "profiles". However, we could rename the function to be explicitly for profiles, as who knows whether there will be additional records in the future that are specific to profiles. It could be related to token parsing, for example. Thus, now that I think about it, it might be best if this function stays in this library but is just renamed to include the word "profile".

Additionally, I do think that the function resolve_zone_file_to_profile belongs in this library, as it's clearly specific to working with profiles.

[shea256]

@muneeb-ali wrote:

I agree. I don't think we should go down that path. It's going to be a large engineering task and we can't do this for each new language. Having a common "sys call" layer might be the right thing to do here.

+1

@muneeb-ali wrote:

I do think that we need to re-factor some components. The downside of re-factoring is that it slows us down and we get into weird version issues. If we are refactoring, I'd say we need:

1. The "sys call package/daemon/background-process" that I think Jude is calling blockstack-runtime
2. The python client, which can be light-weight as most of the work is done by the background process
3. The CLI

+1

@muneeb-ali wrote:

I feel that this hasn't been discussed enough. I'd say that use of blockstack:// is not standard right now. It's in a proposal stage and needs more discussion. I think it's perfect for a RFC and we can discuss it there.

+1

@muneeb-ali wrote:

Can you explain that a bit? You mean without a blockstack-server as well?

Sorry, what I meant by completely client-side is "running in the browser, without the need for a local server". And to clarify further, this doesn't include using the Blockstack server to resolve names to zone files or broadcast transactions (technically one could broadcast to any bitcoind node).

@muneeb-ali wrote:

I think the Python drivers are important because (a) they'll go into blockstack-server which is written in Python and (b) they'll go into the common syscall daemon thing (we really need a name for it) which is also written in Python. If anything we can have separate JS drivers. Ryan is already sort of doing that with S3 data.

We don't need to get rid of the python equivalents do we? I know it will be a bit extra maintenance work but sounds like it's necessary. Thoughts?

[muneeb-ali]

Great, that zone_file is a separate package.

I think the term "profile" is potentially confusing. What's a profile? In the case of the .id namespace it's clear to me what a profile is. What's a profile for the .app namespace? Passing "app.blockstack.org/index.html" to a function called "make_zone_file_for_hosted_data" in a package called "blockstack-profiles" seems confusing to me.

What'd help my understanding is if we can break down the discussion into two parts:

(a) Things specific to the .id namespace.
(b) Things for other namespaces. We don't know what namespaces people want and how they'll evolve. So it'd help if we "future proof" the design a bit. As soon as you started talking about the .app namespace and how the same libraries can be used, my entire mental model broke down and I need to understanding this again :-)

[shea256]

@muneeb-ali wrote:

I think the term "profile" is potentially confusing. What's a profile? In the case of the .id namespace it's clear to me what a profile is. What's a profile for the .app namespace? Passing "app.blockstack.org/index.html" to a function called "make_zone_file_for_hosted_data" in a package called "blockstack-profiles" seems confusing to me.

To clarify, I'm only using the term "profile" to apply to JSON data that describes people, companies, works of art, etc. I am not using it to apply to applications or websites. What I meant by the previous comment was that "make_zone_file_for_hosted_data" could be used in the context of non-profile name resolution.

That said, this function is not specific to profile-centric namespaces and could be applied to any situation where you want a name to resolve to a specific hosted file.

Here's what I would do:

1. Keep the function in this library and change the name to "make_zone_file_for_hosted_profile".
2. Continue to use this library only be used for profile-centric namespaces.
3. Create a JS library for Blockstack app namespaces and have a new function that calls the make_zone_File function in there (this doesn't have to be done until we're ready to create the first app namespace).

This is a clear separation and it's really apparent what each library and each function is for.

[shea256]

@muneeb-ali wrote:

(a) Things specific to the .id namespace.
(b) Things for other namespaces. We don't know what namespaces people want and how they'll evolve. So it'd help if we "future proof" the design a bit. As soon as you started talking about the .app namespace and how the same libraries can be used, my entire mental model broke down and I need to understanding this again :-)

I'd like to propose an alternative distinction in order to facilitate communication. Would love your thoughts on this:

1. Profile JSON-based namespaces (there is currently only one: .id) - the protocol for the browser is to process the JSON object and render a browser-specific template with the JSON passed in - the protocol for non-browsers is to return the raw JSON
2. HTML-based namespaces (there do not exist any of these yet) - the protocol for the browser is to load the file and render the HTML - the protocol for non-browsers is to return the raw HTML
3. Other namespaces (there do not exist any of these yet) - protocols for these namespaces have yet to be defined.

[muneeb-ali]

Sorry, what I meant by completely client-side is "running in the browser, without the need for a local server". And to clarify further, this doesn't include using the Blockstack server to resolve names to zone files or broadcast transactions (technically one could broadcast to any bitcoind node).

OK, makes sense about no local server. I'm of the view that for using the system you probably always need a connection to a blockstack-server. It can be hidden behind some API or resolver, but there is always some blockstack-server that is responding to the client. For writes, you can broadcast directly to bitcoind but for constructing valid transactions you'll need to talk to a blockstack-server (e.g., you need the consensus hash at time of send).

If we establish that clients can't exist without a valid remote server then the question becomes that how do you want to talk to that remote server. Do you want a local proxy / standard syscall or do you want to talk directly to the server over HTTP/RPC.

I've actually seen this issue come up multiple times e.g., in the case of storage drivers. The storage drivers can live at:

1. server side (you just talk to the server over HTTP/RPC -- this is how we started off, but realized that you can't have security credentials at the server.),
2. local process (better for security -- this is the direction Jude and I have been discussing), or
3. at the client (need to reimplement them for each new language -- this is the current model).

[muneeb-ali]

Sounds great about defining communication. We should probably do it in a Github README that is more visible.

Question about:

Profile JSON-based namespaces (there is currently only one: .id) - the protocol for the browser is to process the JSON object and render it in conjunction with a template - the protocol for non-browsers is to return the raw JSON

What are some other examples? .iot? .git? Would it make sense to say "profile of a repository" or "profile of a iot device". Just brain storming. I don't have any suggestions for other names right now.

[shea256]

Sounds great about defining communication. We should probably do it in a Github README that is more visible.

+1

What are some other examples? .iot? .git? Would it make sense to say "profile of a repository" or "profile of a iot device". Just brain storming. I don't have any suggestions for other names right now.

.iot, .company, and .artwork would all be great examples.

[shea256]

OK, makes sense about no local server. I'm of the view that for using the system you probably always need a connection to a blockstack-server. It can be hidden behind some API or resolver, but there is always some blockstack-server that is responding to the client. For writes, you can broadcast directly to bitcoind but for constructing valid transactions you'll need to talk to a blockstack-server (e.g., you need the consensus hash at time of send).

I really like how Bitcoin wallets can use Bitcoin APIs for a very small set of core things that absolutely have to be outsourced to Bitcoin nodes. And then they can optionally use additional methods. The only really core functions for Bitcoin transactions are (1) getting UTXOs and (2) broadcasting transactions.

It seems there's a good parallel here with Blockstack. The only things you really need to use the Blockstack server for are: (1) resolving names to zone files (2) getting the consensus hash (and you could add broadcasting as number 3).

So I'd say that yes, you do still need to communicate with a Blockstack server, but only for those two things. Keeping it light and contained and agnostic of the application is the way to go, in my opinion.

at the client (need to reimplement them for each new language -- this is the current model).

If we're focusing on server-less apps then you only need to implement it in one language - Javascript.

Every other app will be using a server and thus can actually just communicate with a local python process. So with languages like Ruby, we can go with the model of the Ruby client outsourcing functions that we wouldn't want to re-implement in Ruby.

What do you think of that?

[muneeb-ali]

Makes sense about minimal dependence on the server.

JavaScript is an important enough language that it makes sense to have a bunch of JS libraries even if there is functional overlap. Also, multiple implementations don't really hurt us. We can let the implementations evolve organically. We don't know right now which implementation is going to get the most developer attention.

[jcnelson]

It seems there's a good parallel here with Blockstack. The only things you really need to use the Blockstack server for are: (1) resolving names to zone files (2) getting the consensus hash (and you could add broadcasting as number 3).
So I'd say that yes, you do still need to communicate with a Blockstack server, but only for those two things. Keeping it light and contained and agnostic of the application is the way to go, in my opinion.

This is the already status quo with respect to Blockstack server. My general design philosophy for adding features has been "if the client can do it, then it belongs in the client." The reason for this pull request was to establish the boundary between the client and the programs and libraries that use it (or need its functionality).

That said, let me see if I have the action items right:

• I withdraw this pull request.
• We make an RFC for how we expect to structure zonefiles and profiles for arbitrary names (not specific to .id).
• We have the RFC define how namespace-specific implementations may offer extensions or may control the resolution and storage methods, so as to support namespace-specific fields and standards (i.e. we can things like ".id names must have profile data resolvable by an HTTP(S) URI").
• We rename make_zone_file_for_profile to make_zone_file_for_hosted_profile to establish that it operates on zonefiles and profiles in .id-specific ways.