JRFC 27 - Hyper Modular Programming System

[jbenet]

Over the last six months I've crossed that dark threshold where the desire of building a programming language has become an appealing idea. Terrifyingly, I might actually build this some day. Well, not a language, a programming system. The arguments behind its design are long and will be written up some day, but for now I'll just dump the central tenet and core ideas here.

> Hyper Modularity - write symbols once

An Illustration

You open your editor and begin to write a function. The first thing you do is write the (mandatory) doc comment describing what it does, and the type signature (yes, static typing). As you write, your editor suggests lists of functions published to the web (public or private) that match what you're typing. One appears promising, you inspect it. The editor loads the code. If it is exactly what you were going to write. You select it, and you're done.

If no result fits what you want, you continue to write the function implementation. You decompose the problem as much as possible, each time attempting to reuse existing functions. When done, you save it. The editor/compiler/system parses the text, analyzes + compresses the resulting ASG to try to find "the one way" (or good enough) to write the function. This representation is then content addressed, and a module consisting of the compresses representation, the source, and function metadata (doc string, author, version, etc) is published to the (permanent) web, for everyone else to use.

Important Ideas

• exporting a symbol (functions, classes, constants, ...) is the unit of modularity (node)
• system centered around writing functions and writing them once (node)
• stress interfaces, decomposition, abstraction, types (haskell)
• use doc string + function signatures to suggest already published implementations (node, Go)
• content address functions based on compressed representations
• track version history of functions + dependents (in case there are bug fixes, etc). (node)
• if a function has a bug, can crawl the code importing it and notify dependents of bugfix. (node, Go)
• use static analysis to infer version numbers: <interface>.<implementation> (semver inspired)
• when importing, you always bind to a version, but can choose to bind to <interface>/<implementation> or just <interface>
• e.g. factorial = import QmZGhvJiYdp9Q/QmZGhvJiYdp9Q (though editors can soften the ugly hashes) (node + ipfs)
• all modules (functions) are written and published to the (permanent) web (public or private)
• when importing a function, you import using its content address, and bind it to an explicit local name (foo = import <function path> type of thing)
• the registry of all functions is mounted locally and accessible in the filesystem (ipfs style)
• hyper modular means both to "very modular" and "modules are linked and on the web"

Note: this system is not about the language, it is about the machinery and process around producing, publishing, finding, reusing, running, testing, maintaining, auditing, bugfixing, republishing, and understanding code. (It's more about the process of programming, than expressing programs). This means that the system only expresses constraints on language properties, and might work with modified versions of existing languages.

[jviereck]

Nice idea - really like it!

Do you know about http://jsnice.org/? They are indexing source code form GitHub to reverse engineer the variable names. It's not 100% what you are describing here, but the infrastructure and machine learning techniques they use might be useful for your idea as well.

[yuchi]

Subscribed to the thread. Promisingly terrifying.

[sindresorhus]

❤️ I want to hug this idea.

[datagrok]

Possibly related: SLINKY: Static Linking Reloaded (USENIX 2005) describes a mechanism to use content-addressable storage to enable dynamic-like space savings in statically-linked software.

[oliverseal]

Awesome! This sound exactly how I think that the programming Geordi La Forge would do in Engineering worked.

I know. That sounds crazy because it's a fictional character. But when I rationalized how programming might work in the future, I considered it to be a vast array of built-ins that do awesome things and are stored in the whole Starfleet system. When you look at node, Python, even Go, they all take things that were more lines of code and bring them down to more manageable almost one-liners. The idea of having a write-once-use-forever implementation across everything is a futuristic version of this same thing to me.

Everyone thinks about methods/solves differently. This could become very weird engineered form of "Damn you, autocorrect!" and painful to debug.

[fmvilas]

Ufff... Really like it, seriously, but it can drive us crazy.

[per-gron]

This is so cool! This probably sounds weird, but I think a very interesting, more concrete, perspective on how to actually define reusable functions in practice can be found in C++.

Alexander Stepanov is the original author of STL, which is the part of C++'s standard library that has data structures and algorithms. He designed STL based on this idea where he wants the process of programming to consist of identifying the interface of and composing self-contained reusable algorithms: http://www.stepanovpapers.com/DeSt98.pdf

The original ticket says "This means that the system only expresses constraints on language properties, and might work with modified versions of existing languages." When Stepanov talks about generic programming, he says almost exactly this (he happened to pick a subset of C++). The ideas try to solve a similar problem and at a similar level of abstraction.

[dominictarr]

this sort of thing could interface very nicely with tests. say, there are multiple things that have that interface + similar doc string, well, write a test that disambiguates, then you can future versions with interface + test hash - at least, to automatically search for when upgrades may be possible.

[vijayee]

So yeah this sounds to me very much like what Ethereum has going with smart-contracts but with package management going on. I wanted to build something similar to a smart-contract package manager for ethereum contracts but this sounds like a much better idea.

[vijayee]

https://www.nitrous.io/?utm_source=github.com&utm_campaign=Telescope&utm_medium=hackonnitrous

[adamdicarlo]

@vijayee uh... is that... spam? 😦

[jbenet]

@adamdicarlo no, i think it's somewhat relevant. look at the IDE shown there.

[vijayee]

@adamdicarlo I may not always spam github issues but when I do its relevant to distributed programming collaboration

[davidar]

I recall some similar ideas being discussed on the Erlang mailing list a while ago, which I quite liked. In terms of finding "the one way" to write a function, Morte attempts to do this with (a subset of) Haskell. In fact, a surprisingly effective strategy for writing Haskell code is to first write the type signature for the function you want to write, then Hoogle/Hayoo it to see if anyone has already written the corresponding definition. For some functions, it's even possible to automatically generate the definition given the type signature, by taking advantage of the Curry-Howard isomorphism and applying automated theorem-proving techniques. With ipfs/ipfs#4 and a suitable editor, it might even be possible to do a lot of this with unmodified Haskell.

[jbenet]

@davidar this would be great. Would love to hack together a prototype.

[davidar]

Also relevant: http://www.haskellforall.com/2015/05/the-internet-of-code.html

[mtwilliams]

Spit balling (seems to be the theme of this party):

Why not store edits on a AST? Imagine CQRS on the expression level. Sure, pragmatic reasons. (Imagine how unscalable that could be!) However you could do a bunch of crazy awesome stuff with it. For example, you could permute an application from a calendar to a game of pong. Or of more utility, do automatic bug fixes.

You could do similar things at the function level and take the Unix philosophy to the extreme.

[jbenet]

From ipfs/notes#195

Fantastic talk from the deeply thoughtful, idealistic and pragmatic, Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

It captures much of what i dislike about semver. The thoughts arent completed though. Coupled with hashes, static analysis, and freedom from old paradigms (in their case, mvn, jars, namespaces, etc).

It may be that Go's typing and packaging (however stunted...) is enough for something better along these lines. It may be that a new dep field for javascript could be enough. It may be that we need new languages whose import and code loading is structured around these truly semantic versions.

[eparejatobes]

You might like https://github.com/unisonweb/unison/

[sehqlr]

This is something that I've also thought about, but from a different direction. I just wanted to add one piece of it for thought. Have a language ecosystem that has an semi-automated system to add to the standard library. The parts are:

• Language maintainers set up a specification to define requirements for submission, and a test suite that code MUST pass to be considered.
• if a lib passes the tests, then it can be added to the standard library. Whether or not it gets automatically approved or not is up to the language maintainers
• the stdlib site could also double as a social network/payments system for library authors as well

Am I going in a completely wrong direction? Are there parts of this idea that fits with what y'all are saying?

[sesam]

Neat! Software is resources (data, functions), state, storage, business logic and UX, and maybe that's it. Which of these are easier to handle as described above?
Easy early wins could be among: time, currencies, contacts (adress/phone/...), time-space-contexts, all full of exceptions and inconsistencies because life creates complexity.

Let's try imagining an example:
Say I want to build a new Android Home screen or Gear app with nearby people's free/busy schedules.
I could start defining:
1 calendar view of people's busy/free time
1.1 data pagination
1.1.1 get busy/free time data
1.1.1.1 request calendar access
1.1.2 merge busy/free time

Already typing the first line 1. maybe some or most of the points below 1. can be quick to just confirm to get included from a list of suggestions.

Reminds of TDD. Tests with coverage and verified correctness might come included with reusable packages, and can be written in any programming language. The module can run in a lambda or container on whatever hardware I have handy, maybe run it close to the data sources.

[omkarjc27]

I came saw this comment a couple of days ago.
I did something that is not as advanced as described above but ..... it is something.
So please Checkout GitHub Link

[kehiy]

@jbenet That's a super cool idea which I always had similar ideas but on blockchain and smart contracts.

I saw this randomly and I think in these years with additional help of AI in coding, this kind of programming or web based (IPFS) libraries will be a new way to programming.

Here is my previous proposal specified for TON blockchain which is failed:
ton-blockchain/TEPs#154

It had same way to import libraries (import "hash/content-address") and it also was trying to minimize them by merging same codes or even same part of some codes!!!

Also, based on IPFS paper, we can use it as a FS for VM. I think a VM can read some part of its byte code which is imported libraries from IPFS as well.

I will be happy to participate on this idea, a proposal for this or its implementations if any of them is in progress.