talk.asciidoc

Does JavaScript Adopt Functional Programming?

[twitter] @luftyoav

[envelope] [email protected]

Note	Hello! My name is Yoav Luft and today I’m going to talk about whether the JavaScript community adopts functional programming or not.

The Beginning

Note

About five years ago I worked for company that processed huge amounts of data on distributed Python servers. Hoping to improve performance, I came along Erlang and Elixir, which introduced me to functional programming. Combined with my passion for clean and verifiable code, and the trust I put into formal methods, I quickly found myself sucked to the world of functional programming, and especially Haskell and Elm.

The Conflict

Note	But one workplace later I met a group of developers who were not impressed by my Functional-Fu and my declarative-jutsu. In fact, they complained that my code is too declarative! This through me off balance, and made me ask, "Is Functional Programming Really is the future of JavaScript?"

Resolution

?

Note

This talk is based on a research I’m conducting on coding style in Javascript. I conduct the research independently but under the supervision of Prof. Feitelson from the Hebrew University of Jerusalem. The results that I’ll show today are preliminary, and we’ll be subject to more thorough investigation.

Short Version

Note	In case you just can’t contain your curiosity, here’s the short version:

JavaScript becomes more declarative

100 top projects

[plus]

Analyzing the Code

=

JavaScript Becomes Functional Declarative

Note

By looking at Github’s one hundred most popular javascript projects, analyzing their code and dependencies, I feel quite convinced that as the years go on, the Javascript community is adopting a more declarative programming style. How did I manage to get to this conclusion? What it has to do with functional programming, and why even look at javascript and not, say, C#? All of that and more, next. But first,

The End!

Introduction

Note

I’m Yoav Luft, born and raised in Israel, currently living with my partner Dafna in Stockholm Sweden. I’ve have been programming professionally for the last ten or so years, working mostly with web applications but also with games, embedded systems and other bits’n’pieces. Other than programming itself, I’m also involved with the programming community as much as I can. In Israel I’ve founded "FLIP", the first functional programming conference in Israel, now preparing for its second round. I’ve organized the Elm users group meetups for a while, and I was part of the organizing team of "HackExtend", a diversity centered hackathon founded by my partner Dafna and friends. I currently work for the digital agency "Prototyp", a code lab that build code for costumers that want to break new grounds.

Yoav Luft

Web Dev - Games - Embedded

Introduction II

Table of Contents

• Why should we care about JavaScript?

• What "adopting functional programming" even means?

• The Quest for Data

• What’s next?

Note	Why should we care about JavaScript? What "adopting functional programming" even mean? Where is the evidence? What’s next? and finally,

Why should we care about JavaScript?

Note	Why should we even care about JavaScript? And especially, why should we talk about it in a functional programming conference?

The Past and Present of Javascript

Note

Let’s start with a brief history of JavaScript: Javascript was created at netscape in 1995 after Brenden Eich (AiK), who was originally hired to integrate Scheme into the Netscape Navigator, created a prototype in 10 days. It uses a curly-braces like syntax because management had decided that they want a Java-like programming language. It’s goal was to allow running programs in a web-browser, making web pages interactive, and allowing for a ubiquitous platform.

[plus]

[long arrow down]

!

Note

Javascript was created with imperative structural syntax similar to C, using control structures such as if-else blocks, while and for loops. It has functions as first class citizens and supports closures. It is a universally typed language, aka "dynamically" typed, but also supports some version of object oriented structuring with the use of prototypical inheritance.

• Imperative

• Functions are 1st class citizens

• Universally typed

• Prototypical Inheritance

!

var object =
  { name: "Object1",
    action: function() { },
    data: [1, 2, 3]
  }
JSON.stringify(object) ==
  "{\"name\":\"Object1\",\"data\":[1,2,3]}"

Delegation

function Dog() {
  this.voice = "Woof!"
  this.makeSound = function() {
    return this.voice
  }
}
var dog = new Dog
var cat = {voice: "Prrrr",
           makeSound: dog.makeSound}
cat.makeSound() == "Prrrr"

Note

It uses the hashmap as its fundamental data-type, which can also be encoded to a data transfer representation, the JSON format. It uses late-binding, and allows delegation by the use of the "this" keyword, a feature notoriously confusing for programmers who come from other imperative-object-oriented languages such as Java. Ölbaum[https://twitter.com/oscherler/status/660049264903643136?lang=en]

!

Note	Being supported on all web browsers, and also being a common backend language through frameworks such as Node.js has made Javascript extremely popular.

Note

In StackOverflow’s yearly survey of 2018, Javascript was the most popular programming language with 69% of respondents claiming knowledge of the language [https://insights.stackoverflow.com/survey/2018/#most-popular-technologies]. It held this position for the last 6 years. The top 3 most popular frameworks in that same survey are all Javascript frameworks, with Node.js in the lead, which hints to the language’s popularity as a backend language.

!

Note	According the website SimilarTech, node.js is the third most popular server framework, after PHP, ASP.net. The fourth place is occupied by "express", which is a framework built over Node.js. [https://www.similartech.com/categories/framework].

[github]

Note	On Github, javascript has been the number one programming language for new repositories in the last 4 years [https://octoverse.github.com/projects#languages]. Love it or hate it, being the only programming language supported by all web-browsers means that Javascript’s popularity is on the rise.

But why should we care?

Note	But then again, why should we care, when discussing Javascript in the context of functional programming?

JavaScript for Functional Programming

Note	First and foremost, we should note that javascript is a very welcoming language for the flexible functional programmer: It supports anonymous functions and closures; It uses continuation passing and structured modeling of asynchronous code quite often;

Javascript has:

• anonymous functions

• closures

• uses continuation passing

• structured modeling of async (almost monads!)

!

Note

It appears that many members of the EMCA Technical Commitee 39, the committee in charge of the javascript standard, would like to encourage functional programming style in javascript. As people interested in functional programming, I can only awesome that we would all like to use familiar and beloved programming style over an imperative, out-styled, and appearently more error prone style. This is especially true if our daily work requires writing code that would run on web browsers.

Rapidly Evolving Standard

Note

And it’s rapidly evolving standard includes support for things such as shorthand notation for anonymous functions (already commonly used), destructuring assignments, constant references, generator functions, and there are even pending proposals for tail-call optimization, pipe-operator, partial application syntax and pattern matching.

• ✓ anonymous functions shorthand (2015)

• ✓ destructuring assignments (2015)

• ✓ generator functions (2015)

• ❏ tail-call optimization (pending)

• ❏ pipe-operator (pending)

• ❏ partial application syntax (pending)

• ❏ pattern matching (pending)

Part II: What does
"Adopting functional programming"
even mean?

!

Note

Just a short google search with the words "Javascript functional programming" will turn out millions of results, and many of the top results are blog posts about Javascript, or functional programming in javascript. But is that a proof that the community as a whole adopts functional programming? Or are these just functional programming eccentrics howling at the moon?

!

Note	We can break our question to two parts: Let’s start with "adopting"

Adopting     functional programming

Adopting

1. Functional programming was not popular, or formally accepted in Javascript

2. It is now becoming more popular, or being formally accepted

Note	Above is the Merriam-Webster definition for adopting. In adopting some programming style we imply two things:

!

Functional Programming

Note	Now let us look at the second part of my statement: Functional programming. What is, exactly, functional programming? How can we define it, and how can we turn such definition to concrete test cases that we can use to determine "how much" Javascript is functional programming?

How much is JavaScript functional programming

Note	When in doubt regarding definitions, I do what every millennial does: I look it up in wikipedia. The wikipedia definition was a bit mouthful, I will not read it out loud, but just notice the underlined bits:

1. Computation as the evaluation of mathematical functions

2. Avoids changing-state and mutable-data

3. Declarative programming paradigm

Criteria

Note	Let’s see if we can turn them into criteria for estimating how "functional" is a language or a piece of code.

Criterion 1:
Computation of Mathematical Functions

Note

Let’s start with "Computation as the evaluation of mathematical functions". What are "mathematical functions" in this context? Again, from definitions, a mathematical functions is a relation from a set of input to a set of possible outputs where each input is related to exactly one output. OK! That’s tangible! Let’s try and formalize this to a criterion: A piece of code is more "functional programming" if it uses more mathematical functions. A mathematical function is such function that its output depends entirely on its arguments. While simple, alas, determining such thing for arbitrary Javascript code was a bit out of scope for my research.

Mathematical

function append(array, x) {
  return array.concat(x)
}
var a1 = [1, 2]
var a2 = append(a1, 3)
// a1 == [1, 2]; a2 == [1, 2, 3]

Non-mathematical

var array = [1, 2]
function append(x) {
  array.push(x)
}
append(3) // undefined -> no return
          // array == [1, 2, 3]

Criterion 2:
Avoid changing state and mutable-data

Note

I think this one is both simple and complex at the same time. It is simple because mutability and changing state are strongly related, and very intuitive to grasp. This is complex because [a] all useful programs perform state changes; [b] it is difficult to tell apart, when doing static analysis of code, whether a data mutation is done in-order to achieve a desired side-effect or whether it is due to the implementation of an algorithm. Luckily for us, we’ve got some help: Many javascript projects use 3rd party packages to provide immutable data structures. Javascript also added the "const" keyword for immutable reference, and the Object.freeze() method for making objects immutable at runtime. All three can be good and easy to gather indicators for how immutability is common in javascript.

Immutability

Criterion 3:
Declarative Programming Paradigm

Note	And last but not least, we can look at whether projects adopt a declarative style of programming. Again, a definition is in order: While this might sound as vague as the previous definitions, I find that the last bit in there is key:

a style of building the structure and elements of computer programs—that expresses the logic of a computation without describing its control flow.

— Wikipedia

!

[…​] without describing its control flow.

Note	Javascript, being a language that uses C style imperative structures, comes with a nice set of control flow structures:

While loops

while (condition) {
  doAction()
}

do {
  action()
} while (condition)

For loops

Note	Not one, not two, but three different kinds of for loops: Do not worry yourselves about the differences between the two last examples, it is not that important.

for (var i = 0; i < size; i++) {
  use(i)
}
for (var property in object) {
  use(property)
}
for (var index of array) {
  use(index)
}

Branching

Note

In addition, javascript supports if-else statements, switch statements and labels, which are just a different name to C’s good old GOTO statements. Now, how would a declarative piece of javascript code would look like? It would avoid control flow structures, but which? If statements can only be replaces with the trianary operator, which is terse and difficult to read, so if-else are likely to stay. Switch statements are often used where pattern matching would have been used, and given there is no standarized replacement for them, it would be difficult to compare them to something else. Labels are rarely seen, so we are left with looping constructs.

If-else

if (condition) {
  doSomething()
} else {
  doSomethingElse()
}
var a = condition ? val1 : val2

Switch statement

switch (response) {
  case "yes":
    return true
  case "no":
    return false
  default:
    return undefined
}

But Which?

Iteration functions

Note	Luckily for us, looping constructs do have good alternatives: The javascript Array object offers 4 familiar and useful functions: map, filter, forEach and reduce. Here’s a small examples of some of them:

map, filter, forEach and reduce

Examples

Imperative	Declarative
var result for (var i of array) { result[i] = f(array[i]) }	var result = array.map(f)
var acc for (var i of arr) { acc = f(acc, arr[i]) }	var accumulated = arr.reduce(f)

Helper Libraries for Functional Programming

Note

These functions are built in as part of the Array API, and they have common alternative versions in utility libraries such as underscore, lodash and Ramda. Combined with some built functions from the Object class, such as Object.keys which returns an array of all keys, they are complete enough to represent most, if not all, iteration constructs required by javascript developers. Because of these properties I’ve decided that looping constructs are the best candidates for measuring how much does Javascript adopts declarative programming.

Part III: The Quest for Data

Note	I hope I did not bored you with this prolonged introduction. Armed with the understanding that while the question of whether Javascript goes functional or not is complicated, the question of whether it becomes more declarative is easier to answer. Now comes the search for data.

Methods

Note	After consulting with Prof. Feitelson, I’ve decided to focus on projects which are: By looking at how these projects evolve, we can gain insights into general trends in Javascript. I’ve decided to use Github’s one hundred most popular javascript projects, such as:

Projects which are:

1. Open source

2. Have a lot of contributors

3. Have been around for a while

Project examples

Note

Next, I’ve gathered metadata on each project, such as it’s number of stargazers and it’s number of forks, dependencies used and so on. Finally, I’ve looked at the actual code. By selecting the last commit in each of the years 2009, 2012, 2015, and 2018, I would see into some of the trends in the overall Javascript programming style.

Data Collection Process

Note

Once the commits where selected, I’ve downloaded a snapshot of each repository at each of the target commits, and sent the content of the snapshot to a processing pipeline. The processing pipeline uses the Esprima parser to generate an Abstract Syntax Tree for each file, which was then searched for specific patterns.

Data Collection Process 1

Data Collection Process 2

Data Collection Process 3

Terminology

Project = A repository on Github

Sample = Data on patterns from a project’s snapshot

Sample Year = A sample from the specified year

Patterns: Imperative

Note	For each files of valid Javascript, I’ve looked for the following patterns:

• for loop, for..in loops, for..of loops

for (var i = 0; i < size; i++) {...}
for (var i in object) {...}
for (var i of array) {...}

• while and do-while loops

while (cond) {...}

Patterns: Declarative

• forEach calls

• map calls

• filter calls

• reduce calls

Note	After filtering some problematic repositories, I moved into analyzing the results.

The Sample

Projects

Note	This graph shows how many samples we’ve managed to gather in each year, out of our initial one hundred repositories. We had only 3 samples from 2009, so this year will be discarded in future graphs, and we had 79 samples in 2018.

Projects by Age

Note

This brings the question of the age of the projects on Github. We can see that over half of them were created on Github at before 2014. This means they have at least 5 years of development in the public domain, which is desired. It is important to note that not all projects had samples in every year.

Size of Projects

Note

How large were the projects? These two graphs show the total number of parsed Javascript files in each sample year, and the sum of lines of code for each sample. We can see that we’re dealing with magnitude of 3 million lines of code. We can also witness the rapid growth of Javascript from 2012 to 2015, almost tripling the number of lines of code, while doubling the number of files. We can see that from 2015 to 2018 the number of lines of code grows more slowly, but the number of files keeps on growing steadily. This is an indication that code is broken into smaller files, maybe due to refactoring of the projects?

Results

Note

Constructs in Samples

Note

Next, we’ll look at the how common are different code constructs in different samples: In this graph we see the count of each iteration construct across the three sample years. The top for are our impreative constructs, while, for..of, for..in and C-style for-loops. Beneath them are our declarative constructs: forEach, reduce, map and filter. The green line is the sum of all declarative constructs. We can notice that from 2015 to 2018 the total number of iteration constructs had stayed roughly the same, but we already know that the amount of code had increased. We can also see that the declarative part is slightly raising, and also that C-style for loops are the most popular interation construct.

Constructs per LoC

Note

Let’s look at the same data normalized by the number of lines of code in each sample: In this graph it becomes obvious that as time advances, there are less iteration per line of code. Is that because projects use APIs which are even more high-level than the ones I’m sampling, or is because there are less iterations over data in general? We can see that while the total number of iteration constructs shrink, the part of the declarative constructs shrinks more slowly. Let’s have a look at the relative amount of each construct compared to the others:

Constructs in Percentage

Note

Now, here we can see an obvious trend: declarative iteration constructs are on the rise compared to their imperative counterparts. Still, they are only slightly more that 40% of total constructs use, with C-style for loops still in the lead with 34% of all constructs found. while loops are slowly disappearing, dropping from 13% to just 9.6% over the last 6 years, while the use of map is slowly on the rise. Surprisngly enough, the use of reduce, while rare, is quite constants, being around 1% of all samples. It seems like this supports the idea that javascript is moving to more declarative iteration constructs, but it seems like this process is not as fast as I originally had guessed. Let’s try and look what kind of projects change the most.

Conclusions

More Declarative Iterations!

but not that much…​

Individual Projects

Individual Projects

Note

Sixty six of the projects appeared in more than one sample, meaning that we can track their individual development over time. To do that, I started by defining the projects "Declarative Index", which is the ratio of declarative iteration constructs out of the total number of constructs. A declarative index of "one" means that projects uses only declarative iteration constructs, while a "zero" means that only imperative constructs are used. I’ve proceeded with plotting the declarative index for each project over our sampled years. We can see that while most projects create an upwards slope, meaning they go from imperative to declarative, this is not universally true. I’ve plotted the mean declarative index as the magenta line, and we can see that over the last 6 years there’s an obvious trend where projects move to a higher declarative index, from approximately 0.4 in 2012 to almost 0.6 in 2018. Is there anything that can predict whether a project will choose or turn to a more declarative style?

What Kind of Projects Are Declarative?

Age as Estimator

Note

By plotting the declarative index of a project in the last sample in which it appears against different properties of the project, we can hope to achieve some insight into which projects are more declarative. In this graph we see the scatter of the projects index versus age and the regression line. It seems like that the younger the project is, the more likely it is to have an higher declarative index! In other words, newer projects tend to prefer a more declarative programming style.

Number of Forks

Note

Let’s consider the number of forks a project has and it’s index. If you are not familiar with github, a fork is a copy of the project with independent changes that might be merged back into the project. In most projects, the only why for a contributor to contribute code is to first fork the project. Again, we’re look at the scatter-plot with regression. We can see that in this case the declarative index is slightly inversely correlated with the number of forks.

Stargazers vs. Index

Note

Interestingly enough, there is no correlation! But I wouldn’t put to much meaning into this graph, because being a stargazer of a project doesn’t have some inherent well understood meaning, similar to a "like" on twitter or Facebook. Finally, let’s see how the number of contributors correlates with the projects "declarative index":

Project Contributors

Note

Last property of project we’ll see today is the number of contributors, as reported by the projects main Github page. From the number of contributors we can see that there is some correlation between more contributors and more declarative style of code. This might imply that the majority of JavaScript programmers prefer a declarative style of coding.

Conclusions

Supporting Libraries

Note

As we discussed earlier, the question of measuring "functional programming" of a project is more complicated. A possible indication can be the use of libraries that make a functional programming style more accessible in JavaScript, such as "underscore", "lodash" and "Ramda". By looking at the dependencies of different projects, we can see whether they use one of these libraries or not. This graph shows the number of projects in each sample that had such a library as a dependency. We can see that while these libraries are on the rise, they are still being used in less than a quarter of the projects.

What’s Next

What are your options?

Note

Oh! Well, if you don’t like Javascript but need to write code that runs in a web browser there are several nice alterantives: Elm: a small delightful language that IMHO solves many of the problems in web programming ClojureScrip: if you just can live another project without LISP PureScript: a Haskell-like language that compiles to Javascript and has some tweeks for nicer interpolation with Javascript. Reason: an OCaml inspired language that started trending lately, mostly due to it’s easy interop with Javascript. Wait for the Web assembly standard to mature and then just write code in whatever languae you want. 100s of Language X to Javascripts: Hundreds of other programming languages that can compile themselves to Javascript, such as Haskell, Kotlin, Scala, Ruby, Python. Google any word with "Script" as suffix or that is a pun on coffee and there’s probably a Javascript dialect or a language that transpiles to Javascript by that name.

Elm

ClojureScript

PureScript

Reason

WebAssembly

As for me…​

Note

As for my research, there are some obvious steps forward: The most obvious is to look at a larger sample. I can also look at other structures, such as assignments and branching. It might be possible to identify the uses of higher-order functions. Or just examine specific examples in more detail, for example, determining how easy it is to refactor code from imperative to declarative style. We can look at changes over time at the commit level: is code being refactored? And finally, we can see whether projects migrate to functional languages. And of course, there are so many small improvements to make, such as supporting more flavors of JavaScript, and even looking at JavaScript’s typed superset, TypeScript.

• Look at a larger sample

• Examine more structures, e.g. assignments

• Look at the use of higher-order functions

• Examine results into more detail.

• Examine changes of code over time

• Look for projects migrating from Javascript to functional languages

Shameless Plug

Note

If you find this project interesting, you can contribute to the code through it’s repo on github, or by encouraging me to keep researching programming languages. I work for Prototyp, where we <put slogan here>. We’re located in Stockholm and Barcelona and would be happy to help you with your projects.

[twitter] @luftyoav

[envelope] [email protected]

Thank you!