CHANGES

Pike 8.1: Changes since Pike 8.0 (scratch area for future release notes)
------------------------------------------------------------------------

New language features
---------------------

o Unlimited character constant size.

  There is no longer any limit to the size of character constants,
  e.g. 'acdefghijk' creates the bignum 0x61636465666768696a6b.

o Syntax for specifying default values for function arguments added.

  The function definition

    string foo(string bar = "bar") { return bar; }

  will now result in a function(string|void:string) that returns
  "bar" if called with no arguments or UNDEFINED, and its argument
  otherwise. The local variable 'bar' above will be typed string
  (ie not string|void) in the function body.

  The syntax is also valid for implicit create:

    class foo(string bar = "bar") {}

o Syntax for using multi-assign in function calls.

  The function definition

    int add([int a, int b], int c) { return a + b + c; }

  will result in a function that may be called as

    add(({ 1, 2 }), 3);

o Major changes to the compiler type checker.

  - The type for the value 0 or UNDEFINED now needs to be
    declared explicitly. For convenience the symbol 'zero'
    now evaluates to the type 'zero'.

o '__unknown__' type added.

  This is a type that is the inverse of the type mixed|void,
  and is used eg in the type for the empty string
  string(zero:__unknown) and for the generic function type
  function(__unknown__...:mixed).

o 'auto' type added.

  This is much like a strict typed mixed. The actual type is deduced
  compile-time.

  Typical use is as the type for variables in foreach when looping over
  complexly typed values, or as the type in macro-defined functions and
  similar.

  auto can be used as return type for a function, it will be the
  type-union of all the types of the return statements in the
  function.

o typeof(X) can now be used as a type.

  The main use case is in macros, but it might be useful for other
  things as well (as an example in a typedef).

  typedef typeof(Val.true)|typeof(Val.false) bool;

  typeof, when used like this, behaves very much like the C++
  decltype() expression.

o Shorthand for int(val..val) added: int(val) .

  A syntax for when the range of an integer type only
  contains a single value.

o Shorthand for string(val..val) added: string(val) .

  A syntax for when the range of a string type only
  contains a single value.

o Syntax for specifying the length of array and string types.

  Eg string(4bit: 8bit) is 8-bit strings of lengths 0-15.

o ** operator added. It's exponentiation and works with most
  combination of numerical types (int,float,Gmp.mpq,Gmp.mpf,Gmp.mpz)

  `** and ``** operator overloading functions added.

  This introduces one incompatible change: Previously the "pow"
  function called a "_pow" function in the first argument if it was an
  object. It has now been changed to also use `** (or, rather, pow()
  is now implemented using predef::`**()).

* __cast() added.

  This efun performs a cast of a value to a specified type, where the type
  may be specified either as a type value or as a string. It is intended
  to help in implementing lfun::cast().

* Binary or between a multiset and UNDEFINED is now behaves like
  binary or between the multiset and the empty multiset.

o Three pass compiler.

  An extra pass of the compiler has been added. It is used to properly
  resolve types in forward referring expressions, and is only run when
  needed.

o #pragma disassemble

  This directive is now available even in pikes compiled --without-debug.

  The byte code output is now interleaved with the generated machine code
  on selected architectures.

o #pragma no_strict_types

  It is now possible to turn off the strict types mode for a section
  of code.

o Complain about redundant backslash escapes.

o '__weak__' modifier added.

  It is now possible to declare object variables to have weak references.

o Support for arguments to implicit lambdas added.

  Implicit lambdas now behave as if they were declared as
  lambda(mixed ... __ARGS__).

  Support for this feature can be detected with #ifdef __ARGS__.

o Function local function declarations are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

o Anonymous class definitions are now always expressions.

o Function local class definitions are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

o The inherit statement's argument is now a proper
  constant expression. This means that it is possible
  to dynamically (at compile-time) generate the value
  to be inherited, eg with Pike.ProxyFactory().

o Complain about shadowed variant functions.

o Syntax for generator functions added.

  continue is now treated as a modifier, and causes such functions
  to become generator functions, where calling the function sets
  up a restartable function context.

  continue may also preceed return statements, which causes
  the generator function state to be saved before returning.

  Alternatively the "function" Pike.yield() may be used to return
  a value, and restart with a value from the restarter.

  Restartable functions get the type signature

    X fun(mixed|void value, function(mixed|void: void)|void resume_cb)

  where X is the declared return type of the generator function,
  value is the value to be returned by Pike.yield(), and resume_cb
  is a function that will be called at the yield point. resume_cb
  may be used to eg behave as if Pike.yield() had thrown an error.

o Machine code support for more architectures.

  There's now machine code support for arm32 and arm64.

o Fixed multiple integer over- and underflow bugs.

o Extended sscanf %O

  sscanf() is now able to parse all base types that sprintf %O can
  output. It can now also parse most literal constants pike accepts,
  including integers prefixed with 0x and 0b, and floating point
  numbers in scientific notation.

o Returning void

  It is now possible to return void responses from void functions.

    void a() { return; }
    void b() { return a(); }


Bug fixes
---------

o Compiler

  - Constant integers outside the 32-bit signed range not longer
    get their types mapped to int(-0x80000000) or int(0x7fffffff),
    (depending on sign) but instead keep the generic int type.
    This fixes issues where the type resolution system derived
    erroneous types from the truncated types.

  - Under some circumstances the compiler would insert erroneous
    F_ESCAPE_CATCH opcodes. This could cause very strange
    runtime errors.

  - The compiler could get confused by combinations of labels
    with inline functions or constant expressions.

  - The type checker now checks that operator assignment
    expressions are valid.

o Runtime

  The runtime could get confused by PROGRAM_DESTRUCT_IMMEDIATE
  objects having destruct callbacks under some circumstances.

o Operator functions

  - Calling operator functions with more than two arguments will now
    work correctly for objects, where previously only the first two
    objects where added.

  - When adding arrays, multisets and mappings, UNDEFINED will now
    always be ignored. Previously it was only ignored when first in the
    argument list, otherwise an exception would be thrown.

  - Adding UNDEFINED with UNDEFINED now results in UNDEFINED
    (rather than 0).

  - The LFUNs will now be called with a consistent number of
    arguments. Pike implementations rarely implemented varargs on
    operator LFUNs, so this change should address many potential hidden
    errors.

  - Most operators that use LFUNs now have types that check
    that the arguments to the corresponding LFUN are correct.

o Pike.Backend

  Backend callbacks throwing errors could cause backends to
  enter a state where they got stuck throwing "Backend already
  running - cannot reenter.". This was typically triggered via
  the Stdio.sendfile() result callback.

o Protocols.HTTP.Server

  The server module will now read payloads for HTTP PUT requests, just
  as any other method. Previously it would stop reading the body and
  it was up to the caller to read enough data from the socket and
  combine with already read data.

  Setting "connection" header in the "extra_heads" to Request object
  method response_and_finish() will now control if keep-alive should be
  used or not. Otherwise it will be decided by the clients request
  headers, as previously.

  The headers "content-type", "content-range", "content-length",
  "server", "date" and "last-modified" will not be added or
  overwritten if provided in the "extra_heads".

  Header names in "extra_heads" will not have their case modified.

o Standards.JSON and Standards.JSON5

  encode() now allows other threads to run every now and then.

o ADT.OrderedMapping

  Adding duplicate keys now replaces the existing keys.


Incompatible changes
--------------------

o As mentioned above; variables that may be set to 0 or UNDEFINED
  now need to be declared accordingly. Note that this is not needed
  for code that is compiled in compat mode for Pike 8.0 or earlier.

o The new stricter type checker may expose old bugs that the
  old type checker did not find.

o Simplified the Iterator API.

  - Renamed Iterator API functions and made them LFUNs:

    o index() ==> iterator_index()

    o value() ==> iterator_value()

    o next() ==> iterator_next()

  - Iterators now start at the position before the first element
    to be iterated over. This position may be the the same
    position as after the last element.

  - The only function that is required for an Iterator is
    iterator_next(). It should advance the iterator one
    element, and return UNDEFINED if the position is now
    after the last element. It may return any other value
    otherwise. The returned value will be used in place of
    the values returned by iterator_index() and/or
    iterator_value() if they do not exist.

o Sql.Sql is no longer a wrapper class, but a function.

  The wrapper class has been obsoleted by introduction of the
  new base class __builtin.Sql.Connection. Note that it is
  still possible to use Sql.Sql as the type of connection objects.

o Stdio.Port now has a default id of itself (as documented).

  Previously the default id for Stdio.Port objects was 0.

o Named classes are always statements.

  As mentioned above, it is no longer possible to define a named
  class in an expression. Such code has been observed, but none
  where the name was actually used. Use an anonymous class instead.

o Anonymous classes are always expressions.

  It used to be possible (but typically not useful) to define
  an anonymous class as a statement. Add a name for the class.

o Gz.crc32 and Nettle.crc32c now only return positive results.

o glob() has changed.

  The main incompatibilities are that [ and \ are now special
  characters in the pattern, and if you check the return value against
  1 instead of checking if it's true, and use an array as the first
  argument you will have to change your code.

  This is in order to extend the glob function to cover what
  'standard' glob functions do:

  glob() now accepts quotes (\* to match a single *, as an example)
  and handles ranges ([abc] for a, b or c, [a-z0-9] for a single
  character between a and z or 0-9

  You can also negate a range using ^ or ! ([^a-zA-Z] as an example).

  When the first argument (the pattern) to glob is an array, glob now
  returns which pattern in the array matched.

o hash() has been changed to use siphash. The old hash function is
  available under the name hash_8_0(). Note that this is not a
  cryptographic hash function.

o Stdio.UDP()->send() no longer throws errors on EMSGSIZE and EWOULDBLOCK.

o When a thread exits by throwing, the thrown value is propagated to
  wait(). Previously it was sent directly to master()->handle_error().

o String.trim_all_whites() renamed String.trim().

o Floats are no longer by default sorted before added. This may reduce
  the precision but increases the speed of adding large number of
  floats by a factor of 500. Applications handling floats with large
  differences in magnitude need to apply the appropriate sorting
  before arithmetics. As `+ was the only operator that performed
  sorting, and functions like Array.sum did not, this was already a
  concern.

o Returning UNDEFINED from `+ and ``+ is not allowed and will cause an
  exception.

o RegGetValue(), RegGetKeyNames(), RegGetKeyValues(), openlog(),
  syslog() and closelog() have been moved to System.


New modules and functions
-------------------------

o predef::m_add()

  This function adds (as opposed to inserts) an element to a multiset.

o ADT.Scheduler & ADT.TreeScheduler

  These are variants of ADT.Heap where elements typically aren't
  removed from the heap.

o Apple.Keychain

  Parser for Apple Keychain format files (like the ones in
  /System/Library/Keychains/).

o __builtin.Sql

  Generic base classes for Sql API modules.

o Filesystem.Zip
o Function.bind()

  Partially evaluate a function. Similar to Function.curry, but with
  better granularity.

o Standards.HPack (RFC 7541)
o Stdio.FakePipe

  A simulated Stdio.Pipe.

o Parser.Markdown

o Crypto.Checksum

  This module collect non-cryptographic checksums. Support for crc32,
  adler32 and Castagnoli CRC (CRC32C).

  NB: In the future these may be amended to support the Crypto.Hash API.

o Parser.ECMAScript

  This module simply provides a token splitter for
  ECMAScript/JavaScript.

o Pike.Annotations

  Multiple common annotations are available.

o Pike.DestructImmediate

  Objects of classes that inherit Pike.DestructImmediate will be
  destructed immediately on their references reaching zero (rather
  than at some arbitrary time "soon"). This class should be used
  for any classes that are (or hold) locks or similar.

o Pike.InhibitDestruct

  Objects of classes that inherit Pike.InhibitDestruct may
  control whether explicit destruct() calls should succeed
  (or be delayed until later). Note that this class is just
  a convenience API, and that this functionality may also
  be implemented by hand. The intended use is for C-classes
  that do not want their storage to be zapped eg during
  library calls.

  Note that it is not possible to inhibit destruction due
  to running out of references or by the gc.

o Pike.LiveBacktraceFrame

  This is similar to Pike.BacktraceFrame, but refers two-way
  to an active (or not) execution context. It can be used
  to eg examine (and change) local variables, and is intended
  for debugger use.

o Pike.ProxyFactory()

  This is a function that generates a proxy class for objects
  of the given class.

o Process.Process

  Support using Stdio.Fd for "cwd" and "chroot" on OSes that
  have fchdir(2).

o Web.Auth & Web.Api
o Web.EngineIO & Web.SocketIO
o Protocols.HTTP2
o Bittorrent.DHT
o Standards.MsgPack

o Thread.RWMutex

  This class implements a multiple reader / single writer mutex. 


New features
------------

o Command-line options

  - It is now possible to define function-style macros with the
    command-line option '-D'. Eg:

      $ pike '-DFOO(X)=X'

o predef::backtrace()

  predef::backtrace() now takes an optional argument that causes it
  to return LiveBacktraceFrames (instead of BacktraceFrames), which
  may be used to access local variables for the frame while they are
  still in use. This is intended for debugger use.

o predef::destruct()

  lfun::_destruct() may now inhibit explicit destruction.

o predef::equal()

  equal() on functions now checks if their definitions are the same
  identifier in the same program.

o predef::gc()

  gc() called with a weak mapping as argument now removes weak references
  that are only held by that mapping.

o predef::m_clear()

  m_clear() now supports operation on multisets and objects.

o predef::m_delete()

  m_delete() now supports operation on multisets.

o ADT.Heap

  - An indirection object ADT.Heap.Element has been added to make it
    possible to optimize several heap operations.

  - Added low_pop().

o Calendar

  - The timezone expert system has been updated for the first time
    in quite a while.

o Crypto & Nettle

  - Added Curve25519 with EdDSA.

  - Added support for CMAC.

o Filesystem.Monitor

  The filesystem monitoring system now uses accelleration via
  Inotify et al.

o Gmp

  - mpf is now implemented using gmpf if the library is available.

  - Improved support for GMP 5.0 and later.

o GTK2

  Multiple runtime fixes.

o JOSE (JSON Object Signing and Encryption)

  Some low-level API support has been added to the Crypto and Web
  modules to support parts of RFC 7515 - 7520.

o MasterObject

  - Protect against the same file being compiled concurrently
    in multiple threads.

  - cast_to_program() and cast_to_object() should now be thread safe.

o CompilerEnvironment()->lock()

  Access to the compiler lock.

o Crypto.ECC.Curve.Point

  A point on an ECC curve.

o Parser.Pike

  Support new language features.

o Protocols.DNS

  - Protocols.DNS now supports encoding and decoding CAA RRs.

  - Classes derived from Protocols.DNS.server may now override
    report_decode_error() and handle_decode_error() to change how
    errors while decoding a DNS packet are reported and handled.

o Protocols.WebSocket

  Multiple API changes.

o Random rewrite

  The random functions have been rewritten to ensure security by
  default. random_string() and random() now get their data directly
  from the operating system random generator, i.e. /dev/urandom on
  most unixen. This is about half the speed compared with the
  random_string function in Pike 8.0, but is now as secure as the
  system random generator.

  For consumers of random data that have additional requirements,
  different random generators are exposed in the new module
  Random. The module has the following generators:

  - Random.Interface

    This is not actually a generator, but an interface class that is
    inherited into all the other generators. It contains code that can
    turn the output from the random_string method into random numbers
    with different limits without introducing bias. It also contains
    code for all the different variants of random() on different
    types. This is currently not possible to implement in Pike code,
    as the typing is too complicated and it is not possible to access
    private/protected _random methods in objects.

  - Random.System

    This generator maps directly on top of the system random
    generator. This is the default generator used for random() and
    random_string().

  - Random.Deterministic

    This generator creates the same sequence of random numbers for a
    given seed, with good pseudo random properties.

  - Random.Hardware

    This generator accesses the hardware random generator, when
    available.

  - Random.Fast

    This generator takes entropy from the Random.System, but feeds
    it into a cryptographic pseudo random number generator to be
    able to output data fast. This is not the default random number
    generator to avoid loading crypto code on every startup.

  Comparing the different generators with each other gives the
  following approximate speeds on a Linux system with hardware
  random support:

    Random.System           1.0
    Pike 8.0 random_string  0.45
    Random.Hardware         0.25
    Random.Fast             0.20
    Random.Deterministic    0.20

  Objects implementing the _random lfun now get two arguments, the
  current random_string() and random() functions. This is convenient
  for C-level functions in that they don't need to look up functions
  themselves. Note that it is possible for a user to replace these
  with non-conforming functions (returning values of the wrong type,
  strings of the wrong length or shift size, and values outside the
  given range) or even non-functions.

  All code in Pike that use random numbers now use the current random
  functions (though in some cases fixed at object creation). This
  allows for repeatable results if the random functions are replaced
  with a deterministic random generator, such as
  Random.Deterministic. Example:

    Random.Deterministic rnd = Random.Deterministic( seed );
    add_constant( "random_string", rnd->random_string );
    add_constant( "random", rnd->random );

o Sql

  - Most Sql C-modules converted to cmod.

  - All Sql driver modules have __builtin.Sql.Connection as
    a common base class.

  - Added next_result() to support queries returning multiple resultsets.

  - ODBC & tds: Support more datatypes.

  - ODBC: Support big_typed_query().

  - pgsql: Lots of changes and fixes.

  - pgsql: Toggle cache_autoprepared_statements default to off;
    turning it on triggers a bug in PostgreSQL sometimes
    that causes spikes in CPU usage of the database.

o SSL

  - Support session tickets.

  - Support Negotiated FF-DHE.

  - Support Curve25519 key exchange.

  - Support client certificates.

  - Support ALPN.

  - Prefer AEAD suites to CBC suites.

  - SSL.File supports set_buffer_mode().

o Standards.PKCS

  Support PKCS#8 private keys.

o Stdio.Fd

  - Support changing directory relative to an open file.

o String.Buffer & Stdio.Buffer

  Added _search().

o System.LookupAccountName(), etc [NT]

  Several NT-specific functions now support wide strings
  as arguments and as output.

o The self testing framework now supports *.test-files.

o Thread

  - _sprintf() improvements: Thread.Mutex now prints the ID of the thread
    holding the lock, and thread IDs are shown as hexadecimal numbers.

  - Thread.Farm now supports a callback for thread creation and termination
    for the purpose of tracking thread names.

  - Thread.Condition are now bound to the first Thread.Mutex object
    that they are used together with. This ensures that the correct
    Thread.Mutex is aways used.

o sprintf %x

  %X and %x can now be used on 8-bit wide strings to get a hexadecimal
  representation of their contents. Just calling sprintf("%x",data) is
  the same as calling String.string2hex(data).

o Unicode 10.0.0.

o Unicode.is_whitespace()

  This new function returns whether a unicode character is a whitespace
  character or not.


Deprecated symbols and modules
------------------------------

o Sql.mysql_result and Sql.mysqls_result have been deprecated.
  Use Sql.Result instead.

o call_function() has been deprecated. Use `()() instead.


Removed features and modules
----------------------------

o Compatibility for Pike versions before 7.8 is no longer available.

o GTK1 library is deprecated, so glue code has been removed.


C-level API changes
-------------------

o The contract for functions is now changed so that a function is no
  longer required to clean the stack. The topmost value of the stack
  will be regarded as the return value and the rest of the items on
  the stack, compared to before the function arguments were pushed,
  will be popped and discarded. Efuns still have to clean their stack
  as previously.

o Object destructor callbacks may now run during pop_stack() and
  pop_n_elems(). This means that code must be prepared for arbitrary
  state changes after having called them.

o Removed add_function, pike_add_function, pike_add_function2,
  simple_add_variable, map_variable and MAP_VARIABLE. This removes the
  remaining few API:s where text types were used. Use ADD_FUNCTION and
  PIKE_MAP_VARIABLE instead.

o Removed the functions my_rand and my_srand. Use the random functions
  on the stack for _random lfuns, the push_random_string or look up
  the random function from get_builtin_constants(). For deterministic
  pseudo random, create a private Random.Deterministic object.

o The preprocessor has been converted into a cmod, and been modified
  to use more standard Pike datatypes.

o The preprocessor-specific hashtable implementation has been removed.

o The gdb_backtraces() function is now available also --without-debug.

o There's now support to block mapping hashtables from being shrunk
  on map_delete().

o guaranteed_memset() is replaced with secure_zero() which fills a
  buffer with zero. On x86 SSE2 is used to zero the memory without
  loading it into the CPU cache, as this function is typically used
  before calling free() on memory with cryptographic key data.

o The GC marker hash table has been removed. For types which require
  GC markers, they are now allocated as part of the data type. Such
  structures must now start with GC_MARKER_MEMBERS (which includes
  eg the refs field). See "gc_header.h" for details.

  This change significantly improves GC performance (up to a factor
  of 2 in some situations).


Documentation
-------------

o RFC references added.

o Character encoding issues fixed.

o Added @enum/@endenum markup.

o Support undocumented enums with documented constants.

o Add some support for documenting annotations.


Building and installing
-----------------------

o GMP 4.1 or later is now required.

o Nettle 3.5 is now supported.

o C99 assumed

  The configure tests will not check for functions defined in C99
  anymore and C99 compiler support is assumed.