The Windows Update (#4214)

* The Windows Update

This PR includes a major refactor regarding several Windows specific
protocols, particularily SMB2/3 and DCE/RPC. It also touches many parts
of Scapy's core in order to accommodate those changes.

This includes:
  - DCE/RPC:
      - DCERPC_Client with support for NCACN_IP_TCP and NCACN_NP
      - DCERPC_Server with support for NCACN_IP_TCP and NCACN_NP
      - Finish NDR engine !
      - Server/Client Endpoint mapper support
      - Premises of a few special clients (Netlogon, DCOM, ...)
      - A few RPC interfaces (to debug/test the DCE/RPC engine. TODO:
        MORE !)
      - Documentation !
  - SMB2/3:
      - Protocol refactor, many more SMB2/3 structures supported
      - Server (class + 'simple' util smbserver()) (2.0.2 to 3.1.1)
      - Client (class + interactive CLI smbclient()) (2.0.2 only)
      - SMB socket, RPC over SMB socket, etc.
      - Documentation !
  - Kerberos:
      - KerberosSSP to use in SMB/RPC clients/servers
      - Crypto: use cryptography, latest RFC8009, typing, etc.
      - Util functions krb_as_req, krb_tgt_req, kpasswd (both modes), etc.
      - [MS-KILE] variants, SFU and more !
      - Both MIT and Windows variations support
  - NTLM:
      - refactor, clean SSP
      - remove relay (TODO: reimplement with the new system)
  - Extensive GSSAPI / SPNEGO support !
  - Ticketer++
      - CCache support: read/write
      - Kerberos integration: ask/renew/resign/edit tickets
      - Change ticket fields through a GUI ! and more !
  - LDAP
      - Fixes, ASN.1 Windows variation support
      - dclocator, answering machine for "LDAP PING"
  - Automaton:
      - fixes (memory usage on Windows)
      - support for EOF events
      - spawn() mode, better socket.socket support and more !
  - StreamSocket changes, support for TCP reassembly, etc !
  - Unit tests for everything (using samba, etc.)

* CI: remove travis and log smbclient version

* Add parsing of SECURITY_DESCRIPTOR

* Use UPN in NTLMSSP

* Slightly more doc

* More smb error handling

* Minor SMB/Kerberos doc updates

* Minor ticketer improvements and fixes

* Conditional ACEs + SDDL

* Minor SMB client API cleanups for query

* Remove rfc3961 cryptography's hack for DES

* Fix smbserver symlinks + smbclient guest

* PEP8

* Minor doc fix

* Fix wrong SPNEGO auth_type

.config/ci/install.sh

@@ -14,7 +14,7 @@ then
 fi
 
 # Install on osx
-if [ "${OSTYPE:0:6}" = "darwin" ] || [ "$TRAVIS_OS_NAME" = "osx" ]
+if [ "${OSTYPE:0:6}" = "darwin" ]
 then
   if [ ! -z $SCAPY_USE_LIBPCAP ]
   then
@@ -23,30 +23,31 @@ then
   fi
 fi
 
-# Install wireshark data, ifconfig & vcan
-if [ "$OSTYPE" = "linux-gnu" ] || [ "$TRAVIS_OS_NAME" = "linux" ]
+# Install wireshark data, ifconfig, vcan, samba
+if [ "$OSTYPE" = "linux-gnu" ]
 then
   sudo apt-get update
   sudo apt-get -qy install tshark net-tools || exit 1
   sudo apt-get -qy install can-utils || exit 1
   sudo apt-get -qy install linux-modules-extra-$(uname -r) || exit 1
+  sudo apt-get -qy install samba smbclient
   # Make sure libpcap is installed
   if [ ! -z $SCAPY_USE_LIBPCAP ]
   then
     sudo apt-get -qy install libpcap-dev  || exit 1
   fi
 fi
 
-# On Travis, "osx" dependencies are installed in .travis.yml
-if [ "$TRAVIS_OS_NAME" != "osx" ]
-then
-  # Update pip & setuptools (tox uses those)
-  python -m pip install --upgrade pip setuptools --ignore-installed
+# Update pip & setuptools (tox uses those)
+python -m pip install --upgrade pip setuptools wheel --ignore-installed
 
-  # Make sure tox is installed and up to date
-  python -m pip install -U tox --ignore-installed
-fi
+# Make sure tox is installed and up to date
+python -m pip install -U tox --ignore-installed
 
 # Dump Environment (so that we can check PATH, UT_FLAGS, etc.)
 openssl version
+if [ "$OSTYPE" = "linux-gnu" ]
+then
+  smbclient -V
+fi
 set

.config/codespell_ignore.txt

@@ -1,12 +1,14 @@
 aci
 ans
-archtypes
 applikation
+archtypes
 ba
+browseable
 byteorder
 cace
 cas
 componet
+comversion
 cros
 delt
 doas
@@ -18,8 +20,9 @@ funktion
 gost
 hart
 iff
-interaktive
+implementors
 inout
+interaktive
 microsof
 mitre
 nd

.config/mypy/mypy.ini

@@ -6,6 +6,9 @@
 ignore_errors = True
 ignore_missing_imports = True
 
+[mypy-scapy.libs.rfc3961]
+warn_return_any = False
+
 # Layers specific config
 
 [mypy-scapy.arch.*]

.config/mypy/mypy_enabled.txt

@@ -95,6 +95,7 @@ scapy/libs/__init__.py
 scapy/libs/ethertypes.py
 scapy/libs/extcap.py
 scapy/libs/matplot.py
+scapy/libs/rfc3961.py
 scapy/libs/structures.py
 scapy/libs/test_pyx.py
 

doc/scapy/advanced_usage.rst

@@ -730,9 +730,9 @@ The ``START`` event is ``initial=1``, the ``STOP`` event is ``stop=1`` and the `
 Decorators for transitions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Transitions are methods decorated by the result of one of ``ATMT.condition``, ``ATMT.receive_condition``, ``ATMT.timeout``, ``ATMT.timer``. They all take as argument the state method they are related to. ``ATMT.timeout`` and ``ATMT.timer`` also have a mandatory ``timeout`` parameter to provide the timeout value in seconds. The difference between ``ATMT.timeout`` and ``ATMT.timer`` is that ``ATMT.timeout`` gets triggered only once. ``ATMT.timer`` get reloaded automatically, which is useful for sending keep-alive packets. ``ATMT.condition`` and ``ATMT.receive_condition`` have an optional ``prio`` parameter so that the order in which conditions are evaluated can be forced. The default priority is 0. Transitions with the same priority level are called in an undetermined order.
+Transitions are methods decorated by the result of one of ``ATMT.condition``, ``ATMT.receive_condition``, ``ATMT.eof``, ``ATMT.timeout``, ``ATMT.timer``. They all take as argument the state method they are related to. ``ATMT.timeout`` and ``ATMT.timer`` also have a mandatory ``timeout`` parameter to provide the timeout value in seconds. The difference between ``ATMT.timeout`` and ``ATMT.timer`` is that ``ATMT.timeout`` gets triggered only once. ``ATMT.timer`` get reloaded automatically, which is useful for sending keep-alive packets. ``ATMT.condition`` and ``ATMT.receive_condition`` have an optional ``prio`` parameter so that the order in which conditions are evaluated can be forced. The default priority is 0. Transitions with the same priority level are called in an undetermined order.
 
-When the automaton switches to a given state, the state's method is executed. Then transitions methods are called at specific moments until one triggers a new state (something like ``raise self.MY_NEW_STATE()``). First, right after the state's method returns, the ``ATMT.condition`` decorated methods are run by growing prio. Then each time a packet is received and accepted by the master filter all ``ATMT.receive_condition`` decorated hods are called by growing prio. When a timeout is reached since the time we entered into the current space, the corresponding ``ATMT.timeout`` decorated method is called.
+When the automaton switches to a given state, the state's method is executed. Then transitions methods are called at specific moments until one triggers a new state (something like ``raise self.MY_NEW_STATE()``). First, right after the state's method returns, the ``ATMT.condition`` decorated methods are run by growing prio. Then each time a packet is received and accepted by the master filter all ``ATMT.receive_condition`` decorated hods are called by growing prio. When a timeout is reached since the time we entered into the current space, the corresponding ``ATMT.timeout`` decorated method is called. If the socket raises an ``EOFError`` (closed) during a state, the ``ATMT.EOF`` transition is called. Otherwise it raises an exception and the automaton exits.
 
 ::