Use python-requests as HTTP client in Locust

Makefile

@@ -4,5 +4,5 @@ test:
 release:
 	python setup.py sdist upload
 
-docs:
+build_docs:
 	sphinx-build -b html docs/ docs/_build/

docs/api.rst

@@ -8,19 +8,36 @@ Locust class
 
 .. autoclass:: locust.core.Locust
 	:members: tasks, min_wait, max_wait, schedule_task, client
+
+	.. autoattribute:: locust.core.LocustBase.min_wait
+	.. autoattribute:: locust.core.LocustBase.max_wait
+	.. autoattribute:: locust.core.LocustBase.tasks
 
 
-HttpBrowser class
+HttpSession class
 =================
 
-.. autoclass:: locust.clients.HttpBrowser
-	:members: __init__, get, post
+.. autoclass:: locust.clients.HttpSession
+	:members: __init__, request, get, post, delete, put, head, options, patch
 
-HttpResponse class
-==================
+Response class
+==============
 
-.. autoclass:: locust.clients.HttpResponse
-	:members: code, data, url, info
+This class actually resides in the `python-requests <http://python-requests.org>`_ library, 
+since that's what Locust is using to make HTTP requests, but it's included in the API docs 
+for locust since it's so central when writing locust load tests. You can also look at the 
+:py:class:`Response <requests.Response>` class at the 
+`requests documentation <http://python-requests.org>`_.
+
+.. autoclass:: requests.Response
+	:inherited-members:
+	:noindex:
+
+ResponseContextManager class
+============================
+
+.. autoclass:: locust.clients.ResponseContextManager
+	:members: success, failure
 
 
 @require_once decorator

docs/changelog.rst

@@ -1,25 +1,112 @@
-==========
+##########
 Changelog
-==========
+##########
 
 0.6
 ===
 
-Improvements and bug fixes
---------------------------
-
-* Scheduled task callabled can now take keyword arguments.
+.. warning::
+
+    This version comes with non backward compatible changes to the API. 
+    Anyone who is currently using existing locust scripts and want to upgrade to 0.6
+    is adviced to read through these changes. It's nothing major, and the upgrade 
+    should be possible without too much pain.
+
+
+Locust now uses Requests
+------------------------
+
+Locust's own HttpBrowser class (which was typically accessed through *self.client* from within a locust class) 
+has been replaced by a thin wrapper around the requests library (http://python-requests.org). This comes with 
+a number of advantages. Users can  now take advantage of a well documented, well written, fully fledged 
+library for making HTTP requests. However, it also comes with some small API changes wich will require users 
+to update their existing load testing scripts.
+
+Gzip encoding turned on by default
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The HTTP client now sends headers for accepting gzip encoding by default. The **--gzip** command line argument 
+has been removed and if someone want to disable the *Accept-Encoding* that the HTTP client uses, or any 
+other HTTP headers you can do::
+
+    class MyWebUser(Locust):
+        def on_start(self):
+            self.client.headers = {"Accept-Encoding":""}
+
+
+Improved HTTP client
+^^^^^^^^^^^^^^^^^^^^
+
+Because of the switch to using python-requests in the HTTP client, the API for the client has also 
+gotten a few changes.
+
+* Additionally to the :py:meth:`get <locust.clients.HttpSession.get>`, :py:meth:`post <locust.clients.HttpSession.post>`, 
+  :py:meth:`put <locust.clients.HttpSession.put>`, :py:meth:`delete <locust.clients.HttpSession.delete>` and 
+  :py:meth:`head <locust.clients.HttpSession.head>` methods, the :py:class:`HttpSession <locust.clients.HttpSession>` class 
+  now also has :py:meth:`patch <locust.clients.HttpSession.patch>` and :py:meth:`options <locust.clients.HttpSession.options>` methods.
+
+* All arguments to the HTTP request methods, except for **url** and **data** should now be specified as keyword arguments.
+  For example, previously one could specify headers using::
+
+      client.get("/path", {"User-Agent":"locust"}) # this will no longer work
+
+  And should now be specified like::
+
+      client.get("/path", headers={"User-Agent":"locust"})
+
+* In general the whole HTTP client is now more powerful since it leverages on python-requests. Features that we're
+  now able to use in Locust includes file upload, SSL, connection keep-alive, and more.
+  See the `python-requests documentation <http://python-requests.org>`_ for more details.
+
+* The new :py:class:`HttpSession <locust.clients.HttpSession>` class' methods now return python-request 
+  :py:class:`Response <requests.Response>` objects. This means that accessing the content of the response 
+  is no longer made using the **data** attribute, but instead the **content** attribute. The HTTP response 
+  code is now accessed through the **status_code** attribute, instead of the **code** attribute.
+
+
+HttpSession methods' catch_response argument improved and allow_http_error argument removed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* When doing HTTP requests using the **catch_response** argument, the context manager that is returned now
+  provides two functions, :py:meth:`success <locust.clients.ResponseContextManager.success>` and 
+  :py:meth:`failure <locust.clients.ResponseContextManager.failure>` that can be used to manually control 
+  what the request should be reported as in Locust's statistics.
+
+  .. autoclass:: locust.clients.ResponseContextManager
+    :members: success, failure
+    :noindex:
+
+* The **allow_http_error** argument of the HTTP client's methods has been removed. Instead one can use the 
+  **catch_response** argument to get a context manager, which can be used together with a with statement.
+
+  The following code in the previous Locust version::
+
+      client.get("/does/not/exist", allow_http_error=True)
+
+  Can instead now be written like::
+
+      with client.get("/does/not/exist", catch_response=True) as response:
+          response.success()
+
+
+Other improvements and bug fixes
+--------------------------------
+
+* Scheduled task callables can now take keyword arguments and not only normal function arguments.
 * SubLocust classes that are scheduled using :func:`locust.core.Locust.schedule_task` can now take 
   arguments and keyword arguments (available in *self.args* and *self.kwargs*).
 
 
+
 API Changes
 -----------
 
 * Changed signature of :func:`locust.core.Locust.schedule_task`. Previously all extra arguments that
   was given to the method was passed on to the the task when it was called. It no longer accepts extra arguments. 
   Instead, it takes an *args* argument (list) and a *kwargs* argument (dict) which are be passed to the task when 
   it's called.
+* Arguments for :py:class:`request_success <locust.events.request_success>` event hook has been changed. 
+  Previously it took an HTTP Response instance as argument, but this has been changed to take the 
+  content-length of the response instead. This makes it easier to write custom clients for Locust.
 
 
 0.5.1

docs/conf.py

@@ -14,7 +14,7 @@
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ["sphinx.ext.autodoc"]
+extensions = ["sphinx.ext.autodoc", "sphinx.ext.intersphinx"]
 
 # autoclass options
 #autoclass_content = "both"
@@ -32,6 +32,11 @@
 project = 'Locust'
 #copyright = ''
 
+# Intersphinx config
+intersphinx_mapping = {
+    'requests': ('http://requests.readthedocs.org/en/latest/', None),
+}
+
 # The default replacements for |version| and |release|, also used in various
 # other places throughout the built documents.
 #

docs/writing-a-locustfile.rst

@@ -132,32 +132,50 @@ and **/** will be requested twice the amount of times than **/about/**.
 Using the HTTP client
 ======================
 
-Each instance of Locust has an instance of HttpBrowser in the *client* attribute. 
+Each instance of Locust has an instance of :py:class:`HttpSession <locust.clients.HttpSession>` 
+in the *client* attribute. The HttpSession class is actually a subclass of 
+:py:class:`requests.Session` and can be used to  make HTTP requests, that will be reported to Locust's
+statistics, using the :py:meth:`get <locust.clients.HttpSession.get>`, 
+:py:meth:`post <locust.clients.HttpSession.post>`, :py:meth:`put <locust.clients.HttpSession.put>`, 
+:py:meth:`delete <locust.clients.HttpSession.delete>`, :py:meth:`head <locust.clients.HttpSession.head>`, 
+:py:meth:`patch <locust.clients.HttpSession.patch>` and :py:meth:`options <locust.clients.HttpSession.options>` 
+methods. The HttpInstance will preserve cookies between requests so that it can be used to log in to websites 
+and keep a session between requests.
 
-.. autoclass:: locust.clients.HttpBrowser
-    :members: __init__, get, post
-    :noindex:
+Here's a simple example that makes a GET request to the */about* path (in this case we assume *self* 
+is an instance of a :py:class:`Locust <locust.core.Locust>` class::
+
+    response = self.client.get("/about")
+    print "Response status code:", response.status_code
+    print "Response content:", response.content
+
+And here's an example making a POST request::
+
+    response = self.client.post("/login", {"username":"testuser", "password":"secret"})
+
+Manually controlling if a request should be considered successful or a failure
+------------------------------------------------------------------------------
 
 By default, requests are marked as failed requests unless the HTTP response code is ok (2xx). 
-However, one can mark requests as failed, even when the response code is okay, by using the 
-*catch_response* argument with a with statement::
+Most of the time, this default is what you want. Sometimes however - for example when testing 
+a URL endpoint that you expect to return 404, or testing a badly designed system that might 
+return *200 OK* even though an error occurred - there's a need for manually controlling if 
+locust should consider a request as a success or a failure.
+
+One can mark requests as failed, even when the response code is okay, by using the 
+*catch_response* argument and a with statement::
 
-    from locust import ResponseError
-
-    client = HttpBrowser("http://example.com")
     with client.get("/", catch_response=True) as response:
-        if response.data != "Success":
-            raise ResponseError("Got wrong response")
+        if response.content != "Success":
+            response.failure("Got wrong response")
 
-Just as one can mark requests with OK response codes as failures, one can also make requests that 
-results in an HTTP error code still result in a success in the statistics::
+Just as one can mark requests with OK response codes as failures, one can also use **catch_response** 
+argument together with a *with* statement to make requests that resulted in an HTTP error code still 
+be reported as a success in the statistics::
 
-    client = HttpBrowser("http://example.com")
-    response = client.get("/does_not_exist/", allow_http_error=True)
-    if response.exception:
-        print "We got an HTTPError exception, but the request will still be marked as OK"
-
-Also, *catch_response* and *allow_http_error* can be used together.
+    with client.get("/does_not_exist/", catch_response=True) as response:
+        if response.status_code == 404:
+            response.success()
 
 
 The on_start function

examples/basic.py

@@ -1,4 +1,4 @@
-from locust import Locust, require_once
+from locust import Locust, require_once, task
 import random
 
 def login(l):
@@ -20,3 +20,7 @@ class WebsiteUser(Locust):
     tasks = [index, stats]
     min_wait=2000
     max_wait=5000
+
+    @task
+    def page404(self):
+        self.client.get("/does_not_exist")