Back to Table of Contents

Parsing request data from the QUERY_STRING, cookies, and request body

The RqData module is used to extract key/value pairs from the QUERY_STRING, cookies, and the request body of a POST or PUT request.

Hello RqData

Let's start with a simple hello, world! example that uses request parameters in the URL.

> module Main where
> import Happstack.Server (ServerPart, look, nullConf, simpleHTTP, ok)
> helloPart :: ServerPart String
> helloPart =
>     do greeting <- look "greeting"
>        noun     <- look "noun"
>        ok $ greeting ++ ", " ++ noun
> main :: IO ()
> main = simpleHTTP nullConf $ helloPart

[Source code for the app is here.]

Now if we visit http://localhost:8000/?greeting=hello&noun=rqdata, we will get the message hello, rqdata

we use the look function to look up some keys by name. The look function has the type:

> look :: (Functor m, Monad m, HasRqData m) => String -> m String

Since we are using look in the ServerPart monad it has the simplified type:

> look :: String -> ServerPart String

The look function looks up a key and decodes the associated value as a String. It assumes the underlying ByteString was utf-8 encoded. If you are using some other encoding, then you can use lookBS to construct your own lookup function.

If the key is not found, then look will fail. In ServerPart that means it will call mzero.

Handling Submissions

In the previous example we only looked at parameters in the URL. Looking up values from a form submission (a POST or PUT request) is almost the same. The only difference is we need to first decode the request body using decodeBody:

> {-# LANGUAGE OverloadedStrings #-}
> import Control.Monad                      (msum)
> import Happstack.Server                   ( Response, ServerPart, Method(POST)
>                                           , BodyPolicy(..), decodeBody, defaultBodyPolicy
>                                           , dir, look, nullConf, ok, simpleHTTP
>                                           , toResponse, method
>                                           )
> import Text.Blaze                         as B
> import Text.Blaze.Html4.Strict            as B hiding (map)
> import Text.Blaze.Html4.Strict.Attributes as B hiding (dir, label, title)
> main :: IO ()
> main = simpleHTTP nullConf $ handlers
> myPolicy :: BodyPolicy
> myPolicy = (defaultBodyPolicy "/tmp/" 0 1000 1000)
> handlers :: ServerPart Response
> handlers =
>     do decodeBody myPolicy
>        msum [ dir "hello" $ helloPart
>             , helloForm
>             ]
> helloForm :: ServerPart Response
> helloForm = ok $ toResponse $
>     html $ do
>       B.head $ do
>         title "Hello Form"
>       B.body $ do
>         form ! enctype "multipart/form-data" ! B.method "POST" ! action "/hello" $ do
>              B.label "greeting: " >> input ! type_ "text" ! name "greeting" ! size "10"
>              B.label "noun: "     >> input ! type_ "text" ! name "noun" ! size "10"
>              input ! type_ "submit" ! name "upload"
> helloPart :: ServerPart Response
> helloPart =
>     do method POST
>        greeting <- look "greeting"
>        noun     <- look "noun"
>        ok $ toResponse (greeting ++ ", " ++ noun)

[Source code for the app is here.]

Why is decodeBody even needed?

The body of the HTTP request is ignored unless we call decodeBody. The obvious question is, Why isn't the request body automatically decoded?.

If servers had unlimited RAM, disk, CPU and bandwidth available, then automatically decoding the body would be a great idea. But, since that is generally not the case, we need a way to limit or ignore form submission data that is considered excessive.

A simple solution would be to impose a static quota an all form data submission server-wide. But, in practice, you might want finer granularity of control. By explicitly calling decodeBody you can easily configure a site-wide static quota. But you can also easily adapt the quotas depending on the user, particular form, or other criteria.

In this example, we keep things simple and just call decodeBody for all incoming requests. If the incoming request is not a PUT or POST request with multipart/form-data then calling decodeBody has no side-effects.

Using BodyPolicy and defaultBodyPolicy to impose quotas

The only argument to decodeBody is a BodyPolicy. The easiest way to define a BodyPolicy is by using the defaultBodyPolicy function:

> defaultBodyPolicy :: FilePath  -- ^ directory to *temporarily* store uploaded files in
>                   -> Int64     -- ^ max bytes to save to disk (files)
>                   -> Int64     -- ^ max bytes to hold in RAM (normal form values, etc)
>                   -> Int64     -- ^ max header size (this only affects headers
>                                --                    in the multipart/form-data)
>                   -> BodyPolicy

In the example, we define this simple policy:

> myPolicy :: BodyPolicy
> myPolicy = (defaultBodyPolicy "/tmp/" 0 1000 1000)

Since the form does not do file uploads, we set the file quota to 0. We allow 1000 bytes for the two form fields and 1000 bytes for overhead in the multipart/form-data encoding.

Using decodeBody

Using decodeBody is pretty straight-forward. You simple call it with a BodyPolicy. The key things to know are:

  1. You must call it anytime you are processing a POST or PUT request and you want to use look and friends
  2. decodeBody only works once per request. The first time you call it the body will be decoded. The second time you call it, nothing will happen, even if you call it with a different policy.

Other tips for using <form>

When using the <form> element there are two important recommendations you should follow:

  1. Set the enctype to multipart/form-data. This is especially important for forms which contain file uploads.
  2. Make sure to set method to POST or the form values will show up in the URL as query parameters.

File Uploads

The lookFile function is used to extract an uploaded file:

> lookFile :: String -> RqData (FilePath, FilePath, ContentType)

It returns three values:

  1. The location of the temporary file which holds the contents of the file
  2. The local filename supplied by the browser. This is typically the name of the file on the users system.
  3. The content-type of the file (as supplied by the browser)

The temporary file will be automatically deleted after the Response is sent. Therefore, it is essential that you move the file from the temporary location.

In order for file uploads to work correctly, it is also essential that your <form> element contains the attributes enctype="multipart/form-data" and method="POST"

The following example has a form which allows a user to upload a file. We then show the temporary file name, the uploaded file name, and the content-type of the file. In a real application, the code should use System.Directory.renameFile (or similar) to move the temporary file to a permanent location. This example looks a bit long, but most of the code is just HTML generation using BlazeHtml. The only really new part is the use of the lookFile function. Everything else should already have been covered in previous sections. So if you don't understand something, try looking in earlier material.

> {-# LANGUAGE OverloadedStrings #-}
> import Control.Monad                      (msum)
> import Happstack.Server                   ( Response, ServerPart, defaultBodyPolicy
>                                           , decodeBody, dir, lookFile, nullConf, ok
>                                           , simpleHTTP, toResponse )
> import Text.Blaze                         as B
> import Text.Blaze.Html4.Strict            as B hiding (map)
> import Text.Blaze.Html4.Strict.Attributes as B hiding (dir, title)
> main :: IO ()
> main = simpleHTTP nullConf $ upload
> upload :: ServerPart Response
> upload =
>     do decodeBody (defaultBodyPolicy "/tmp/" (10*10^6) 1000 1000)
>        msum [ dir "post" $ post
>             , uploadForm
>             ]
> uploadForm :: ServerPart Response
> uploadForm = ok $ toResponse $
>     html $ do
>       B.head $ do
>         title "Upload Form"
>       B.body $ do
>         form ! enctype "multipart/form-data" ! B.method "POST" ! action "/post" $ do
>              input ! type_ "file" ! name "file_upload" ! size "40"
>              input ! type_ "submit" ! value "upload"
> post :: ServerPart Response
> post =
>    do r <- lookFile "file_upload"
>       ok $ toResponse $
>          html $ do
>            B.head $ do
>              title "Post Data"
>            B.body $ mkBody r
>     where
>       mkBody (tmpFile, uploadName, contentType) = do
>                 p (toHtml $ "temporary file: " ++ tmpFile)
>                 p (toHtml $ "uploaded name:  " ++ uploadName)
>                 p (toHtml $ "content-type:   " ++ show contentType)

[Source code for the app is here.]

File uploads important reminder

Remember that you must move the temporary file to a new location or it will be garbage collected after the Response is sent. In the example code we do not move the file, so it is automatically deleted.

Limiting lookup to QUERY_STRING or request body

By default, look and friends will search both the QUERY_STRING the request body (aka, POST/PUT data) for a key. But sometimes we want to specify that only the QUERY_STRING or request body should be searched. This can be done by using the body and queryString filters:

> body :: (HasRqData m) => m a -> m a
> queryString ::  (HasRqData m) => m a -> m a

Using these filters we can modify helloPart so that the greeting must come from the QUERY_STRING and the noun must come from the request body:

> helloPart :: ServerPart String
> helloPart =
>     do greeting <- queryString $ look "greeting"
>        noun     <- body        $ look "noun"
>        ok $ greeting ++ ", " ++ noun

queryString and body act as filters which only pass a certain subset of the data through. If you were to write:

> greetingRq :: ServerPart String
> greetingRq =
>     body (queryString $ look "greeting")

This code would never match anything because the body filter would hide all the QUERY_STRING values, and the queryString filter would hide all the request body values, and hence, there would be nothing left to search.

Using the RqData Monad for better error reporting

So far we have been using the look function in the ServerPart monad. This means that if any look fails, that handler fails. Unfortunately, we are not told what parameter was missing -- which can be very frustrating when you are debugging your code. It can be even more annoying if you are providing a web service, and whenever a developer forgets a parameter, they get a 404 with no information about what went wrong.

So, if we want better error reporting, we can use functions like look in the RqData Applicative Functor.

We can use getDataFn to run the RqData:

> getDataFn :: (HasRqData m, ServerMonad m, MonadIO m) =>
>              RqData a
>           -> m (Either [String] a)
> module Main where
> import Control.Applicative ((<$>), (<*>))
> import Happstack.Server (ServerPart, badRequest, nullConf, ok, simpleHTTP)
> import Happstack.Server.RqData (RqData, look, getDataFn)
> helloRq :: RqData (String, String)
> helloRq =
>     (,) <$> look "greeting" <*> look "noun"
> helloPart :: ServerPart String
> helloPart =
>     do r <- getDataFn helloRq
>        case r of
>          (Left e) ->
>              badRequest $ unlines e
>          (Right (greet, noun)) ->
>              ok $ greet ++ ", " ++ noun
> main :: IO ()
> main = simpleHTTP nullConf $ helloPart

[Source code for the app is here.]

If we visit http://localhost:8000/?greeting=hello&noun=world, we will get our familiar greeting hello, world.

But if we leave off the query parameters http://localhost:8000/, we will get a list of errors:

Parameter not found: greeting
Parameter not found: noun

We could use the Monad instance RqData to build the request. However, the monadic version will only show us the first error that is encountered. So would have only seen that the greeting was missing. Then when we added a greeting we would have gotten a new error message saying that noun was missing.

In general, improved error messages are not going to help people visiting your website. If the parameters are missing it is because a form or link they followed is invalid. There are two places where there error messages are useful:

  1. When you are developing and debugging your site
  2. Reporting errors to users of your web service API

If you are providing a REST API for developers to use, they are going to be a lot happier if they get a detailed error messages instead of a plain old 404.

Handling non-String request parameters

So far we have only tried to look up String values using look. But the RqData module provides a variety of ways to work with values besides Strings.

Using Read

For some types, it is sufficient to use Read to parse the String into a value. RqData provides functions such as lookRead to assist with this. The advantage of using lookRead instead of calling look and applying read yourself is that lookRead ties into the RqData error handling system neatly.

> lookRead :: (Functor m, Monad m, HasRqData m, Read a) => String -> m a

Here is a trivial example where we create a lookInt function which looks for an Int parameter named int.

> module Main where
> import Control.Applicative ((<$>), (<*>))
> import Happstack.Server (ServerPart, badRequest, nullConf, ok, simpleHTTP)
> import Happstack.Server.RqData (RqData, lookRead, getDataFn)
> lookInt :: RqData Int
> lookInt = lookRead "int"
> intPart :: ServerPart String
> intPart =
>     do r <- getDataFn lookInt
>        case r of
>          (Left e) ->
>              badRequest $ unlines e
>          (Right i) ->
>              ok $ "Read the int: " ++ show i
> main :: IO ()
> main = simpleHTTP nullConf $ intPart

[Source code for the app is here.]

Now if we visit http://localhost:8000/?int=1, we will get the message:

Read the int: 1

If we visit http://localhost:8000/?int=apple, we will get the error:

Read failed while parsing: apple

Using checkRq

Sometimes the representation of a value as a request parameter will be different from the representation required by Read. We can use checkRq to lift a custom parsing function into RqData.

> checkRq :: (Monad m, HasRqData m) => m a -> (a -> Either String b) -> m b

In this example we create a type Vote with a custom parsing function:

> module Main where
> import Control.Applicative ((<$>), (<*>))
> import Happstack.Server (ServerPart, badRequest, nullConf, ok, simpleHTTP)
> import Happstack.Server.RqData (RqData, checkRq, getDataFn, look, lookRead)
> data Vote = Yay | Nay deriving (Eq, Ord, Read, Show, Enum, Bounded)
> parseVote :: String -> Either String Vote
> parseVote "yay" = Right Yay
> parseVote "nay" = Right Nay
> parseVote str   = Left $ "Expecting 'yay' or 'nay' but got: " ++ str
> votePart :: ServerPart String
> votePart =
>     do r <- getDataFn (look "vote" `checkRq` parseVote)
>        case r of
>          (Left e) ->
>              badRequest $ unlines e
>          (Right i) ->
>              ok $ "You voted: " ++ show i
> main :: IO ()
> main = simpleHTTP nullConf $ votePart

[Source code for the app is here.]

Now if we visit http://localhost:8000/?vote=yay, we will get the message:

You voted: Yay

If we visit http://localhost:8000/?vote=yes, we will get the error:

Expecting 'yay' or 'nay' but got: yes

Other uses of checkRq

Looking again at the type for checkRq we see that function argument is fairly general -- it is not restricted to just string input:

> checkRq :: RqData a -> (a -> Either String b) -> RqData b

So, checkRq is not limited to just parsing a String into a value. We could use it, for example, to validate an existing value. In the following example we use lookRead "i" to convert the value i to an Int, and then we use checkRq to ensure that the value is within range:

> module Main where
> import Control.Applicative ((<$>), (<*>))
> import Happstack.Server (ServerPart, badRequest, nullConf, ok, simpleHTTP)
> import Happstack.Server.RqData (RqData, checkRq, getDataFn, look, lookRead)
> inRange :: (Show a, Ord a) => a -> a -> a -> Either String a
> inRange lower upper a
>     | lower <= a && a <= upper = Right a
>     | otherwise =
>         Left (show a ++ " is not between " ++ show lower ++ " and " ++ show upper)
> oneToTenPart :: ServerPart String
> oneToTenPart =
>     do r <- getDataFn (lookRead "i" `checkRq` (inRange (1 :: Int) 10))
>        case r of
>          (Left e) ->
>              badRequest $ unlines e
>          (Right i) ->
>              ok $ "You picked: " ++ show i
> main :: IO ()
> main = simpleHTTP nullConf $ oneToTenPart

[Source code for the app is here.]

Now if we visit http://localhost:8000/?i=10, we will get the message:

 $ curl http://localhost:8000/?i=10
You picked: 10

But if we pick an out of range value http://localhost:8000/?i=113, we will get the message:

 $ curl http://localhost:8000/?i=113
113 is not between 1 and 10

Looking up optional parameters

Sometimes query parameters are optional. You may have noticed that the RqData module does not seem to provide any functions for dealing with optional values. That is because we can just use the Alternative class from Control.Applicative which provides the function optional for us:

> optional :: Alternative f => f a -> f (Maybe a)

Here is a simple example where the greeting parameter is optional:

> module Main where
> import Control.Applicative ((<$>), (<*>), optional)
> import Happstack.Server (ServerPart, look, nullConf, ok, simpleHTTP)
> helloPart :: ServerPart String
> helloPart =
>     do greet <- optional $ look "greeting"
>        ok $ (show greet)
> main :: IO ()
> main = simpleHTTP nullConf $ helloPart

[Source code for the app is here.]

If we visit http://localhost:8000/?greeting=hello, we will get Just "hello".

if we leave off the query parameters we get http://localhost:8000/, we will get Nothing.

Working with Cookies

What are Cookies?

HTTP is a stateless protocol. Each incoming Request is processed with out any memory of any previous communication with the client. Though, from using the web, you know that it certainly doesn't feel that way. A website can remember that you logged in, items in your shopping cart, etc. That functionality is implemented by using Cookies.

When the server sends a Response to the client, it can include a special Response header named Set-Cookie, which tells the client to remember a certain Cookie. A Cookie has a name, a string value, and some extra control data, such as a lifetime for the cookie.

The next time the client talks to the server, it will include a copy of the Cookie value in its Request headers. One possible use of cookies is to store a session id. When the client submits the cookie, the server can use the session id to look up information about the client and remember who they are. Sessions and session ids are not built-in to the HTTP specification. They are merely a common idiom which is provided by many web frameworks.

Simple Cookie Demo

The cookie interface is pretty small. There are two parts to the interface: setting a cookie and looking up a cookie.

To create a Cookie value, we use the mkCookie function:

> -- | create a 'Cookie'
> mkCookie  :: String -- ^ cookie name
>           -> String -- ^ cookie value
>           -> Cookie

Then we use the addCookie function to send the cookie to the user. This adds the Set-Cookie header to the Response. So the cookie will not actually be set until the Response is sent.

> -- | add the 'Cookie' to the current 'Response'
> addCookie :: (MonadIO m, FilterMonad Response m) => CookieLife -> Cookie -> m ()

The first argument of addCookie specifies how long the browser should keep the cookie around. See the cookie lifetime section for more information on CookieLife.

To lookup a cookie, we use some HasRqData functions. There are only three cookie related functions:

> -- | lookup a 'Cookie'
> lookCookie :: (Monad m, HasRqData m) =>
>               String -- ^ cookie name
>            -> m Cookie
> -- | lookup a 'Cookie' and return its value
> lookCookieValue :: (Functor m, Monad m, HasRqData m) =>
>                    String -- ^ cookie name
>                 -> m String
> -- | look up a 'Cookie' value and try to convert it using 'read'
> readCookieValue :: (Functor m, Monad m, HasRqData m, Read a) =>
>                    String -- ^ cookie name
>                 -> m a

The cookie functions work just like the other HasRqData functions. That means you can use checkRq, etc.

The following example puts all the pieces together. It uses the cookie to store a simple counter specifying how many requests have been made:

> module Main where
> import Control.Monad.Trans ( liftIO )
> import Control.Monad       ( msum, mzero )
> import Happstack.Server    ( CookieLife(Session), Request(rqPaths), ServerPart
>                            , addCookie , askRq, look, mkCookie, nullConf
>                            , ok, readCookieValue, simpleHTTP )
> homePage :: ServerPart String
> homePage =
>     msum [ do rq <- askRq
>               liftIO $ print (rqPaths rq)
>               mzero
>          , do requests <- readCookieValue "requests"
>               addCookie Session (mkCookie "requests" (show (requests + (1 :: Int))))
>               ok $ "You have made " ++ show requests ++ " requests to this site."
>          , do addCookie Session (mkCookie "requests" (show 2))
>               ok $ "This is your first request to this site."
>          ]
> main :: IO ()
> main = simpleHTTP nullConf $ homePage

[Source code for the app is here.]

Now if you visit http://localhost:8000/ you will get a message like:

This is your first request to this site.

If you hit reload you will get:

You have made 3 requests to this site.

Now wait a second! How did we go from 1 to 3, what happened to 2? The browser will send the cookie with every request it makes to the server. In this example, we ignore the request path and send a standard response to every request that is made. The browser first requests the page, but it also requests the favicon.ico for the site. So, we are really getting two requests everytime we load the page. Hence the counting by twos. It is important to note that the browser does not just send the cookie when it is expecting an html page -- it will send it when it is expecting a jpeg, a css file, a js, or anything else.

There is also a race-condition bug in this example. See the cookie issues section for more information.

Cookie Lifetime

When you set a cookie, you also specify the lifetime of that cookie. Cookies are referred to as session cookies or permanent cookies depending on how their lifetime is set.

session cookie
A cookie which expires when the browser is closed.
permanent cookie
A cookie which is saved (to disk) and is available even if the browser is restarted. The expiration time is set by the server.

The lifetime of a Cookie is specified using the CookieLife type:

> -- | the lifetime of the cookie
> data CookieLife
>   = Session          -- ^ expire when the browser is closed
>   | MaxAge Seconds   -- ^ expire after the specified number of seconds
>   | Expires UTCTime  -- ^ expire at a specific date and time
>   | Expired          -- ^ expire immediately

If you are intimately familiar with cookies, you may know that cookies have both an expires directive and a max-age directive, and wonder how they related to the constructors in CookieLife. Internet Explorer only supports the obsolete expires directive, instead of newer max-age directive. Most other browser will honor the max-age directive over expires if both are present. To make everyone happy, we always set both.

So, when setting CookieLife you can use MaxAge or Expires -- which ever is easiest, and the other directive will be calculated automatically.

Deleting a Cookie

There is no explicit Response header to delete a cookie you have already sent to the client. But, you can convince the client to delete a cookie by sending a new version of the cookie with an expiration date that as already come and gone. You can do that by using the Expired constructor. Or, you can use the more convenient, expireCookie function.

> -- | Expire the cookie immediately and set the cookie value to ""
> expireCookie :: (MonadIO m, FilterMonad Response m) => 
>                 String  -- ^ cookie name
>              -> m () 

Cookie Issues

Despite their apparently simplicity, Cookies are the source of many bugs and security issues in web applications. Here are just a few of the things you need to keep in mind.

Security issues

To get an understanding of cookie security issues you should search for cookie security issues and cookie XSS

One important thing to remember is that the user can modify the cookie. So it would be a bad idea to do, addCookie Session (mkCookie "userId" "1234") because the user could modify the cookie and change the userId at will to access other people's accounts.

Also, if you are not using https the cookie will be sent unencrypted.

Delayed Effect

When you call addCookie the Cookie will not be available until after that Response has been sent and a new Request has been received. So the following code will not work:

> do addCookie Session (mkCookie "newCookie" "newCookieValue")
>    v <- look "newCookie"
>    ...

The first time it runs, look will fail because the cookie was not set in the current Request. Subsequent times look will return the old cookie value, not the new value.

Cookie Size

Browsers impose limits on how many cookies each site can issue, and how big those cookies can be. The RFC recommends browsers accept a minimum of 20 cookies per site, and that cookies can be at least 4096 bytes in size. But, implementations may vary. Additionally, the cookies will be sent with every request to the domain. If your page has dozens of images, the cookies will be sent with every request. That can add a lot of overhead and slow down site loading times.

A common alternative is to store a small session id in the cookie, and store the remaining information on the server, indexed by the session id. Though that brings about its own set of issues.

One way to avoid having cookies sent with every image request is to host the images on a different sub-domain. You might issues the cookies to, but host images from Note that you do not actually have to run two servers in order to do that. Both domains can point to the same IP address and be handled by the same application. The app itself may not even distinguish if the requests were sent to images or www.

Server Clock Time

In order to calculate the expires date from the max-age or the max-age from the expires date, the server uses getCurrentTime. This means your system clock should be reasonably accurate. If your server is not synchronized using NTP or something similar it should be.

Cookie Updates are Not Atomic

Cookie updates are not performed in any sort of atomic manner. As a result, the simple cookie demo contains a race condition. We get the Cookie value that was included in the Request and use it to create an updated Cookie value in the Response. But remember that the server can be processing many requests in parallel and the browser can make multiple requests in parallel. If the browser, for example, requested 10 images at once, they would all have the same initial cookie value. So, even though they all updated the counter by 1, they all started from the same value and ended with the same value. The count could even go backwards depending on the order Requests are received and Responses are processed.

Other Cookie Features

The mkCookie function uses some default values for the Cookie. The Cookie type itself includes extra parameters you might want to control such as the cookie path, the secure cookie option, etc.

Next: Serving Files from Disk