Beating Download Manager Protection

by Straightface  (straightfacegangsta@excite.com)

While searching for interesting files on the net you may encounter a file that has been "Download Manager Blocked," meaning that you must use a browser to get the file.

If you attempt to download the file with a download manager, you will receive a lovely text message in place of the file you desired informing you of your "mistake."  Some may feel defeated, but with a little slight of hand you can use a download manager to retrieve the file.

The initial question we have to ask ourselves is, "How in the world does the server know whether the program making the download request is a browser or not?!?"

The answer can be found by analyzing the HTTP headers the browser sends in its request for the file.  The server attempts to protect itself from download managers by checking for particular HTTP headers.  Usually it checks the User-Agent header and can also check for a cookie or referring page header.

First we must fill our tool box with the proper tools.  We will need a packet sniffer to learn how the browser is communicating with the server.

Sniffit is a nice one for Linux   If using Windows, WinDump works well.

Be aware the WinPcap libraries are needed for WinDump to work properly and can be found on the WinDump web site.  I also employ the Windows program Dice to read the raw files WinDump creates.  We are also going to need a nice customizable download manager.  For this I choose GNU Wget.  It is available for both Linux and Windows, free, and has a very small footprint.

Once we have all the tools ready we can begin to collect the proper HTTP headers.

Start up the browser of your choice and bring it to the web page with the link of the file you want to download.  Make sure you have your cookies enabled on the browser.  Now it is time to start up our packet sniffer.  Make sure you are sniffing the right interface.  In this example the interface is: ppp0

WinDump requires you to first run it with the -D option for a list of interfaces and then you must choose the proper one.  See the documentation for full details.

Using Sniffit: sniffit -t @ -F ppp0

Using WinDump: windump -w output.cap -i 1

Now we are all set to capture the headers.

Go back to your browser and click on the proper link for the file.  Choose a place for it to reside and start the download.  Let the file download a few kilobytes, then stop it.

Now let's look at the packets we captured.

Sniffit will leave behind some files with names like 65.23.29.34.33265-208.48.67.24.80 which you can view with your favorite text editor.

When using WinDump, opening the output file with Dice will give you a list of all the packets you caught.

The packets of interest are usually the first few leaving your machine.  You can tell it is leaving as the first IP address' port number is pretty large, such as in the example file name above.

Find the HTTP request the browser sent.  It will look something like this:

GET /myDLmanagerblockedfile.avi HTTP/1.0 Connection: Keep-Alive User-Agent: Mozilla/4.78 [en] (Linux 2.4.8 i686) Host: example.com Accept: image/gif, image/jpeg, image/pjpeg, image/png, */* Accept-Encoding: gzip Accept-Language: en Accept-Charset: iso-8859-1,*,utf-8 Cookie: f908dkl=93 Referer: http://www.example.com/video5.html

Ah ha!  There are some odd HTTP headers in the request.  The two lines we want to pay attention to are the Referer and Cookie lines.  We also need to include the User-Agent header in our download manager's request.  Now we know how to emulate the browser!

Finally, lets set GNU Wget to retrieve the file.  The GNU Wget command using the above captured packets will look like this:

$ wget --user-agent='Mozilla/4.78 [en] (Linux 2.4.8 1686)' --header='Cookie: f908dkl=93' --header='Referer: http://www.example.com/video5.html' http://www.example.org/myDLmanagerblockedfile.avi

The file should begin to download properly.

If it gives you the "No Download Managers" message you might have missed another abnormal HTTP header.  You can sniff the browser's request for the file and then sniff GNU Wget request and see how they differ to find your missing header.

Simply include the missing header in your GNU Wget command with the --header option.

For serious downloading, GNU Wget has options to download a list of files, but I usually just set up a bunch of GNU Wget commands in a batch file.

Have fun with your knowledge of packet sniffing and HTTP headers!

They are great tools for your own personal toolbox...

URLs Used

Dice: www.ngthomas.co.uk/dice.htm

Sniffit: reptile.rug.ac.be/~coder/sniffit/sniffit.html

GNU Wget (Linux): www.gnu.org/software/wget/wget.html

Wget (Win32): space.tin.it/computer/hherold/

WinDump: www.winpcap.org/windump