YOUTUBE CACHING USING SQUID IN TRANSPARENT MODE

Learn | Teach Open Source Technologies

YOUTUBE CACHING USING SQUID IN TRANSPARENT MODE

YouTUBE Caching using Squid as a Transparent Proxy server

Umair Junaid

In this tutorial we will be creating a youtube caching server using squid and apache server

We will be using squid proxy server and apache php web server to redirect and store youtubevideos.
We will need

Server Machine with two NIC’s

eth0 (192.168.1.111, 255.255.255.0, gw 192.168.1.1) (eth0 WAN interface )

eth1 (192.168.0.254, 255.255.255.0) ( eth1 LAN interface for local clients)
Centos base OS installation ver 5

 

LAB Setup

STEP 1 (SQUID SERVER) eth0 192.168.1.111
Squid version used
squid-3.1.8-1.el5.i386.rpm (at the time of writing it is the latest rpm it can be downloaded from squid website I have not tested on older versions and cannot say they will work or not).
wget http://people.redhat.com/jskala/squid/squid-3.1.8-1.el5/x86_64/squid-3.1.8-1.el5.x86_64.rpm

rpm -ivh squid-3.1.8-1.el5.x86_64.rpm

Step 2 (Apache Server with PHP support) eth0 192.168.1.111

NOTE: PREFRABBLY APACHE AND SQUID CAN BE ON DIFFERENT MACHINES THEN YOU HAVE TO CHANGE THE REDIRECTION IP IN PER.PHP TO THE IP OF APACHE SERVER MACHINE.

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
yum install mysql mysql-bench mysql-server mysql-devel mysqlclient10 php-mysql httpd gcc pcre-devel php-gd gd mod_ssl glib2-devel gcc-c++ libpcap-devel php php-pear
Note: we will not use mysql directly but some of the libraries will be required for the program so that why we are installing these.

chkconfig httpd on
service httpd start
STEP 3 SQUID.CONF
Then replace squid.conf with the attached squid.conf
add place phpredir.php   in /etc/squid/  with permissions 770 owner root and group squid =  phpredir.php

STEP 4 APACHE ADDITIONS
Then add per.php in /var/www/html/
add a videos folder in /var/www/html       with permissions      drwxrwxrwx 2 root root  videos
at the time youtube caching is being done  only
i am still looking to make metacafe and blip.tv to work and some other after that if possible

STEP 5 WIN CLIENT SETTINGS

User Client Settings

Ip address    192.168.0.___

Netmask       255.255.255.0

Gateway       192.168.0.254

 

For Now as Youtube caching is done with the help of apache server

Squid access logs will show TCP_MISS when accessing cached videos but don’t worry just ignore this warning

You will be able to verify the youtube caching is working properly by replaying already played video in the client’s browser.

And no functionality exists for deleting videos that are not accessed often.

If someone can guide in this regard it will be very helpful indeed.

 

 

 

per.php

 

<?php$file_path=”/var/www/html/videos”;$logfile=”$file_path/cache.log”;$url=urldecode($_GET[‘url’]);$urlptr=fopen($_GET[‘url’],”r”);$blocksize=32*1024;//attempt to get. a 404 shouldn’t happen, but…if($urlptr===FALSE){

header(“Status: 404 Not Found”);

die();

}

//echo $_GET[‘url’];

//find content type and length

foreach($http_response_header as $line){

//echo $line;

if(substr_compare($line,’Content-Type’,0,12,true)==0)

$content_type=$line;

else if(substr_compare($line,’Content-Length’,0,14,true)==0){

$content_length=$line;

}

}

/**Youtube will detect if requests are coming form the wrong ip (ie, if only video requests are redirected, so, we must redirect all requests to youtube.

As such, we must capture all requests t youtube. Most are unimportant, so we can pass them straight through **/

if(!preg_match(“@.*youtube.*videoplayback.*@”,$url)){

fpassthru($urlptr);

fclose($urlptr);

exit(0);

}

//send content type and length

header($content_type);

header($content_length);

//find youtube id;

$url_exploded=explode(‘&’,$url);

$id=””;

foreach($url_exploded as $line){

if(substr($line,0,3)===’id=’)

$id=substr($line,3);

}

//Get the supposed file size

$length=intval(substr($content_length,16));

file_put_contents($logfile,”\nFound id=$id, content-type: $content_type content-length=$content_length\n”,FILE_APPEND);

//Do we have it? delivar if we do

$fname=”$file_path/$id-$length”;

//Check if we have the file, and it is the correct size. incorrect size implies corruption

//print ($fname);

//echo $fname;

//echo $length;

//echo filesize($fname);

 

 

if(file_exists($fname) &&filesize($fname)==$length){

readfile($fname);

//  echo $fname;

logdata(“HIT”,$url,$fname);

exit(0);

}

//file not in cache? Get it, send it & save it

logdata(“What the hell”,$url,$fname);

$fileptr=fopen($fname,”w”);

//no validity check, simply don’t write the file if we can’t open it. prevents noticeable failure/

while(!feof($urlptr)){

$line=fread($urlptr,$blocksize);

echo $line;

if($fileptr) fwrite($fileptr,$line);

}

fclose($urlptr);

if($fileptr) fclose($fileptr);

function logdata($type,$what, $fname){

$file_path=”/var/www/html/videos”;

$logfile=”$file_path/cache.log”;

$line=”@ “.time().”Cache $type url: $what file: $fname client:”.$_SERVER[‘REMOTE_ADDR’].”\n”;

file_put_contents($logfile,$line,FILE_APPEND);

}

?>

 

 

phpredir.php

#!/usr/bin/php<?phpwhile ( $input = fgets(STDIN) ) {// Split the output (space delimited) from squid into an array.$input=explode(” “,$input);if(preg_match(“@youtube@”,$input[0])){$input[0]=urlencode($input[0]);

$input= implode(” “,$input);

echo “http://192.168.1.111/per.php?url=$input”; //URL of my web server  // change this ip as per apache server

}else

echo “”; // empty line means no re-write by Squid.

}

 

squid.conf

## Recommended minimum configuration:#acl manager proto cache_objectacl localhost src 127.0.0.1/32 ::1acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1# Example rule allowing access from your local networks.

# Adapt to list your (internal) IP networks from where browsing

# should be allowed

acl localnet src 10.0.0.0/8     # RFC1918 possible internal network

acl localnet src 172.16.0.0/12  # RFC1918 possible internal network

acl localnet src 192.168.0.0/24# my internal networks

acl localnet src fc00::/7       # RFC 4193 local private network range

acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

 

acl SSL_ports port 443

acl Safe_ports port 80          # http

acl Safe_ports port 21          # ftp

acl Safe_ports port 443         # https

acl Safe_ports port 70          # gopher

acl Safe_ports port 210         # wais

acl Safe_ports port 1025-65535  # unregistered ports

acl Safe_ports port 280         # http-mgmt

acl Safe_ports port 488         # gss-http

acl Safe_ports port 591         # filemaker

acl Safe_ports port 777         # multiling http

acl CONNECT method CONNECT

visible_hostname squidpxy.umair.com

#

# Recommended minimum Access Permission configuration:

#

# Only allow cachemgr access from localhost

 

http_access allow manager localhost

#http_access deny manager

 

# Deny requests to certain unsafe ports

http_access deny !Safe_ports

 

# Deny CONNECT to other than secure SSL ports

http_access deny CONNECT !SSL_ports

 

# We strongly recommend the following be uncommented to protect innocent

# web applications running on the proxy server who think the only

# one who can access services on “localhost” is a local user

#http_access deny to_localhost

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

#

 

# Example rule allowing access from your local networks.

# Adapt localnet in the ACL section to list your (internal) IP networks

# from where browsing should be allowed

http_access allow localnet

http_access allow localhost

 

# And finally deny all other access to this proxy

http_access deny all

 

# Squid normally listens to port 3128

http_port 3128 transparent

 

acl youtube dstdomain .youtube.com .googlevideo.com

cache deny youtube

acl youtubeip dst 74.125.15.0/24

cache deny youtubeip

acl youtubeip2 dst 74.125.236.0/24

cache deny youtubeip2

 

maximum_object_size 13107200 KB

maximum_object_size_in_memory 102400 KB

cache_mem 100 MB

 

# pass requests

url_rewrite_program /etc/squid/phpredir.php

url_rewrite_access allow youtube

 

# leave caching up to the local web server

cache deny youtube

cache deny youtubeip

cache deny youtubeip2

 

# We recommend you to use at least the following line.

hierarchy_stoplist cgi-bin ?

 

# Uncomment and adjust the following to add a disk cache directory.

#cache_dir ufs /var/spool/squid 100 16 256

 

# Leave coredumps in the first cache dir

coredump_dir /var/spool/squid

 

# Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:     144020%  10080

refresh_pattern ^gopher:   14400%   1440

refresh_pattern -i (/cgi-bin/|\?) 0  0%   0

refresh_pattern .          0    20%  4320

 

STEP 6 IPTABLES RULES

 

Configure the iptables as shown below.

And it will make your server act as a router for clients.

Add the following rules in vi /etc/rc.local to make squid act as transparent proxy server on startup.

 

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADEiptables -A FORWARD  -i eth1 -j ACCEPTecho 1 > /proc/sys/net/ipv4/ip_forwardiptables -t nat -A PREROUTING -i  eth1 -p tcp –dport 80 -j REDIRECT –to-port 3128

 

 

You can always contact me on [email protected]

This tutorial was inspired and taken from http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube