YouTUBE Caching using Squid as a Transparent Proxy server

Umair Junaid

In this tutorial we will be creating a youtube caching server using squid and apache server

We will be using squid proxy server and apache php web server to redirect and store youtubevideos.
We will need

Server Machine with two NIC’s

eth0 (192.168.1.111, 255.255.255.0, gw 192.168.1.1) (eth0 WAN interface )

eth1 (192.168.0.254, 255.255.255.0) ( eth1 LAN interface for local clients)
Centos base OS installation ver 5

 

LAB Setup

STEP 1 (SQUID SERVER) eth0 192.168.1.111
Squid version used
squid-3.1.8-1.el5.i386.rpm (at the time of writing it is the latest rpm it can be downloaded from squid website I have not tested on older versions and cannot say they will work or not).
wget http://people.redhat.com/jskala/squid/squid-3.1.8-1.el5/x86_64/squid-3.1.8-1.el5.x86_64.rpm

rpm -ivh squid-3.1.8-1.el5.x86_64.rpm

Step 2 (Apache Server with PHP support) eth0 192.168.1.111

NOTE: PREFRABBLY APACHE AND SQUID CAN BE ON DIFFERENT MACHINES THEN YOU HAVE TO CHANGE THE REDIRECTION IP IN PER.PHP TO THE IP OF APACHE SERVER MACHINE.

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
yum install mysql mysql-bench mysql-server mysql-devel mysqlclient10 php-mysql httpd gcc pcre-devel php-gd gd mod_ssl glib2-devel gcc-c++ libpcap-devel php php-pear
Note: we will not use mysql directly but some of the libraries will be required for the program so that why we are installing these.

chkconfig httpd on
service httpd start
STEP 3 SQUID.CONF
Then replace squid.conf with the attached squid.conf
add place phpredir.php   in /etc/squid/  with permissions 770 owner root and group squid =  phpredir.php

STEP 4 APACHE ADDITIONS
Then add per.php in /var/www/html/
add a videos folder in /var/www/html       with permissions      drwxrwxrwx 2 root root  videos
at the time youtube caching is being done  only
i am still looking to make metacafe and blip.tv to work and some other after that if possible

STEP 5 WIN CLIENT SETTINGS

User Client Settings

Ip address    192.168.0.___

Netmask       255.255.255.0

Gateway       192.168.0.254

 

For Now as Youtube caching is done with the help of apache server

Squid access logs will show TCP_MISS when accessing cached videos but don’t worry just ignore this warning

You will be able to verify the youtube caching is working properly by replaying already played video in the client’s browser.

And no functionality exists for deleting videos that are not accessed often.

If someone can guide in this regard it will be very helpful indeed.

 

 

 

per.php

 

<?php$file_path=”/var/www/html/videos”;$logfile=”$file_path/cache.log”;$url=urldecode($_GET[‘url’]);$urlptr=fopen($_GET[‘url’],”r”);$blocksize=32*1024;//attempt to get. a 404 shouldn’t happen, but…if($urlptr===FALSE){

header(“Status: 404 Not Found”);

die();

}

//echo $_GET[‘url’];

//find content type and length

foreach($http_response_header as $line){

//echo $line;

if(substr_compare($line,’Content-Type’,0,12,true)==0)

$content_type=$line;

else if(substr_compare($line,’Content-Length’,0,14,true)==0){

$content_length=$line;

}

}

/**Youtube will detect if requests are coming form the wrong ip (ie, if only video requests are redirected, so, we must redirect all requests to youtube.

As such, we must capture all requests t youtube. Most are unimportant, so we can pass them straight through **/

if(!preg_match(“@.*youtube.*videoplayback.*@”,$url)){

fpassthru($urlptr);

fclose($urlptr);

exit(0);

}

//send content type and length

header($content_type);

header($content_length);

//find youtube id;

$url_exploded=explode(‘&’,$url);

$id=””;

foreach($url_exploded as $line){

if(substr($line,0,3)===’id=’)

$id=substr($line,3);

}

//Get the supposed file size

$length=intval(substr($content_length,16));

file_put_contents($logfile,”\nFound id=$id, content-type: $content_type content-length=$content_length\n”,FILE_APPEND);

//Do we have it? delivar if we do

$fname=”$file_path/$id-$length”;

//Check if we have the file, and it is the correct size. incorrect size implies corruption

//print ($fname);

//echo $fname;

//echo $length;

//echo filesize($fname);

 

 

if(file_exists($fname) &&filesize($fname)==$length){

readfile($fname);

//  echo $fname;

logdata(“HIT”,$url,$fname);

exit(0);

}

//file not in cache? Get it, send it & save it

logdata(“What the hell”,$url,$fname);

$fileptr=fopen($fname,”w”);

//no validity check, simply don’t write the file if we can’t open it. prevents noticeable failure/

while(!feof($urlptr)){

$line=fread($urlptr,$blocksize);

echo $line;

if($fileptr) fwrite($fileptr,$line);

}

fclose($urlptr);

if($fileptr) fclose($fileptr);

function logdata($type,$what, $fname){

$file_path=”/var/www/html/videos”;

$logfile=”$file_path/cache.log”;

$line=”@ “.time().”Cache $type url: $what file: $fname client:”.$_SERVER[‘REMOTE_ADDR’].”\n”;

file_put_contents($logfile,$line,FILE_APPEND);

}

?>

 

 

phpredir.php

#!/usr/bin/php<?phpwhile ( $input = fgets(STDIN) ) {// Split the output (space delimited) from squid into an array.$input=explode(” “,$input);if(preg_match(“@youtube@”,$input[0])){$input[0]=urlencode($input[0]);

$input= implode(” “,$input);

echo “http://192.168.1.111/per.php?url=$input”; //URL of my web server  // change this ip as per apache server

}else

echo “”; // empty line means no re-write by Squid.

}

 

squid.conf

## Recommended minimum configuration:#acl manager proto cache_objectacl localhost src 127.0.0.1/32 ::1acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1# Example rule allowing access from your local networks.

# Adapt to list your (internal) IP networks from where browsing

# should be allowed

acl localnet src 10.0.0.0/8     # RFC1918 possible internal network

acl localnet src 172.16.0.0/12  # RFC1918 possible internal network

acl localnet src 192.168.0.0/24# my internal networks

acl localnet src fc00::/7       # RFC 4193 local private network range

acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

 

acl SSL_ports port 443

acl Safe_ports port 80          # http

acl Safe_ports port 21          # ftp

acl Safe_ports port 443         # https

acl Safe_ports port 70          # gopher

acl Safe_ports port 210         # wais

acl Safe_ports port 1025-65535  # unregistered ports

acl Safe_ports port 280         # http-mgmt

acl Safe_ports port 488         # gss-http

acl Safe_ports port 591         # filemaker

acl Safe_ports port 777         # multiling http

acl CONNECT method CONNECT

visible_hostname squidpxy.umair.com

#

# Recommended minimum Access Permission configuration:

#

# Only allow cachemgr access from localhost

 

http_access allow manager localhost

#http_access deny manager

 

# Deny requests to certain unsafe ports

http_access deny !Safe_ports

 

# Deny CONNECT to other than secure SSL ports

http_access deny CONNECT !SSL_ports

 

# We strongly recommend the following be uncommented to protect innocent

# web applications running on the proxy server who think the only

# one who can access services on “localhost” is a local user

#http_access deny to_localhost

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

#

 

# Example rule allowing access from your local networks.

# Adapt localnet in the ACL section to list your (internal) IP networks

# from where browsing should be allowed

http_access allow localnet

http_access allow localhost

 

# And finally deny all other access to this proxy

http_access deny all

 

# Squid normally listens to port 3128

http_port 3128 transparent

 

acl youtube dstdomain .youtube.com .googlevideo.com

cache deny youtube

acl youtubeip dst 74.125.15.0/24

cache deny youtubeip

acl youtubeip2 dst 74.125.236.0/24

cache deny youtubeip2

 

maximum_object_size 13107200 KB

maximum_object_size_in_memory 102400 KB

cache_mem 100 MB

 

# pass requests

url_rewrite_program /etc/squid/phpredir.php

url_rewrite_access allow youtube

 

# leave caching up to the local web server

cache deny youtube

cache deny youtubeip

cache deny youtubeip2

 

# We recommend you to use at least the following line.

hierarchy_stoplist cgi-bin ?

 

# Uncomment and adjust the following to add a disk cache directory.

#cache_dir ufs /var/spool/squid 100 16 256

 

# Leave coredumps in the first cache dir

coredump_dir /var/spool/squid

 

# Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:     144020%  10080

refresh_pattern ^gopher:   14400%   1440

refresh_pattern -i (/cgi-bin/|\?) 0  0%   0

refresh_pattern .          0    20%  4320

 

STEP 6 IPTABLES RULES

 

Configure the iptables as shown below.

And it will make your server act as a router for clients.

Add the following rules in vi /etc/rc.local to make squid act as transparent proxy server on startup.

 

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADEiptables -A FORWARD  -i eth1 -j ACCEPTecho 1 > /proc/sys/net/ipv4/ip_forwardiptables -t nat -A PREROUTING -i  eth1 -p tcp –dport 80 -j REDIRECT –to-port 3128

 

 

You can always contact me on umairjunaid137@gmail.com

This tutorial was inspired and taken from http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube

Recommended For You.

https://www.youtube.com/watch?v=ojZT1sKCGQs  

84 Replies to “High Availability Kubernetes on Bare Metal [A] – Muhammad Kamran Azeem & Henrik Høegh, Praqma”

  1. Hello there, just became aware of your blog through Google, and found that it
    is really informative. I’m gonna watch out for brussels. I will be grateful if
    you continue this in future. Lots of people
    will be benefited from your writing. Cheers!

  2. When I initially commented I clicked the “Notify me when new comments are added” checkbox and
    now each time a comment is added I get several e-mails with
    the same comment. Is there any way you can remove me from that
    service? Thanks a lot!

  3. facebook find love –
    Hmm is anyone else encountering problems with the images on this blog loading?

    I’m trying to figure out if its a problem on my end or if it’s the blog.
    Any responses would be greatly appreciated.

  4. We are a group of volunteers and opening a new scheme in our community.
    Your site offered us with helpful info to work on. You’ve performed an impressive process and our whole neighborhood can be
    thankful to you.

  5. When I initially left a comment I appear to have clicked the -Notify
    me when new comments are added- checkbox and now whenever a comment is added
    I get 4 emails with the exact same comment. Perhaps there
    is an easy method you are able to remove me from that service?
    Kudos!

  6. Aw, this was a very nice post. Taking a few minutes and actual effort to generate a really good article…
    but what can I say… I procrastinate a whole lot and never seem to get anything done.

  7. Many people notice this to get more relaxing than taking dance lessons on the own. The privilege to download other stuff like games, music,
    and TV episode shows, all-inclusive within the
    one-time subscription fees. In case of outdoor photography one must choose the right time of day to adopt
    a snap.

  8. Oh my goodness! Incredible article dude! Many thanks, However I am
    experiencing troubles with your RSS. I don’t understand the reason why I can’t join it.
    Is there anyone else having identical RSS problems? Anyone that knows the solution will you kindly respond?
    Thanx!!

  9. I really adore your website and find most of your blog posts to be exactly what I’m interested in. Do you offer guest writers to write content for you? I would not mind creating a post about how to watch free movies online or even on most of the topics you write about on this site. Cool site!

  10. I want to express my admiration for your kindness supporting visitors who really want assistance with this concern. Your very own commitment to getting the message up and down ended up being really beneficial and have usually enabled women just like me to reach their desired goals. This insightful guideline indicates a whole lot a person like me and a whole lot more to my peers. Thanks a lot; from each one of us.

  11. What i do not realize is actually how you’re not really much more well-liked than you might be right now. You’re very intelligent. You realize thus considerably relating to this subject, produced me personally consider it from a lot of varied angles. Its like women and men aren’t fascinated unless it is one thing to accomplish with Lady gaga! Your own stuffs excellent. Always maintain it up!

  12. I do not even understand how I stopped up right here, but I thought this publish used
    to be great. I don’t know who you’re however
    definitely you are going to a well-known blogger if you happen to are not already.
    Cheers!

  13. Hello There. I discovered your blog the usage of msn. That is an extremely neatly written article.
    I will be sure to bookmark it and return to read
    more of your helpful info. Thank you for the post.
    I will definitely return.

  14. It’s a shame you don’t have a donate button! I’d certainly donate to this brilliant blog!
    I suppose for now i’ll settle for book-marking and adding your RSS feed to
    my Google account. I look forward to brand new updates and will share this blog with my Facebook group.
    Talk soon!

  15. That image is associated with a certain historic period of piracy
    that has caught hold of the imagination. There is even a greater probability that these have
    been formerly exhibited inside the malls and however remain unknown to the public.
    However, some of these proxy sites are a bit inconsistent and
    often cause your net speed to go down a bit.

  16. I came over here via another web address on dental bridge cost and imagined I might as well consider this. I like what I see therefore now I”m following you. Getting excited about checking out your website again.

  17. I am seriously enjoying the theme of your internet site. Do you ever run into any kind of web browser compatibility troubles? A few of the website readers have lamented about my porcelain veneers website not operating effectively in Internet Explorer but seems very good in Chrome. Are there any tips to assist repair that problem?

  18. Hello, you are definitely correct. I frequently read through your site content carefully. I’m likewise focused on emergency dentist, maybe you might discuss that at times. I’ll be back soon!

  19. I know this if off topic but I’m looking into starting
    my own weblog and was curious what all is needed to get set up?
    I’m assuming having a blog like yours would cost a pretty penny?

    I’m not very web savvy so I’m not 100% certain. Any suggestions or advice would be
    greatly appreciated. Kudos

  20. It’s really a nice and useful piece of info.

    I am satisfied that you just shared this useful info with us.
    Please keep us up to date like this. Thank you for sharing.

  21. Great beat ! I wish to apprentice whilst you amend your site, how could
    i subscribe for a weblog web site? The account aided me a
    applicable deal. I were tiny bit acquainted of this your broadcast provided brilliant transparent concept

  22. Very nice post. I just stumbled upon your weblog and wished to say that I’ve truly enjoyed browsing your blog posts. In any case I will be subscribing to your feed and I hope you write again very soon!

  23. Hi there! Someone in my Myspace group shared this site with us so I came to give it a look. I’m definitely loving the information. I’m book-marking and will be tweeting this to my followers! Superb blog and excellent design and style.

  24. Hello, Neat post. There is an issue together with your web site in internet explorer, may
    test this? IE still is the market chief and a large portion of people will leave out your excellent writing because of this
    problem.

  25. I’m truly loving the theme/design of your information site. Do you ever come across any kind of internet browser interface issues? A lot of my website audience have complained regarding my how to watch movies online website not operating properly in Explorer but seems excellent in Safari. Do you have any kind of solutions to aid fix this matter?

  26. I’m definitely loving the theme/design of your site. Do you come across any kind of browser interface troubles? Some of the website visitors have complained regarding my free movie website not working the right way in Explorer yet seems awesome in Safari. Do you have any recommendations to aid repair the issue?

  27. Simply wish to say your article is as astonishing.
    The clarity to your submit is just cool and i could
    suppose you’re an expert on this subject. Well along
    with your permission let me to grab your RSS feed to stay up to date
    with impending post. Thanks one million and please carry on the enjoyable work.

  28. Oh my goodness! Incredible article dude! Thank you,
    However I am going through problems with your RSS. I don’t know why I cannot join it.

    Is there anyone else having the same RSS problems?
    Anyone who knows the answer can you kindly respond?
    Thanks!!

  29. I don’t even know how I ended up here, but I thought this post was great.
    I don’t know who you are but definitely you are going to a famous blogger
    if you are not already 😉 Cheers!

  30. Hi there, I’m really thrilled I discovered your blog page, I basically encountered you by mistake, while I was browsing on Bing for mesothelioma law cases. Anyways I’m here now and would really enjoy to say kudos for a tremendous write-up and the all-round fun site (I also enjoy the design), I don’t have time to go through it entirely at the moment but I have book-marked it and moreover added in your RSS feed, so whenever I have sufficient time I will be back to look over more. Make sure you do keep up the amazing work.

  31. I came over here from some other page on the subject of free mesothelioma advice and considered I should look at this. I enjoy the things I see thus I am just following you. Getting excited about checking out the blog again.

  32. Fantastic website you have here but I was curious
    about if you knew of any forums that cover the same topics talked about here?
    I’d really love to be a part of community where I can get opinions from other experienced individuals that share the same interest.
    If you have any suggestions, please let me know.
    Cheers!

  33. Hey! Quick question that’s totally off topic.
    Do you know how to make your site mobile
    friendly? My blog looks weird when viewing from my iphone4.
    I’m trying to find a theme or plugin that might be
    able to resolve this problem. If you have any recommendations,
    please share. Thanks!

  34. I’ve tried restarting my telephone, uninstalling/reinstalling Battle Royale, setting my Play Retailer to my
    private e-mail, and changing my google play sign-in on CRoy to my non-public gmail, so it’s undoubtedly set onto it.
    Although it doesn’t have quite a bit in the sense of should – see monuments, should you dig deeper you’ll
    really discover some hidden (and not so hidden) gems away from
    the everyday touristy spots.

  35. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you can do with some pics to drive the message home a bit, but instead of that, this is fantastic blog. A great read. I’ll certainly be back.

  36. I don’t know whether it’s just me or if perhaps everyone else experiencing issues with your blog.
    It looks like some of the text in your content are running off the screen. Can someone
    else please provide feedback and let me know if this is
    happening to them too? This might be a problem with my internet
    browser because I’ve had this happen previously. Thank you

  37. I like the helpful information you provide in your articles. I will bookmark your blog and check again here frequently. I’m quite certain I will learn many new stuff right here! Best of luck for the next!

  38. Hello! I know this is kinda off topic but I was wondering if you knew where I could find a captcha plugin for my comment form? I’m using the same blog platform as yours and I’m having problems finding one? Thanks a lot!

  39. This design is incredible! You certainly know how to keep a reader entertained. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job. I really loved what you had to say, and more than that, how you presented it. Too cool!

  40. Hey there, I think your site might be having browser compatibility issues. When I look at your blog site in Safari, it looks fine but when opening in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up! Other then that, amazing blog!

  41. Hello, Neat post. There’s an issue along with your website in web explorer, might test this… IE nonetheless is the market chief and a good part of people will leave out your excellent writing because of this problem.

  42. I was curious if you ever considered changing the page layout of your site?
    Its very well written; I love what youve got to say. But maybe you could a little more
    in the way of content so people could connect with
    it better. Youve got an awful lot of text for only having
    one or 2 images. Maybe you could space it out better?

  43. I’m extremely pleased to discover this great site. I want to to thank
    you for your time for this wonderful read!! I definitely savored every
    part of it and I have you bookmarked to check out new stuff in your website.

  44. You can certainly see your expertise in the article you write.
    The sector hopes for even more passionate writers such as you who aren’t afraid to mention how they believe.
    All the time go after your heart.

  45. Hello There. I found your weblog using msn. That is a
    really smartly written article. I’ll make sure to bookmark it and come back
    to read extra of your useful info. Thanks for the post.
    I’ll certainly comeback.

  46. You’re so interesting! I don’t suppose I have read through a single thing like that
    before. So nice to find somebody with a few unique thoughts on this subject.
    Really.. thanks for starting this up. This website is one
    thing that is needed on the internet, someone with
    some originality!

  47. I’d like to thank you for the efforts you’ve put in penning
    this website. I am hoping to see the same high-grade content by you later on as well.
    In fact, your creative writing abilities has motivated me to get my own site now 😉

  48. What i do not realize is in reality how you are
    not actually a lot more smartly-liked than you might be now.
    You’re so intelligent. You realize thus significantly on the subject of this topic,
    produced me in my view believe it from so many various angles.
    Its like men and women are not involved unless it is one thing to
    accomplish with Woman gaga! Your individual stuffs outstanding.
    Always deal with it up!

  49. Hey! I’ve been following your website for a while now and
    eventually got the nerve to go ahead and give you a shoutout from the
    USA! Just wanted to say carry on with the fantastic
    work!

  50. I’m amazed, I must say. Seldom do I come across a blog that’s equally educative and entertaining,
    and let me tell you, you have hit the nail on the head.
    The issue is an issue that too few folks are speaking intelligently about.
    I am very happy I came across this in my hunt for something concerning this.

  51. I don’t even know how I ended up here, but I thought this post
    was great. I don’t know who you are but certainly you are going to a famous blogger
    if you aren’t already 😉 Cheers!

  52. I have been surfing on-line more than three hours these days, but I never found any attention-grabbing article like yours.
    It is lovely price sufficient for me. In my opinion, if all site owners
    and bloggers made just right content material as you did,
    the internet can be much more helpful than ever before.

Leave a Reply

Your email address will not be published.