Using wget to mirror a website

Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.

curl can probably do this too, and supports authentication, but it wasn't obvious.

So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.

Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):

wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://web_site_name_to_mirror/

where:

• -e robots=off obviously disables robots


• --wait 1 forces a pause between gets (so the site doesn't get hammered)


• --user and --password: self-evident


• -x creates a local directory structure which "mirrors" (see what I did there?) the directory structure on the site you're mirroring


• -m turns on mirror mode: "turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings" (from the man page)


• -k converts links after download so that URLs in the mirrored files reference local files

Don't use it carelessly on someone else's website, as they might get angry...

The Linux File System - Explained

[youtube=http://youtu.be/2qQTXp4rBEE]

Published on Nov 17, 2012
In Linux, everything is a file. All the files the kernel can understand are displayed.

How To Install LAMP (Linux, Apache, MySQL, PHP) On Ubuntu/Linux Mint

If you develop web applications and scripts, it will be nice testing them locally in your own computer before launching them online. This will require the installation of a webserver on your computer.LAMP (Linux, Apache, MySQL, PHP) is one of the easiest and perfect environment where you can test all your PHP codes. In this tutorial, we will help you install the LAMP webserver in the following Ubuntu/Linux Mint distributions:

• Ubuntu 13.04/12.1012.04/11.10 or older


• Linux Mint 14/13/12 or older

LAMP Installation

The LAMP webserver can be installed easily with this command (the caret (^) is required, don't exclude it):

sudo apt-get install lamp-server^

During the installation, you will be asked to enter a new root password for the MySQL database, submit it and press Enter:

You will be prompted to enter the password again for confirmation. Wait now until the installation is complete. You have now installed the LAMP webserver on Ubuntu 11.04/11.10. Let's now go to the next step.

Testing Apache

Launch your web browser (Firefox, Google Chrome, etc.) and open one of these addresses (or provide your server IP address if needed):

http://localhost/

or

http://127.0.0.1/

If you get this page, then Apache is started:

Otherwise try to restart Apache with this command:

sudo /etc/init.d/apache2 restart

Then give it another try.

Testing PHP

Let's now test PHP. You need to create an empty PHP file in /var/www  and insert this snippet of code into it:

<?php phpinfo(); ?>

You can easily do it with these two commands via the terminal:

echo "<?php phpinfo(); ?>" | sudo tee /var/www/test.php

sudo /etc/init.d/apache2 restart

Then open this address:

http://localhost/test.php

You should see a page like this:

Configuring MySQL

Since you are using the LAMP webserver locally, your MySQL database must uses the same IP address of your localhost which is: 127.0.0.1. Via the Terminal, run this command to verify it:

cat /etc/hosts | grep localhost

Here is the correct output you must get:

~$ cat /etc/hosts | grep localhost
127.0.0.1 localhost
::1     ip6-localhost ip6-loopback


Also verify that the bind address is set correctly by running this command:

cat /etc/mysql/my.cnf | grep bind-address

You should get this output:

~$ cat /etc/mysql/my.cnf | grep bind-address
bind-address = 127.0.0.1

If you get a different IP address, then edit the my.cnf file with this command:

sudo gedit /etc/mysql/my.cnf

Search for the line containing "bind-address" and correct its address by replacing it with 127.0.0.1.

phpMyAdmin Installation

If you want an easy GUI for managing your MySQL databases, you can install phpMyAdmin with this command:

sudo apt-get install libapache2-mod-auth-mysql phpmyadmin

During the installation you will be asked to select a web server that will be configured automatically to run phpMyAdmin. Select apache2 using your spacebar and press Enter:

You will be asked next to configure a database for phpmyadmin with dbconfig-common, select Yes and press Enter:

In the next screen, enter the MySQL password you have submitted before and press Enter:

[
OR,
Select  NO and press ENTER
And, Type below command in terminal

sudo cp /etc/phpmyadmin/apache.conf /etc/apache2/conf.d

restart your apache server using

sudo /etc/init.d/apache2 restart

]

Congratulation! phpMyAdmin is now installed in your system. To test it, open simply this address via your web browser:

http://localhost/phpmyadmin/

Login to phpMyAdmin using root as username and the password you created earlier:

You have now successfully installed LAMP on your system. All your projects and files must be placed in /var/www so that you can run them.

Removing LAMP & phpMyAdmin

To uninstall the LAMP web server and phpMyAdmin, open the terminal and run this command:

for pkg in `dpkg -l *apache* *mysql* phpmyadmin | grep ^ii | awk '{ print $2 }'`; do sudo apt-get -y purge --auto-remove $pkg; done;

That's it!
Enjoy!

Install WordPress 3.5 'Elvin' On Ubuntu 12.10/12.04 and Linux Mint 14/13

WordPress has been updated to version 3.5 having this codename 'Elvin' in honor of Elvin Jones (drummer). This new version comes with a new media manager and Retina support. Users of WordPress will have a new experience when uploading images and creating galleries thanks to the new image uploader.

WordPress 3.5 comes also with various tweaks and a new default theme (Twenty Twelve). You can find full features here. In this tutorial, we will see how to install this blogging and CMS platform under the following Ubuntu/Linux Mint distributions:

• Ubuntu 12.10/12.04/11.10 or older


• Linux Mint 14/13/12 or older


• Any other Ubuntu-based System

  1. Requirements

To be able to install WordPress 3.5 in Ubuntu/Linux Mint, we need to install a web server (Apache, PHP, MySQL, etc.). If you are following my previous tutorials, you will notice that I prefer to use the LAMP server, you can find here detailed instructions for installing it.

   2. Creating A MySQL Database For WordPress

For GUI mode, I recommend that you use phpMyAdmin to create a MySQL database and user for WordPress. For CLI mode, you can run this sequence of commands (change text in red to reflect your own database configuration):

## Connect to MySQL Server & Enter Password (if any or leave blank)## 
mysql -u root -p
Enter password:

## Creating New User for WordPress Database ##
CREATE USER wordpress@localhost IDENTIFIED BY "12345";

## Create New Database ##
create database wordpress;

## Grant Privileges to Database ##
GRANT ALL ON wordpress.* TO wordpress@localhost;

## FLUSH privileges ##
FLUSH PRIVILEGES;

## Exit ##
exit

The commands above will create these details:

1. Hostname: localhost


2. Database: wordpress


3. Database User: wordpress


4. Database User Password: 12345

3. WordPress 3.5 Installation

Via the terminal, download and move the WordPress folder to /var/www/ with these commands:

cd /tmp

wget -c http://wordpress.org/latest.zip

sudo unzip -q latest.zip -d /var/www/

sudo chmod 777 -R /var/www/wordpress

sudo /etc/init.d/apache2 restart

Open now this link to start the web-based installation:

      http://localhost/wordpress

NOTE: You can replace localhost with your server IP address.

In the first page, click Create a Configuration File:


Then follow setup instructions:

That’s it!
Enjoy!

Recover Grub2 after installing Windows

[Recover grub back again after if it is lost of these reasons-- windows 7/xp/vista/8 installation, uninstalled package grub from Linux system.]

So you were enjoying life using Ubuntu and then one day you had to install Windows for that one application your boss makes you use and now no more Ubuntu. When we need a dual boot system it’s always best to install Windows first then Ubuntu due to Windows stripping out the boot loader but what if you have no choice? or a crash that corrupted the boot loader. Boot-repair  to the rescue.

So what is Boot-Repair?

Its a small graphical tool used to restore access to Ubuntu and other OS’s such as Windows when disaster strikes, it has two basic options.(See Figure)  and follow steps below-

Boot-repair

•  run Ubuntu  media in live mode.


•  connect Ubuntu to Internet source


• open up a terminal and type the following:



• sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt-get update


• sudo apt-get install -y boot-repair && boot-repair 


• click "Recommended repair" and apply.  make sure you leave the "Reinstall GRUB" check-box ticked. Now reboot your system. now you can see GRUB back.