From the BeautifulSoup website:
Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:
1. Beautiful Soup won’t choke if you give it bad markup. It yields a parse tree that makes approximately as much sense as your original document. This is usually good enough to collect the data you need and run away.
2. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. You don’t have to create a custom parser for each application.
3. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don’t have to think about encodings, unless the document doesn’t specify an encoding and Beautiful Soup can’t autodetect one. Then you just have to specify the original encoding.
Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. You can tell it “Find all the links”, or “Find all the links of class externalLink”, or “Find all the links whose urls match “foo.com”, or “Find the table heading that’s got bold text, then give me that text.”
Valuable data that was once locked up in poorly-designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup.
Getting Started
1. Install BeautifulSoup
2. Create a script that uses BeautifulSoup
from BeautifulSoup import BeautifulSoup
import re
import urllib2
# download the page
response = urllib2.urlopen("http://www.boattrader.com/search-results/Type-any/Make-mako/Length-17,25/Zip-02445/Radius-100/Sort-Length:DESC/ ")
html = response.read()
# create a beautiful soup object
soup = BeautifulSoup(html)
# all links to detailed boat information have class lfloat
links = soup.findAll("a", { "class" : "lfloat" })
for link in links:
print link['href']
print link.string
# all prices are spans and have the class rfloat
prices = soup.findAll("span", { "class" : "rfloat" })
for price in prices:
print price
print price.string
# all boat images have attribute height=105
images = soup.findAll("img",height="105")
for image in images:
print image # print the whole image tag
print image['src'] # print the url of the image only
MySQL Show or Desc(ribe) All Tables at Once
I was recently getting up to speed on a few new applications and needed to understand the database schema. I found myself repeatedly using the MySQL show tables and desc tables command line queries. After doing this a few times I got a little frustrated so I wrote this handy little shell script that will generate HTML output of an entire database schema to db_schema.html on your desktop.
Example Output - db_schema.html
This script uses the mysql command line “SHOW TABLES” and “DESC <TABLE>” commands.
For those not familiar with the “SHOW tables” command here’s what the output looks like-
mysql> show tables;
+---------------------------+
| Tables_in_mysql |
+---------------------------+
| columns_priv |
| db |
| event |
| func |
| general_log |
| help_category |
For those not familiar with the “DESC Table” command here’s what the output looks like.
mysql> desc user;
+-----------------------+-----------------------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+-----------------------------------+------+-----+---------+-------+
| Host | char(60) | NO | PRI | | |
| User | char(16) | NO | PRI | | |
| Password | char(41) | NO | | | |
| Select_priv | enum('N','Y') | NO | | N | |
This script assumes that:
if [ -f ~/Desktop/db_schema.html ]; then
rm ~/Desktop/db_schema.html
fi
for i in `mysql -B -N -uroot dbname -e ’show tables;’`; do
echo "<a href=’#$i’>$i</a><br>" >> /tmp/$$.nav.html
echo "<a name=’$i’></a>" >> /tmp/$$.tables.html
echo "<h2>$i</h2>" >> /tmp/$$.tables.html
mysql –html –column-names -uroot dbname -e "desc $i;" >> /tmp/$$.tables.html
echo -n >> /tmp/$$.tables.html
done;
cat /tmp/$$.nav.html /tmp/$$.tables.html > ~/Desktop/db_schema.html