I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPError. So I catch that. But then I get a urllib2.URLError, so
I catch that too. The next day, it encounters a urllib2.HTTPError, then a
IOError, a socket.timeout, httplib.InvalidURL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

class URLError(IOError):
class HTTPError(URLError, addinfourl):
class GopherError(URLError):

This suggests that catching URLError should have caught your HTTPError,
so you might have the chronology backwards above.

E.g.:

py> class BobError(Exception): pass
...
py> class CarolError(BobError): pass
...
py> try:
...   raise CarolError
... except BobError:
...   print 'got it'
...
got it

Now,

% cat httplib.py | grep -e '^\s*class'

produces the following at one point in its output:

class HTTPException(Exception):
class NotConnected(HTTPException):
class InvalidURL(HTTPException):
class UnknownProtocol(HTTPException):
class UnknownTransferEncoding(HTTPException):
class UnimplementedFileMode(HTTPException):
class IncompleteRead(HTTPException):
class ImproperConnectionState(HTTPException):
class CannotSendRequest(ImproperConnectionState):
class CannotSendHeader(ImproperConnectionState):
class ResponseNotReady(ImproperConnectionState):
class BadStatusLine(HTTPException):

Which suggests that "try: except HTTPException:" will be specific enough
as a catchall for this module.

The following, then, should catch everything you mentioned except the
socket timeout:

try:
   whatever()
except URLError, HTTPException:
   alternative()

But it seems to me that working with the internet as you are doing is
fraught with peril anyway.

except (URLError, HTTPException):

to catch both.

Most of the time there isn't a whole lot of real "handling" going on in
an exception handler, but merely something like logging and/or reporting
it onscreen in a cleaner fashion than a traceback, then failing anyway.
  This is one reason Java does get it wrong: 95% of exceptions don't
need and shouldn't have special handling anyway.

Good code should probably have a very small set of real exception
handling cases, and one or two catchalls at a higher level to avoid
barfing a traceback at the user.

try:
    process_things()
except ExpectedErrors:
    recover_from_error_gracefully()
except ErrorsThatCantHappen:
    print "Congratulations! You have found a program bug!"
    print "For a $327.68 reward, please send the following " \
    "traceback to Professor Donald Knuth."
    raise
except:
    print "An unexpected error occurred."
    print "This probably means the Internet is broken."
    print "If the bug still occurs after fixing the Internet, " \
    "it may be a program bug."
    log_error()
    sys.exit()

The way to deal with it is to add another except clause to deal with the
KeyboardInterrupt, or to have recover_from_error_gracefully() deal with
it. The design pattern still works. I don't know if it has a fancy name,
but it is easy to describe:-

catch specific known errors that you can recover from, and recover from
them whatever way you like (including, possibly, re-raising the exception
and letting higher-level code deal with it);

then catch errors that cannot possibly happen unless there is a bug,
and treat them as a bug;

and lastly catch unexpected errors that you don't know how to handle and
die gracefully.

My code wasn't meant as production level code, nor was ExpectedErrors
meant as an exhaustive list. I thought that was too obvious to need
commenting on.

Oh, in case this also wasn't obvious, Donald Knuth won't really pay
$327.68 for bugs in your Python code. He only pays for bugs in his own
code. *wink*

I put up a proposal for adding an AsynchronousException class to
contain all of these types of exceptions, so you can check for that.

Reraise = (LookupError, ArithmeticError, AssertionError) # And then some

try:
    process_things()
except Reraise:
    raise
except:
    log_error()

CLASSES
    exceptions.Exception
        HTTPException
            BadStatusLine
            ImproperConnectionState
                CannotSendHeader
                CannotSendRequest
                ResponseNotReady
            IncompleteRead
            InvalidURL                             <-------  HERE
            NotConnected
            UnimplementedFileMode
            UnknownProtocol
            UnknownTransferEncoding
        HTTPException
            BadStatusLine
            ImproperConnectionState
                CannotSendHeader
                CannotSendRequest
                ResponseNotReady
            IncompleteRead
            InvalidURL
            NotConnected
            UnimplementedFileMode
            UnknownProtocol
            UnknownTransferEncoding
    HTTP
    HTTPConnection
        HTTPSConnection
    HTTPResponse

(Yes, it appears twice, don't ask me why...)

--
Jorge Godoy      <go...@ieee.org>

--------------------------------------------------
#!/usr/bin/env python                                                                              

import socket

try:
    x = []
    y = x[42]
except StandardError, foo:
    print "Caught a StandardError: ", foo
except Exception, foo:
    print "Caught something else: ", foo

try:
    socket.socket (9999)
except StandardError, foo:
    print "Caught a StandardError: ", foo
except Exception, foo:
    print "Caught something else: ", foo

try:
    raise "I'm a string pretending to be an exception"
except StandardError, foo:
    print "Caught a StandardError: ", foo
except Exception, foo:
    print "Caught something else: ", foo
--------------------------------------------------

What I meant was, I'm calling robotparser, and there's no mention of
exceptions on http://docs.python.org/lib/module-robotparser.html

"""
The class hierarchy for built-in exceptions is:

    Exception
     +-- StandardError
     |    +-- KeyboardInterrupt
     |    +-- ImportError
     |    +-- EnvironmentError
     |    |    +-- IOError
"""
http://www.python.org/doc/current/lib/module-exceptions.html

try:
   whatever()
except EnvironmentError:
   ...
except StandardError:
   ...
except Exception:
   ...

or something like that.

I do agree with you that there is some value in Java's "must catch or
re-export all exceptions" semantics, and this would be one of those places
where it would be useful.  In general, however, I've always found it to be
a major pain in the butt, to the point where I sometimes just punt and
declare all my methods to "throw Exception" (or whatever the correct syntax
is).  Not to mention that with a dynamic language like Python, it's
probably impossible to implement.