abc ! @` s d d l m Z m Z m Z d d l m Z m Z d d l m Z m Z d d l
Z
d d l Z d d l m
Z
d d l m Z m Z m Z m Z d d l m Z d d l m Z d d
l m Z y d d l m Z Wn e k
r e Z n Xe g e D] Z e j d ^ q Z e g e D] Z e j d ^ q" Z e g e D] Z e j d ^ qJ Z e e d
d g BZ d Z! e j" re! d d k re! j# d d k st$ e j% e! d e& d d Z' n e j% e! Z' e( d d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 g Z) e j% d3 Z* i Z+ d4 e, f d5 YZ- d6 Z. d7 e, f d8 YZ/ d9 e/ f d: YZ0 d; e1 f d< YZ2 d= e, f d> YZ3 d? e, f d@ YZ4 dA Z5 d S(B i ( t absolute_importt divisiont unicode_literals( t text_typet binary_type( t http_clientt urllibN( t webencodingsi ( t EOFt spaceCharacterst asciiLetterst asciiUppercase( t ReparseException( t _utils( t StringIO( t BytesIOu asciit >t t | j | | k rd | t | j | 8} | d 7} q' W| | g | _ d S( Ni i ( t _bufferedBytest AssertionErrorR R R ( R R t offsett i( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt seekL s c C` sp | j s | j | S| j d t | j k r_ | j d t | j d k r_ | j | S| j | Sd S( Ni i i( R t _readStreamR R t _readFromBuffer( R t bytes( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt readU s
c C` s& t g | j D] } t | ^ q
S( N( t sumR R ( R t item( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR ^ s c C` sL | j j | } | j j | | j d c d 7 Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
u i i u utf-8u certainN(
R
t supports_lone_surrogatest Nonet reportCharacterErrorsR t characterErrorsUCS4t characterErrorsUCS2t newLinest lookupEncodingt charEncodingt
openStreamt
dataStreamt reset( R R? ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` sC d | _ d | _ d | _ g | _ d | _ d | _ d | _ d S( Nu i ( R t chunkSizet chunkOffsett errorst prevNumLinest prevNumColsRF t _bufferedCharacter( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRO s c C` s( t | d r | } n t | } | S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R: R ( R R? R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRM s c C` st | j } | j d d | } | j | } | j d d | } | d k r\ | j | } n | | d } | | f S( Nu
i ii ( R t countRS t rfindRT ( R R R t nLinest positionLinet lastLinePost positionColumn( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt _position s
c C` s& | j | j \ } } | d | f S( u: Returns (line, col) of the current position in the stream.i ( R\ RQ ( R t linet col( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` sL | j | j k r% | j s% t Sn | j } | j | } | d | _ | S( uo Read one character from the stream or queue if available. Return
EOF when EOF is reached.
i ( RQ RP t readChunkR R ( R RQ t char( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR` s
c C` sO | d k r | j } n | j | j \ | _ | _ d | _ d | _ d | _ | j j | } | j
r | j
| } d | _
n
| s t St | d k r t
| d } | d k s d | k o d k n r | d | _
| d } q n | j r| j | n | j d d } | j d
d } | | _ t | | _ t S( Nu i i ii
i i u
u
u
( RF t _defaultChunkSizeR\ RP RS RT R RQ RN R$ RU R9 R t ordRG t replacet True( R RP R( t lastv( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR_ s0
(
c C` s: x3 t t t j | D] } | j j d q Wd S( Nu invalid-codepoint( t rangeR t invalid_unicode_ret findallRR R' ( R R( t _( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRH % s "c C` s t } x t j | D] } | r( q n t | j } | j } t j | | | d ! r t j | | | d ! } | t k r | j
j d n t } q | d k r | d k r | t
| d k r | j
j d q t } | j
j d q Wd S( Ni u invalid-codepointi i i ( R9 Rg t finditerRb t groupt startR
t isSurrogatePairt surrogatePairToCodepointt non_bmp_invalid_codepointsRR R' Rd R ( R R( t skipt matcht codepointR t char_val( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRI ) s c
C` s} y t | | f } Wn t k
r x& | D] } t | d k s+ t q+ Wd j g | D] } d t | ^ qZ } | s d | } n t j d | } t | | f c B` sb e Z d Z d d d d d e d Z d Z d Z e d Z d Z d Z
d Z RS(
u Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
u windows-1252c C` s | j | | _ t j | | j d | _ d | _ | | _ | | _ | | _ | | _ | | _
| j | | _ | j d d k s t | j d S( u Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i id i N( RM t rawStreamR= R t numBytesMetat numBytesChardett override_encodingt transport_encodingt same_origin_parent_encodingt likely_encodingt default_encodingt determineEncodingRL RF R RO ( R R? R R R R R t
useChardet( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` s3 | j d j j | j d | _ t j | d S( Ni u replace( RL t
codec_infot streamreaderR RN R= RO ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRO s "c C` sU t | d r | } n t | } y | j | j Wn t | } n X| S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R: R R R R ( R R? R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRM s c C` s! | j d f } | d d k r& | St | j d f } | d d k rO | St | j d f } | d d k rx | S| j d f } | d d k r | St | j d f } | d d k r | d j j d r | St | j d f } | d d k r| S| ry d d l
m } Wn t k
r4qXg } | } x[ | j
s| j j | j } t | t szt | sPn | j | | j | qGW| j t | j d } | j j d | d k r| d f Sn t | j d f } | d d k r| St d d f S( Nu certaini u tentativeu utf-16( t UniversalDetectoru encodingu windows-1252( t detectBOMRF RK R R t detectEncodingMetaR t namet
startswithR t chardet.universaldetectorR t ImportErrort doneR R$ R R4 R# R R' t feedt closet resultR R ( R t chardetRL R t bufferst detectorR t encoding( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR sR '
c C` s | j d d k s t t | } | d k r5 d S| j d k re t d } | d k s t nr | | j d k r | j d d f | _ nF | j j d | d f | _ | j t d | j d | f d S( Ni u certainu utf-16beu utf-16leu utf-8i u Encoding changed from %s to %s( u utf-16beu utf-16le( RL R RK RF R R R RO R ( R t newEncoding( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt changeEncoding s
c C` s i d t j 6d t j 6d t j 6d t j 6d t j 6} | j j d } t | t s_ t
| j | d } d } | s | j | } d } | s | j | d } d } q n | r | j j | t
| S| j j d d
Sd
S( u Attempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Noneu utf-8u utf-16leu utf-16beu utf-32leu utf-32bei i i i N( t codecst BOM_UTF8t BOM_UTF16_LEt BOM_UTF16_BEt BOM_UTF32_LEt BOM_UTF32_BER R$ R4 R# R t getR RK RF ( R t bomDictt stringR R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s&
c C` s | j j | j } t | t s* t t | } | j j d | j } | d k r| | j
d k r| t d } n | S( u9 Report the encoding declared by the meta element
i u utf-16beu utf-16leu utf-8N( u utf-16beu utf-16le( R R$ R R4 R# R t EncodingParserR t getEncodingRF R RK ( R R t parserR ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR 9 s N( R1 R2 R3 RF Rd R RO RM R R R R ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR> s ( > "t
EncodingBytesc B` s e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
e e
e Z d Z
e e
Z e d
Z d Z d Z d
Z RS( u String-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C` s+ t | t s t t j | | j S( N( R4 R# R t __new__t lower( R t value( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR L s c C` s
d | _ d S( Ni( R\ ( R R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR P s c C` s | S( N( ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt __iter__T s c C` sS | j d } | _ | t | k r/ t n | d k rD t n | | | d !S( Ni i ( R\ R t
StopIterationR< ( R t p( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt __next__W s c C` s
| j S( N( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt next_ s c C` sY | j } | t | k r$ t n | d k r9 t n | d | _ } | | | d !S( Ni i ( R\ R R R< ( R R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt previousc s c C` s+ | j t | k r t n | | _ d S( N( R\ R R ( R R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt setPositionl s c C` s< | j t | k r t n | j d k r4 | j Sd Sd S( Ni ( R\ R R RF ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt getPositionq s
c C` s | | j | j d !S( Ni ( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt getCurrentByte{ s c C` sc | j } xJ | t | k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( u Skip past a list of charactersi N( R R R\ RF ( R R{ R R| ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRp s c C` sc | j } xJ | t | k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( Ni ( R R R\ RF ( R R{ R R| ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt skipUntil s c C` sQ | j } | | | t | !} | j | } | rM | j t | 7_ n | S( u Look for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone( R R R ( R R# R R( R, ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt
matchBytes s c C` sh | | j j | } | d k r^ | j d k r= d | _ n | j | t | d 7_ t St d S( u Look for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchii i N( R t findR\ R Rd R ( R R# t newPosition( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt jumpTo s ( R1 R2 R3 R R R R R R R R t propertyR R t currentBytet spaceCharactersBytesRp R R R ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR H s R c B` s_ e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
d Z RS(
u? Mini parser for detecting character encoding from meta elementsc C` s t | | _ d | _ d S( u3 string - the data to work on for encoding detectionN( R R( RF R ( R R( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` s d | j f d | j f d | j f d | j f d | j f d | j f f } xv | j D]k } t } xR | D]J \ } } | j j | rk y | } PWq t k
r t } Pq Xqk qk W| sX PqX qX W| j
S( Ns ( R( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` sK | j j t k r t St } d } x"t rF| j } | d k rG t S| d d k r | d d k } | rC| d k rC| | _ t Sq% | d d k r | d } t | } | d k rC| | _ t Sq% | d d k r% t t
| d } | j } | d k rCt | } | d k r@| r4| | _ t S| } q@qCq% q% Wd S( Ni s
http-equivi s content-typet charsett content( R( R R Rd R9 RF t getAttributeR RK t ContentAttrParserR t parse( R t hasPragmat pendingEncodingt attrt tentativeEncodingt codect
contentParser( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s:
c C` s
| j t S( N( t handlePossibleTagR9 ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` s t | j | j t S( N( R R( R Rd ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s
c C` s | j } | j t k r9 | r5 | j | j n t S| j t } | d k ra | j n+ | j } x | d k r | j } qp Wt S( NR (
R( R t asciiLettersBytesR R Rd R t spacesAngleBracketsR RF ( R t endTagR( R| R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s
c C` s | j j d S( NR ( R( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s c C` s | j } | j t t d g B } | d k sI t | d k sI t | d k rY d Sg } g } x t r
| d k r | r Pnz | t k r | j } Pn^ | d
k r d j | d f S| t k r | j
| j n | d k r d S| j
| t | } qh W| d k r7| j
d j | d f St | | j } | d k r| } x t rt | } | | k rt | d j | d j | f S| t k r| j
| j qb| j
| qbWn^ | d k rd j | d f S| t k r | j
| j n | d k r-d S| j
| x} t rt | } | t k rwd j | d j | f S| t k r| j
| j q=| d k rd S| j
| q=Wd S( u_ Return a name,value pair for the next attribute in the stream,
if one is found, or Nonet /i R t =R) t 't "N( R N( R R ( R R ( R( Rp R t frozensetRF R R Rd R* t asciiUppercaseBytesR' R R R R ( R R( R| t attrNamet attrValuet quoteChar( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR sh $
( R1 R2 R3 R R R R R R R R R ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR s $ R c B` s e Z d Z d Z RS( c C` s" t | t s t | | _ d S( N( R4 R# R R( ( R R( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR f s c C` s: y!| j j d | j j d 7_ | j j | j j d k sH d S| j j d 7_ | j j | j j d k r | j j } | j j d 7_ | j j } | j j | r | j | | j j !Sd SnP | j j } y( | j j t | j | | j j !SWn t k
r | j | SXWn t k
r5d SXd S( NR i R R R ( R R ( R( R R Rp R RF R R R ( R t quoteMarkt oldPosition( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR j s.
( R1 R2 R R ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR e s c C` sw t | t r: y | j d } Wq: t k
r6 d SXn | d k ro y t j | SWqs t k
rk d SXn d Sd S( u{ Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.u asciiN( R4 R t decodet UnicodeDecodeErrorRF R t lookupt AttributeError( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRK s
(6 t
__future__R R R t pip._vendor.sixR R t pip._vendor.six.movesR R R Rv t pip._vendorR t constantsR R R
R R R) R
t ioR R R R R& t encodeR R R R t invalid_unicode_no_surrogateRE RV R Rw t evalRg t setRo t ascii_punctuation_reRt t objectR RD R= R> R# R R R RK ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt sR "
((( +
J h'