3
|`hGt @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl m Z ddl
mZ ddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZm Z m Z m Z m Z m Z m!Z!m"Z" dd l#m$Z$ dd
l%m&Z& dd l'm(Z(m)Z)m*Z*m+Z+ djZ,ej- Z.dddZ/ej0dkr2dd Z1dd Zdd Z2dd Z3dkdd Z4d d Z5d d Z6d!d" Z7d#d$ Z8d%d& Z9d'd( Z:dld)d*Z;d+d, Zd1d2 Z?d3d4 Z@d5d6 ZAd7d8 ZBd9d: ZCeDdmZEd=d> ZFd?d@ ZGdAdB ZHdCdD ZIdEdF ZJdGdH ZKejLdIdJ ZMdKdL ZNdndMdNZOdOdP ZPdodRdSZQdTdU ZRdVdW ZSdXjTdYZUeUdZ ZVeUd[ ZWd\d] ZXd^d_ ZYd`da ZZej[dbZ\ej[dcZ]ddde Z^dfdg Z_dhdi Z`dS )pz
requests.utils
~~~~~~~~~~~~~~
This module provides utility functions that are used within Requests
that are also useful for external consumption.
N ) __version__)certs)to_native_string)parse_http_list)quoteurlparsebytesstr OrderedDictunquote
getproxies proxy_bypass
urlunparse
basestring
integer_typesis_py3proxy_bypass_environmentgetproxies_environmentMapping)cookiejar_from_dict)CaseInsensitiveDict)
InvalidURL
InvalidHeaderFileModeWarningUnrewindableBodyError.netrc_netrcP i )ZhttpZhttpsZwin32c C s yt rdd l}ndd l}W n tk
r. dS X y6|j|jd}t|j|dd }|j|dd }W n tk
rz dS X | s| rdS |j d}xX|D ]P}|dkrd| krd S |j
dd
}|j
d d }|j
d
d}t j || t j
rd S qW dS )Nr Fz;Software\Microsoft\Windows\CurrentVersion\Internet SettingsZ ProxyEnableZ
ProxyOverride;z.Tz\.*z.*?)r winreg_winreg ImportErrorOpenKeyHKEY_CURRENT_USERintZ QueryValueExOSErrorsplitreplacerematchI)hostr# ZinternetSettingsZ proxyEnableZ
proxyOverrideZtest r0 /usr/lib/python3.6/utils.pyproxy_bypass_registry/ s8
r2 c C s t rt| S t| S dS )zReturn True, if the host should be bypassed.
Checks proxy settings gathered from the environment, if specified,
or the registry.
N)r r r2 )r/ r0 r0 r1 r V s r c C s t | dr| j } | S )z/Returns an internal sequence dictionary update.items)hasattrr3 )dr0 r0 r1 dict_to_sequenceb s
r6 c C s2 d }d}t | dr t| }nbt | dr.| j}nPt | dr~y | j }W n tjk
rZ Y n$X tj|j}d| jkr~t j
dt t | dry | j }W n$ t
tfk
r |d k r|}Y n\X t | do|d kry&| jdd | j }| j|pd W n t
tfk
r d}Y nX |d kr$d}td|| S )
Nr __len__lenfilenoba% Requests has determined the content-length for this request using the binary size of the file: however, the file has been opened in text mode (i.e. without the 'b' flag in the mode). This may lead to an incorrect content-length. In Requests 3.0, support will be removed for files in text mode.tellseek )r4 r8 r9 ioUnsupportedOperationosfstatst_sizemodewarningswarnr r; r) IOErrorr< max)oZ total_lengthZcurrent_positionr9 r0 r0 r1 super_lenk s@
rI Fc C s yddl m }m} d}xJtD ]B}ytjjdj|}W n tk
rJ dS X tjj|r |}P q W |dkrndS t | }|j
}y6||j |} | r| d rdnd}
| |
| d fS W n |t fk
r |r̂ Y nX W n t
tfk
r Y nX dS )z;Returns the Requests tuple auth for a given url from netrc.r )netrcNetrcParseErrorNz~/{}r r= )rJ rK NETRC_FILESr@ path
expanduserformatKeyErrorexistsr hostnameZauthenticatorsrF r% AttributeError) urlZ raise_errorsrJ rK Z
netrc_pathflocZrir/ r Zlogin_ir0 r0 r1 get_netrc_auth s2
rW c C sB t | dd}|r>t|tr>|d dkr>|d dkr>tjj|S dS )z0Tries to guess the filename of the given object.nameNr )getattr
isinstancer r@ rM basename)objrX r0 r0 r1 guess_filename s r` c C s t jj| r| S t jj| \}}x4|rTt jj| rTt jj|\}}dj||g}q"W tj|sd| S tj|}||j kr~| S t j
}t jj|f|jd }t jj|s|j ||d}|S )zReplace nonexistent paths that look like they refer to a member of a zip
archive with the location of an extracted copy of the target, or else
just return the provided path unchanged.
/)rM ) r@ rM rQ r* joinzipfileZ
is_zipfileZZipFileZnamelisttempfileZ
gettempdirextract)rM archivememberprefixZzip_fileZtmpZextracted_pathr0 r0 r1 extract_zipped_paths s
ri c C s. | dkr dS t | ttttfr&tdt| S )a Take an object and test to see if it can be represented as a
dictionary. Unless it can not be represented as such, return an
OrderedDict, e.g.,
::
>>> from_key_val_list([('key', 'val')])
OrderedDict([('key', 'val')])
>>> from_key_val_list('string')
ValueError: need more than 1 value to unpack
>>> from_key_val_list({'key': 'val'})
OrderedDict([('key', 'val')])
:rtype: OrderedDict
Nz+cannot encode objects that are not 2-tuples)r] r
r boolr(
ValueErrorr )valuer0 r0 r1 from_key_val_list s
rm c C s@ | dkr dS t | ttttfr&tdt | tr8| j } t| S )a Take an object and test to see if it can be represented as a
dictionary. If it can be, return a list of tuples, e.g.,
::
>>> to_key_val_list([('key', 'val')])
[('key', 'val')]
>>> to_key_val_list({'key': 'val'})
[('key', 'val')]
>>> to_key_val_list('string')
ValueError: cannot encode objects that are not 2-tuples.
:rtype: list
Nz+cannot encode objects that are not 2-tuples) r] r
r rj r( rk r r3 list)rl r0 r0 r1 to_key_val_list s
ro c C s\ g }xRt | D ]F}|dd |dd ko4dkn rJt|dd }|j| qW |S )a Parse lists as described by RFC 2068 Section 2.
In particular, parse comma-separated lists where the elements of
the list may include quoted-strings. A quoted-string could
contain a comma. A non-quoted string could have quotes in the
middle. Quotes are removed automatically after parsing.
It basically works like :func:`parse_set_header` just that items
may appear multiple times and case sensitivity is preserved.
The return value is a standard :class:`list`:
>>> parse_list_header('token, "quoted value"')
['token', 'quoted value']
To create a header from the :class:`list` again, use the
:func:`dump_header` function.
:param value: a string with a list header.
:return: :class:`list`
:rtype: list
Nr "r[ r[ )_parse_list_headerunquote_header_valueappend)rl resultitemr0 r0 r1 parse_list_header2 s (rv c C s| i }xrt | D ]f}d|kr$d||< q|jdd\}} | dd | dd koVdkn rlt| dd } | ||< qW |S )a^ Parse lists of key, value pairs as described by RFC 2068 Section 2 and
convert them into a python dict:
>>> d = parse_dict_header('foo="is a fish", bar="as well"')
>>> type(d) is dict
True
>>> sorted(d.items())
[('bar', 'as well'), ('foo', 'is a fish')]
If there is no value for a key it will be `None`:
>>> parse_dict_header('key_without_value')
{'key_without_value': None}
To create a header from the :class:`dict` again, use the
:func:`dump_header` function.
:param value: a string with a dict header.
:return: :class:`dict`
:rtype: dict
=Nr rp r[ r[ )rq r* rr )rl rt ru rX r0 r0 r1 parse_dict_headerR s ( rx c C s^ | rZ| d | d ko dkn rZ| dd
} | sF| dd dkrZ| j ddj ddS | S ) zUnquotes a header value. (Reversal of :func:`quote_header_value`).
This does not use the real unquoting but what browsers are actually
using for quoting.
:param value: the header value to unquote.
:rtype: str
r r rp Nr= z\\\z\"r[ r[ )r+ )rl Z is_filenamer0 r0 r1 rr u s
$ rr c C s" i }x| D ]}|j ||j< q
W |S )zReturns a key/value dictionary from a CookieJar.
:param cj: CookieJar object to extract cookies from.
:rtype: dict
)rl rX )cj cookie_dictZcookier0 r0 r1 dict_from_cookiejar s
r| c C s
t || S )zReturns a CookieJar from a key/value dictionary.
:param cj: CookieJar to insert cookies into.
:param cookie_dict: Dict of key/values to insert into CookieJar.
:rtype: CookieJar
)r )rz r{ r0 r0 r1 add_dict_to_cookiejar s r} c C sT t jdt tjdtjd}tjdtjd}tjd}|j| |j| |j| S )zlReturns encodings from given content string.
:param content: bytestring to extract encodings from.
zIn requests 3.0, get_encodings_from_content will be removed. For more information, please see the discussion on issue #2266. (This warning should only appear once.)z!])flagsz+]z$^<\?xml.*?encoding=["\']*(.+?)["\'>])rD rE DeprecationWarningr, compiler. findall)contentZ
charset_reZ pragma_reZxml_rer0 r0 r1 get_encodings_from_content s
r c
C s | j d}|d j |dd }}i }d}xh|D ]`}|j }|r2|d }}|jd} | dkr|d| j|}|| d d j|}|||j < q2W ||fS ) zReturns content type and parameters from given header
:param header: string
:return: tuple containing content type and dictionary of
parameters
r r r Nz"' Trw r[ )r* stripfindlower)
headertokens content_typeparamsZ params_dictZitems_to_stripparamkeyrl Zindex_of_equalsr0 r0 r1 _parse_content_type_header s
r c C sD | j d}|sdS t|\}}d|kr4|d jdS d|kr@dS dS )z}Returns encodings from given HTTP Header Dict.
:param headers: dictionary to extract encoding from.
:rtype: str
z content-typeNcharsetz'"textz
ISO-8859-1)getr r )headersr r r0 r0 r1 get_encoding_from_headers s
r c c sr |j dkr"x| D ]
}|V qW dS tj|j dd}x | D ]}|j|}|r:|V q:W |jddd}|rn|V dS )zStream decodes a iterator.Nr+ )errors T)final)encodingcodecsgetincrementaldecoderdecode)iteratorrru decoderchunkrvr0 r0 r1 stream_decode_response_unicode s
r c c sL d}|dks|dkr t | }x*|t | k rF| ||| V ||7 }q W dS )z Iterate over slices of a string.r N)r8 )stringZ slice_lengthposr0 r0 r1 iter_slices s r c C sv t jdt g }t| j}|rJy t| j|S tk
rH |j| Y nX yt| j|ddS t k
rp | jS X dS )zReturns the requested content back in unicode.
:param r: Response object to get unicode content from.
Tried:
1. charset from content-type
2. fall back and replace all unicode characters
:rtype: str
zIn requests 3.0, get_unicode_from_response will be removed. For more information, please see the discussion on issue #2266. (This warning should only appear once.)r+ )r N)
rD rE r r r r
r UnicodeErrorrs TypeError)r Ztried_encodingsr r0 r0 r1 get_unicode_from_response s
r Z4ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzz0123456789-._~c
C s | j d}xtdt|D ]}|| dd }t|dkr|j rytt|d}W n tk
rt td| Y nX |tkr||| dd ||< qd|| ||< qd|| ||< qW dj |S ) zUn-escape any percent-escape sequences in a URI that are unreserved
characters. This leaves all reserved, illegal and non-ASCII bytes encoded.
:rtype: str
%r r r= z%Invalid percent-escape sequence: '%s'N )
r* ranger8 isalnumchrr( rk r UNRESERVED_SETrb )uripartsihcr0 r0 r1 unquote_unreserved/ s
r c C s: d}d}yt t| |dS tk
r4 t | |dS X dS )zRe-quote the given URI.
This function passes the given URI through an unquote/quote cycle to
ensure that it is fully and consistently quoted.
:rtype: str
z!#$%&'()*+,/:;=?@[]~z!#$&'()*+,/:;=?@[]~)ZsafeN)r r r )r Zsafe_with_percentZsafe_without_percentr0 r0 r1 requote_uriG s r c C sl t jdtj| d }|jd\}}t jdtjtt|d }t jdtj|d |@ }||@ ||@ kS )zThis function allows you to check if an IP belongs to a network subnet
Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24
returns False if ip = 192.168.1.1 and net = 192.168.100.0/24
:rtype: bool
z=Lr ra )structunpacksocket inet_atonr* dotted_netmaskr( )ZipZnetZipaddrZnetaddrbitsZnetmaskZnetworkr0 r0 r1 address_in_network] s
r c C s& ddd| > d A }t jtjd|S )zConverts mask from /xx format to xxx.xxx.xxx.xxx
Example: if mask is 24 function returns 255.255.255.0
:rtype: str
l r z>I)r Z inet_ntoar pack)maskr r0 r0 r1 r l s r c C s* yt j| W n t jk
r$ dS X dS )z
:rtype: bool
FT)r r error)Z string_ipr0 r0 r1 is_ipv4_addressw s
r c C s | j ddkryt| jdd }W n tk
r8 dS X |dk sJ|dkrNdS ytj| jdd W q tjk
r| dS X ndS dS )zV
Very simple check of the cidr format in no_proxy variable.
:rtype: bool
ra r Fr r T)countr( r* rk r r r )Zstring_networkr r0 r0 r1
is_valid_cidr s r c c sT |dk }|r"t jj| }|t j| < z
dV W d|rN|dkrDt j| = n
|t j| < X dS )zSet the environment variable 'env_name' to 'value'
Save previous value, yield, and then restore the previous value stored in
the environment variable 'env_name'.
If 'value' is None, do nothingN)r@ environr )Zenv_namerl Z
value_changedZ old_valuer0 r0 r1 set_environ s
r c C s, dd }|}|dkr |d}t | }|jdkr2dS |rdd |jdd jd
D }t|jrxz|D ],}t|rt|j|rdS qb|j|krbdS qbW nD|j}|jr|d j|j7 }x&|D ] }|jj |s|j |rdS qW t
d|6 yt |j}W n t t
jfk
r d }Y nX W dQ R X |r(dS d S )
zL
Returns whether we should bypass proxies or not.
:rtype: bool
c S s t jj| pt jj| j S )N)r@ r r upper)kr0 r0 r1 s z'should_bypass_proxies..Nno_proxyTc s s | ]}|r|V qd S )Nr0 ).0r/ r0 r0 r1 s z(should_bypass_proxies.. r ,z:{}F)r rR r+ r* r r r ZportrO endswithr r r r Zgaierror) rT r Z get_proxyZ no_proxy_argparsedZproxy_ipZhost_with_portr/ Zbypassr0 r0 r1 should_bypass_proxies s<
r c C s t | |dri S t S dS )zA
Return a dict of environment proxies.
:rtype: dict
)r N)r r
)rT r r0 r0 r1 get_environ_proxies s r c C sv |pi }t | }|jdkr.|j|j|jdS |jd |j |jd|j dg}d}x |D ]}||krX|| }P qXW |S )zSelect a proxy for the url, if applicable.
:param url: The url being for the request
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
Nallz://zall://)r rR r scheme)rT ZproxiesZurlpartsZ
proxy_keysproxyZ proxy_keyr0 r0 r1 select_proxy s
r python-requestsc C s d| t f S )zO
Return a string representing the default user agent.
:rtype: str
z%s/%s)r )rX r0 r0 r1 default_user_agent s r c C s t t djddddS )z9
:rtype: requests.structures.CaseInsensitiveDict
z, gzipdeflatez*/*z
keep-alive)z
User-AgentzAccept-EncodingZAcceptZ
Connection)r r )r r rb r0 r0 r0 r1 default_headers s
r c C s g }d}| j |} | s|S xtjd| D ]}y|jdd\}}W n tk
r^ |d }}Y nX d|j di}xP|jdD ]B}y|jd\}} W n tk
r P Y nX | j |||j |< qzW |j| q(W |S ) zReturn a list of parsed link headers proxies.
i.e. Link: ; rel=front; type="image/jpeg",; rel=back;type="image/jpeg"
:rtype: list
z '"z, * '"rw )r r, r* rk rs ) rl ZlinksZ
replace_charsvalrT r linkr r r0 r0 r1 parse_header_links( s&
r asciir= c C s | dd }|t jt jfkr dS |dd t jkr6dS |dd t jt jfkrRdS |jt}|dkrhd S |dkr|ddd tkrd
S |d dd tkrd S |dkr|dd t krd
S |d d t krdS dS )z
:rtype: str
N zutf-32r z utf-8-sigr= zutf-16r zutf-8z utf-16-ber z utf-16-lez utf-32-bez utf-32-le)
r BOM_UTF32_LE BOM_UTF32_BEBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BEr _null_null2_null3)dataZsampleZ nullcountr0 r0 r1 guess_json_utfS s*
r c C s8 t | |\}}}}}}|s$|| }}t||||||fS )zGiven a URL that may or may not have a scheme, prepend the given scheme.
Does not replace a present scheme with the one provided as an argument.
:rtype: str
)r r )rT Z
new_schemer netlocrM r queryfragmentr0 r0 r1 prepend_scheme_if_neededs s
r c C sB t | }yt|jt|jf}W n ttfk
r< d}Y nX |S )z{Given a url with authentication components, extract them into a tuple of
username,password.
:rtype: (str,str)
r )r r )r r ZusernameZpasswordrS r )rT r Zauthr0 r0 r1 get_auth_from_url s
r s ^\S[^\r\n]*$|^$z^\S[^\r\n]*$|^$c C sf | \}}t |trt}nt}y|j|s4td| W n* tk
r` td||t|f Y nX dS )zVerifies that header value is a string which doesn't contain
leading whitespace or return characters. This prevents unintended
header injection.
:param header: tuple, in the format (name, value).
z7Invalid return character or leading space in header: %sz>Value for header {%s: %s} must be of type str or bytes, not %sN)r] r _CLEAN_HEADER_REGEX_BYTE_CLEAN_HEADER_REGEX_STRr- r r type)r rX rl Zpatr0 r0 r1 check_header_validity s
r c C sF t | \}}}}}}|s"|| }}|jddd }t|||||dfS )zW
Given a url remove the fragment and the authentication part.
:rtype: str
@r r r[ )r rsplitr )rT r r rM r r r r0 r0 r1
urldefragauth s
r c C s` t | jdd}|dk rTt| jtrTy|| j W q\ ttfk
rP tdY q\X ntddS )zfMove file pointer back to its recorded starting position
so it can be read again on redirect.
r< Nz;An error occurred when rewinding request body for redirect.z+Unable to rewind request body for redirect.)r\ Zbodyr] Z_body_positionr rF r) r )Zprepared_requestZ body_seekr0 r0 r1 rewind_body s r )r r )F)FzBABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~)N)r )a__doc__r
contextlibr> r@ r, r r sysrd rD rc r r r Z_internal_utilsr compatr rq r r r r
r r r
r r r r r r r r Zcookiesr Z
structuresr
exceptionsr r r r rL whereZDEFAULT_CA_BUNDLE_PATHZ
DEFAULT_PORTSplatformr2 r6 rI rW r` ri rm ro rv rx rr r| r} r r r r r r frozensetr r r r r r r contextmanagerr r r r r r r encoder r r r r r r r r r r r r0 r0 r0 r1 s D
' =
- #
% =
&