a < b/v@sdZddlmZddlmZddlmZmZddlZddlm Z ddl Z ddl m Z mZddlmZmZddlZdd lmZmZdd lmZdd lmZddlmmmZdd lm Z d ddddZ!ddddddZ"GdddZ#GdddZ$Gddde ej%Z&dS)a Read SAS7BDAT files Based on code written by Jared Hobbs: https://bitbucket.org/jaredhobbs/sas7bdat See also: https://github.com/BioStatMatt/sas7bdat Partial documentation of the file format: https://cran.r-project.org/package=sas7bdat/vignettes/sas7bdat.pdf Reference for binary data compression: http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/1992/9210/ross/ross.htm ) annotations)abc)datetime timedeltaN)cast)FilePath ReadBuffer)EmptyDataErrorOutOfBoundsDatetime) DataFrameisna) get_handle)Parser) ReaderBasefloatstrZ sas_datetimeunitcCsVt|rtjS|dkr,tdddt|dS|dkrJtdddt|dStddS)Ns)secondsd)dayszunit must be 'd' or 's')r pdZNaTrr ValueErrorrrf/Users/vegardjervell/Documents/master/model/venv/lib/python3.9/site-packages/pandas/io/sas/sas7bdat.py_parse_datetime1srz pd.Series) sas_datetimesrreturncCsHztj||ddWStyB|jt|d}ttj|}|YS0dS)a Convert to Timestamp if possible, otherwise to datetime.datetime. SAS float64 lacks precision for more than ms resolution so the fit to datetime.datetime is ok. Parameters ---------- sas_datetimes : {Series, Sequence[float]} Dates or datetimes in SAS unit : {str} "d" if the floats represent dates, "s" for datetimes Returns ------- Series Series of datetime64 dtype or datetime.datetime. z 1960-01-01)roriginrN)r to_datetimer applyrrSeries)rrZs_seriesrrr_convert_datetimes?s   r&c@sBeZdZUded<ded<ded<ded<dddddddZd S) _SubheaderPointerintoffsetlength compressionptyper)r*r+r,cCs||_||_||_||_dSNr-)selfr)r*r+r,rrr__init___sz_SubheaderPointer.__init__N__name__ __module__ __qualname____annotations__r0rrrrr'Ys r'c@sVeZdZUded<ded<ded<ded<ded<ded <ddddddd d d Zd S)_Columnr(col_idz str | bytesnamelabelformatbytesctyper*r7r8r9r:r<r*cCs(||_||_||_||_||_||_dSr.r=)r/r7r8r9r:r<r*rrrr0ns z_Column.__init__Nr1rrrrr6fs r6c@seZdZUdZded<ded<dVdd d d Zd d ddZd d ddZd d ddZdd ddZ dd ddZ ddZ dddddZ dddddd Z ddd!d"d#Zdd d$d%Zd&d d'd(Zd)d*Zdd d+d,Zd-dd.d/d0Zddd1d2d3d4Zdd-d5d6d7Zdd1dd8d9d:Zdddd;dd?Zdddd;d@dAZdddd;dBdCZdddd;dDdEZdddd;dFdGZdddd;dHdIZdddd;dJdKZdWdLdMdNdOdPZdQdRZ dSd dTdUZ!dS)XSAS7BDATReadera Read SAS files in SAS7BDAT format. Parameters ---------- path_or_buf : path name or buffer Name of SAS file or file-like object pointing to SAS file contents. index : column identifier, defaults to None Column to use as index. convert_dates : bool, defaults to True Attempt to convert dates to Pandas datetime values. Note that some rarely used SAS date formats may be unsupported. blank_missing : bool, defaults to True Convert empty strings to missing values (SAS uses blanks to indicate missing character variables). chunksize : int, defaults to None Return SAS7BDATReader object for iterations, returns chunks with given number of lines. encoding : string, defaults to None String encoding. convert_text : bool, defaults to True If False, text variables are left as raw bytes. convert_header_text : bool, defaults to True If False, header text, including column names, are left as raw bytes. r( _int_lengthz bytes | None _cached_pageNTzFilePath | ReadBuffer[bytes]) path_or_bufc Cs||_||_||_||_||_||_||_d|_d|_g|_ g|_ g|_ g|_ g|_ d|_g|_g|_g|_d|_d|_d|_t|ddd|_|jj|_z||Wnty|Yn0dS)Nzlatin-1rrbF)Zis_text)index convert_dates blank_missing chunksizeencoding convert_textconvert_header_textdefault_encodingr+column_names_strings column_namescolumn_formatscolumns%_current_page_data_subheader_pointersr@_column_data_lengths_column_data_offsets _column_types_current_row_in_file_indexZ_current_row_on_page_indexr handleshandle _path_or_buf_get_properties_parse_metadata Exceptionclose) r/rArDrErFrGrHrIrJrrrr0s:    zSAS7BDATReader.__init__z np.ndarray)r cCstj|jtjdS)z5Return a numpy int64 array of the column data lengthsdtype)npasarrayrQint64r/rrrcolumn_data_lengthssz"SAS7BDATReader.column_data_lengthscCstj|jtjdS)z0Return a numpy int64 array of the column offsetsr\)r^r_rRr`rarrrcolumn_data_offsetssz"SAS7BDATReader.column_data_offsetscCstj|jtddS)zj Returns a numpy character array of the column types: s (string) or d (double) ZS1r\)r^r_rSr]rarrr column_typesszSAS7BDATReader.column_typesNonecCs|jdSr.)rUr[rarrrr[szSAS7BDATReader.closecCs|jd|jd|_|jdttjtjkr|tj?|_@|tjA|tjB}|$d|_C|j&r|jC'|j(p|j)|_C|tjD|tjE}|$d|_F|j&r2|jF'|j(p,|j)|_F|tjG|tjH}|$d|_I|j&rr|jI'|j(pl|j)|_I|tjJ|tjK}|$d}t|dkr|'|j(p|j)|_Ln@|tjM|tjN}|$d|_L|j&r|jL'|j(p|j)|_LdS)Nri z'magic number mismatch (not a SAS file?)rrTF<>zunknown (code=)1unix2Zwindowsunknown rrrr"z*The SAS7BDAT file appears to be truncated.)OrWseekreadr@lenconstmagicr _read_bytesZalign_1_offsetZalign_1_lengthZu64_byte_checker_valueZ align_2_valueU64r?Zpage_bit_offset_x64_page_bit_offsetZsubheader_pointer_length_x64_subheader_pointer_lengthZpage_bit_offset_x86Zsubheader_pointer_length_x86Zalign_2_offsetZalign_2_lengthZalign_1_checker_valueZendianness_offsetZendianness_length byte_orderZencoding_offsetZencoding_lengthZencoding_names file_encodingZplatform_offsetZplatform_lengthplatformZdataset_offsetZdataset_lengthrstripr8rJdecoderHrKZfile_type_offsetZfile_type_length file_typer _read_floatZdate_created_offsetZdate_created_lengthrZ to_timedeltaZ date_createdZdate_modified_offsetZdate_modified_lengthZ date_modified _read_intZheader_size_offsetZheader_size_length header_lengthZpage_size_offsetZpage_size_length _page_lengthZpage_count_offsetZpage_count_lengthZ _page_countZsas_release_offsetZsas_release_lengthZ sas_releaseZsas_server_type_offsetZsas_server_type_lengthZ server_typeZos_version_number_offsetZos_version_number_length os_versionZos_name_offsetZos_name_lengthos_nameZos_maker_offsetZos_maker_length)r/Zalign1Zalign2bufZ total_alignepochxrrrrXs                             zSAS7BDATReader._get_propertiescCs*|j|jp dd}|dur&|t|S)Nr)nrows)rsrGr[ StopIteration)r/darrr__next__hs zSAS7BDATReader.__next__)r)widthcCsJ|dvr|td|||}|dkr0dnd}t|j||dS)N)rhrgzinvalid float widthrhfrrr[rrwstructunpackr{)r/r)rrfdrrrrps  zSAS7BDATReader._read_float)r)rr cCsP|dvr|td|||}ddddd|}t|j||d}|S)N)rrhrgzinvalid int widthbhlqrr)r/r)rritZivrrrrys zSAS7BDATReader._read_int)r)r*cCs|jdurX|j||j|}t||krT|d|dd|dd}t||S||t|jkrz|td|j|||SdS)NzUnable to read rz bytes from file position .zThe cached page is too small.)r@rWrrrsrtr[r)r/r)r*rmsgrrrrws    zSAS7BDATReader._read_bytescCsNd}|sJ|j|j|_t|jdkr(qJt|j|jkr@td|}qdS)NFrz2Failed to read a meta data page from the SAS file.)rWrsrr@rtr_process_page_meta)r/donerrrrYszSAS7BDATReader._parse_metadataboolcCsZ|tjtjgtj}|j|vr,||jtj@}|jtjv}t|pV|pV|j gkSr.) _read_page_headerrupage_meta_typeZ page_amd_typepage_mix_types_current_page_type_process_page_metadatapage_data_typerrP)r/pt is_data_pageZ is_mix_pagerrrrs   z!SAS7BDATReader._process_page_metacCsX|j}tj|}||tj|_tj|}||tj|_tj |}||tj |_ dSr.) ryruZpage_type_offsetrZpage_type_lengthrZblock_count_offsetZblock_count_lengthZ_current_page_block_countZsubheader_count_offsetZsubheader_count_length_current_page_subheaders_count)r/ bit_offsetZtxrrrrs   z SAS7BDATReader._read_page_headercCsp|j}t|jD]Z}|tj||}|jdkr2q|jtjkr@q| |j }| ||j|j }| ||qdS)Nr)ryranger_process_subheader_pointersruZsubheader_pointers_offsetr*r+Ztruncated_subheader_id_read_subheader_signaturer)_get_subheader_indexr,_process_subheader)r/ripointersubheader_signaturesubheader_indexrrrrs     z%SAS7BDATReader._process_page_metadatar;) signaturer cCs`tj|}|dur\|tjkp$|dk}|tjk}|jdkrL|rL|rLtjj}n|t d|S)NrrBzUnknown subheader signature) ruZsubheader_signature_to_indexgetZcompressed_subheader_idZcompressed_subheader_typer+SASIndexdata_subheader_indexr[r)r/rr+r,rDf1f2rrrrs   z#SAS7BDATReader._get_subheader_indexr')r)subheader_pointer_indexr c Cst|j}|||}|||j}||j7}|||j}||j7}||d}|d7}||d}t||||} | S)Nr)rzrr?r') r/r)rZsubheader_pointer_lengthZ total_offsetZsubheader_offsetZsubheader_lengthZsubheader_compressionZsubheader_typerrrrrs     z*SAS7BDATReader._process_subheader_pointers)r)r cCs|||j}|Sr.)rwr?)r/r)rrrrrsz(SAS7BDATReader._read_subheader_signature)rrr cCs|j}|j}|tjjkr |j}n|tjjkr4|j}n|tjjkrH|j }n|tjj kr\|j }nt|tjj krp|j }n`|tjjkr|j}nL|tjjkr|j}n8|tjjkr|j}n$|tjjkr|j|dStd|||dS)Nzunknown subheader index)r)r*rurZrow_size_index_process_rowsize_subheaderZcolumn_size_index_process_columnsize_subheaderZcolumn_text_index_process_columntext_subheaderZcolumn_name_index_process_columnname_subheaderZcolumn_attributes_index#_process_columnattributes_subheaderZformat_and_label_index_process_format_subheaderZcolumn_list_index_process_columnlist_subheaderZsubheader_counts_index_process_subheader_countsrrPappendr)r/rrr)r* processorrrrrs.          z!SAS7BDATReader._process_subheader)r)r*r cCs|j}|}|}|jr&|d7}|d7}n|d7}|d7}||tj|||_||tj|||_||tj|||_ ||tj |||_ tj |}|||||_ ||d|_||d|_dS)Niiibizr)r?rxrruZrow_length_offset_multiplierZ row_lengthZrow_count_offset_multiplier row_countZcol_count_p1_multiplier col_count_p1Zcol_count_p2_multiplier col_count_p2Z'row_count_on_mix_page_offset_multiplierZ_mix_page_row_count_lcs_lcp)r/r)r*int_lenZ lcs_offsetZ lcp_offsetZmxrrrr s0  z)SAS7BDATReader._process_rowsize_subheadercCsT|j}||7}||||_|j|j|jkrPtd|jd|jd|jddS)Nz Warning: column count mismatch (z + z != z) )r?r column_countrrprint)r/r)r*rrrrr(s z,SAS7BDATReader._process_columnsize_subheadercCsdSr.rr/r)r*rrrr3sz(SAS7BDATReader._process_subheader_countsc Cs||j7}||tj}|||}|d|d}|}|jrR||jpN|j }|j |t |j dkrd}tj D]}||vrx|}qx||_||j8}|d} |jr| d7} || |j}|d}|dkrd|_|d} |jr| d7} || |j}|d|j|_n|tjkrR|d } |jr2| d7} || |j}|d|j|_nH|jdkrd|_|d} |jr|| d7} || |j}|d|j|_|jrt|d r|j|jp|j |_dS) NrrqrrBrh ( creator_proc)r?rruZtext_block_size_lengthrwr~rJrrHrKrLrrtZcompression_literalsr+rxrrrZrle_compressionhasattr) r/r)r*Ztext_block_sizerZ cname_rawcnameZcompression_literalZclZoffset1rrrr6sZ           z,SAS7BDATReader._process_columntext_subheaderc Cs|j}||7}|d|dd}t|D]}|tj|dtj}|tj|dtj}|tj|dtj}||tj} ||tj } ||tj } |j | } |j | | | | q*dS)Nr rgr)r?rruZcolumn_name_pointer_lengthZ!column_name_text_subheader_offsetZcolumn_name_offset_offsetZcolumn_name_length_offsetrZ!column_name_text_subheader_lengthZcolumn_name_offset_lengthZcolumn_name_length_lengthrLrMr) r/r)r*rZcolumn_name_pointers_countrZtext_subheaderZcol_name_offsetZcol_name_lengthidx col_offsetZcol_lenZname_strrrrrjs>     z,SAS7BDATReader._process_columnname_subheaderc Cs|j}|d|d|d}t|D]}||tj||d}|d|tj||d}|d|tj||d}|||} |j| ||tj } |j | ||tj } |j | dkrdndq&dS)Nrrrgrds) r?rruZcolumn_data_offset_offsetZcolumn_data_length_offsetZcolumn_type_offsetrrRrZcolumn_data_length_lengthrQZcolumn_type_lengthrS) r/r)r*rZcolumn_attributes_vectors_countrZcol_data_offsetZ col_data_lenZ col_typesrrrrrs*     z2SAS7BDATReader._process_columnattributes_subheadercCsdSr.rrrrrrsz,SAS7BDATReader._process_columnlist_subheadercCsl|j}|tjd|}|tjd|}|tjd|}|tjd|}|tjd|}|tjd|} ||tj } t | t |j d} ||tj } ||tj} ||tj}t |t |j d}||tj}|| tj}|j |}||||}|j | }|| | | }t |j}t||j||||j||j|}|j||j|dS)Nr)r?ruZ)column_format_text_subheader_index_offsetZcolumn_format_offset_offsetZcolumn_format_length_offsetZ(column_label_text_subheader_index_offsetZcolumn_label_offset_offsetZcolumn_label_length_offsetrZ)column_format_text_subheader_index_lengthminrtrLZcolumn_format_offset_lengthZcolumn_format_length_lengthZ(column_label_text_subheader_index_lengthZcolumn_label_offset_lengthZcolumn_label_length_lengthrOr6rMrSrQrNr)r/r)r*rZtext_subheader_formatZcol_format_offsetZcol_format_lenZtext_subheader_labelZcol_label_offsetZ col_label_lenrZ format_idxZ format_startZ format_lenZ label_idxZ label_startZ label_lenZ label_namesZ column_labelZ format_namesZ column_formatZcurrent_column_numbercolrrrrsL    z(SAS7BDATReader._process_format_subheaderz int | NonezDataFrame | None)rr cCs|dur|jdur|j}n|dur(|j}t|jdkrF|td|j|jkrVdS|j|j}||krn|}|jd}|jd}tj ||ft d|_ tj |d|ftj d|_d|_t|}|||}|jdur||j}|S)NrzNo columns to parse from filerrr\rg)rGrrtrSr[r rTcountr^emptyobject _string_chunkZzerosZuint8 _byte_chunk_current_row_in_chunk_indexrrs_chunk_to_dataframerDZ set_index)r/rmndnsprsltrrrrss.       zSAS7BDATReader.readcCsg|_|j|j|_t|jdkr(dSt|j|jkrf|dt|jdd|jdd}t|||j }|t j kr| |t j @}t j gt j}|s|j |vr|SdS)NrTz-failed to read complete page from file (read rz of z bytes)F)rPrWrsrr@rtr[rrrrurrrr_read_next_page)r/rZ page_typerrrrrrs,   zSAS7BDATReader._read_next_pager c Cs|j}|j}t|||}i}d\}}t|jD]\}|j|}|j|dkr|j|ddfj|jdd} t j | t j |d||<|j r|j|tjvrt||d||<n"|j|tjvrt||d||<|d7}q0|j|dkrnt j |j|ddf|d ||<|jr<|jdur<||j|jp4|j||<|jrd||jd k} t j||| <|d7}q0|td t|j|q0t||j|d d } | S)Nrfrrr\)r]rDrrr)rDrzunknown column type F)rOrDcopy) rrTrrrMrSrviewr{rr%r^Zfloat64rErNruZsas_date_formatsr&Zsas_datetime_formatsrrIrHrrrKrFrtnanr[rreprr ) r/nrixrZjsZjbjr8Zcol_arriiZdfrrrrs<       z"SAS7BDATReader._chunk_to_dataframe)NTTNNTT)N)"r2r3r4__doc__r5r0rbrcrdr[rXrrrrwrYrrrrrrrrrrrrrrrrsrrrrrrr>sP 0       4 1"r>)'r __future__r collectionsrrrrtypingrZnumpyr^Zpandas._typingrrZ pandas.errorsr r Zpandasrr r Zpandas.io.commonr Zpandas.io.sas._sasrZpandas.io.sas.sas_constantsioZsasZ sas_constantsruZpandas.io.sas.sasreaderrrr&r'r6Iteratorr>rrrrs&