gddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z dZd!dZdZdZdZd Zd Zd Zd Zd ZdZdZdZdZdZdZdZdZGddZ dZ!d"dZ"dZ#dZ$dZ%dZ&dZ'd Z(dS)#N) DirectMap IDSelectorIDSelectorArrayIDSelectorBatchOperatingPointsRangeSearchResult rev_swig_ptrswig_ptrtry_extract_index_ivfct|jdkrtddd|jtj|S)Nuint8zInput argument codesz, must be ndarray of dtype uint8, but found )dtype TypeErrornpascontiguousarray)rs R/var/www/html/mpstechhub/venv/lib/python3.11/site-packages/faiss/class_wrappers.py_check_dtype_uint8r'sJ {gi29''5;;HII I  & &&Fc t||}n#t$r|rYdSwxYw|jd|zkrdSt||dz|t|||dS)z Replaces a method in a class with another version. The old method is renamed to method_name_c (because presumably it was implemented in C) N replacement__c)getattrAttributeError__name__setattr) the_classname replacementignore_missing orig_methods rreplace_methodr".si..    FF ~444 Itd{K000 It[)))))s  $$cZdd}dd}t|d|t|d|dS)Nc\|j\}}tj|d}||jksJ|Xtj|d}|j|fksJ||t ||t |dS||t ||dS)aPerform clustering on a set of vectors. The index is used for assignment. Parameters ---------- x : array_like Training vectors, shape (n, self.d). `dtype` must be float32. index : faiss.Index Index used for assignment. The dimension of the index should be `self.d`. weights : array_like, optional Per training sample weight (size n) used when computing the weighted average to obtain the centroid (default is 1 for all training vectors). float32rNshaperrdtrain_cr )selfxindexweightsnr)s rreplacement_trainz,handle_Clustering..replacement_train@sw1  ) 4 4 4DF{{{{  *7)DDDG=QE)))) LLHQKK0A0A B B B B B LLHQKK / / / / /rc |j\}}t|}||ksJ|j|jksJ|Yt j|d}|j|fksJ||t|||t|dS||t|||dS)a Perform clustering on a set of compressed vectors. The index is used for assignment. The decompression is performed on-the-fly. Parameters ---------- x : array_like Training vectors, shape (n, codec.code_size()). `dtype` must be `uint8`. codec : faiss.Index Index used to decode the vectors. Should have dimension `self.d`. index : faiss.Index Index used for assignment. The dimension of the index should be `self.d`. weights : array_like, optional Per training sample weight (size n) used when computing the weighted average to obtain the centroid (default is 1 for all training vectors). Nr%r&)r(r sa_code_sizer)rrtrain_encoded_cr )r+r,codecr-r.r/r)s rreplacement_train_encodedz4handle_Clustering..replacement_train_encodedWs w1 q ! !E&&((((((w%'!!!!  *7)DDDG=QE))))  HQKK!&(9(9 ; ; ; ; ;  HQKK > > > > >rtrain train_encodedNr")rr0r5s rhandle_Clusteringr:>sX0000.????89g'89999o/HIIIIIrc.d}t|d|dS)Nc|j\}}tj|d}||jksJ||t |dS)zPerform clustering on a set of 1D vectors. Parameters ---------- x : array_like Training vectors, shape (n, 1). `dtype` must be float32. r%r&N)r(rrr) train_exact_cr r+r,r/r)s rreplacement_train_exactz4handle_Clustering1D..replacement_train_exactysUw1  ) 4 4 4DF{{{{ 1hqkk*****r train_exactr9)rr?s rhandle_Clustering1DrAws, + + +9m-DEEEEErc~d}d}d}t|d|t|d|t|d|dS)Nc|j\}}tj|d}||jksJ||t |dS)z Train the quantizer on a set of training vectors. Parameters ---------- x : array_like Training vectors, shape (n, self.d). `dtype` must be float32. r%r&Nr'r>s rr0z+handle_Quantizer..replacement_trainsSw1  ) 4 4 4DF{{{{ Q $$$$$rc|j\}}tj|d}||jksJtj||jfd}|t|t|||S)al Compute the codes corresponding to a set of vectors. Parameters ---------- x : array_like Vectors to encode, shape (n, self.d). `dtype` must be float32. Returns ------- codes : array_like Corresponding code for each vector, shape (n, self.code_size) and `dtype` uint8. r%r&r )r(rrr)empty code_sizecompute_codes_cr )r+r,r/r)rs rreplacement_compute_codesz3handle_Quantizer..replacement_compute_codessww1  ) 4 4 4DF{{{{!T^,G<<< Xa[[(5//1=== rc|j\}}t|}||jksJtj||jfd}|t|t|||S)aJReconstruct an approximation of vectors given their codes. Parameters ---------- codes : array_like Codes to decode, shape (n, self.code_size). `dtype` must be uint8. Returns ------- Reconstructed vectors for each code, shape `(n, d)` and `dtype` float32. r%r&)r(rrFrrEr)decode_cr )r+rr/csr,s rreplacement_decodez,handle_Quantizer..replacement_decodesp 2"5))T^#### Ha[ 2 2 2 huoox{{A666rr6 compute_codesdecoder9)rr0rHrLs rhandle_QuantizerrOsp % % %*&9g'89999o/HIII9h(:;;;;;rc.d}t|d|dS)NcJ|j\}}||jksJ|jdksJ|jd|ksJ|jd}tj|d}tj|d}||t |t ||dS)Nrr%r&int64)r(r)ndimrrbuild_cr )r+r,graphr/r)Ks rreplacement_buildz%handle_NSG..replacement_buildsw1DF{{{{zQ{1~"""" KN  ) 4 4 4$U'::: Q Xe__a88888rbuildr9)rrYs r handle_NSGr[s,9999g'899999rcd}d}d2d}d}ddddd}dddddd}d ddddd d }d }d2d } d2d} d3d} d} ddd} ddddd}ddd}d2d}d2d}d2d}d}t|d|t|d|t|d|t|d|t|d|t|d |t|d!| t|d"| t|d#| t|d$| t|d%| d&'t|d(|d&'t|d)|d&'t|d*|d&'t|d+|d&'t|d,|t|d-|t|d.|t|d/|d&'d0}d1}||_||_dS)4Nc|j\}}||jksJtj|d}||t |dS)aAdds vectors to the index. The index must be trained before vectors can be added to it. The vectors are implicitly numbered in sequence. When `n` vectors are added to the index, they are given ids `ntotal`, `ntotal + 1`, ..., `ntotal + n - 1`. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. r%r&N)r(r)rradd_cr r>s rreplacement_addz%handle_Index..replacement_addsSw1DF{{{{  ) 4 4 4 1hqkk"""""rc|j\}}||jksJtj|d}tj|d}|j|fks Jd||t |t |dS)aWAdds vectors with arbitrary ids to the index (not all indexes support this). The index must be trained before vectors can be added to it. Vector `i` is stored in `x[i]` and has id `ids[i]`. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. ids : array_like Array if ids of size n. The ids must be of type `int64`. Note that `-1` is reserved in result lists to mean "not found" so it's better to not use it as an id. r%r&rTnot same nb of vectors as idsN)r(r)rradd_with_ids_cr r+r,idsr/r)s rreplacement_add_with_idsz.handle_Index..replacement_add_with_idssw1DF{{{{  ) 4 4 4"3g666yQE!!!#B!!! Ax{{HSMM:::::rc,|j\}}||jksJtj|d}|#tj||ftj}n|j||fksJ||t|t|||S)aFind the k nearest neighbors of the set of vectors x in the index. This is the same as the `search` method, but discards the distances. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. labels : array_like, optional Labels array to store the results. Returns ------- labels: array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 r%r&)r(r)rrrErTassign_cr r+r,klabelsr/r)s rreplacement_assignz(handle_Index..replacement_assigns(w1DF{{{{  ) 4 4 4 >Xq!fBH555FFs rr0z'handle_Index..replacement_trainsSw1DF{{{{  ) 4 4 4 Q $$$$$r)paramsDIc |j\}}tj|d}||jksJ|dksJ|#tj||ftj}n|j||fksJ|#tj||ftj}n|j||fksJ||t||t|t||||fS)aFind the k nearest neighbors of the set of vectors x in the index. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. params : SearchParameters Search parameters of the current search (overrides the class-level params) D : array_like, optional Distance array to store the result. I : array_like, optional Labels array to store the results. Returns ------- D : array_like Distances of the nearest neighbors, shape (n, k). When not enough results are found the label is set to +Inf or -Inf. I : array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 r%r&r) r(rrr)rEr%rTsearch_cr )r+r,rirmrnror/r)s rreplacement_searchz(handle_Index..replacement_search,s6w1  ) 4 4 4DF{{{{1uuuu 9!Qrz222AA7q!f$$$$ 9!Qrx000AA7q!f$$$$ a!a!hqkk6JJJ!t r)rmrnroRc P|j\}}||jksJtj|d}|dksJ|#tj||ftj}n|j||fksJ|#tj||ftj}n|j||fksJ|$tj|||ftj}n|j|||fksJ||t||t|t|t|||||fS)aFind the k nearest neighbors of the set of vectors x in the index, and return an approximation of these vectors. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. params : SearchParameters Search parameters of the current search (overrides the class-level params) D : array_like, optional Distance array to store the result. I : array_like, optional Labels array to store the result. R : array_like, optional reconstruction array to store Returns ------- D : array_like Distances of the nearest neighbors, shape (n, k). When not enough results are found the label is set to +Inf or -Inf. I : array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 R : array_like Approximate (reconstructed) nearest neighbor vectors, shape (n, k, d). r%r&r) r(r)rrrEr%rTsearch_and_reconstruct_cr ) r+r,rirmrnrorsr/r)s r"replacement_search_and_reconstructz8handle_Index..replacement_search_and_reconstructZs:>w1DF{{{{  ) 4 4 41uuuu 9!Qrz222AA7q!f$$$$ 9!Qrx000AA7q!f$$$$ 9!Q"*555AA7q!Qi'''' %% x{{ x{{ QKK!f   !QwrF)include_listnosrmrnrorc |j\}} | |jksJtj|d}|dksJ|#tj||ftj}n|j||fksJ|#tj||ftj}n|j||fksJ|j} |r| |z } |$tj||| ftj }n|j||| fksJ| |t||t|t|t||||||fS)aFind the k nearest neighbors of the set of vectors x in the index, and return the codes stored for these vectors Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. params : SearchParameters Search parameters of the current search (overrides the class-level params) include_listnos : bool, optional whether to include the list ids in the first bytes of each code D : array_like, optional Distance array to store the result. I : array_like, optional Labels array to store the result. codes : array_like, optional codes array to store Returns ------- D : array_like Distances of the nearest neighbors, shape (n, k). When not enough results are found the label is set to +Inf or -Inf. I : array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 R : array_like Approximate (reconstructed) nearest neighbor vectors, shape (n, k, d). r%r&r) r(r)rrrEr%rTrFcoarse_code_sizer search_and_return_codes_cr ) r+r,rirwrmrnrorr/r) code_size_1s r#replacement_search_and_return_codesz9handle_Index..replacement_search_and_return_codessfFw1DF{{{{  ) 4 4 41uuuu 9!Qrz222AA7q!f$$$$ 9!Qrx000AA7q!f$$$$n  3 40022 2K =HaK0AAAEE;1a"55555 && x{{ x{{ QKK%/     !U{rct|tr|}n|jdksJt|}t j|d}|r=|jjtj kr#t|j t|}n"t|j t|}||S)aRemove some ids from the index. This is a O(ntotal) operation by default, so could be expensive. Parameters ---------- x : array_like or faiss.IDSelector Either an IDSelector that returns True for vectors to remove, or a list of ids to reomove (1D array of int64). When `x` is a list, it is wrapped into an IDSelector. Returns ------- n_remove: int number of vectors that were removed rSrTr&) isinstancerrUr rr direct_maptyper Hashtablersizer r remove_ids_c)r+r,sel index_ivfs rreplacement_remove_idsz,handle_Index..replacement_remove_idss a $ $ ;CC6Q;;;;-d33I$Qg666A ;Y16):MMM%afhqkk::%afhqkk::  %%%rc|&tj|jtj}n|j|jfksJ||t ||S)aiApproximate reconstruction of one vector from the index. Parameters ---------- key : int Id of the vector to reconstruct x : array_like, optional pre-allocated array to store the results Returns ------- x : array_like reconstructed vector, size `self.d`, `dtype`=float32 Nr&)rrEr)r%r( reconstruct_cr r+keyr,s rreplacement_reconstructz-handle_Index..replacement_reconstructs[ 9rz222AA7tvj(((( 3 ,,,rc"tj|d}|j\}|(tj||jftj}n|j||jfksJ||t|t||S)aApproximate reconstruction of several vectors from the index. Parameters ---------- key : array of ints Ids of the vectors to reconstruct x : array_like, optional pre-allocated array to store the results Returns ------- x : array_like reconstrcuted vectors, size `len(key), self.d` rTr&)rrr(rEr)r%reconstruct_batch_cr )r+rr,r/s rreplacement_reconstruct_batchz3handle_Index..replacement_reconstruct_batch s"3g666 Y 9!TVBJ777AA7q$&k))))   HSMM8A;;???rrc|dkr |j|z }|(tj||jftj}n|j||jfksJ|||t||S)a'Approximate reconstruction of vectors `n0` ... `n0 + ni - 1` from the index. Missing vectors trigger an exception. Parameters ---------- n0 : int Id of the first vector to reconstruct (default 0) ni : int Number of vectors to reconstruct (-1 = default = ntotal) x : array_like, optional pre-allocated array to store the results Returns ------- x : array_like Reconstructed vectors, size (`ni`, `self.d`), `dtype`=float32 rNr&)ntotalrrEr)r%r(reconstruct_n_cr r+n0nir,s rreplacement_reconstruct_nz/handle_Index..replacement_reconstruct_n#sx$ 88r!B 9"dfRZ888AA7r46l**** RXa[[111rc|j}|j|fksJ|j||jfksJtj|d}tj|d}||t |t |dS)Nr%r&rT)rr(r)rrupdate_vectors_cr )r+keysr,r/s rreplacement_update_vectorsz0handle_Index..replacement_update_vectors?s IzaU""""w1df+%%%%  ) 4 4 4#D888 a$!=====rrmc|j\}}||jksJtj|d}t |}t |}||t||||t|j |dz }t|d}t|j | } t|j | } || | fS)aSearch vectors that are within a distance of the query vectors. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. thresh : float Threshold to select neighbors. All elements within this radius are returned, except for maximum inner product indexes, where the elements above the threshold are returned params : SearchParameters Search parameters of the current search (overrides the class-level params) Returns ------- lims: array_like Starting index of the results for each query vector, size n+1. D : array_like Distances of the nearest neighbors, shape `lims[n]`. The distances for query i are in `D[lims[i]:lims[i+1]]`. I : array_like Labels of nearest neighbors, shape `lims[n]`. The labels for query i are in `I[lims[i]:lims[i+1]]`. r%r&rSr)r(r)rrfloatrrange_search_cr r limscopyint distancesrj) r+r,threshrmr/r)resrndrnros rreplacement_range_searchz.handle_Index..replacement_range_searchHs8w1DF{{{{  ) 4 4 4v"" Ax{{FC@@@CHa!e,,1133 b]]  + + 0 0 2 2 R ( ( - - / /Qzrc |j\}} tj|d}| |jksJ|dksJ|#tj||ftj}n|j||fksJ|#tj||ftj}n|j||fksJtj|d}| Jd|j||jfksJ|(tj|d}|j|jksJ||t||t|t|t|t|d||fS)aFind the k nearest neighbors of the set of vectors x in an IVF index, with precalculated coarse quantization assignment. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. Dq : array_like, optional Distance array to the centroids, size (n, nprobe) Iq : array_like, optional Nearest centroids, size (n, nprobe) params : SearchParameters Search parameters of the current search (overrides the class-level params) D : array_like, optional Distance array to store the result. I : array_like, optional Labels array to store the results. Returns ------- D : array_like Distances of the nearest neighbors, shape (n, k). When not enough results are found the label is set to +Inf or -Inf. I : array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 r%r&rNrTparams not supportedF) r(rrr)rEr%rTnprobesearch_preassigned_cr ) r+r,riIqDqrmrnror/r)s rreplacement_search_preassignedz4handle_Index..replacement_search_preassignedrsu@w1  ) 4 4 4DF{{{{1uuuu 9!Qrz222AA7q!f$$$$ 9!Qrx000AA7q!f$$$$  !"G 4 4 4~~5~~~xAt{+++++ >%b :::B8rx'''' !! x{{ RLL(2,, QKK!     !t rc |j\}}||jksJtj|d}tj|d}| Jd|j||jfksJ|(tj|d}|j|jksJt |}t |}||t||t|t||t|j |dz } t| d} t|j |  } t|j|  } | | | fS)aSearch vectors that are within a distance of the query vectors. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. thresh : float Threshold to select neighbors. All elements within this radius are returned, except for maximum inner product indexes, where the elements above the threshold are returned Iq : array_like, optional Nearest centroids, size (n, nprobe) Dq : array_like, optional Distance array to the centroids, size (n, nprobe) params : SearchParameters Search parameters of the current search (overrides the class-level params) Returns ------- lims: array_like Starting index of the results for each query vector, size n+1. D : array_like Distances of the nearest neighbors, shape `lims[n]`. The distances for query i are in `D[lims[i]:lims[i+1]]`. I : array_like Labels of nearest neighbors, shape `lims[n]`. The labels for query i are in `I[lims[i]:lims[i+1]]`. r%r&rTNrrSr)r(r)rrrrrrange_search_preassigned_cr r rrrrrj r+r,rrrrmr/r)rrrrnros r$replacement_range_search_preassignedz:handle_Index..replacement_range_search_preassignedsk@w1DF{{{{  ) 4 4 4  !"G 4 4 4~~5~~~xAt{+++++ >%b :::B8rx''''v"" '' x{{F RLL(2,,    CHa!e,,1133 b]]  + + 0 0 2 2 R ( ( - - / /Qzrcr|j\}}||jksJtj|d}|5tj||ftj}n!|j||fksJ||t|t||SNr%r&) r(r)rrrEr2r sa_encode_cr )r+r,rr/r)s rreplacement_sa_encodez+handle_Index..replacement_sa_encodesw1DF{{{{  ) 4 4 4 =Ha!2!2!4!45RXFFFEE;1d&7&7&9&9"::::: HQKK%999 rcJ|j\}}||ksJt|}|(tj||jftj}n|j||jfksJ||t|t||SNr&) r(r2rrrEr)r% sa_decode_cr )r+rr,r/rKs rreplacement_sa_decodez+handle_Index..replacement_sa_decodes 2T&&(((((("5)) 9!TVBJ777AA7q$&k)))) HUOOXa[[999rc|j\}}||ksJt|}||j|fksJt|}||t||dSr8)r(r2rr add_sa_codes_c)r+rrdr/rKs rreplacement_add_sa_codesz.handle_Index..replacement_add_sa_codess 2T&&(((((("5)) ?9$$$$3--C Ax44444rc|j\}||jksJtj|d}|t j|dSNrTr&)r(rrrpermute_entries_cfaissr )r+permr/s rreplacement_permute_entriesz1handle_Index..replacement_permute_entriessW ZDK#D888 u~d3344444radd add_with_idsassignr6search remove_ids reconstructreconstruct_batch reconstruct_n range_searchupdate_vectorsTr search_and_reconstructsearch_and_return_codessearch_preassignedrange_search_preassigned sa_encode sa_decode add_sa_codespermute_entriescRdtj|iS)Nthis)rserialize_indextobytesr+s rindex_getstatez$handle_Index..index_getstate7s$-d33;;==>>rcztjtj|dd}|j|_dS)Nrr r&)rdeserialize_indexr frombufferr)r+stindex2s rindex_setstatez$handle_Index..index_setstate:s2(r&z)Q)Q)QRRK rr8rrN)r" __getstate__ __setstate__)rr_rerkr0rrrvr|rrrrrrrrrrrrrrs r handle_Indexrs###$;;;*@ % % %26,,,,,\BFQUY]99999z"$$$dBBBBBH&&&8,08>>>=A(((((TFJTUY>>>>>@QU88888t        5555555 9e_5559n.FGGG9h(:;;;9g'89999h(:;;;9l,BCCC9m-DEEE9102229o/HIII9n.FGGG9.0J"&((((965dLLLL976tMMMM921$HHHH987NNNN9k+@AAA9k+@AAA9n.FGGG9/1L"&((((???   ,I+Ircd}d}d}d}dd}d}d }d }dd d } d } dd} t|d|t|d|t|d|t|d|t|d| t|d|t|d|t|d|t|d| t|d|dt|d| ddS)Nc|j\}}t|}||jksJ||t |dSr8)r(rrFr^r r>s rr_z+handle_IndexBinary..replacement_addDsMw1 q ! !DN"""" 1hqkk"""""rc |j\}}t|}tj|d}||jksJ|j|fks Jd||t |t |dS)NrTr&ra)r(rrrrFrbr rcs rrez4handle_IndexBinary..replacement_add_with_idsJsw1 q ! !"3g666DN""""yQE!!!#B!!! Ax{{HSMM:::::rc|j\}}t|}||jksJ||t |dSr8)r(rrFr*r r>s rr0z-handle_IndexBinary..replacement_trainRsMw1 q ! !DN"""" Q $$$$$rctj|jtj}||t ||Sr)rrErFr rr rs rrz3handle_IndexBinary..replacement_reconstructXs; HT^28 4 4 4 3 ,,,rrrc|dkr |j|z }|(tj||jftj}n|j||jfksJ|||t||S)Nrr&)rrrErFr r(rr rs rrz5handle_IndexBinary..replacement_reconstruct_n]sx 88r!B 9"dn-RX>>>AA7r4>22222 RXa[[111rc nt|}|j\}}||jksJ|dksJtj||ftj}tj||ftj}||t||t|t|||fS)Nrr&) rr(rFrrEint32rTrqr )r+r,rir/r)rrjs rrrz.handle_IndexBinary..replacement_searchhs q ! !w1DN""""1uuuuHaV28444 1a&111 a!),,v&& ( ( (&  rc P|j\}}t|}||jksJ|dksJtj||ftj}tj||ftj}tj|d}|j||jfksJ|(tj|d}|j|jksJ| |t||t|t|t|t|d||fS)Nrr&rTrF) r(rrFrrErrTrrrr ) r+r,rirrr/r)rnros rrz:handle_IndexBinary..replacement_search_preassignedts"w1 q ! !DN""""1uuuu HaV28 , , , HaV28 , , ,  !"G 4 4 4xAt{+++++ >%b888B8rx'''' !! x{{ RLL(2,, QKK!     !t rc|j\}}t|}||jksJt|}||t |||t |j|dz}t|d}t |j |}t |j |} ||| fS)NrSr) r(rrFrrr r rrrrrj) r+r,rr/r)rrrrnros rrz4handle_IndexBinary..replacement_range_searchsw1 q ! !DN"""""" Ax{{FC888CHa!e,,1133 b]]  + + 0 0 2 2 R ( ( - - / /Qzrrc |j\}}t|}||jksJtj|d}| Jd|j||jfksJ|(tj|d}|j|jksJt |}t|}||t||t|t||t|j |dz } t | d} t|j |  } t|j|  } | | | fS)NrTr&rrrSr)r(rrFrrrrrrr r rrrrjrs rrz@handle_IndexBinary..replacement_range_search_preassignedsdw1 q ! !DN""""  !"G 4 4 4~~5~~~xAt{+++++ >%b888B8rx''''V"" '' x{{F RLL(2,,    CHa!e,,1133 b]]  + + 0 0 2 2 R ( ( - - / /Qzrct|tr|}nE|jdksJtj|d}t |jt|}||SNrSrTr&) r~rrUrrrrr r)r+r,rs rrz2handle_IndexBinary..replacement_remove_idssj a $ $ 7CC6Q;;;;$Qg666A!!&(1++66C  %%%rc.|j\}}t|}||jksJ|dksJ|#tj||ftj}n|j||fksJ||t|t|||S)aFind the k nearest neighbors of the set of vectors x in the index. This is the same as the `search` method, but discards the distances. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be uint8. k : int Number of nearest neighbors. labels : array_like, optional Labels array to store the results. Returns ------- labels: array_like Labels of the nearest neighbors, shape (n, k). When not enough results are found, the label is set to -1 rNr&)r(rrFrrErTrgr rhs rrkz.handle_IndexBinary..replacement_assigns(w1 q ! !DN""""1uuuu >Xq!fBH555FF.apply_methodsvw1  ) 4 4 4DI~~~~ Ha_BJ 7 7 7 1hqkk8A;;777rc|j\}}tj|d}||jksJtj||jftj}||t|t||Sr) r(rrrrErr%reverse_transform_cr rs rreplacement_reverse_transformz=handle_VectorTransform..replacement_reverse_transformsvw1  ) 4 4 4DJ Ha^2: 6 6 6   HQKK!===rc|j\}}tj|d}||jksJ||t |dSr)r(rrrr*r r>s rreplacement_vt_trainz4handle_VectorTransform..replacement_vt_trainsQw1  ) 4 4 4DI~~~~ Q $$$$$rr6reverse_transform)r"apply_pyapply)rrrrs rhandle_VectorTransformrsw%%% 9g';<<<%I"IO91022222rcVd}d}t|d|t|d|dS)Nc|r|j|jksJ|j\|_|_||j|rt |ndt |dSr8)r(nqgt_nnnset_groundtruth_cr r+rnros rreplacement_set_groundtruthz=handle_AutoTuneCriterion..replacement_set_groundtruthst  &7ag%%%% w  K3!tXa[[ B B B B Brc|j|jksJ|j|j|jfksJ|t |t |Sr8)r(rnnn evaluate_cr r s rreplacement_evaluatez6handle_AutoTuneCriterion..replacement_evaluatesSw!'!!!!w47DH-----x{{HQKK888rset_groundtruthevaluater9)rr rs rhandle_AutoTuneCriterionrsRBBB999 9/1LMMM9j*>?????rc.d}t|d|dS)Nc|j|j|jfksJtj|d}t }|||jt||||Sr)r(rr)rrr explore_cr )r+r-xqcritopss rreplacement_explorez2handle_ParameterSpace..replacement_explore slxDGUW-----  !"I 6 6 6 udgx||S " " " rexplorer9)rrs rhandle_ParameterSpacers,9i)<=====rc.|jfd}||_dS)Nct|jdksJtj|d}||jd|jdt |dS)NrRr%r&rrS)lenr(rrr )r+m original_inits rreplacement_initz,handle_MatrixStats..replacement_init-s_17||q     ) 4 4 4 dAGAJ HQKK@@@@@r)__init__)rr"r!s @rhandle_MatrixStatsr$*s9&MAAAAA *Ircd}||_dS)z add a write_bytes method cP|t|dt|SNrS)r r)r+bs r write_bytesz$handle_IOWriter..write_bytes7s"tHQKKCFF+++rN)r))rr)s rhandle_IOWriterr*5s!,,,(Ircd}||_dS)z add a read_bytes method ct|}|t|dt|}t|d|Sr') bytearrayr rbytes)r+totszbufwas_reads r read_bytesz#handle_IOReader..read_bytes@sD4 q#c((33S(^$$$rN)r2)rr2s rhandle_IOReaderr3=s!%%% &Irc.d}t|d|dS)Nc|j\}}||jksJtj|d}||t |dS)aTrains the index on a representative set of vectors inplace. The index must be trained before vectors can be added to it. This call WILL change the values in the input array, because of two scaling proceduces being performed inplace. Parameters ---------- x : array_like Query vectors, shape (n, d) where d is appropriate for the index. `dtype` must be float32. r%r&N)r(r)rrtrain_inplace_cr r>s rreplacement_train_inplacez.replacement_train_inplaceIsUw1DF{{{{  ) 4 4 4 Q ,,,,,r train_inplacer9)rr7s rhandle_IndexRowwiseMinMaxr9Hs,---$9o/HIIIIIrcVd}d}t|d|t|d|dS)Nc|j|jfksJ|j\}}||jksJd|cxkr||jzksnJ|t ||t j|dS)Nr)r(rF block_sizenvecpack_1_cr r)r+r,offsetblocknblockr<s rreplacement_pack_1z-handle_CodePacker..replacement_pack_1`sw4>+++++"[ T_,,,,F3333Z$)3333333 hqkk65>%+@+@AAAAArc|j\}}||jksJd|cxkr||jzksnJtj|jd}|tj||t||S)Nrr r&) r(r<r=rzerosrF unpack_1_crr )r+r@r?rAr<r,s rreplacement_unpack_1z/handle_CodePacker..replacement_unpack_1gs"[ T_,,,,F3333Z$)3333333 HT^7 3 3 3 u--vx{{CCCrpack_1unpack_1r9)rrBrFs rhandle_CodePackerrI^sQBBB9h(:;;;9j*>?????rcVd}d}t|d|t|d|dS)Nc|j\}|f|jksJ||t|t|dSr8)r(r^r )r+rvalsr/s rreplacement_map_addz0handle_MapLong2Long..replacement_map_addysH Zttz!!!! 1htnnhtnn55555rc|j\}tj|d}||t |t ||Sr)r(rrEsearch_multiple_cr )r+rr/rLs rreplacement_map_search_multiplez.replacement_map_search_multiple~sH Zx))) q(4..(4..AAA rrsearch_multipler9)rrMrPs rhandle_MapLong2LongrRwsY666  9e%89999/244444rcnt|ds |g|_dS|j|dS)Nreferenced_objects)hasattrrTappend)r+refs radd_to_referenced_objectsrXsB 4- . .,#&% &&s+++++rc$eZdZdZdZdZdZdS)RememberSwigOwnershipaJ SWIG's seattr transfers ownership of SWIG wrapped objects to the class (btw this seems to contradict https://www.swig.org/Doc1.3/Python.html#Python_nn22 31.4.2) This interferes with how we manage ownership: with the referenced_objects table. Therefore, we reset the thisown field in this context manager. c||_dSr8)obj)r+r\s rr#zRememberSwigOwnership.__init__s rcdt|jdr|jj|_dSd|_dS)Nthisown)rUr\r^ old_thisownrs r __enter__zRememberSwigOwnership.__enter__s7 48Y ' ' $#x/D   #D   rc:|j|j|j_dSdSr8)r_r\r^)r+ignoreds r__exit__zRememberSwigOwnership.__exit__s&   '#/DH    ( 'rN)r __module__ __qualname____doc__r#r`rcrrrZrZsK$$$ 00000rrZc2|j|_d}||_dS)a  this wrapper is to enable initializations of the form SearchParametersXX(a=3, b=SearchParamsYY) This also requires the enclosing class to keep a reference on the sub-object, since the C++ code assumes the object ownwership is handled externally. ch||D]\}}t||sJt|5t |||dddn #1swxYwYt |t tttfvrt||dSr8) r!itemsrUrZrrrrboolstrrX)r+argsrivs rr"z1handle_SearchParameters..replacement_inits JJLL 3 3DAq4## # # #&q)) $ $a### $ $ $ $ $ $ $ $ $ $ $ $ $ $ $AwwsE4555)$222  3 3sA--A1 4A1 Nr#r!)rr"s rhandle_SearchParametersrps.(0I333*IrTc<|j|_fd}||_dS)Nct|dkrQ|\}rtj|d}t|tj|f}st |||j|dSr)rrrrr rXr!)r+rmsubset class_owns force_int64s rr"z1handle_IDSelectorSubset..replacement_inits t99>>GF E-fGDDDKK!7!78D 8)$777D!!!!rro)rrtrur"s `` rhandle_IDSelectorSubsetrvs='0I " " " " " "*Irc0dd}t|d|dS)Nc.|j\}}||jksJtj|tj}|tj|t }n|j|fksJ||t|t||Sr) r(r)rrr rErkinsert_cr )r+rinsertedr/r)s rreplacement_insertz*handle_CodeSet..replacement_inserts{1DF{{{{$U"(;;;  x...HH>aU**** a%(8*<*<===rinsertr8r9)rr{s rhandle_CodeSetr}s1    9h(:;;;;;rcF|j|_d}d}||_||_dS)Nct|dkrd|\}|j\}}|||tjt j||jdS|j|dSr') rr(r!rcopy_array_to_vectorrrravelrn)r+rmarrayr/r)s rr"z)handle_Tensor2D..replacement_inits t99>>FE;DAq   q! $ $ $  &$U++1133TV = = = = = D  % % % %rctjdtj}tjtj||j|jtj|j  |d|dS)NrRr&rrS) rrDrTrmemcpyr r(nbytesvector_to_arrayrnreshape)r+r(s rnumpyzhandle_Tensor2D..numpyse"(+++ U^E**DJ EEE$TV,,44U1XuQxHHHr)r#r!r)rr"rs rhandle_Tensor2DrsC'0I&&&III *IIOOOrc`j_fd}d}d}|_|_|_dS)Nct|dks|djkr |j|dS|d}||j|j||dSNrSr)r __class__r!num_embeddings embedding_dim from_torch)r+rmembrs rr"z*handle_Embedding..replacement_initsu t99>>T!W.);; D  % % F1g 3-s/@AAA rc|jj|j|jfksJt jt j|jj |jdS)z# copy weights from torch.Embedding N) weightr(rrrrrrdatar)r+rs rrz$handle_Embedding..from_torch sizD$79K#LLLLL "   1 1 7 7 9 94; H H H H Hrc|j|j|jfksJtjt j||jdSz copy weights from numpy array N) r(rrrrrrrr)r+rs r from_arrayz$handle_Embedding..from_arrays]{t2D4FGGGGG "   ' ' - - / / > > > > >rr#r!rrrr"rrs` rhandle_Embeddingrsh'0IHHH >>> &I%I)Ircbj_fd}d}dd}|_|_|_dS)Nct|dks|djkr |j|dS|d}|jdu}||j|j|||dSr)rrr!bias in_features out_featuresr)r+rmlinearrrs rr"z'handle_Linear..replacement_inits t99>>T!W.);; D  % % Fa{$& 6-v/BDIII rc|jj|j|jfksJt j|jj|j|j P|j j|jfksJt j|j j|j dSdS)z copy weights from torch.Linear N) rr(rrrrrrrr)r+rs rrz!handle_Linear..from_torch's}"t'8$:J&KKKKK " M  $ $ & & , , . .  = = = ; ";$):(<<<<<  &v{'7'='='?'? K K K K K # "rc|j|j|jfksJtjt j||j|/|j|jfksJtj||j dSdSr) r(rrrrrrrrr)r+rrs rrz!handle_Linear..from_array0s{t0$2BCCCCC "   ' ' - - / / > > >  :$"3!55555  &tTY 7 7 7 7 7  rr8rrs` r handle_Linearrsm'0I     LLL8888*I%I%IrcLj_fd}d}|_|_dS)Nct|dks|djkr |j|dS|d}||j|j|j|j||dSr)rrr!r)rXLhr)r+rmsteprs rr"z*handle_QINCoStep..replacement_initDs{ t99>>T!W.);; D  % % FAw 46464646::: rc|j|j|j|jf|j|j|j|jfksJ|j|j|j|jt|jD]d}|j|}| |}|j |d|j |dedS)z# copy weights from torch.QINCoStep rrRN) r)rXrrcodebookr MLPconcatrangeresidual_blocksget_residual_blocklinear1linear2)r+rlsrcdests rrz$handle_QINCoStep..from_torchMs/DFDFDFDF3SSSSS   /// !!$.111tv , ,A&q)C**1--D L # #CF + + + L # #CF + + + +  , ,rr#r!rrr"rs` rhandle_QINCoSteprAsN'0I , , ,*I%IrcLj_fd}d}|_|_dS)Nct|dks|djkr |j|dS|d}||j|j|j|j|j||dSr) rrr!r)rXrMrr)r+rmqincors rr"z&handle_QINCo..replacement_init`s t99>>T!W.);; D  % % FQ 57EGUWeguwGGG rcf|j|j|j|j|jf|j|j|j|j|jfksJ|j|jt|jdz D]5}|||j |6dS)z copy weights from torch.QINCo rSN) r)rXrrr codebook0rrget_stepsteps)r+rr s rrz handle_QINCo..from_torchjsWeguw 9 VTVTVTVTV 4 5 5 5 5 !!%/222uw{## 8 8A MM!   ' ' A 7 7 7 7 8 8rrrs` r handle_QINCor]sN'0I888*I%Ir)F)T))inspectrrr faiss.loaderrrrrrrr r r rr"r:rArOr[rrrrrr$r*r3r9rIrRrXrZrprvr}rrrrrrgrrrss                        8''' * * * * 6J6J6JrFFF$9<9<9>>***(((&&&JJJ,@@@2444.,,, 00000000.********"<<<,,***:!&!&!&N&&&8&&&&&r