"one_hot": Encodes each individual element in the input into anĪrray the same size as the vocabulary, containing a 1 at the element "int": Return the raw integer indices of the input tokens. Or "tf_idf" configuring the layer as follows: Values can be "int", "one_hot", "multi_hot", "count", output_mode: Specification for the output of the layer.Map indices to vocabulary items instead of mapping vocabulary items to invert: Only valid when output_mode is "int"."tf_idf", this argument must be supplied. If the vocabulary argument is set, and output_mode is Will be multiplied by per sample term counts for the final tf_idf A tuple, list,ġD numpy array, or 1D tensor or the same length as the vocabulary,Ĭontaining the floating point inverse document frequency weights, which idf_weights: Only valid when output_mode is "tf_idf".This argument is set, there is no need to adapt() the layer. Path, the file should contain one line per term in the vocabulary. Or 1D tensor containing the string vocbulary terms. If passing an array, can pass a tuple, list, 1D numpy array, Either an array of strings or a string path to a oov_token: Only used when invert is True.Instances of the mask token in the input will be dropped. Other output modes, the token will not appear in the vocabulary and "int", the token is included in vocabulary and mapped to index 0. mask_token: A token that represents masked inputs.If this value is 0, OOV inputs will cause an error when calling Value is more than 1, OOV inputs are hashed to determine their OOV num_oov_indices: The number of out-of-vocabulary tokens to use.Note that this size includes the OOV and mask tokens. If None, there is no cap on the size of the Only be specified when adapting the vocabulary or when setting max_tokens: Maximum size of the vocabulary for this layer.
FILE EXTENSION LOOKUP FULL
OOV indices and instances of the mask token will be dropped.įor an overview and full list of preprocessing layers, see the preprocessing Is "multi_hot", "count", or "tf_idf" the vocabulary will begin with Is "int", the vocabulary will begin with the mask token (if set), followedīy OOV indices, followed by the rest of the vocabulary. The position of these tokens in the vocabulary is fixed. (which can optionally occupy multiple indices in the vocabulary, as set The vocabulary can optionally contain a mask token as well as an OOV token When output_mode is "multi_hot", "count", or "tf_idf", input stringsĪre encoded into an array where each dimension corresponds to an element in Input strings are converted to their index in the vocabulary (an integer). There are two possible output modes for the layer. Tokens will be used to create the vocabulary and all others will be treated If the vocabulary is capped in size, the most frequent During adapt(), the layer will analyze a data set,ĭetermine the frequency of individual strings tokens, and create a
The vocabulary for the layer must be either supplied on construction or Natural language, see the TextVectorization layer. This layer translates a set of arbitrary strings into integer output via a
StringLookup ( max_tokens = None, num_oov_indices = 1, mask_token = None, oov_token = "", vocabulary = None, idf_weights = None, encoding = "utf-8", invert = False, output_mode = "int", sparse = False, pad_to_max_tokens = False, ** kwargs )Ī preprocessing layer which maps string features to integer indices.
0 kommentar(er)
