Notfittederror: Tfidfvectorizer - Vocabulary Wasn't Fitted

I am trying to build a sentiment analyzer using scikit-learn/pandas. Building and evaluating the model works, but attempting to classify new sample text does not. My code: import

Solution 1:

You've fitted a vectorizer, but you throw it away because it doesn't exist past the lifetime of your vectorize function. Instead, save your model in vectorize after it's been transformed:

self._vectorizer = vectorizer

Then in your classify function, don't create a new vectorizer. Instead, use the one you'd fitted to the training data:

input_counts = self._vectorizer.transform(input_text)

Solution 2:

Save vectorizer as a pickle or joblib file and load it when you want to predict.

pickle.dump(vectorizer, open("vectorizer.pickle", "wb")) //Save vectorizer
pickle.load(open("models/vectorizer.pickle", 'rb'))     // Load vectorizer

Solution 3:

You can save both the model and the vectorizer and use them later on as well: here is how I did it:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.svm import LinearSVC
import pickle


# Train the classification modeldeftrain_model():
    df = pd.read_json('intent_data.json')

    X_train, X_test, y_train, y_test = train_test_split(df['Utterance'], df['Intent'], random_state=0)

    count_vect = CountVectorizer()
    X_train_counts = count_vect.fit_transform(X_train)
    tfidf_transformer = TfidfTransformer()
    X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

    model = LinearSVC().fit(X_train_tfidf, y_train)

    # Save the vectorizer
    vec_file = 'vectorizer.pickle'
    pickle.dump(count_vect, open(vec_file, 'wb'))

    # Save the model
    mod_file = 'classification.model'
    pickle.dump(model, open(mod_file, 'wb'))


# Load the classification model from disk and use for predictionsdefclassify_utterance(utt):
    # load the vectorizer
    loaded_vectorizer = pickle.load(open('vectorizer.pickle', 'rb'))

    # load the model
    loaded_model = pickle.load(open('classification.model', 'rb'))

    # make a predictionprint(loaded_model.predict(loaded_vectorizer.transform([utt])))