Do not fit transformers on testing data

[suhaibmujahid]

Currently, we fit the transformers before splitting the data, which could leak testing data into the training data, especially when using something like TF-IDF in the extraction pipeline:

Transforming:

bugbug/bugbug/model.py

Line 352 in 02906a6

X = self.extraction_pipeline.fit_transform(X_gen)

Splitting:

bugbug/bugbug/model.py

Line 368 in 02906a6

X_train, X_test, y_train, y_test = self.train_test_split(X, y)

[suhaibmujahid]

Moving transformers that require fitting to the clf pipeline could fix the leaking issue:

bugbug/bugbug/models/defect.py

Lines 313 to 341 in 49e79c8

	self.clf = Pipeline(
	[
	(
	"tokenizer",
	HuggingfacePretrainedTokenizer(
	"distilbert-base-uncased", max_length=512
	),
	),
	(
	"classifier",
	NeuralNetClassifier(
	DistilBertModule,
	module__name="distilbert-base-uncased",
	module__num_labels=2,
	module__last_layer_only=last_layer_only,
	optimizer=torch.optim.AdamW,
	lr=6e-5,
	max_epochs=2,
	criterion=nn.CrossEntropyLoss,
	batch_size=4,
	iterator_train__shuffle=True,
	device=get_training_device(),
	callbacks=[
	ProgressBar(),
	],
	),
	),
	]
	)

However, with this solution, the extraction pipeline can no longer be treated as a transformers pipeline (transformers should support fit and transform).

[suhaibmujahid]

The ideal solution for now will be to have two pipelines, one for extracting, and one for transforming. The extraction pipeline should not allow using fit().