DataSet Utils

used to export the detected data which can used for retrain/fine tune the model

class eyewitness.dataset_util.BboxDataSet(dataset_folder, dataset_name, valid_labels=None)

Bases: object

generate DataSet with same format as VOC object detections:

<dataset_folder>/Annotations/<image_name>.xml

<dataset_folder>/JPEGImages/<image_name>.jpg

<dataset_folder>/ImageSets/Main/trainval.txt

<dataset_folder>/ImageSets/Main/test.txt

convert_into_darknet_format()

dataset_iterator(with_gt_objs=True, mode='TEST_ONLY')

dataset_type

generate_train_test_list(overwrite=True, train_ratio=0.9)

generate train and test list

Parameters:

• overwrite (bool) -- if overwrite and file not exit will regenerate the train, test list
• train_ratio (float) -- the ratio used to sample train, test list, should between 0~1

get_selected_images(mode='TEST_ONLY')

get_valid_labels()

ground_truth_iterator(selected_images)

ground_truth interator

Parameters:mode (str) -- the mode to iterate the dataset Returns:gt_object_generator -- ground_truth_object generator, with first item if the ImageId Return type:Generator[(ImageId, List[BoundedBoxObject])]

image_obj_iterator(selected_images)

generate eyewitness Image obj from dataset

Parameters:mode (str) -- the mode to iterate the dataset Returns:image_obj_generator -- eyewitness Image obj generator Return type:Generator[eyewitness.image_utils.Image]

store_and_convert_darknet_bbox_tuples(dataset_file, selected_images, images_dir, labels_dir, label2idx, logging_frequency=100)

testing_set

training_and_validation_set

classmethod union_bbox_datasets(datasets, output_dataset_folder, dataset_name, filter_labels=None, remove_empty_labels_file=False)

union bbox datasets and copy files to the given output_dataset

valid_labels

the valid_labels in the dataset

eyewitness.dataset_util.add_filename_prefix(filename, prefix)

eyewitness.dataset_util.copy_image_to_output_dataset(filename, src_dataset, jpg_images_folder, anno_folder, file_fp, filter_labels=None, remove_empty_labels_file=False)

move annotation, jpg file from src_dataset to file destination, add prefix to filename and print to id list file

Parameters:

• filename (str) -- ori filename
• src_dataset (BboxDataSet) -- source dataset
• jpg_images_folder (str) -- destination jpg file folder
• anno_folder (str) -- destination annotation file folder
• file_fp -- the file pointer used to export the id list
• filter_labels (Optional[set[String]]) -- used for filtering label for the destination dataset

eyewitness.dataset_util.create_bbox_dataset_from_eyewitness(database, valid_classes, output_dataset_folder, dataset_name)

generate bbox dataset from eyewitness requires:

• FalseAlertFeedback table: remove images with false-alert feedback
• BboxDetectionResult: get images with selected classes objects

eyewitness.dataset_util.generate_etree_obj(image_id, detected_objects, dataset_name)

Parameters:

• image_id (str) -- image_id as filename
• detected_objects -- detected_objects obj from detected_objects table
• dataset_name (str) -- dataset_name

eyewitness.dataset_util.parse_xml_obj(obj)

eyewitness.dataset_util.read_ori_anno_and_store_filered_result(ori_anno_file, dest_anno_file, filter_labels, remove_empty_labels_file)

read the original annotation file, filter objects with valid labels export to the dest_anno_file

Parameters:

• ori_anno_file (str) -- original annotation file
• dest_anno_file (str) -- destination annotation file
• filter_labels (Optional[set[String]]) -- filter the labels
• remove_empty_labels_file (bool) -- remove the image if it don't have obj