Using python package to build your custom dataset — MLDatasetBuilder

As an ML Newbie, I need to figure out the best way to prepare the dataset for our machine learning training model. As per my last article, I came up with a Python package for this process!

Whenever you are training a custom model the important thing is the dataset. Yes, of course, the dataset plays the main role in deep learning. The accuracy of your model will be based on the dataset. So, before you train a custom model, you need to plan how to build a dataset? Here, I’m going to share my ideas on the easy way to build your dataset.

MLDatasetBuilder-Version 1.0.0

A Python package to build Dataset for Machine Learning Whenever we begin a machine learning project, the first thing that we need is a dataset. Datasets will be the pillar of the training model. You can build the dataset either automatically or manually. MLDatasetBuilder is a python package which is helping to prepare the image for your ML dataset.

Installation

We can install MLDatasetBuilder package using the below command

pip install MLDatasetBuilder

How to test

When you run python3 in the terminal, it will produce output like this:

Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Run the following code to you can get the Initialize process output for the MLDatasetBuilder package.

>>> from MLDatasetBuilder import *
>>> MLDatasetBuilder()

Available Operations

PrepareImage — Remove unwanted format images and Rename your images

#PrepareImage(folder_name, image_name)
PrepareImage('images', 'dog')

ExtractImages — Extract images from video file

#ExtractImages(video_path, file_name, frame_size)
ExtractImages('video.mp4', 'frame', 10)
#OR
#ExtractImages(video_path, filename)
ExtractImages('video.mp4', 'frame')
#Default FPS will be 5

Step1 — Get images from google

Yes, we can get images from Google. Using the Download All Images browser extension we can easily get images in a few minutes. You can check out here for more details about this extension!

Step2 — Create a Python file

Once you have downloaded the images using this extension, you can create a python file called test.py the same directory as below.

download_image_folder/
|   _14e839ba-9691-11ea-a968-2ed746e9a968.jpg
|   5e5f7af12600004018b602c0.jpeg
|   A471529_Alice_b-1.jpg
|   image1.png
|   image2.png
|   ...
test.py

Inside the images folder, you can see lots of PNG images and random filenames.

Step3 — PrepareImage

MLDatasetBuilder provides a method called PrepareImage. Using this method to we can remove the unwanted images and rename your image files which are already you have downloaded from the browser’s extensions.

PrepareImage(folder_path, class_name)
#PrepareImage('download_image_folder', 'dog')

As per the above code, we need to mention the image folder path and class name.

After completing the process your image folder structure will look like below

download_image_folder/
|   dog_0.jpg
|   dog_1.jpg
|   dog_2.jpg
|   dog_3.png
|   dog_4.png
|   ...
test.py

This process very helps to annotate your images while labeling. And of course, it will be like one of the standardized things.

Step4 — ExtractImage

MLDatasetBuilder also provides a method called ExtractImages. Using this method we can extract the images from the video files.

download_image_folder/
video.mp4
test.py

As per the below code, we need to mention the video path, folder name, and framesize. Folder name will the class name and framesize’s default value 5 and it’s not mandatory.

ExtractImages(video_path, folder_name, framesize)
#ExtractImages('video.mp4', 'frame', 10)
ExtractImages(video_path, folder_name)
#ExtractImages('video.mp4', 'frame')

After completing the process your image folder structure will look like below

download_image_folder/
dog/
|   dog_0.jpg
|   dog_1.jpg
|   dog_2.jpg
|   dog_3.png
|   dog_4.png
|   ...
dog.mp4
test.py

What is version 2.0.0?

I have planned to release version 2.0.0 on next month, This will include some additional features.

I mean this package will provide more than 100 objects images with annotations file :)

Contributing

All issues and pull requests are welcome! To run the code locally, first, fork the repository and then run the following commands on your computer:

git clone https://github.com/<your-username>/ML-Dataset-Builder.git
cd ML-Dataset-Builder
# Recommended creating a virtual environment before the next step
pip3 install -r requirements.txt

When adding code, be sure to write unit tests where necessary.