tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[12] = 12 is not in [0, 0)

[magedhelmy1]

I am trying to write an alternative to xml_to_csv.py for TFRecord as I am using VGG Image Annotator and extracted the annotations as JSON.

Here is a sample JSON file

{
  "0.jpg59329": {
    "filename": "0.jpg",
    "size": 59329,
    "regions": [{
      "shape_attributes": {
        "name": "rect",
        "x": 412,
        "y": 130,
        "width": 95,
        "height": 104
      },
      "region_attributes": {}
    }, {
      "shape_attributes": {
        "name": "rect",
        "x": 521,
        "y": 82,
        "width": 126,
        "height": 106
      },
      "region_attributes": {}
    }
}

With VGG Annotator, you have a directory for images and a directory for annotation in separate folders.

Here is my code for creating TF records:

# Ref 1: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md
# Ref 2: https://github.com/datitran/raccoon_dataset/blob/master/generate_tfrecord.py


import json
import glob
from object_detection.utils import dataset_util
import tensorflow as tf
from pathlib import Path

flags = tf.compat.v1.app.flags
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
FLAGS = flags.FLAGS


def json_to_tf(jsonFile, im):
    with open(im, "rb") as image:
        encoded_image_data = image.read()

    with open(jsonFile) as json_file:
        data = json.load(json_file)

        for key, value in data.items():
            width = 1920
            height = 1080
            filename = value["filename"]
            filename = filename.encode('utf8')
            image_format = b'jpeg'
            xmins = []
            xmaxs = []
            ymins = []
            ymaxs = []
            classes_text = []
            classes = []

            for x in value["regions"]:
                xmins.append(x["shape_attributes"]['x'])
                xmaxs.append(x["shape_attributes"]['width'] + x["shape_attributes"]['x'])
                ymins.append(x["shape_attributes"]['y'])
                ymaxs.append(x["shape_attributes"]['height'] + x["shape_attributes"]['y'])
                classes_text.append("car".encode('utf8'))
                classes.append(1)

            tf_example = tf.train.Example(features=tf.train.Features(feature={
                'image/height': dataset_util.int64_feature(height),
                'image/width': dataset_util.int64_feature(width),
                'image/filename': dataset_util.bytes_feature(filename),
                'image/source_id': dataset_util.bytes_feature(filename),
                'image/encoded': dataset_util.bytes_feature(encoded_image_data),
                'image/format': dataset_util.bytes_feature(image_format),
                'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
                'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
                'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
                'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
                'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
                'image/object/class/label': dataset_util.int64_list_feature(classes),
            }))
            writer.write(tf_example.SerializeToString())

writer = tf.compat.v1.python_io.TFRecordWriter("train.record")

for fn in glob.glob("..\\annotation_refined\\*.json"):
    for img in glob.glob("..\\images\\*.jpg"):
        if Path(fn).stem == Path(img).stem:
            tf_example_1 = json_to_tf(fn, img)

writer.close()

The error I get:

tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: : The system cannot find the path specified.
; No such process

I am not sure if it is related to the tf_record or something else, any ideas @abdelrahman-gaber ?

[magedhelmy1]

I realized this problem was due to passing incorrect filepaths, however after fixing that I get

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[12] = 12 is not in [0, 0)

I believe something is wrong in my JSON to TF records conversion but I am not sure what.