Visual Question Answering for Response Synthesis Based on Spatial Actions

The paper considers the automatic analysis problem of a user’s natural language query from an image. The mechanism synthesizes a logically correct non-binary response. Synthesis is carried out on the basis of combining the results of convolutional and recurrent networks and projection on a set of valid answers. A three-dimensional data set has been developed to search for an answer in a complex environment using a robotic arm. Similar systems examples and their comparison are given. The experiments results showed that our method is able to achieve indicators comparable with known models. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.