| 摘要: |
| 为了解决现有视频问答模型认知推理能力不足的问题,引入旁观者记忆模块,提出了基于多角度融合与联合记忆网络的机器认知模型.该模型根据问题定位目标对象,获得视频中对应的区域特征,同时联合视频的运动特征和外观特征,通过加入时间注意力机制的门控循环单元,有效地融合问题特征和视频特征,用于答案的生成,以提高模型认知推理能力.实验结果表明：相比于现有的视频问答模型,该模型的准确率更高,尤其对于推理难度较大的信念推理问题,该模型体现出了更好的推理能力及泛化性能. |
| 关键词: 认知推理 注意力机制 记忆网络 视频问答 |
| DOI：10.20192/j.cnki.JSHNU（NS）.2024.05.003 |
| 分类号:TP301 |
| 基金项目:国家自然科学基金（6210020445）；上海市自然科学基金（21ZR1446900,21511100102） |
|
| A cognitive model of video QA based on multi-angle fusion and joint memory network |
|
NI Qin1, LIU Shuang2, YU Yangze2, LIN Xin3, DENG Ciping4
|
|
1.School of Education, Shanghai International Studies University, Shanghai 201620, China;2.College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai 201418, China;3.School of Computer Science and Technology, East China Normal University, Shanghai 200062, China;4.The School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China
|
| Abstract: |
| In order to solve the problem of insufficient cognition and reasoning ability in existing video question answering models, an observer memory module was introduced, and a machine cognition model based on multi-angle fusion and joint memory network was proposed. The target object was located based on the problem and the corresponding regional features in the video were obtained by this model. At the same time, the motion and appearance features of the video were combined. By adding a gated loop unit with time attention mechanism, the problem features and video features were integrated more effectively for answer generation, which improved the model’s cognitive reasoning ability. The experimental results showed that compared to existing video QA models, this model had higher accuracy, which demonstrated better reasoning ability and generalization ability especially for belief reasoning problems with greater difficulty in cognitive reasoning task. |
| Key words: cognitive reasoning attention mechanism memory network video QA |
