Toward a Neural-Symbolic Framework for Automated Workflow Analysis in Surgery

Hirenkumar Nakawala, Elena De Momi, Roberto Bianchi, Michele Catellani, Ottavio De Cobelli, Pierre Jannin, Giancarlo Ferrigno, Paolo Fiorini

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Learning production rules from continuous data streams, e.g. surgical videos, is a challenging problem. To learn production rules, we present a novel framework consisting of deep learning models and inductive logic programming (ILP) system for learning surgical workflow entities that are needed in subsequent surgical tasks, e.g. “what kind of instruments will be needed in the next step?” As a prototypical scenario, we analyzed the Robot-Assisted Partial Nephrectomy (RAPN) workflow. To verify our framework, first consistent and complete rules were learnt from the video annotations which can classify RAPN surgical workflow and temporal sequence at high-granularity e.g. steps. After we found that RAPN workflow is hierarchical, we used combination of learned predicates, presenting workflow hierarchy, to predict the information on the next step followed by a classification of step sequences with deep learning models. The predicted rules on the RAPN workflow was verified by an expert urologist and conforms with the standard workflow of RAPN.

Original languageEnglish
Title of host publication15th Mediterranean Conference on Medical and Biological Engineering and Computing – MEDICON 2019 - Proceedings of MEDICON 2019
EditorsJorge Henriques, Paulo de Carvalho, Nuno Neves
PublisherSpringer
Pages1551-1558
Number of pages8
ISBN (Print)9783030316341
DOIs
Publication statusPublished - Jan 1 2020
Event15th Mediterranean Conference on Medical and Biological Engineering and Computing, MEDICON 2019 - Coimbra, Portugal
Duration: Sep 26 2019Sep 28 2019

Publication series

NameIFMBE Proceedings
Volume76
ISSN (Print)1680-0737
ISSN (Electronic)1433-9277

Conference

Conference15th Mediterranean Conference on Medical and Biological Engineering and Computing, MEDICON 2019
CountryPortugal
CityCoimbra
Period9/26/199/28/19

    Fingerprint

Keywords

  • Deep learning
  • Inductive logic programming
  • Production rules
  • Robot-Assisted Partial Nephrectomy
  • Surgical workflow

ASJC Scopus subject areas

  • Bioengineering
  • Biomedical Engineering

Cite this

Nakawala, H., De Momi, E., Bianchi, R., Catellani, M., De Cobelli, O., Jannin, P., Ferrigno, G., & Fiorini, P. (2020). Toward a Neural-Symbolic Framework for Automated Workflow Analysis in Surgery. In J. Henriques, P. de Carvalho, & N. Neves (Eds.), 15th Mediterranean Conference on Medical and Biological Engineering and Computing – MEDICON 2019 - Proceedings of MEDICON 2019 (pp. 1551-1558). (IFMBE Proceedings; Vol. 76). Springer. https://doi.org/10.1007/978-3-030-31635-8_192