{"id":9677,"date":"2024-04-30T17:25:47","date_gmt":"2024-04-30T15:25:47","guid":{"rendered":"https:\/\/dev.29forward.com\/bez-kategorii\/generating-interpretation-of-graphs-in-scientific-articles-using-deep-learning"},"modified":"2024-08-15T10:27:00","modified_gmt":"2024-08-15T08:27:00","slug":"generating-interpretation-of-graphs-in-scientific-articles-using-deep-learning","status":"publish","type":"post","link":"https:\/\/29forward.com\/pl\/artykul-specjalistyczny\/generating-interpretation-of-graphs-in-scientific-articles-using-deep-learning","title":{"rendered":"Generating Interpretation of Graphs in Scientific Articles Using Deep Learning"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Graue\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Home<\/a><\/span><\/span><\/p>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Artyku\u0142 specjalistyczny<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Deep Learning<\/h1>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tGenerating Interpretation of Graphs in Scientific Articles Using Deep Learning\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Introduction Motivation
<\/strong>Data is exponentially growing worldwide. 90 % of it has been generated in the recent two years alone1<\/sup>. This trend will continue to manifest in the same speed in the next few years and will generate vast amounts of data unimaginable to the human brain. This inevitably makes us wonder how we will deal with data in the future and what it will look like.<\/p>\n

Balnojan2<\/sup>, as many other academics and data specialists, predicts that a lot of the data, we will be working with in 6 years\u2019 time, will be in the image form. The scope of this data will mainly be to provide consumers an easy way to find desired products online through images on their mobile devices. We can make sense of such image data through computer vision and deep learning technologies. These technologies have improved immensely in the recent years \u2013 they even surpass human vision in many cases. This means, that we are probably in the right path to handling this vast amount of data in real time through AI-technologies.<\/p>\n

Computer Vision\u2019s most impressive use cases range from cancer detection in medicine, performance assessment in sports, to autonomous vehicles3<\/sup>. Inspired by the advancement in the deep learning technologies that enable us to improve our lives, I too, would like to report on an interesting use case of deep learning technologies on image data. <\/p>\n

This use case came about as part of my master thesis at the Berlin School of Economics and Law<\/a>. Similar to the aforementioned exponential growth of data, the number of scientific papers published worldwide is also growing rapidly4<\/sup>. Thus, academics face the challenge of an exhaustive literature review. In fact, this is not a challenge anymore, but has become a nearly impossible task. When working on scientific research, we need to make sure that we go through all the available scientific work, and as a result differentiate our work from what has already been contributed. This becomes even harder when we take into consideration that only approximately 30 % of the delivered results on online libraries, are relevant to the searched topic.<\/p>\n

This is where the CauseMiner5<\/sup> software comes to help. It is a software, developed by M\u00fcller and H\u00fctteman5<\/sup> that uses natural language processing, linguistic rules, and text mining to analyse the contents of scientific articles and finally extract the main ideas behind the research. This way, researchers can quickly decide whether a scientific paper is relevant and thus worth their time.<\/p>\n

But the main hypothesis or ideas behind scientific work are not only expressed through text, often researchers also provide visual graphs that concisely express the main hypothesis and their relationships through nodes and edges (see figure 1). These graphs are also known as graphical research models. Having the contents of these graphs as an output of the CauseMiner Software would be a good extension. This was the goal of the master thesis, to take the image data of these graphical research models and extract the information utilizing deep learning models.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\u2013\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Figure 1: Example of a graphical research model retrieved from 6<\/ssup><\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Methodology<\/strong>
In this article, I would like to show step by step, how you can utilize image data to extract insightful information from it through deep learning.<\/p>\n

The main goal of the thesis was to assess the capabilities of deep learning technologies to extract the structure of graphical research models in image form. I would like to point out that we are only interested in the structure of these graphs. We can extract the text provided in them through other computer vision technologies, such as Tesseract7<\/sup>.<\/p>\n

I needed image data extracted from research papers. Clark and Divvala8<\/sup> have developed a framework that extracts all image data from a corpus of papers. This provides us with a good amount of \u201creal\u201d graphical research models. I had to go through all these images and extract those that could be classified as graphical research models. These images then would become part of the dataset that I used to run my deep learning model on.<\/p>\n

Unfortunately, my dataset was not ready for the deep learning models yet. While doing my research, I found out that the best deep learning models out there, that could extract information from the graphs in text form, were image captioning models<\/em> and instance segmentation models<\/em>.<\/p>\n

Image captioning models generate a textual description of images. For this I could generate the needed data using a Python library: Graphviz9<\/sup>. instead of writing a description to every image myself.<\/p>\n

For the image segmentation model, on the other hand, I could not generate any data and had to annotate it myself. Instance Segmentation Models are deep learning models that detect objects on images and generate masks on top of them. In other words, they assign every pixel to a class. I decided to detect only a few classes as the amount of data available was limited. The classes were nodes, edges (lines), arrows, and edge labels. This was done with an open-source tool, the VGG Image Annotator10<\/sup>.<\/p>\n

After having my datasets ready, I was finally free to experiment with deep learning models. As an image captioning model, I chose to use a captioning model with semantic attention developed by Xu et al.11<\/sup>. This approach turned out to not be successful with my data. Therefore, I would like to not further dive into explanations on this, but would like to point out, that this technology is not able to generate the structure of graphical research models.<\/p>\n

The instance segmentation model, on the other hand, delivered very promising results. I utilized Mask R-CNN12<\/sup>, which is an extension of an object detection model Faster R-CNN12<\/sup>. I chose this model as it was one of the state-of-the-art solutions by the time I was working on the thesis and due to all the literature available on it. <\/p>\n

A short explanation of the Mask R-CNN model architecture:
<\/strong>An image is processed as follows: firstly, it is passed to a Convolutional Neural Network (CNN), the backbone network for feature extraction. Convolutional Neural Networks are multi-layer neural networks, that are applied to visual imagery and can learn local patterns or features in images that they can later recognize again in different locations13<\/sup>. The extracted features are fed to the Region Proposal Network, which creates anchor boxes that contain potential objects to detect. The ROI Align layers maintains the spatial orientation. Then, the fully connected network (FCN), processes the proposed regions and passes them to two different fully connected layers for object detection and bounding box refinement. The same output is processed in parallel by the Mask R-CNN branch, which generates the segmentation masks14<\/sup>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"generated\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Figure 2: The architecture of Mask R-CNN14<\/sup><\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I trained the model on 245 annotated images of graphs, or 435 real and augmented images. I was able to achieve an overall accuracy of 87 %. This was a very satisfying result when we consider the limited amount of training data available.<\/p>\n

In figure 3 we demonstrate the tasks performed by Mask R-CNN, that is object detection, semantic segmentation, and instance segmentation.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Graphs\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Figure 3: Tasks performed my Mask R-CNN (Object detection, semantic segmentation, instance segmentation)<\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As you might have already noticed, this deep learning model, would only deliver me the objects with their segmentation masks and classes, not with the desired graph structure. I could, however, use the information provided to me by Mask R-CNN to generate the graph structures (see figure 4)<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"h\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Figure 4: An annotated graph to generate the structure of graph retrieved from 15<\/sup><\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWith the segmentation masks provided by the model, one could tell just by looking at an image with the masks visualized on top, that these masks intersect with each other (see figure 4). This information can lead us to the structure of the graph: one node intersects with a line, the line with an arrow and so on. In Python we can use Shapely16<\/sup>, a package for \u201cmanipulation of planar geometric objects\u201d, to identify whether our generated segmentation masks intersect with each other. After obtaining shapely polygons out of the segmentation masks, we save them in a dictionary that contains a generated element id, the class name of the element, the polygons, and the cropped image of every node and edge label. We also generate a set of intersecting ID-Pairs using Shapely.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Grafik\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Figure 5: Annotated graph with polygons and their id-s retrieved from [14]<\/b><\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe use the cropped image of every node and edge label to pass it through Tesseract. Tesseract returns the text of every node and edge label. We save this information in the dictionary again. Now we have all the information we need for the structure generation of our graphical research models: the intersecting ID-Pairs, the classes, and the text on nodes and edge labels. Through the intersecting ID-Pairs, we can identify the structure of a graph. We can follow the path from one element to another. For example, if elements 1 and 2 intersect, but also elements 2 and 5 intersect, we can conclude that element 1 leads to element 2 and element 2 to element 5. This is how we come to the full structure of our graphs. However, developing an algorithm that does this properly, would be very time-consuming. Thus, we make use of Networkx17<\/sup>, a \u201cpython package for creation, manipulation, and the analysis of the structure of complex networks\u201c. With the help of Networkx, we can generate all possible paths in a graphical research model. All these possible paths are our structure.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Let\u2019s wrap up the main steps we took to come to our generated graph structures:
<\/strong><\/p>\n

    \n
  1. Generating instance segmentation masks with Mask R-CNN.<\/li>\n
  2. Converting the generated masks to polygons.<\/li>\n
  3. Creating dictionaries that contain the polygons, an identifying key, and their classes (node, arrow, edge label, line).<\/li>\n
  4. Using Tesseract to detect the text on every node and attaching this text to the corresponding element in the dictionary.<\/li>\n
  5. Determining which elements (polygons) intersect with each other. If two elements intersect with each other, then they are connected.<\/li>\n
  6. Creating id-pairs of intersecting elements and identifying all the possible network paths in the graph. These paths are the generated structure of the graphs<\/li>\n<\/ol>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    This is what we have retrieved from the graph shown in Figure 5:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"graph-to-generate-the-structure-of-graph\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    Figure 4: Generated Structure<\/h6>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Note that the order of line and arrow, defines the direction of the connection as shown through the red arrows on Figure 6.
    Now that we have finally extracted the information behind graphical research models, with a little refinement we can now add it to the CauseMiner Software.<\/p>\n

    Final Words
    <\/span><\/strong>We\u2019ve gone through all the steps of solving a problem through deep learning: from creating an appropriate dataset, to training a deep learning model and using its output to generate insightful information for us. I would like to point out, how powerful data can be and how much knowledge we can extract from it. With a little imagination, we can probably find many use cases for different datasets and who knows what we can achieve\u2026<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tReferences:<\/strong>\n
    1<\/sup> D. Reinsel, J. Gantz, J. Rydning,<\/a> „The Digitization of the World From Edge to Core” 2018. Retrieved May 5, 2022<\/h6>\n
    2<\/sup> S. Balnojan,<\/a> „The Future Of Good Data – What You Should Know Now!” 2020. Retrieved May 5, 2022<\/h6>\n
    3<\/sup> V. Meel,<\/a> „87 Most Popular Computer Vsion Application in 2022” 2022. Retrieved May 5, 2022<\/h6>\n
    4<\/sup> A. E. Jinha,<\/a> \u201cArticle 50 million: an estimate of the number of scholarly articles in existence,\u201d Learned Publishing, vol. 23, no. 3, pp. 258\u2013263, 2010.<\/h6>\n
    5<\/sup> R. M. Mueller and S. Huttemann,<\/a> \u201cExtracting Causal Claims from Information Systems \u00a8 Papers with Natural Language Processing for Theory Ontology Learning,\u201d in Proceedings of the 51st Hawaii International Conference on System Sciences, pp. 5295\u20135304, 2018.<\/h6>\n
    6<\/sup> V. Cova, V. Abbas,<\/a> „The cultural aspect in the relationship customer-place: Proposal and test of an integrated model” 2018. Retrieved May 5, 2022<\/h6>\n
    7<\/sup> A. Rosebrock,<\/a> \u201cOpenCV OCR and text recognition with Tesseract – PyImageSearch,\u201d 2018. Retrieved August 8, 2020,<\/h6>\n
    8<\/sup> C. Clark and S. Divvala,<\/a> \u201cPDFFigures 2.0: Mining figures from research papers,\u201d in Proceedings of the 16th ACM\/IEEE-CS on Joint Conference on Digital Libraries – JCDL \u201916, pp. 143\u2013152, ACM Press, 2016.<\/h6>\n
    9<\/sup> \u201cGraphviz<\/a> – graph visualization software.\u201d<\/h6>\n
    10<\/sup> A. Dutta and A. Zisserman,<\/a> \u201cThe VIA annotation software for images, audio and video,\u201d in Proceedings of the 27th ACM International Conference on Multimedia, pp. 2276\u2013 2279, ACM.<\/h6>\n
    11<\/sup> K. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, \u201cShow, attend and tell: Neural image caption generation with visual attention,\u201d in Proceedings of the 32nd International Conference on International Conference on Machine Learning – Volume 37, ICML\u201915, p. 2048\u20132057, JMLR.org, 2015.<\/h6>\n
    12<\/sup> K. He, G. Gkioxari, P. Dollar, and R. Girshick,<\/a> \u201cMask R-CNN,\u201d in \u00b4 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980\u20132988.<\/h6>\n
    13<\/sup> F. Chollet, Deep learning with Python. New York: Manning Publications Co, 2018. OCLC: ocn982650571<\/h6>\n
    14<\/sup> J. Hui,<\/a> \u201cImage segmentation with mask R-CNN,\u201d 2019. Retrieved July 29, 2020<\/h6>\n
    15<\/sup> A. Azadegan and G. Kolfschoten,<\/a> \u201cAn assessment framework for practicing facilitator,\u201d Group Decision and Negotiation, pp. 1013\u2014-1045, 2012.<\/h6>\n
    16<\/sup> \u201cShapely<\/a> \u2014 shapely 1.8dev documentation.\u201d<\/h6>\n
    17<\/sup> \u201cNetworkX<\/a> \u2014 NetworkX documentation.\u201d<\/h6>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tWszystkie artyku\u0142y<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t
    \n\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    Masz pytania, chcia\u0142by\u015b om\u00f3wi\u0107 z nami sw\u00f3j projekt lub szukasz wsparcia technicznego? Z niecierpliwo\u015bci\u0105 czekamy na rozmow\u0119.<\/h5>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Um\u00f3w si\u0119 na spotkanie ju\u017c teraz<\/h2>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t
    \n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t