Advertisement

Deep understanding of 3-D multimedia information retrieval on social media: implications and challenges

  • Ritika WasonEmail author
  • Vishal Jain
  • Gagandeep Singh Narula
  • Anupam Balyan
Original Article

Abstract

With the recent penetration and proliferation of social networks into our lives, human choices and preferences have become more socially accessible. This easy accessibility of private data in different formats has opened many new initiatives. The big explosion of multimedia data on the web has enabled social networks to gauge user likes, dislikes, and needs. This has imposed high demands on multimedia information retrieval (MIR) techniques. This manuscript illustrates the MIR concept in terms of its application to social media. It further positions the current research in the field of 3D MIR. Further it highlights the challenges in 3-D MIR on social media and finally translates them into significant research directions.

Keywords

Multimedia information retrieval (MIR) 3D multimedia information retrieval (3D MIR) Social media Multimedia data Deep learning 

1 Introduction

Advancements in technology are fast penetrating into our lives, digitizing our personalities in smart ways. Social networks (Twitter, Facebook, Snapchat, Instagram, etc.) today are proficient enough to learn our likes, dislikes, needs and preferences. The core idea behind this intelligence of social networks is an underlying neural network that accumulates data and utilizes it for other processes such as analysis and prediction [1, 2, 3, 4, 5, 6, 7, 8]. However, the intriguing question here is how do social networks manage to look into our private, less comprehensible, and concealed selves. The technique behind this novel intelligence of social networks is what is technically called deep learning [9]. The science of deep learning simply explained divides ideas as layers of definitions [10]. Hence, smaller concepts collectively abstract to larger ones and so on. Hence, with enough input information, a sufficiently layered neural net can learn quite deeply similar to the human brain [11, 12, 13].

Many varied deep learning technologies have been successfully applied onto social networks [3], [14, 15, 16]. This deep intelligence has been successfully applied to an almost infinite collection of consumer data accumulated by the world’s largest social networks for varied useful tasks [16, 17, 18, 19]. Table I below highlights the significant applications of deepnet onto popular and large social networks.

Table 1 above is by no means a complete essence of successful deep learning applications to gauge social network data. However, it gives a strong view of the fact that deep learning frameworks have been successfully applied for effective usage of multimedia big data available from social networks for different business domains [34].
Table 1

Significant deep learning applications to social networks

S.no.

Application

Description

Key research domain

1.

Social network search based on semantic analysis and learning [1]

Summarizes semantics of a complete social network search framework

Deep semantic learning of social networks

2.

Attributed social network embedding [20]

Summarizes a generic social network embedding (SNE) outline that learns representations for social actors while preserving attribute and structural proximity

Social network embedding

3.

Personality prediction system from Facebook users [2]

Summarizes a prediction system to automatically forecast a user’s personality through Facebook user information. successfully applies deep learning architectures

Personality prediction

4.

Deep learning for network analysis [21]

Discusses advances in application of deep learning frameworks for identification of patterns, characteristics, and anomalies in networks

Network anomaly detection

5.

Age-group classification in social networks using deep learning [4]

Applies deep convolutional networks to extract user age from the social network data and apply it for useful sentiment analysis

Age-group classification

6.

Ontology-based deep learning for human behavior prediction with explanations in health social networks [6]

Proposes ORBM+ to achieve human behavior prophecy through undirected and nodes-attributed graphs. The method is based on the illustrious deep learning-restricted Boltzmann machine concept

Behavior prediction

7.

Detecting spammers on social networks [7]

Increasing popularity and usage of social networks has also exposed significant incidences of spammers leading to user inconvenience. Proposes a SVM-based spammer detection algorithm for detecting spammers on Sina Weibo data

Spammer detection

8.

Cloud-based big data analytics framework for face recognition in social networks using machine learning [8]

Proposes a big data, cloud computing, machine learning and social networks based approach for performing face tagging for social networks running on big data

Face recognition

9.

Learning deep architectures for AI [10]

Summarizes the principles related to learning algorithms for deep architectures to construct deeper models such as deep belief networks for application to different domains

Deep model design

10.

Fusing social networks with deep learning for volunteerism tendency prediction [22]

Summarizes a framework for multiple social network learning that fuses social networks through deep learning with consistency regularization and source confidence for predicting user penchant for volunteerism

Volunteerism tendency prediction

11.

Applications of online deep learning for crisis response using social media [23]

Applies stochastic gradient descent-based deep neural networks for classification of informal tweets to topical classes especially during disaster circumstances

Crisis response

12.

Link prediction based on common neighbors for dynamic social networks [24]

Novel approach for link prediction using dynamic topology of a social network. Utilizes time-varied weight, change measure of common neighbor and relationship between common neighbors for edge prediction

Link prediction

13.

Data crawler for Sina Weibo based on Python [25]

Proposes a data crawler based upon keyword matching and parallelization technology to extract fan information of different Weibo users from Weibo data in real time

Data crawler

14.

Online social network image classification and application based on deep learning [26]

Proposes image classification on data obtained from the Sina blog by combining text and content features from deep learning architectures

Image classification

15.

Text normalization algorithm on twitter in complaint category [27]

Normalization algorithm for unstructured Twitter complaint text in Indonesian language

Language processing

16.

Enhancing deep learning sentiment analysis with ensemble techniques in social applications [3]

Integrates deep learning techniques with traditional surface-based approaches for sentiment analysis to propose a sentiment classifier that surpasses the existing surface classifiers

Sentiment analysis

17.

Deep learning model for sentiment analysis in arab social media [19]

Automated deep learning-based information processing system in arabic language domain

Sentiment analysis

18.

Deep learning model for sentiment analysis on Thai Twitter data [18]

Automated deep learning-based sentiment classifier for thai twitter data

Sentiment analysis

19.

Deep learning Twitter sentiment analysis [28]

Analysis of efficacy of deep learning models for Twitter sentiment analysis

Sentiment analysis

20.

Ontology-based deep learning for human behavior prediction in health social networks [14]

Proposes ontology-based restricted Boltzmann machine (ORBM) framework for human conduct calculation in health social networks

Health informatics

21.

Information assimilation framework for event detection in multimedia surveillance systems [29]

Proposes multimedia information assimilation framework based on hierarchical probabilistic integration to sense atomic events

Information assimilation

22.

Real-time speaker tracking particle filter sensor fusion [30]

Applies sensor fusion for real-time speaker identification and tracking

Object detection

23.

Content-based image retrieval—approaches and trends of the new age [31]

Recent advances in image reclamation and automated image annotation discussed

Image retrieval

24.

Content-based 3D object retrieval [32]

Two novel approaches for benchmarking 3D object retrieval systems

3D object retrieval

25.

Companies applying deep learning [33]

Affectiva: applies deep networks for real-time analysis of emotions from video or images

Gridspace: applies deep networks for sophisticated speech recognition systems for speaker identification

Ditto Labs: deep learning network based detection system for identification of brands and logos from social media data

Nervana: applies deep framework named neon to allow users create their own cloud-based deep data

Deep genomics: applies deep networks to predict natural and therapeutic genetic variations in cellular processes

Deep instinct: applies deep learning to identify, predict and prevent advanced, persistent cyber threats in real time

Industry

The flow of this manuscript is as follows: Sect. 2 elaborates on how Deep Multimedia Information Retrieval Techniques are being applied for managing the biggest big data available on social networks. Section 3 summarizes the techniques available for 3D-multimedia information. Section 4 highlights the challenges for MIR techniques while giving significant future research directions.

2 Deep MIR on social networks

As per records there are already more than 1.8 billion Facebook users of which around 800 million spend close to 40 min per day on the social network each day [1]. From these big numbers it is not difficult to analyze that the amount of multimedia data available on Facebook alone shall be somewhere between some significant tera, peta or exabyte numbers growing each day. Multimedia data from all social networks combined alone are rightly termed as the biggest available big data of present times [34].

To effectively handle this vast multimedia data, multimedia information retrieval (MIR) techniques have become a necessity. To understand and appreciate the need of MIR systems we first need to clarify the meaning of few common terms being used in this manuscript. First, multimedia data implies one or more primary data types such as audio, video, images, text, graphics, animation, etc. [35]. Multimedia information retrieval implies the mechanism of searching for and locating multimedia documents [36] to develop multimedia search engines. An important challenge for MIR is the fact that the input query here itself might be multimedia. However, considerable research has been performed in development of effective MIR systems to allow optimum usage of vast multimedia data on the web [37]. With the growth of available multimedia data on social networks different MIR techniques have also been applied for effective management of this social multimedia data.

It may be noted here that though social media has penetrated into our lives in a big way, majority of the multimedia data here is unstructured [38]. Conventional MIR systems are limited in their ability to identify and extract unusual information [39]. Majority of this deals with feature extraction/retrieval from this data [34]. To optimize performance of these techniques recent efforts have shifted towards application of deep learning-based techniques. In the recent times, deep learning has made its mark in a number of varied domains. It has also successfully affected MIR techniques applied to social networks for effective information gathering and dissemination [1]. Many varied Deep Learning techniques have also given hopeful results in large-scale multimedia processing and reclamation [34]. As deep learning systems are capable of learning features in a hierarchy, thus they can prove useful in reducing the semantic gap in multimedia data on social networks.

Conventional multimedia computing was restricted in capturing multifaceted multimedia content such as text, images, audio, and video as it was restricted to specific domain knowledge [34]. Deep Learning enables automatic representation learning to model the multi-modal data. Many significant deep learning-based MIR systems are being successfully applied to social networks. Table II below lists some important deep learning-based MIR systems successfully being applied to social network multimedia data.

Table 2 above elaborates how deep learning has empowered MIR systems to effectively be applied for revolutionary advances in varied domains such as speech recognition [52], image analysis [53], information retrieval [36, 54], recommendation [48], and natural language processing [22], etc. However, this successful application of deep frameworks does not imply the end of road has been achieved. Table 2 also highlights the various unanswered research challenges and future research directions that are still open in almost every domain.
Table 2

Deep learning-based significant MIR systems

S.no.

Application

Description

Deep learning framework implemented

Challenge

1

Deep learning and music adversaries [40], [41]

Proposes deep learning system application to image–object recognition. It exploits system constraints to locate minimal changes of the input image to enable system misclassify the same with high confidence

Convolutional neural network architectures

High recorded throughput of these systems does not ensure that they are trained to perfectly resolve high-level problems

2

A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme [42]

Investigates if deep features from MR images can generate radiomics signature for prediction of overall survival in patients suffering from glioblastoma multiforme

Deep feature extraction through transfer learning

Generalization of the radiomics model questionable and requires a more large-scale multicenter analysis. Further the radiomics–genomics correlation needs to be evaluated

3

VulDeePecker: a deep learning-based system for vulnerability detection [43]

Vulnerability deep pecker applies code gadgets for program transformation to vectors to enable automatic detection of software vulnerabilities

Bidirectional long short-term memory (LSTM) neural network

Detection of vulnerabilities in executables is still unanswered. Further, proposed VulDeePecker design is only limited to vulnerabilities in library/API function calls in C/C++ programs and is not generic

4

Feature learning for chord recognition: the deep chroma extractor [44]

Proposes learned chroma feature extractor

Three-layered deep neural network with 512 rectifier units each

DNN ensemble usage could result in better results

5

Query by singing/humming system based on deep learning [45]

Proposes a deep learning-based query by singing/humming system for proficient indexing and recovery of desired music from a huge music database

Two-layered deep belief network with top layer composed of undirected binary chart and lower layer a directed sigmoid belief network

More accurate deep framework-based pitch extractor still requires work

6

Cardiac arrhythmia detection using deep learning [46]

Proposes efficient automatic cardiac arrhythmia detection method based upon transferred deep learning

AlexNet-feature extraction; back-propagation neural network: classifier

Realization of a viable computer-aided diagnostic system well recognized in clinical practice to aid clinicians and patients alike

7

Deep learning-based automated segmentation of macular edema in optical coherence tomography [47]

Proposes a convolutional neural network (CNN) that identifies intraretinal fluid (IRF) on OCT

Convolutional neural network

Generalizable studies missing

8

Deep content-based music recommendation [48]

Proposes a model for suggestion, and forecast of latent factors from music audio when they cannot be retrieved from conventional data

Deep convolutional neural networks

Generic model missing

9

Metric learning-based data augmentation for environmental sound classification [49]

Proposes a framework for data augmentation through metric learning. The idea is to first discover a metric from the original training data, and then use it to filter out augmented data samples that are far from original ones in the same class

Convolutional neural network

Generative model for augmentation schemes

10

Geometric deep learning [50]

Proposes a unified framework allowing to generalize CNN architectures to non-Euclidean realms (graphs and manifolds) and learn local, stationary, and compositional task-specific features

Adv: deformable 3D shape analysis applications, the key advantage of our approach is that it is intrinsic and thus deformation invariant by construction

Convolutional neural network on spatial domain model

Application of proposed model to computational social sciences

11

Graph-based classification of omni-directional images [51]

Proposes image classification task by taking into account the specific geometry of omni-directional cameras with graph-based representations

Graph convolutional network

Extension of proposed framework to generic geometries of the camera lenses

3 3-D MIR on social networks

The previous sections successfully documented the successful materialization of deep learning-based MIR systems into varied domains of computing. Recent trends have now shifted from single mode information retrieval to 3-D multimedia information retrieval (3D MIR) based systems. 3D multimedia is an amalgamation of 3D models, images, motion data, movies, etc [55]. Single-dimensional data retrieval as well as 2-D retrieval mechanisms cannot be directly generalized to 3-D retrieval mechanisms [56]. Hence, deep research is now directed towards designing 3D MIR systems that can accurately recover 3D multimedia data analogous to user query component structure in real time. This 3D multimedia data have big potential to represent complex information [32, 56, 57, 58, 59, 60, 61]. Successful application of 3D-content-based searching from large 3D object repositories has been performed in numerous domains such as CAD/CAM [62], biomedical, virtual reality, military, entertainment, etc. Table 3 below elaborates some significant work in the same.
Table 3

Significant 3D MIR systems

S.no.

3D-MIR system

Description

Challenge

1.

Automatic Alzheimer’s disease recognition from MRI data using deep learning method [53]

Proposes an automatic deep learning-based AD recognition algorithm that uses 3D brain MRI

Data processing is costly and requires more efficient capability

2.

3D multimedia data search system based on stochastic ARG matching method [55]

Treats 3D multimedia data through a MIR framework designed on stochastic attribute relational graph

High calculation cost of ARG matching can be reduced by parallel processing, further the proposed system needs to be generalized to support various file types

3.

Real-time facial segmentation and performance capture from RGB input [63]

Proposes real-time 3D facial routine confine through overt semantic segmentation in the RGB input

Limited application to only facial recognition; should be generalized to other body parts

4

Multimodal location based services semantic 3D city data as virtual and augmented reality [64]

Proposes novel mobile service for investigating urban frameworks at diverse level of details (LoDs) through popular standards such as CityGML. Facilitates researchers and city planners to discover energy datasets through virtual globes, virtual reality and augmented reality

Should be generalized with different data sources such as the internet of things (IoT) simulation to provide real-time data

5

Deep shape-aware descriptor for nonrigid 3D object retrieval [15], [50], [65]

Proposes 3D shape recovery through a multi-level facet learning method

High computation time for training the model attributes can be warded off by use of GPU. SA-BoF also requires testing on 3D large-scale shape benchmarks

6

On-the-fly learning for visual search of large-scale image and video datasets [66]

Proposes to enable run-time visual search of large-scale video datasets for semantic entities through a text query

Developing varied available competencies for visual search rather than conventional text-based search

7

3-D content-based retrieval [32] [56]

Automated 3-D object retrieval systems based on similarity measures

Choice of an effective and optimum predefined similarity algorithm to enable automatic similarity assessment between 3-D object pairs

8

3D model search engines [67], [68]

Propose shape-based search techniques for effectively searching 3D data on the web

Though speedy, however, high computational costs involved

9

Princeton shape benchmark [69]

Freely accessible collection of 3D models, software tools, and software for 3D shape matching

High storage and computational costs involved; most shape matching algorithms do not perform generically well on all objects

10

Comparison framework for 3D object classification methods [70]

Proposes structure to evaluate effectiveness of diverse query-to-class membership measures

Can be generalized to definition of classifiers based on model prototypes. Reduction of search space for object recovery can also be computed

11

Stratified point sampling of 3D models [71]

Procedure to generate a stratified sampling of 3D models applicable across many domains

Effectiveness and computation costs may vary across domains

Table 3 above enlists certain significant implementations of 3D MIR systems. However, it is important to note that 3-D MIR is the most challenging MIR system as it involves mapping similarity-based searching among 3D multimedia data [72]. Certain individual challenges of each application have been outlined in Table 3 above. However, Sect. 4 elaborates the significant challenges faced by 3-D MIR techniques.

4 Challenges for 3-D MIR techniques

Deep Learning algorithms are powerful enough to enable MIR systems understand the multimedia data at different semantic levels and extract user choice of data from the same. Quite significant work has been done on applying deep MIR technologies to diverse real world usage of big data generated through social networks. However, despite its successful applications to varied business domains many significant open challenges and research areas still require optimal solutions. Simply put the core issues with multimedia data on social networks are optimizing storage, processing, indexing and searching of this data. 3D multimedia data storage, processing, indexing as well as searching is even more challenging as it deals with multimodal data. Simply put, 3D information is not effortless to recover [56]. This section lists some significant open challenges for 3D MIR techniques [73, 74, 75]:
  1. i.

    Effective acquirement and representation of available spatio-temporal data in social networks [1]: acquiring social network data involves cleaning and filtration of data followed by feature extraction and effective storage and management. As the storage space required for such big data is immense, computation cost involved rises exponentially. To apply cost-friendly techniques for effective as well as optimal storage, representation and management of spatio-temporal multimedia data from social networks are important concerns.

     
  2. ii.
    Scalable and efficient storage and processing framework [34]: 3D multimedia information recovery demands a scalable and proficient storage and processing framework for the enormous amount of data. This challenge demands architectures more robust and capable than the conventional database management systems. MapReduce Framework has established itself as the real, practical yardstick for batch processing of big data. The Hadoop distributed file system (HDFS) is one significant technology implementing the MapReduce framework in an efficient mechanism. Here, data in a Hadoop cluster are stored in blocks and disseminated all through the cluster. Further the map and reduce functions are run on smaller subsets of a larger dataset. Some significant work has been carried out recently on the indexing, searching, and analysis of multimedia big data using MapReduce framework. Table 4 below highlights some of the notable works:
    Table 4

    Significant works on multimedia big data

    S.no.

    Work

    Description

    Challenge

    1

    Indexing and searching 100 M images with map reduce [76]

    Proves greater scalabilty in indexing large volume of images by applying MapReduce framework through Hadoop

    Inputting the data under consideration into the grid/cloud through restricted bandwidth is a realistic hitch

    2

    Implementation and performance evaluation of a hybrid-distributed system for storing and processing images from the web [77]

    Proposes design and implementation of a hybrid-distributed architecture realized through Hadoop distributed file system for processing images crawled from the web

    Testing if the architecture is plausible with video content

    3

    Mars: a MapReduce framework on graphic processors [78]

    Binds GPU power for MapReduce. Abstracts the programming complexity of GPU through the interface of MapReduce

    Integrating Mars into existing MapReduce implementations

    4

    A multimedia parallel processing approach on GPU MapReduce framework [79]

    Proposes a parallel processing technique for multiple multimedia processing programs

    Cannot support dynamic multimedia processing programs

    5

    Large-scale multimedia data mining using MapReduce framework [80]

    Explores MapReduce for large-scale data mining

    Successful on image categorization and video-event recognition, however, implementation on multimodal data to be tested

    6

    Towards fast multimedia feature extraction: Hadoop or storm [81]

    Evaluates Apache Hadoop and Apache storm for multi-processor task distribution

    Apache Hadoop was found to be inappropriate for cases where data were externally stored. Suggests integration of both frameworks for a more generically viable scenario

     
As we can observe from Table 4 above to Map Reduce Framework has proven itself as the framework of choice for big data processing and storage. However, there are still unresolved issues and challenges that still require research effort [82, 83].
  1. iii.

    Multi-modal study and recovery algorithms [15]: evaluating the synergy between the varied media including text and context information is a significant challenge for MIR systems. Success has been noted on single nodal media as well as limited multimedia data. However, the same when applied to 3D multimedia data becomes more challenging due to the inherent complexities of the media involved.

     
  2. iv.

    Bridging the semantic gap [75, 82]: this implies effectively mapping the low-level multimedia features to high-level user terms or queries. Enabling meaningful semantic search directed at detection of concepts in media with multifaceted backgrounds is an important challenge. In 3D-multimedia data this becomes even more rigorous as the variety of underlying data increases.

     
  3. v.

    Absence of Benchmarked standardized datasets for testing algorithms [75, 84]: varied media-related standardized databases on which diverse groups can test their algorithms are required. For text retrieval, it is uncomplicated to acquire big compilation of old newspaper texts as the copyright owners do not consider the same to be of much value; however, image, video, and speech library owners value their collections and, therefore, are more guarded in sharing their data. While not exactly a research challenge, accessing big 3D-multimedia compilations for extensive assessment benchmarking is a realistic and significant issue to be addressed. Competitions and events such as TRECVID [75] are aimed at making such task-related data accessible.

     
  4. vi.

    Implementation of efficient similarity search algorithms [85, 86, 87, 88, 89]: similarity matching of 3D shapes is a challenging task as they require modes for effective representation of the skeletal and topological structure of the data involved [85]. Establishing accurate correspondence between parts of a 3D object is a difficult task. Hence, searching and matching different 3D objects in a data set requires selection and implementation of efficient similarity search algorithms at different levels of matching. Varied techniques have successfully been applied on 2D multimedia data. However, these techniques cannot be directly applied to 3D data. When dealing with 3D data, different techniques have been successfully applied to different types of data [62, 87, 88, 89, 90]. However, a generic efficient similarity search algorithm for all types of 3D data is still a distant dream. Further issues such as sensitivity of noise, surface undulations need to be appropriately handled on 3D data [58, 70, 91, 92, 93, 94].

     
  5. vii.

    Ability to perform partial matching [95, 96]: 3D multimedia data may not require to be matched completely. For many real world applications such as 3D scene searching partial matching of 3D multimedia data may also serve the purpose [95, 96]. Hence, partial matching between 3D objects can also prove to be useful if done appropriately. However, the implemented approaches for such matching still suffers from a number of limitations such as noise, low resolution, etc.

     
  6. viii.

    Lack of a sustainable 3D file format: currently, there exists no generic as well as widespread 3D file format. All popular 3D tools have their own proprietary formats. Hence, what is required is a fundamental file format that congregates characteristics of all 3D representations into a customizable as well as extensible encoding [97, 98].

     
  7. ix.

    The absence of canonical 3D representations: 3D data have no canonical representation. Hence, 3D representations are generally layered across surfaces, volumes, and structures. As a result there are no representation-independent, stable 3D markup, retrieval or query techniques available [58, 82, 99].

     

The above listed are a few main challenges that need to be addressed significantly to enable effective and smooth knowledge transfer from social networks to commercial user realistically plausible [74, 100, 101].

5 Conclusion and future directions

Multimedia information retrieval technology for big data on social networks has made momentous progress in recent times. The rapid improvements in this technology have largely concentrated on development of deep learning MIR systems that have resulted in groundbreaking improvements in the accuracy of storage, processing, indexing and searching of user desired multimedia data from social networks.

However, despite tremendous effort directed at evolving and establishing deep techniques for effective multimedia, multimodal information retrieval from popular social networks there still remain significant challenges that should be addressed before we can claim fully optimal, generic and effective commercially viable multimedia information retrieval. This manuscript has analyzed 3D MIR techniques and their applications and has tried to highlight the challenges still faced by these systems. From our reasoning we can foresee that 3D multimedia data are gaining momentum and shall soon result into a huge 3D online data set in the near future. To use this large set of 3D documents though there shall be multiple specialized tools but notion of generalized treatment of 3D documents is still far from reality. Though 3D shall soon have penetrated significantly into our lives making the technology scalable and manageable at the large scale of its growth shall remain a challenge.

Notes

References

  1. 1.
    Kou, F., Du, J., He, Y., Ye, L.: Social network search based on semantic analysis and learning. CAAI Trans. Intell. Technol. 1(4), 293–302 (2016)Google Scholar
  2. 2.
    Tandera, T., Hendro, Suhartono, D., Wongso, R., Prasetio, Y.L.: Personality prediction system from Facebook users. Proc. Comput. Sci. 116, 604–611 (2017)Google Scholar
  3. 3.
    Araque, O., Corcuera-Platas, I., Sánchez-Rada, J.F., Iglesias, C.A.: Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Expert Syst. Appl. 77, 236–246 (2017)Google Scholar
  4. 4.
    Guimaraes, R.G., Rosa, R.L., De Gaetano, D., Rodriguez, D.Z., Bressan, G.: Age groups classification in social network using deep learning. IEEE Access 5(c), 10805–10816 (2017)Google Scholar
  5. 5.
    Najafabadi, M.M., Villanustre, F., Khoshgoftaar, T.M., Seliya, N., Wald, R., Muharemagic, E.: Deep learning applications and challenges in big data analytics. J. Big Data 2(1), 1–21 (2015)Google Scholar
  6. 6.
    Phan, N., Dou, D., Wang, H., Kil, D., Piniewski, B.: Ontology-based deep learning for human behavior prediction with explanations in health social networks. Inf. Sci. (NY) 384, 298–313 (2017)Google Scholar
  7. 7.
    Zheng, X., Zeng, Z., Chen, Z., Yu, Y., Rong, C.: Detecting spammers on social networks. Neurocomputing 159(1), 27–34 (2015)Google Scholar
  8. 8.
    Vinay, A., Shekhar, V.S., Rituparna, J., Aggrawal, T., Murthy, K.N.B., Natarajan, S.: Cloud based big data analytics framework for face recognition in social networks using machine learning. Proc. Comput. Sci. 50, 623–630 (2015)Google Scholar
  9. 9.
    Arel, I., Rose, D.C., Karnowski, T.P.: Deep machine learning—a new frontier in artificial intelligence research. IEEE Comput. Intell. Mag. 5(4), 13–18 (2010)Google Scholar
  10. 10.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends® Mach. Learn. 2(1), 1–127 (2009)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Seraj, R.M.: Multi-task learning (2014)Google Scholar
  12. 12.
    Chen, X.-W., Lin, X.: Big data deep learning: challenges and perspectives. IEEE Access 2, 514–525 (2014)Google Scholar
  13. 13.
    Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I.J., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., Vincent, P., Courville, A., Bergstra, J.: Unsupervised and transfer learning challenge: a deep learning approach. JMLR W& CP Proc. Unsuperv. Transf. Learn. Chall. Work. 27, 97–110 (2012)Google Scholar
  14. 14.
    Phan, N., Dou, D., Wang, H., Kil, D., Piniewski, B.: Ontology-based deep learning for human behavior prediction in health social networks. In: Proceedings of the 6th ACM Conference on Bioinformatics, Computational Biology Health Informatics, pp. 433–442 (2015)Google Scholar
  15. 15.
    Ghodrati, H., Ben Hamza, A.: Deep shape-aware descriptor for nonrigid 3D object retrieval. Int. J. Multimed. Inf. Retr. 5(3), 151–164 (2016)Google Scholar
  16. 16.
    Ain, Q.T., Ali, M., Riaz, A., Noureen, A., Kamran, M., Hayat, B., Rehman, A.: Sentiment analysis using deep learning techniques: a review, IJACSA). Int. J. Adv. Comput. Sci. Appl. 8(6), 424–433 (2017)Google Scholar
  17. 17.
    Chen, T., Xu, R., He, Y., Wang, X.: Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Syst. Appl. 72, 221–230 (2017)Google Scholar
  18. 18.
    Vateekul, P., Koomsubha, T.: A study of sentiment analysis using deep learning techniques on Thai Twitter data. In: 2016 13th International Joint Conference on Computer Science Software Engineering, pp. 1–6 (2016)Google Scholar
  19. 19.
    Abbes, M., Kechaou, Z., Alimi, A.M.: Enhanced deep learning models for sentiment analysis in arab social media. In: Lecture Notes in Computer Science (including subseries Lecture Notes Artificial Intelligence, Lecture Notes Bioinformatics), vol. 10638, pp. 667–676 (2017)Google Scholar
  20. 20.
    Liao, L., He, X., Zhang, H., Chua, T.-S.: Attributed Social Network Embedding, vol. 14(8), pp. 1–12 (2017)Google Scholar
  21. 21.
    Pal, S., Dong, Y., Thapa, B., Chawla, N.V., Swami, A., Ramanathan, R.: Deep learning for network analysis: problems, approaches and challenges. In: MILCOM 2016—2016 IEEE Military Communications Conference, pp. 588–593 (2016)Google Scholar
  22. 22.
    Jia, Y., Song, X., Zhou, J., Liu, L., Nie, L., Rosenblum, D.S.: Fusing social networks with deep learning for volunteerism tendency prediction. In: Proceedings of the 13th AAAI Conference on Artificial Intelligence, pp. 165–171 (2016)Google Scholar
  23. 23.
    Nguyen, D.T., Joty, S., Imran, M., Sajjad, H., Mitra, P.: Applications of Online Deep Learning for Crisis Response Using Social Media Information (2016)Google Scholar
  24. 24.
    Yao, L., Wang, L., Pan, L., Yao, K.: Link prediction based on common-neighbors for dynamic social network. Proc. Comput. Sci. 83, 82–89 (2016)Google Scholar
  25. 25.
    Zhou Zhonghua, X.J., Huiran, Z.: Data crawler for Sina Weibo based on Python. J. Comput. Appl. 34(11), 3131–3134 (2014)Google Scholar
  26. 26.
    Yang, C., Wang, Y.: Online social network image classification and application based o n deep learning. ICETA 2016, 41–46 (2016)Google Scholar
  27. 27.
    Hanafiah, N., Kevin, A., Sutanto Fiona, C., Arifin, Y., Hartanto, J.: Text normalization algorithm on Twitter in complaint category. Proc. Comput. Sci. 116, 20–26 (2017)Google Scholar
  28. 28.
    Lu, Y., Sakamoto, K., Shibuki, H., Mori, T.: Are deep learning methods better for Twitter sentiment analysis ? 言語処理学会 第23回年次大会 発表論文集 C, 787–790 (2017)Google Scholar
  29. 29.
    Atrey, P.K., Kankanhalli, M.S., Jain, R.: Information assimilation framework for event detection in multimedia surveillance systems. Multimed. Syst. 12(3), 239–253 (2006)Google Scholar
  30. 30.
    Chen, Y., Rui, Y.: Real-time speaker tracking using particle filter sensor fusion. Proc. IEEE 92(3), 485–494 (2004)Google Scholar
  31. 31.
    Datta, R., Li, J., Wang, J.Z.: Content-based image retrieval—approaches and trends of the new age. In: Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval—MIR’05, p. 253 (2005)Google Scholar
  32. 32.
    Schreck, T.: 3D object. IEEE Comput. Graph. Appl. 27(4), 22–27 (2007)Google Scholar
  33. 33.
    Murnane, K.: Thirteen Companies that Use Deep Learning to Produce Actionable Results (online). https://www.forbes.com/sites/kevinmurnane/2016/04/01/thirteen-companies-that-use-deep-learning-to-produce-actionable-results/#127cea8033b8. Accessed: 16 Jan 2018 (2016)
  34. 34.
    Trovati, M., Hill, R., Anjum, A., Zhu, S.Y., Liu, L.: Big-data analytics and cloud computing: theory, algorithms and applications. Big-Data Anal. Cloud Comput. Theory, Algorithms Appl. 2016, i–xvi (2016)Google Scholar
  35. 35.
    Hanjalic, A.: New grand challenge for multimedia information retrieval: bridging the utility gap. Int. J. Multimed. Inf. Retr. 1(3), 139–152 (2012)Google Scholar
  36. 36.
    Rüger, S.: Multimedia Information Retrieval, vol. 1(1) (2010)Google Scholar
  37. 37.
    Raieli, R.: Multimedia Information Retrieval : Theory and Techniques. Chandos Publishing, Oxford (2013)Google Scholar
  38. 38.
    Zhu, W.: Visions and views multimedia big data computing. IEEE Multimed. 22(3), 96–105 (2015)Google Scholar
  39. 39.
    Abraham, A.: Computational social networks: mining and visualization. Comput. Soc. Netw. Min. Vis. 9781447140, 1–385 (2012)Google Scholar
  40. 40.
    Lee, H., Largman, Y., Pham, P., Ng, A.: Unsupervised feature learning for audio classification using convolutional deep belief networks. Adv. Neural Inf. Process. Syst. 22, 1096–1104 (2009)Google Scholar
  41. 41.
    Kereliuk, C., Sturm, B.L., Larsen, J.: Deep learning and music adversaries. IEEE Trans. Multimed. 17(11), 2059–2071 (2015)Google Scholar
  42. 42.
    Lao, J., Chen, Y., Li, Z.C., Li, Q., Zhang, J., Liu, J., Zhai, G.: A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci. Rep. 7(1), 1–8 (2017)Google Scholar
  43. 43.
    Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., Zhong, Y.: VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. NDSS (2018)Google Scholar
  44. 44.
    Korzeniowski, F., Widmer, G.: Feature Learning for Chord Recognition: The Deep Chroma Extractor (2016)Google Scholar
  45. 45.
    Sun, J.-Q., Lee, S.-P.: Query by singing/humming system based on deep learning. Int. J. Appl. Eng. Res. 12(13), 973–4562 (2017)Google Scholar
  46. 46.
    Isin, A., Ozdalili, S.: ScienceDirect cardiac arrhythmia detection using deep learning. Proc. Comput. Sci. 120, 268–275 (2017)Google Scholar
  47. 47.
    Lee, C.S., Tyring, A.J., Deruyter, N.P., Wu, Y., Rokem, A., Lee, A.Y.: Deep-learning based, automated segmentation of macular edema in optical coherence tomography. Biomed. Opt. Express 8(7), 3440 (2017)Google Scholar
  48. 48.
    Zhang, S., Yao, L., Sun, A.: Deep learning based recommender system: a survey and new perspectives. ACM Comput. Surv. 1(1), 1–35 (2017)Google Scholar
  49. 49.
    Paltz, N.: Metric learning based data augmentation for environmental sound classification. Department of Electronics, State Key Lab of Intelligent Technologies and Systems Tsinghua National Laboratory for Information Science and Technology (TNList), Beijing, pp. 1–5 (2017)Google Scholar
  50. 50.
    Monti, F., Boscaini, D., Masci, J., Rodolà, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs and manifolds using mixture model CNNs (2016)Google Scholar
  51. 51.
    Khasanova, R., Frossard, P.: Graph-Based Classification of Omnidirectional Images 1 Introduction 2 Related work, pp. 869–878Google Scholar
  52. 52.
    Raffel, C.: Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching, p. 222 (2016)Google Scholar
  53. 53.
    Luo, S., Li, X., Li, J.: Automatic Alzheimer’s disease recognition from MRI data using deep learning method. J. Appl. Math. Phys. 5(9), 1892–1898 (2017)Google Scholar
  54. 54.
    Saiyeda, A.: Cloud computing for deep learning analytics : a survey of current trends and challenges. Int. J. Adv. Res. Comput. Sci. 8(2), 68–72 (2017)Google Scholar
  55. 55.
    Nakamura, N., Takano, S., Okada, Y.: 3D multimedia data search system based on stochastic ARG matching method. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence, Lecture Notes in Bioinformatics), vol. 5371, pp. 379–389 (2009)Google Scholar
  56. 56.
    Tangelder, J.W.H., Veltkamp, R.C.: A survey of content based 3D shape retrieval methods. Multimed. Tools Appl. 39(3), 441–471 (2008)Google Scholar
  57. 57.
    Iyer, N., Jayanti, S., Lou, K., Kalyanaraman, Y., Ramani, K.: Three-dimensional shape searching: state-of-the-art review and future trends. Comput. Des. 37(5), 509–530 (2005)Google Scholar
  58. 58.
    Havemann, S., Fellner, D.W.: Seven research challenges of generalized 3D documents. IEEE Comput. Graph. Appl. 27(3), 70–76 (2007)Google Scholar
  59. 59.
    Bustos, B., Keim, D., Saupe, D., Schreck, T., Vrani, D.: An experimental effectiveness comparison of methods for 3D similarity search. Int. J. Digit. Libr. 6(1), 39–54 (2006)Google Scholar
  60. 60.
    Böhm, C., Berchtold, S., Keim, D.A.: Searching in high-dimensional spaces: index structures for improving the performance of multimedia databases. ACM Comput. Surv. 33(3), 322–373 (2001)Google Scholar
  61. 61.
    Bustos, B., Keim, D.A., Saupe, D., Schreck, T., Vranić, D.V.: Feature-based similarity search in 3D object databases. ACM Comput. Surv. 37(4), 345–387 (2005)Google Scholar
  62. 62.
    Rea, H.J., Corney, J.R., Clark, D.E.R., Taylor, N.K.: A surface partitioning spectrum (SPS) for retrieval and indexing of 3D CAD models. In: Proceedings of the 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004, pp. 167–174Google Scholar
  63. 63.
    Saito, S., Li, T., Li, H.: Real-time facial segmentation and performance capture from RGB input. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence, Lecture Notes in Bioinformatics), vol. 9912(1), pp. 244–261 (2016)Google Scholar
  64. 64.
    Santana, J.M., Wendel, J., Trujillo, A., Suárez, J.P., Simons, A., Koch, A.: Progress in Location-Based Services 2016, pp. 329–353 (2017)Google Scholar
  65. 65.
    Castellani, U., Cortelazzo, G.M., Cristani, M., Delponte, E., Fusiello, A., Giachetti, A., Mizzaro, S., Odone, F., Puppo, E., Scateni, R., Zanuttigh, P.: 3-SHIRT : three-dimensional shape indexing and retrieval techniques. In: Eurographics Italian Chapter Conference, pp. 113–120 (2008)Google Scholar
  66. 66.
    Chatfield, K., Arandjelović, R., Parkhi, O., Zisserman, A.: On-the-fly learning for visual search of large-scale image and video datasets. Int. J. Multimed. Inf. Retr. 4(2), 75–93 (2015)Google Scholar
  67. 67.
    Funkhouser, T., Min, P., Kazhdan, M., Chen, J., Halderman, A., Dobkin, D., Jacobs, D.: A search engine for 3D models. ACM Trans. Graph. 22(1), 83–105 (2003)Google Scholar
  68. 68.
    Min, P., Funkhouser, T.: A 3D model search engine. Comput. Sci. 2004, 139 (2004)Google Scholar
  69. 69.
    Shilane, P., Min, P., Kazhdan, M., Funkhouser, T.: The Princeton shape benchmark. Proc. SMI 8540, 167–178 (2004)Google Scholar
  70. 70.
    Biasotti, S., Giorgi, D., Marini, S., Spagnuolo, M., Falcidieno, B.: A Comparison Framework for 3D Object Classification MethodsGoogle Scholar
  71. 71.
    Nehab, D., Shilane, P.: Stratified point sampling of 3D models. In: Proceedings of the First Eurographics Conference on Point-Based Graphics, pp. 49–56 (2004)Google Scholar
  72. 72.
    Besl, P.J., Jain, R.C.: Three-dimensional object recognition. ACM Comput. Surv. 17(1), 75–145 (1985)Google Scholar
  73. 73.
    Guo, Y., Liu, Y., Georgiou, T., Lew, M.S.: A review of semantic segmentation using deep neural networks. Int. J. Multimed. Inf. Retr. 7, 1–7 (2017)Google Scholar
  74. 74.
    Kankanhalli, M.S., Rui, Y.: Application potential of multimedia information retrieval. Proc. IEEE 96(4), 712–720 (2008)Google Scholar
  75. 75.
    Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: state of the art and challenges. ACM Trans. Multimed. Comput. Commun. Appl. 2(1), 1–19 (2006)Google Scholar
  76. 76.
    Moise, D., Shestakov, D., Gudmundsson, G., Amsaleg, L.: Indexing and searching 100 M images with map-reduce. In: Proceedings of the 3rd ACM International Conference on Multimedia Retrieval—ICMR’13, p. 17 (2013)Google Scholar
  77. 77.
    Krishna, M., Kannan, B., Ramani, A., CloudCom, 2010 IEEE, and undefined 2010: Implementation and performance evaluation of a hybrid distributed system for storing and processing images from the web. ieeexplore.ieee.orgGoogle Scholar
  78. 78.
    He, B., Fang, W., Luo, Q., Govindaraju, N.K., Wang, T.: Mars: a MapReduce framework on graphics processors. In: Proceedings of the 17th International Conference on Parallel Architecture and Compilation Techniques—PACT’08, p. 260 (2008)Google Scholar
  79. 79.
    Chen, S.Y., Lai, C.F., Hwang, R.H., Chao, H.C., Huang, Y.M.: A multimedia parallel processing approach on GPU MapReduce framework. In: Proceedings of the 2014 7th International Conference on Ubi-Media Computing and Workshops, U-MEDIA 2014, pp. 154–159 (2014)Google Scholar
  80. 80.
    Wang, H., Shen, Y., Wang, L., Zhufeng, K., Wang, W., Cheng, C.: /12/$31.00 ©2012 IEEE large-scale multimedia data mining using MapReduce framework. In: 2012 IEEE 4th International Conference on Cloud Computing Technology Science, pp. 978–1 (2012)Google Scholar
  81. 81.
    Mera, D., Batko, M., Zezula, P.: Towards fast multimedia feature extraction: Hadoop or storm. In: Proceedings of the 2014 IEEE International Symposium Multimedia, ISM 2014, pp. 106–109 (2015)Google Scholar
  82. 82.
    Zhang, C., Chen, T.: Indexing and retrieval of 3D models aided by active learning. In: Proceedings of the Ninth ACM International Conference on Multimedia—MULTIMEDIA’01, p. 615 (2001)Google Scholar
  83. 83.
    Gao, Y., Dai, Q.: View-Based 3-D Object Retrieval. Elsevier, New York (2014)Google Scholar
  84. 84.
    Vranic, D.V., Saupe, D., Richter, J.: Tools for 3D-object retrieval: Karhunen–Loeve transform and spherical harmonics. In: 2001 IEEE Fourth Workshop on Multimedia Signal Processing (Cat. No. 01TH8564), pp. 293–298 (2001)Google Scholar
  85. 85.
    Hilaga, M., Shinagawa, Y., Kohmura, T., Kunii, T.L.: Topology matching for fully automatic similarity estimation of 3D shapes. In: Proceedings of the 28th Annual Conference on Computing Graphics Interactive Techniques—SIGGRAPH’01, pp. 203–212 (2001)Google Scholar
  86. 86.
    Laga, H., Takahashi, H., Nakajima, M.: Geometry image based similarity estimation for 3D model retrieval. In: Nicograph International 2004, Taiwan, pp. 133–138 (2004)Google Scholar
  87. 87.
    Choi, S.-M., Kim, Y.-G.: Similarity Estimation of 3D Shapes Using Modal Strain Energy, pp. 206–212. Springer, Berlin (2005)Google Scholar
  88. 88.
    Sánchez-Cruz, H., Bribiesca, E.: A method of optimum transformation of 3D objects used as a measure of shape dissimilarity. Image Vis. Comput. 21(12), 1027–1036 (2003)Google Scholar
  89. 89.
    Shum, H.-Y., Hebert, M., Ikeuchi, K.: On 3D Shape Similarity (1995)Google Scholar
  90. 90.
    Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)Google Scholar
  91. 91.
    Cass, T.A.: Robust affine structure matching for 3D object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1265–1274 (1998)Google Scholar
  92. 92.
    Flynn, P.J., Jain, A.K.: BONSAI: 3D object recognition using constrained search. IEEE Trans. Pattern Anal. Mach. Intell. 13(10), 1066–1075 (1991)Google Scholar
  93. 93.
    Wang, W., Iyengar, S.S.: Efficient data structures for model-based 3-D object recognition and localization from range images. IEEE Trans. Pattern Anal. Mach. Intell. 14(10), 1035–1045 (1992)Google Scholar
  94. 94.
    Zhou, Y., Kaufman, A., Toga, A.W.: Three-dimensional skeleton and centerline generation based on an approximate minimum distance field. Vis. Comput. 14(7), 303–314 (1998)Google Scholar
  95. 95.
    Modemlis, A., Daras, P., Tzovaras, D., Strintzis, M.G. :On 3D partial matching of meaningful parts. In: Proceedings of the International Conference on Image Processing ICIP, vol. 2, pp. 1–4 (2007)Google Scholar
  96. 96.
    Funkhouser, T., Shilane, P.: Partial matching of 3D shapes with priority-driven search. In: Proceedings of the fourth Eurographics Symposium on Geometry Processing, pp. 131–142 (2006)Google Scholar
  97. 97.
    Ioannides, M., Quak, E.: 3D research challenges in cultural heritage : a roadmap in digital heritage preservationGoogle Scholar
  98. 98.
    Münster, S., Pfarr-Harfst, M., Kuroczyński, P., Ioannides, M.: 3D research challenges in cultural heritage II : how to manage data and knowledge related to interpretative digital 3D reconstructions of cultural heritageGoogle Scholar
  99. 99.
    Dugelay, J.-L., Baskurt, A., Daoudi, M.: 3D Object Processing : Compression, Indexing and Watermarking. Wiley, New York (2008)Google Scholar
  100. 100.
    Kuroczyński, P., Hauck, O., Dworak, D.: 3D Models on Triple Paths—New Pathways for Documenting and Visualizing Virtual Reconstructions, pp. 149–172 (2016)Google Scholar
  101. 101.
    von Schwerin, J., Lyons, M., Loos, L., Billen, N., Auer, M., Zipf, A.: Show Me the Data!: Structuring Archaeological Data to Deliver Interactive, Transparent 3D Reconstructions in a 3D WebGIS, pp. 198–230 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Bharati Vidyapeeth’s Institute of Computer Applications and Management (BVICAM)New DelhiIndia
  2. 2.CSIR-NISTADSNew DelhiIndia

Personalised recommendations