Agora Center Director, Prof. Pekka Neittaanmäki	Dr. Mikko Kovalainen	Anna Konarzewska & Michal Swiatkowski	Asko Suokas & Eetu Ojanen
Researcher Kam	Computing Group	Researchers

InBCT 2004: Industrial IT & Paper IT

Research team: Project leaders: prof. Pekka Neittaanmäki, prof. Heikki Lyytinen and prof. Vagan Terziyan; prof. S. Repin, prof. Olivier Pironneua, prof. S. Turek, Dr. Narciso González Vega, dos. Tuomo Rossi, FM Hermanni Hyytiälä, FL Eetu Ojanen, ins. Juhani Forsman, FM Pavlo Turzin

Main goals in 2004:

• Integration of relevant results of this and other projects to support other activities in Jyväskylä area in Paper IT,
• Develop prototypes of future industrial IT concepts with intelligent agents,
• Develop assessment and support technologies of human performance in challenging working life conditions (such as driving and man-machine interaction in the paper industry) . These mean conditions which contain high mental load and need to maintain the full capability where errors and skill-and state-related unoptimalities (such as drowseness and even temporary reduction of sensory or cognitive capabilities) may have seriously damaging consequences.

Three subprojects will be realized:

Virtual reality in collaborative modelling and design
Human performance in simulation environment
Semantic Web and industrial ontologies

Virtual Reality in Collaborative Modelling and Design

Strategic view

The main goal of this subproject is the development of a state-of-the-art Collaborative Virtual Environment that would substantially facilitate the modelling, simulation and analysis of industrial applications, considerably speed up the design of products and improve their quality. This kind of research is very essential to make Finland - and in this case Jyväskylä region - competitive with other technologically developed countries, where the industrial use of Virtual Reality techniques for modelling and design is already a common practice. A multidisciplinary Virtual Reality Laboratory AC-CAVE at Agora Center will start 12/2003. An important prerequisite for the success of the project is an active collaboration with other research groups working in this field. Contacts to a number of leading universities in Europe and in the USA have already been established. These contacts will be realized during special meetings and workshops in ECCOMAS 2004 conference held at University of Jyväskylä (http://www.mit.jyu.fi/eccomas2004/ ).

Background and significance of the topic

The modern development of information technology gives us an opportunity to implement sophisticated distributed systems for collaborative design. Persons with different interests and competencies (i.e., engineers and managers) can at least theoretically be brought together in a distributed design space where a virtual model is designed and functionally evaluated. A design space built in virtual reality environment will enable us to realistically simulate the form, functionality, and the use of the desired model. In terms of presenting and interacting with displayed information a virtual environment potentially provides a more intuitive means of communicating with a computer. Virtual reality allows for a better understanding of the three-dimensional structure of the dataset, as well as the spatial scale of features within it. This makes a virtual reality an ideal environment to perform tasks such as modelling and design.

Collaborative environments are on-line workplaces that have been equipped with multimodal communications and collaborative work interfaces. These environments can enable people to see and hear each other, as well as to share computer applications such as drawing tools and desktop applications. The motivation for building and using collaborative environments comes from the need to support remote collaborations by connecting people to data, images, simulations, and each other via a network. Additionally, the collaborative environment promotes application sharing and the exchange of control between remote users. Often, groups are able to work together at a distance with the effectiveness of collocation. A Collaborative Virtual Environment (CVE) is one that actively supports human-human communication in addition to human-machine communication and which uses a virtual environment as the user interface. This is a field with much potential for inter-disciplinary collaboration particularly in the fields of computer science, psychology, sociology, artificial intelligence, and in computer-aided design and testing of new products.

The design process itself has already been affected by computer aided design systems greatly, and this attitude has become even more important with the current trend towards concurrent engineering. The latter involves parallel development across product life-cycle activities by applying technologies like computer-aided design and modelling or optimal design of complex industrial plants like paper mills. Using shared information, engineers, designers, customers, and product managers can simultaneously and effectively collaborate during design and modelling processes. Valuable extensions to current practices could probably be provided by virtual reality. Studying the wide range of virtual reality applications it can be concluded that, nowadays, virtual reality is primarily used as a simulation tool for experiencing and evaluating virtual objects and designs. By this, expensive physical models can largely be eliminated resulting in shorter development time and lower costs. By using virtual reality during the design process a user can interpret experimental results and behaviour of a model in a more natural and understandable manner.

Introducing virtual reality into the computer aided design and modelling process could bring a number of advantages. This is, for example, an improved visualization of product, since the user is allowed to co-exist in the same space as the product model, in this way gaining a better appreciation of product geometry and aesthetics. Another advantage is the improved interaction with design in term of more intuitive model manipulation and functional experimentation. The latter means that the designer could effectively interact with the product model directly rather than using the traditional means (e.g., 2-D mouse and cursor). In the future, virtual reality computer aided design systems could be useful at all the stages of the design and modelling process, such as conceptual design, functional experimentation, and making final modifications and refinements. In addition to the above-mentioned advantages of virtual reality, let us emphasize that virtual environments can allow multiple participants to share the same virtual space. Thus, it would be natural to use this advantage to allow the whole design team to occupy a virtual environment containing a product model to collaborate in the design process. In collaborative virtual environment design systems the number of participants need not to be limited to the designer alone. In a virtual environment, physical location is not a barrier to effective communication. Allowing the whole design team to collaborate in virtual environments would aid the concurrent engineering technology and also reduce the product time-to-market.

Nevertheless, the collaborative environments, illustrating the potential of virtual environments, where designers can construct and evaluate designs have already been proposed, they still remain quite basic and there exists a necessity for further research. By and large, it means that the research topic presented briefly in this section will remain important, perspective and promising for many years to come.

Agora Center Virtual Reality Laboratory (AC-CAVE)

The research will be done at the VR laboratory of Agora Center (AC-CAVE). AC-CAVE has been built up in collaboration with department of Physics where first installation was made in 1998. Since 2003 all VR research activities have been concentrated in Agora Center. The current laboratory has two separate single-screen immersive VR displays driven by SGI Onyx visual supercomputer. User interaction is done with pen like stylus connected to magnetic tracking equipment. Acoustic rendering for 3D virtual acoustics is also done with SGI visual supercomputer. For computationally hard applications our laboratory has IBM RS6000/SP supercomputer, which can be coupled with visualization system allowing real-time modifications of large geometries and data sets.

For multidisciplinary research needs evolved from within University of Jyväskylä, industry (e.g., Metso and Nokia) and Air Force an extensive redesign process has been started. This contains both hardware and software aspects of the VR laboratory in order to do required upgrade during 2004 and expand VR activities to multidisciplinary research.

During 2004 one of the single-screen VR displays will be upgraded to cube like 5-screen multi-display VR environment because the level of immersion provided by single-screen VR display is not deep enough for e.g. visualization of very large objects and for psychological research. The unique design of this cube is made in-house so that the ceiling display can be removed easily away from optical tracking and other interaction and research equipment needed. Commercially available CAVE-like systems do not offer this kind of possibility. As the number of VR display screens will increase we will have to also upgrade our visual supercomputer during 2004. During this upgrade the user interaction means will be also re-engineered to fulfil needs of multidisciplinary research in our VR laboratory. All tracking methods and interaction devices will be connected to dedicated computer handling all I/O devices needed in various applications. This will allow the development of general application program interface for different I/O devices and will allow hiding peculiarities of devices behind the communication protocol between VR applications and computer handling all I/O devices. This will also speed up the development of the VR application programs. The magnetic tracking will stay as a part of user tracking system. Parallel to it user will have an option to use optical tracking if the application of this wireless method is more suitable than magnetic tracking. I/O computer will also have analogue and digital inputs and outputs for connecting of different steering devices and simulator cockpits. The same computer will also have steering option for VR applications with speech-recognition software.

The University has already invested 650 000 euros into VR laboratory, which will have total costs of 1 500 000 including planned upgrade during 2004. Research facilities funding will be applied among others from Academy of Finland in the end of 2003 for upgrade from both academic and industrial sources and partners.

Figure 1 AC-CAVE facilities.

Strategic goal of the proposal

Finland is much behind from other technologically developed countries in applying VR in modelling and design. High costs and lack of educated researchers as well as too weak signals from industry have reasons to postpone full-scale VR installation and organizing education of experts specializing in VR technology. Technological products, software and living spaces are touching people's lives ever more intimately, and the demands on designers are increasing sharply. The goal of the proposal is to create a multidisciplinary and multi-technology research unit around AC-CAVE (see Figure 1). An essential emphasis is on human aspects and collaborative elements in the design and take-into-use of virtual reality environments. The foremost aim of the research consortium is to enable / reinforce the use of VR technologies in the multi-disciplinary research projects and also to find novel and challenging application fields in the area of virtual reality design and modelling.

This project has three goals:

1. To build multidisciplinary and multi-technological VR-technology-based research unit and
2. To create active research collaboration with most active and leading groups working in VR-based modelling and design in Europe and USA
3. To start AC-CAVE multidiscipline research projects with the following concrete pioneering applications:
   • Pre- and post-processing and simulation,
   • Interactive computer-aided optimal shape/structural design,
   • Collaborative working in VR environment.

Human performance in simulation environment

Background

Simulation systems have become an important element in technology and product development processes in high-tech industry. Industry has adopted simulation and modeling tools also for investigating human behavior and performance in new environments and complex situations. Product usability testing, air pilot testing, pilot education, automotive safety testing - these are some of examples how industry uses these new tools. Transportation, both professional and casual, is one of the areas where human performance is very important factor.

The problems are: how to detect and prove skill-related unoptimalities associated particularly with safety, and/or how to detect decreased human performance and prevent this condition. The first environment chosen for this study is car driving simulation. Other environments of interest are paper machine monitoring situations. Some of the Finnish companies (e.g. two Jyväskylä based companies) are developing new services and products to solve these problems, and they will be using driving and other simulation systems for developing new commercial products. Human performance is the point of interest in these cases.

Purpose, Methods and Execution of Project

This research of Agora Center will be linked to Wellness Dream Lab -program (WDL), which is a new product development and commercialization program for wellness technology industry in Jyväskylä region. In WDL-program most of the development projects will be company based and therefore WDL-program will invest into new driving simulation system for company purposes above. This research project is linked to a Mind-Tech research group from Scientific Computing and experts of experimental psychophysiology of the senior psychologists working in the Center of Exellence funded by the Finnish Academy.

The long tradition, expertise, and quality in the psychophysiological research field of these researchers is a guaranty of the achievement of the research goals proposed here. This part aims to find answers to the scientific hypothesis: monitoring human psychophysiological and behavioral performance in simulation environment by means of a composite of two or three measures, is sufficiently reliable to reveal the starting and progressive development of sub-optimal alertness or readiness states on the driver which, are incompatible with safe driving. Consequently, solutions can be provided to the practical problem of accidents that result from drowsy and other reduced performance capacity states.

In the research developing these applications measurements will be made with a full arsenal of psychophysiological, i.e., EEG and ERPs, such as MMN and P300, EOG, SPL/SPR, HR, RR, and other psychological variables, i.e., PERCLOS, head nodding movement, leg-pressure on the seat, and of course also driving performance measures, such as RT on a concurrent visual enviroment. Video recording of the participants's head and body movements and face gestures will be used for off-line analysis. These variables will be measured while the participants perform a simulator-based driving task's: in a monotonous scenario and during a long period of time, and multiple task scenarios. This experimental procedure on which the prototyped detection method is based has been successfully tested in a previous and extensive pilot study we have just completed.

Using different procedures using advanced statistical techniques involving one to three most sensitive components of the prototyped measures we will test their predictive power of reduced alertness/attention preceding behaviorally observable performance impairment. New computational tools will be applied to the electrophysiologial signals to extract information not available through classical averaging techniques. The practical application of the answers obtained will lead to the specification of a procedure to warn and awake the performer who has trespassed the optimal/sub-optimal alertness threshold. The procedure will be empirically validated against a proven laboratory-based vigilance/drowsy stage detection method. Generalization of these results to a professional performers sample will increase the reliability of this procedure. Likewise, after being contrasted in this high-end simulator, the procedure can be contrasted in real-life contexts. The latter will be addressed in the next phases of this research.

Schedule and objectives of the research

Upon delivery of the high-end simulator, to be provided by Jyväskylä Science Park (WDL-program), standard scenarios of the simulated driving tasks will be developed for the first experiment to enable the realisation of the experiments. The first experiment to be run in this environment will measure voluntary students from the University of Jyväskylä. The results will be analysed and published by the end of 2004.

The goals of this research are the following:

1. Develop simulation scenarios in simulator to realise the empirical work,
2. select reliable and valid procedures to distinguish human performance stage in which the driver demonstrates full behavioural and cognitive capacity from other lower vigilance states, e.g., drowsiness, on which the capacity of the driver is severely hindered, emphasis is made on psychophysiological and their behavioural correlates,
3. develop means to define and monitor on-line the deterioration of driving performance in a driving simulator environment
4. develop practical means to stop the unwanted change in state and performance deterioration,

Potential industrial developments resulting from this research will be specified during Q2-Q3 2004 by Jyväskylä Science Park.

Research environment and equipment

This research will be performed at the University of Jyväskylä, in the multidisciplinary environment of Psykocenter /Agora Center, and in Viveca - center of wellness industry and research. The psychophysiological hardware (amplifier and other interfaces) needed for this study has been developed and tested extensively throughout the vast tradition in psychophysiological research in the University of Jyväskylä, Department of Psychology. Mathematical data processing, such as different statistical procedures and modeling, will be made in Agora Center, Department of Information Technology and Department of Psychology by the Mind-Tech group.

Recording of the physiological variables such as, EEG, EOG, HR, SPL/SPR, respiration rate, head movement -by means of the accelerometer ICSensor interfaced through the amplifier- during the experiment, is performed by means of the DSAMP software of the Mind-Tech group. This is an in-house developed recording software for experiments involving psychophysiological measures, which interfaces with the hardware.

Perceptual, auditory stimuli are administered and the visual environment monitored for experimental purposes by means of the E-Prime software. This is a general purpose software package for the implementation of performance measurements such as reaction times used in psychological research. This software interfaces with DSAMP to synchronise the contingency of the presented stimuli and recording of the EEG and other ANS signals.

Video cameras record the driving scenario and the participant's head and body for intensive behavioural assessments but the measurement of the most significant aspects of performance is automatized at best online. The video signals of the cameras plus the video signal from the DSAMP screen, displaying the most important physiological channels in real time, are mixed by means of video mixers into a single synchronised video recording which can be analysed off-line.

Research results and their significance

Results will be used for the development of an on-line performance deterioration assessment system. The system could detect drivers' reduced alertness and performance deterioration early enough so as to be able to prevent motor-vehicle accidents due this cause. Potential exploitation can be in the form of industrial production of the system and use by current professional and non-professional drivers. At the same time the technology needed for assessing drivers skills affected by more long term influences such as aging is under development in our Mind-Tech group.

Industrial Ontologies and Semantic Web

Strategic view

The Semantic Web is an initiative of the World Wide Web Consortium (W3C), with the goal of extending the current Web to facilitate Web automation, universally accessible content and promotion of existing Web to qualitatively new and higher level, utilizing machine-processable metadata associated with Web resources. Semantic Web approaches to development of global environment on top of Web with interoperable heterogeneous applications, web services, data repositories, humans, etc. The Semantic Web it is a new context within which one should rethink and re-interpret his existing businesses, resources, services, technologies, processes, environments, products etc. to raise them to totally new level of performance.

In 2004, the results of our previous research activities (see below) will be applied for creating of a concrete application of Semantic Web technology. Project has title "Semantic Facilitators for Web Information Retrieval" and its overall objective for 2004 is to explore possibilities to enhance information retrieval methods implementing software based on the Semantic Web technology

The main goal of this subproject is to advance the development of the Semantic Web-enabled tools for end-users, software developers and to facilitate integration of the semantic technology into existing information management systems. Research within the project concerns applications of the Semantic Web technology for improving existing information search systems adding semantic enabled extensions (semantic wrappers) that allow making resources, that are "behind" the front-end of information retrieval system, virtually accessible as resources in the Semantic Web. Presentation of "semantic-enabled" resources introduces benefits of the Semantic Web technology: a possibility to perform a semantic search, integration of heterogeneous data, use of semantically annotated search results by software.

As experts say, the main challenge of the Semantic Web, which is semantically annotated web content, will be met only in 5-7 years, whereas we assume that the use of "wrappers" or "adapters" for existing resources, which are not annotated yet, allows benefiting from the Semantic Web technology in already 1-2 years. For example, PaperIT project can use semantic wrappers for data sources that are to be integrated into the information sharing environment because a wrapper provides unified interface to various kinds of information retrieval (search) systems. In the future semantic wrappers will transform into personal portals for using the Semantic Web (virtually all available resources).

The overall goal is to bring Finland to the top level of world community research in Semantic Web area and in the same time develop a prototype of "killer" application of the semantic technology.

Background and significance of the topic

During 2004 we will develop general concept of a semantic wrapper for existing heterogeneous non-Semantic Web resources. Such a wrapper will be represented by Generic Semantic Search Facilitator software for upgrade of existing search services with advanced ontology-based information retrieval capabilities and facilitation of better search results within heterogeneous information repositories. A concrete software prototype will be developed and delivered:

• Prototype of Semantic Search Facilitator for Internet-search engine (e.g. Google) that enhances performance of user interface for semantic search as well as enables software applications to automatically use semantic search facilities.

Existing search systems usually provide keyword-based search services and do not store metadata that allows complex querying using concepts. Taking into account that nowadays data is not yet stored along with its semantic annotation (metadata) according to the Semantic Web's vision, there is still a need for advanced data search in the Web and in variety of information systems. A semantic-based approach for data management will be used in development of the future information retrieval systems that use ontologies for descriptions of concepts and relations between data objects that exist in a certain domain.

In 2004 we will develop generic add-on for search services. Semantic Search Facilitator transforms concept-based queries on the intermediate stage of query execution into a set of simple (keyword-based) queries to the used search services of various types (Internet search-engine, database, intranet, etc.). Such "semantic facilitator" uses knowledge of how to translate semantic queries into the query formats of several different search services and can perform routine actions that most of users do in order to achieve better performance and get more relevant results.

The results of the project can be applied in the development of next-generation Semantic Web-enabled ICT products that require advanced search capabilities. The issues that are under consideration in the project conform much to the growing market of inter-enterprise resource sharing, information retrieval and search. Potential users of "semantic facilitators" are users of the Web (to use them for a search), providers of the web services and developers of web applications (to improve them with search capabilities). Project results can be also applied in the software applications that automatically invoke search services.

Figure 2 Semantic search of non-semantic enabled resources via existing search facilities

Industrial Ontologies Group
Project team ("Industrial Ontologies Group"):

Industrial Ontologies Group hase no doubts about the possibilities, which Semantic Web opens for industry. That is why one important objective of our activities is to study appropriate industrial cases, collect arguments, launch industrial projects and develop prototypes for the industrial companies to not only believe together with us but also benefit from the Semantic Web.

Developments during 2004 will be based on scientific results obtained during 2003, and published in about 20 research papers, reports, and theses. Main results were:

• New approach to development of industrial asset management systems based on Semantic Web, Intelligent Web Services, Peer-to-Peer and Agent technologies;
• New approach for development of personal information management system based on Resource Description Framework, ontology personalization and dynamic user interface;
• Concepts of:
  • OntoServ.Net, an Semantic Web based environment for integration of industrial maintenance web-services;
  • GUN - Global Understanding eNvironment, a heterogeneous Semantic Web-based environment for agent-based resource management;
  • OntoAdapter (semantic adapter), a generic software component for connection of resources to semantic-enabled environments;
  • OntoShell, a service platform for hierarchical service integration;
  • Mobile Resource, a technical approach for delivering agent-based services in distributed environments.
  • Semantic wrapper, an add-on to search services, internet search engines and information retrieval systems that facilitates better search on the internet/intranet.

The concept of semantic wrappers (adapters) is being developed by the research group and recently we have started several initiatives of implementation of this concept in various domains. We intend to use semantic wrappers for creation semantic-enabled environments for software applications (P2P & Semantic Web research) and industrial device.

Strategic goal of the proposal

This project has three goals:

• To provide arguments for Finnish industry in favour of the Semantic Web-based products and
• To promote multidisciplinary and multi-technological research in the field of the semantic-based technologies
• To create a prototype of semantic-enabled search system based on existing non-semantic search engines

Overall objective for 2004: "Semantic Facilitators for Web Information Retrieval";

• During 2004 we will develop general concept of a semantic wrapper for existing heterogeneous non-Semantic Web resources. Such a wrapper will be represented by Generic Semantic Search Facilitator software for upgrade of existing search services with advanced ontology-based information retrieval capabilities and facilitation of better search results within heterogeneous information repositories. A concrete software prototype will be developed and delivered:
• Prototype of Semantic Search Facilitator for Internet-search engine (e.g. Google) that enhances performance of user interface for semantic search as well as enables software applications to automatically use semantic search facilities.
• Business plan for new ideas and tools of this project

Lisätietoja yms.:
More Information etc.:

InBCT 3.1 Research Plan (DOC, 363 KB)

Presentations:

German N., Zhovtobryukh D., Multiservice Provision in Wireless Mobile Environments (PPT, 253.5 KB)

Presentations and Research Papers of Industrial Ontologies Group:

http://www.cs.jyu.fi/ai/OntoGroup/index.htm

Semantic Web: The Future Starts Today (PPT, 1.2 MB)

Terziyan V., Advanced Diagnostics Algorithms in Online Field Device Monitoring

Publications with papers on-line:

http://www.cs.jyu.fi/ai/vagan/papers.html

Terziyan V., Vitko O., Probabilistic Metanetworks for Intelligent Data Analysis, Artificial Intelligence, Donetsk Institute of Artificial Intelligence, Vol. 3, 2002, ISSN 1561-5359, pp. 188-197.
URL: http://www.cs.jyu.fi/ai/papers/IAI-02.pdf

Terziyan V., Intelligent "Mirror Web Browsing" vs. Pull/Push Technology, Eastern-European Journal of Enterprise Technologies, V. 1 No. 1, 2003, "Technology Center" Foundation, Kharkov, Ukraine, pp.4-14.
URL: http://www.cs.vu.nl/~atjournal/V1/etj_1_1_1.pdf

Ryabov V., Terziyan V., Industrial Diagnostics Using Algebra of Uncertain Temporal Relations, In: Proceedings of the 21-st IASTED International Multi-Conference on Applied Informatics (AI-2003), February 10-13, 2003, Innsbruck, Austria, ACTA Press, ISBN 0-88986-341-5, ISSN 1027-2666, pp.351-356.
URL: http://www.cs.jyu.fi/ai/papers/AI-2003.pdf

Veijalainen J., Terziyan V., Tirri H., Transaction Management for M-Commerce at a Mobile Terminal, In: "Mobile Commerce: Core Business Technology and Intelligent Support", Proceedings of the 36-th Annual Hawaii International Conference on System Sciences, 6-9 January 2003, Hilton Waikoloa Village, Hawaii, 10 pp.
URL: http://cosco.hiit.fi/Articles/hiccs36.pdf

Terziyan V., Ontological Modelling of E-Services to Ensure Appropriate Mobile Transactions, In: Proceedings of the Workshop in conjunction with ES2002: the 22nd Annual International Conference of the British Computer Society's Specialist Group on Artificial Intelligence, Peterhouse College, Cambridge, UK December 10th 2002, pp. 13-26.
URL: http://www.csd.abdn.ac.uk/~apreece/es2002/workshop/proc.pdf

Narciso Gonzalez Vega: Deliverable: WDL:n Ajosimulaattorin selvitystyö (DOC, 1.2 MB)

Human performance in simulation environment - Report Of Activity (DOC, 69.5 KB)

Industrial Ontologies Group - Project Report May 2004 (DOC, 368.5 KB)

Industrial Ontologies Group - Results, achieved during the first period of the project (1.1.2004 - 31.5.2004), comparing with the goals of the research plan (DOC, 47 KB)

Semantic Search Facilitator: Concept and Current State of Development (PPT, 3 MB)