SmartDataLake - SYNYO GmbH

SmartDataLake: A framework for making data lakes smart

leo — Fri, 31 Dec 2021 07:42:46 +0000

SmartDataLake

SmartDataLake: A framework for making data lakes smart

Data lakes are raw data ecosystems, where large amounts of diverse data are retained and coexist. They facilitate self-service analytics for flexible, fast, ad hoc decision making. SmartDataLake enables extreme-scale analytics over sustainable big data lakes.

SmartDataLake provides an adaptive, scalable and elastic data lake management system that offers: (a) data virtualization for abstracting and optimizing access and queries over heterogeneous data, (b) data synopses for approximate query answering and analytics to enable interactive response times, and (c) automated placement of data in different storage tiers based on data characteristics and access patterns to reduce costs.

The data lake’s contents are modelled and organised as a heterogeneous information network, containing multiple types of entities and relations. Efficient and scalable algorithms are provided for: (a) similarity search and exploration for discovering relevant information, (b) entity resolution and ranking for identifying and selecting important and representative entities across sources, (c) link prediction and clustering for unveiling hidden associations and patterns among entities, and (d) change detection and incremental update of analysis results to enable faster analysis of new data.

Interactive and scalable visual analytics are provided to include and empower the data scientist in the knowledge extraction loop. This includes functionalities for: (a) visually exploring and tuning the space of features, models and parameters, and (b) enabling large-scale visualizations of spatial, temporal and network data. The results of the project are evaluated in real-world use cases from the business intelligence domain, including scenarios for portfolio recommendation, production planning and pricing, and investment decision making.

SDL-Virt

SDL-Virt: This layer is responsible for delivering efficient data access over a data lake containing large volumes of heterogeneous data. It encapsulates and abstracts all issues related to the efficient placement, distribution, management, and retrieval of data, providing upper layers homogeneous access to data through an SQL-like query language. Specifically, the main offered functionalities include: (i) data virtualization over different data types and formats, (ii) automated placement of data over different storage tiers to optimize the trade-off between storage cost and speed of retrieval, (iii) approximate query processing that can speed up data analysis by enabling approximate answers with theoretical guarantees for accuracy.

SDL-HIN

SDL-HIN: This layer is dedicated to searching and analysing the contents of a data lake, being represented in the form of a Heterogeneous Information Network (HIN), i.e., a graph consisting of entities and relations of different types. The offered functionalities include: (i) discovering similar or near-duplicate entities under different similarity criteria and matching conditions, (ii) ranking of entities based on the structure of the graph, as well as on other domain-specific criteria such as properties of geospatial regions, (iii) predicting or suggesting links between entities based on their attributes and position in the network, (iv) detecting communities of entities, including potentially overlapping or hierarchical communities, and (iv) detecting changes in evolving communities and in data represented as time series.

SDL-Vis

SDL-Vis: This layer includes the human in the data analysis loop through offering visual analytics capabilities building on top of the functionalities implemented in the two lower layers. Using a visual analytics model at its core to determine user interactions, it offers visualizations for supporting data profiling and parameter tuning. It also offers custom visualizations for specific types of data, including graph, spatial, and temporal. Further, individual components typically also include some sort of UI which enables the user to interact with the component directly. This is primarily to support the idea of keeping the component of being capable of both working in unity, but also individually, thus reducing the interdependencies between the components.

Links

https://smartdatalake.eu/

https://github.com/smartdatalake

Keywords

Big data, virtualization, data visualization, heterogenous graphs, meta-path, scalability

GreenEsteem: Project for sustainable Nature-Based Solutions through Co-Creation Launched

ReIncluGen: European Research Project Maps New Ways to Strengthen Support for Migrant Women

METEOR Programme development and work package and outreach updates

August 1, 2025

The post SmartDataLake: A framework for making data lakes smart first appeared on SYNYO GmbH.

The post SmartDataLake: A framework for making data lakes smart appeared first on SYNYO GmbH.

SmartDataLake: Pilot C overview, main components and data

leo — Thu, 01 Jul 2021 07:05:56 +0000

SmartDataLake

SmartDataLake: Pilot C overview, main components and data

In order to provide a balanced focus on both research and innovation in the SmartDataLake project, the project included three large scale piloting use-cases, executed by the three industry partners of the projects. These pilots focused on providing three different perspectives and application of the results of the SmartDataLake project. The article below describes the third pilot.

Throughout the third pilot use-case of the SmartDataLake project, SYNYO applied selected components from the SmartDataLake toolbox to complement the end-to-end workflow of open data processing, analysis and converting open data into insights and actionable information. This was performed primarily on datasets surrounding the European research landscape, e.g., projects performed under the H2020 programme, and the organisation participating in executing them, with a focus of applying key SmartDataLake functionalities:

Flexibly and interactively working with the data using RAW.
Enriching the core dataset by joining it with other independent datasets through entity resolution.
Analysing and evaluating the data with support from the SciNeM, Loci and SimSearch components.
Integrating the gained insights into an end-user targeting solution.

While the first two action points made working with the data easier and more flexible, e.g., by providing a flexible SQL like interface to raw data, the analysis made it possible evaluate the relevance of the data, by e.g.,

Ranking of individual entities based on their interconnection to identify dominant one.
Identifying communities of entities belonging together.
Comparing entities based on their similarity.
Applying the above in a geographical context.

The main insights gained from the analysis were then selected and, either by persisting the gained information or by directly integrating the components and their configuration, integrated into an end-user targeting application that enables a user to navigate the European research landscape, by leveraging their own knowledge as a starting point. Specifically, this includes, reflecting the above capabilities:

Ranking of organisations, based on their activity in the European research landscape.
Identifying communities of organisations based on how frequently the collaborate.
Identifying similar organisation, projects and topics to identify future collaborations.

Keywords

Big data, data lake, smart, pilot, research landscape

OpenMusE: Project Finale Sets the Stage for Data-Driven European Music Pathways

July 1, 2025

METEOR kicks off: EU-funded programme to transform doctoral training and researcher skills

BOND: Outcomes in Advancing Education, Tolerance and Heritage Preservation to combat Antisemitism

January 1, 2025

The post SmartDataLake: Pilot C overview, main components and data first appeared on SYNYO GmbH.

The post SmartDataLake: Pilot C overview, main components and data appeared first on SYNYO GmbH.

SmartDataLake: Validation of the system and individual components through long term piloting

leo — Fri, 01 Jan 2021 08:43:07 +0000

SmartDataLake

SmartDataLake: Validation of the system and individual components through long term piloting

Throughout the SmartDataLake project a strong focus has been on the combination of driving research results forward, while at the same time collecting industry-oriented feedback. In order to expand on this feedback and to proactively validate the solution, the project includes a ten months long piloting phase, during which all the individual components are applied to use-cases provided by the project’s industry partner.

Overview

In short, the pilot consists of the various components of the SmartDataLake project that are used by the three pilot partners; SpazioDati, SPRING TECHNO and SYNYO, to solve their data related challenges. In line with the rest of the project structure, the pilot focuses on the three main functionalities of the SmartDataLake project, namely; data management, data exploration and analysis and interactive visual analytics.

Overall, the main use-cases have been defined and detailed by specifying the challenges and the relevant key performance indicators as well as been mapped to the relevant SmartDataLake component that may be used to achieve these use-cases. The intermediate use-cases are as follows:

Assembling entity profiles – As even structured and properly annotated data may be distributed across different systems and instances, this functionality allows to operate on distributed data as if it was provided as a single instance of a database.
Matching company profiles across sources – When working on uncorrelated datasets, the challenge is to accurately combine these datasets into a single one, by matching the individual entities.
Computing descriptive analytics – helpful when working on a newly acquired data to gain insights into the data quality.
Finding similar entity profiles – Contrary to identifying exact matches, similar entities are helpful when identifying alternatives or just expanding the selection based on certain known parameters.
Predicting potential links – Similar to finding similar entities, but applies the relation of entities rather than the attributes of a given entity as the decisive metric.
Storage tiering for historical data – While not all data is always relevant, this use-case supports the autonomous decision-making for when what segment of data may be stored in cold storage.
Detection of similar and correlated time series – This use-case identifies similar trends in time series data.
Detection of seasonal patterns – Building on top of the previous use-case, the focus is to be able to add additional meta-information such as seasonality, when performing the analysis.
Detection of changes – As a lot of useful data is constantly evolving, monitoring the changes over time provides insights into which entities are being impacted.
Community detection and ranking of different types of entities – While investigating large data set, a helpful functionality is to be able to identify the most prominent entities, and the dominating communities. This is particular interesting when combined with change detection, in order to monitor how these evolve.
Visual analytics accompanying all the above use-cases to provide visual feedback during the analysis.

Throughout the pilots the project partners will be collaborating on executing and evaluating each of the above use-case and the components that support their implementation. The final outcomes, insights and results will be summarized and published in a project report by the end of the project, specifically in the beginning of January 2022, together with the revised repositories, where applicable, on GitHub, containing the developed components.

Links

https://github.com/smartdatalake

Keywords

Big data, data lake, smart, piloting, feedback

OpenMusE: Live Music Census tunes into health of European music scene

ReIncluGen: Participatory Action Research. An Evaluation of CSO Practices on Empowerment and Inclusion.

FU-TOURISM: Acceleration Programme – 20.000 Euros for Innovative SMEs

The post SmartDataLake: Validation of the system and individual components through long term piloting first appeared on SYNYO GmbH.

The post SmartDataLake: Validation of the system and individual components through long term piloting appeared first on SYNYO GmbH.

SmartDataLake: Data virtualization, heterogenous network mining, visual analytics and how they relate to each other

leo — Wed, 01 Jul 2020 07:47:45 +0000

SmartDataLake

SmartDataLake: Data virtualization, heterogenous network mining, visual analytics and how they relate to each other

The SmartDataLake project targets to solve key challenges when it comes to the handling, processing and analysing large data sets. These challenges include handling and processing of the data, the analysis and insight generation, and finally a visual analytics layer that facilitates the communication of the results. Together, they enable the user to rapidly go from raw data to actual insights.

System Architecture

The system architecture for the SmartDataLake project consists of three key components, each broken down into individual modules, providing their dedicated functionality. Generally, each module is implemented as a micro-service, with the intention of allowing the user to mix and match the functionality as and when they need it, while the component attribution is primarily used to provide a logical breakdown of the functionality into appropriate activities that the user is assumed to be able to perform. An overview of the components and their individual modules, or microservices where applicable, are illustrated in the figure and key functionality is described below.

Each of the three components, reflects a key step of the data processing pipeline; data ingestion/handling, data analysis and data visualization, and improves on each of the activities. Finally, as illustrated on the right-hand side of the figure, the project also considers three business use-cases, which have been used to derive requirements for the various components, and will be used as validation during the piloting part of the project, to be executed in the final phase of the project. When combined as a processing pipeline, the component enables a user to perform end-to-end discover and investigation based semi-structured, raw data, gaining early insights.

Adaptive data virtualization and storage tiering

The data management layer, also called SDL-Virt, focuses on three main aspects; 1) the ingestion of arbitrary data formats, e.g., XML, JSON, CSV, so generally applied data format, and enabling the user to operate on them though an SQL-like language. 2) to efficiently using the available hardware, CPU and GPU, to perform queries and 3) to efficiently create query plans on the fly. This layer essentially enables the user to ingest data and enable them to work on the data efficiently and through means that they are familiar with.

Heterogeneous information network mining

The analysis layer, also referred to as SDL-HIN, converts the data into a heterogeneous information network that then enables the user to perform various analytical tasks on the data, such as joining entities across datasets, ranking of the entities based on their connectivity, searching for similar entities, based on their attributes or similarities of their inter-connectivity. Therefore, the main purpose of this layer is to efficiently generate insights into the ingested data.

Visual analytics

Finally, there is the data visualisation layer, which provides the user with interactive feedback and results based on their actions and parameter choices, which are propagated to the analytics modules, in order to provide continuous feedback on the intuition that is being applied and allows the user to explore that data upon which they are working on.

Next steps

Over the course of the project, each of the mentioned components will be further developed and their performance improved, and, as mentioned, evaluated and benchmarked on an individual basis during a ten-month long pilot. The pilot, which will be executed during the last phase of the project, will enable the use-case partners of the project to apply the components of their use-cases and to provide feedback on the functional aspects of the components, their usability, in terms of e.g., manual effort and the intuitiveness of the tools, and to a large extend to evaluate the performance of the components when use to evaluate large scale datasets.

Further details about the system architecture and the components may be found in Deliverable D1.2: System architecture, available at the link below, while each of the three components will be document in their respective public deliverables by the end of the project, and will be made available on the project website, also listed below.

Links

https://smartdatalake.eu

https://smartdatalake.eu/wp-content/uploads/2013/09/SmartDataLake-D1.2-System_architecture.pdf

Keywords

Big data, data lake, smart, system architecture, microservices, distributed

CircularCityChallenge: Final project results boost innovation for sustainable development education

BioBeo: Digital Handbook empowers education stakeholders to bring innovation to schools

BOND: Toolbox for dialogue released

The post SmartDataLake: Data virtualization, heterogenous network mining, visual analytics and how they relate to each other first appeared on SYNYO GmbH.

The post SmartDataLake: Data virtualization, heterogenous network mining, visual analytics and how they relate to each other appeared first on SYNYO GmbH.

SmartDataLake: Stakeholder survey and key project insights

leo — Wed, 01 Jan 2020 08:26:54 +0000

SmartDataLake

SmartDataLake: Stakeholder survey and key project insights

In an era of data-driven decision-making, a key requirement is high-quality data upon which the necessary insights may be based. While SmartDataLake intends to improve the rapidness at which deep insights may be achieved when operating on new, large-scale datasets, the question is how this workflow can even be improved and what the specific requirements are.

Overall, the SmartDataLake project consists of both research partners, driving the research and innovation forward, as well as industrial partners, providing use-case driven requirements and evaluation of the developed solutions and components through feedback. As part of the requirements elicitation methodology, the use-case partners are involved throughout the whole project, starting with the elaboration of the use-cases and the derivation of the requirements, as well as the execution of a series of piloting activities, during which the feedback is collected based on the hands-on observations that were made. However, while the three use-case partners represent three versatile applications of the SmartDataLake solution, external inputs have also been collected in order to broaden the scope and to identify the needs from a broad range of relevant stakeholders.

The collection of additional requirements was facilitated through a questionnaire, targeting the two primary stakeholder groups of the SmartDataLake project, namely data analysts and data scientists, and went through the processes of collection feedback on their daily challenges and frequency of various data-related tasks and activities. Majority of the questions were focusing on acquiring insights in a) the various challenges they are facing, in order to identify gaps, b) categorize the tools that are being used, in order to streamline the development to be as much inline with the currently applied processes as possible, and c) collect input on the importance of the various challenges, in order to be able to prioritize accordingly.

Here, we reproduce a few key insights that were gained as part of the analysis of the answers received to the questionnaire.

While most of the insights achieved generally confirmed the overall industry trends, where, e.g., Python is dominating the landscape, due to the broad range of excellent frameworks and libraries available, it was positive to see that there is also a strong application of open and web data, as illustrated in the below figure. This could imply that past efforts to make open data available and findable are creating benefits for the end-user and emphasizing the continuation of these efforts.

Proprietary data created by your organization
Proprietary data generated by your organization’s systems/products
Proprietary data purchased from third parties
Open/Web/Crowdsourced data

Another aspect emphasized, or rather reaffirming as mentioned before, by the answers was that the questionnaire participants were still primarily using traditional and well-established tool; e.g., a combination of Python based tools such as Jupyter, Pandas and Matplotlib, and less frequently than their higher-level derivatives such as Seaborn and other tools built on top of them.

From a process point of view, majority of the participants responded that their data collection, processing and analysis work is mainly performed on an ad-hoc basis. While not highly surprising, this trend does highlight and emphasize the need for a high intuition of the tools to be developed, such that the consequence of switching between tasks is reduced. In the SmartDataLake, besides focusing on the documentation of the various tools and components, this is materialized through a strong focus on visual support being baked in into most of the components, thus, maintaining a certain level of familiarity, independent of the dataset being investigated. Further, partially driven by the feedback collection, the individual components are intended to work both in unison, as an overarching data ingestion, processing and analysis pipeline, but also as a self-contained unit, for use-cases where specific functionality is needed and integrated into new or existing systems.

Finally, when asked about concerns and barriers with regard to integrating new systems into existing workflows, the highest rated concerns included the cost of the migration to a new tool, both in terms of time and resources. Both aspects are approached by the project by focusing on integration of existing tools, e.g., Jupyter Notebook to a large extend, and to make the component as flexible as possible while at the same time act as an efficient replacement for activities that typically require manual effort.

The questionnaire will be open throughout the full duration of the project, in order to monitor how the landscape develops over time, and can be accessed using the link below.

Links

https://www.surveyly.com/p/index.php/survey/index/sid/515498/newtest/Y/lang/en

Keywords

Big data, data lake, smart, questionnaire, requirements

FU-TOURISM: FU-TOURISM participates in Second Anniversary Celebration of the Tourism Pathway in Europe

June 1, 2024

ALLIES: Pilot and Validation Workshop: A Pioneering Step for Online Safety

June 1, 2024

BioBeo: Bioeconomy Festival held in Brussels

May 1, 2024

The post SmartDataLake: Stakeholder survey and key project insights first appeared on SYNYO GmbH.

The post SmartDataLake: Stakeholder survey and key project insights appeared first on SYNYO GmbH.

SmartDataLake: Improving data lakes through better data management, early insights and visual analytics

leo — Mon, 01 Jul 2019 08:24:57 +0000

SmartDataLake

SmartDataLake: Improving data lakes through better data management, early insights and visual analytics

Data, without context and purpose, is largely without value, sheer rows and columns of characters and numbers and attributes. Data lakes, combine data from arbitrary sources, and pool them up in one big lake of bits and bytes, different formats, sizes and of largely varying quality. SmartDataLake, on the other hand, provides the means and the tools to find the needle in the hay stack and convert data, into information.

While the overall research strategy and goals have been rather well-defined from the very beginning of the project, a strong focus has been put on shaping the details and steering the implementation of the various components. Firstly, the overarching goals have been classified into five key challenges: handling data heterogeneity, reducing storage, making sense of the data, monitoring changes and, last but not least, keeping the human in the loop during these steps. These goals have then been derived to specific challenges, derived by both the research and industry partners, into specific use cases, as outlined in the following.

Efficient data storage and management

Handling data is managed and improved through storage tiering for historical data, where infrequently used data may be autonomously assigned to cheaper cold storages, as well as providing data virtualization functionality, that essentially enables the user to work directly on heterogenous data types and formats and to use a tool that they are already familiar with.

Scalable data preparation and analytics

Regarding making sense of the data, SmartDataLake defines a set of methodologies that facilitate typical data processing techniques but are optimized for large data sets. These include assembling entity profiles, e.g., where an entity’s attributes may be merged across different datasets, through applying entity resolution, computing and presenting descriptive attributes for the combined dataset, and then identifying similar entities for applications that require e.g., providing alternative opportunities or identifying potential, promising entities. As a related activity, one of the components focuses is to accurately predict a potential relation between two entities by looking at the relations of the given entity rather than comparing the specific attributes with regard to their similarity. The application of comparing entities based on their similarity is applied on both the individual attributes of the entity, such as their classifications or tags, their numerical values, such as the number of employees in a given organisation, as well as the geographical location, and secondly on their timely properties, e.g., when the data is provided as a time series, though time series forecasting and detection and correlation of different time series in order to support finding and identifying seasonal patterns.

SmartDataLake further provides high-level analytics functionality, which builds on the previously defined components, in order to convert the previously created rich sets of data into actionable information. These use-cases consist of ranking of different types of entities, in order to identify the most prominent ones, community detection for identifying groups of entities with a strong bond as well as detecting changes in the data, which may be used for identifying outliers and anomalies.

Visual analytics and intuitive insights

Finally, in a strive to keep the user in the loop, the primary use-cases that are in focus, and reflect the previously specified use-cases, facilitate the visual representation of the various individual steps along the processing pipeline, e.g., presentation of the computed attributes and presenting the outcomes and results of the individual components, through e.g., visual analytics of time series, data profiling and descriptive statistics, providing insights into the considered data, and predictive parameter based visualisations, that show insights into how changing a parameter would change the outcomes by showing the incremental changes side by side.

In summary, SmartDataLake makes working with heterogenous datasets, and thus gaining early insights, more efficient, faster, and, not least, smarter.

Keywords

Big data, data lake, smart, data analytics, data visualization, insights, data driven

CircularCityChallenge: Successful School Workshops to Empower Youth for Circular Futures

May 1, 2024

ReIncluGen: On what it means to be a woman. Interviews, focus groups and photography.

BOND: National Reports on Antisemitism released

January 1, 2024

The post SmartDataLake: Improving data lakes through better data management, early insights and visual analytics first appeared on SYNYO GmbH.

The post SmartDataLake: Improving data lakes through better data management, early insights and visual analytics appeared first on SYNYO GmbH.

SmartDataLake: Project on sustainable data lakes for extreme-scale analytics

leo — Tue, 01 Jan 2019 08:26:56 +0000

SmartDataLake

SmartDataLake: Project on sustainable data lakes for extreme-scale analytics

In the era of Big Data, data-driven decision-making processes are becoming increasingly data-intensive. The data lake approach refers to assembling large amounts of diverse data from a multitude of data sources, allowing the data collected to retain their original model and format, and allowing users to query and analyse them in situ. Thus, it promises to enable ad hoc, self-service analytics and to reduce the required time from data to insights and pushing premature design choices to stages where they belong.

Introduction and overview

SmartDataLake targets at designing, developing and evaluating novel approaches, techniques and tools for extreme-scale analytics over Big Data Lakes. It tackles the challenges of reducing costs and extracting value from Big Data Lakes by providing solutions for virtualized and adaptive data access, automated and adaptive data storage tiering, smart data discovery, exploration and mining, monitoring and assessing the impact of changes as well as empowering the data scientist in the loop through scalable and interactive data visualizations.

The results of the SmartDataLake project will be evaluated in real-world use cases from the Business Intelligence domain, including scenarios for portfolio recommendation, production planning and pricing, and investment decision making, thus concluding the relevance and applicability of the developed solution both within and beyond the scientific community.

Project objectives

The SmartDataLake project tackles the data lake object through three primary layers, each addressed by expert partners within their respective fields;

Adaptive data virtualization and storage tiering improving the efficiency of working and accessing large datasets.
Heterogenous information network mining to support the user in rapidly gaining insights into the data, though entity ranking, evaluation of similarity between entities and identifying communities.
Visual analytic over spatial, temporal and graph data in order to provide intuitive representation of the outcomes and results of the above data processing algorithms, and to ensure that the user is kept in the loop.

These high-level domains are further broken down into five specific challenges, namely;

Challenge 1: Handling data heterogeneity – How to operate on data from different sources and in different formats.
Challenge 2: Reducing storage costs – How to autonomously optimize how and where infrequently used data is stored.
Challenge 3: Making sense of the data – How to combine datasets and how to learn from them.
Challenge 4: Monitoring changes – How to identify how insights change over time.
Challenge 5: Support the human in the loop – How to intuitively keep the user in the loop.

Project structure

The SmartDataLake consortium consists of a total of eight partners from five European countries; Switzerland, The Netherlands, German, Italy and Austria. Furthermore, the consortium is a combination of five research organisations, focusing on the research activities; Athena RC, who is also the coordinator, EPFL, Technical University Eindhoven and University of Konstance, as well as three industry partners, SpazioDati, Spring Techno and SYNYO, who are providing the requirements derived from their day to day data analytics and data processing challenges, and, at a later stage of the project, by providing feedback on the various components developed during the project based on hands on experience, when applied to solved the previously mentioned challenges.

The project, which is funded by the European Commission through the Horizon 2020 Framework Programme, will be running for 36 months, starting from January 1^st, 2019, until December, 2021. In order to achieve its goals, the project is divided into three main phases; a requirements elicitation and elaboration phase, a research and development phase, and finally a long-term piloting phase during which the developed components are evaluated and improved based on iterative feedback from, primarily, the three industry partners.

From a project structure point of view, the project is broken down accordingly top the main challenges. The vertical requirements engineering is focused in WP1 – Requirements, Architecture and Integration, the key functional layers are each developed in WP2 – Adaptive Data Virtualization and Storage Tiering, WP3 – Heterogeneous Information Network Mining, and WP4 – Scalable and Interactive Visual Analytics, respectively. WP5 – Pilot Testing and Evaluation is responsible for the planning, execution and evaluation piloting actives, while WP6 – Communication, Dissemination and Exploitation and WP7 – Management and Coordination are responsible for the project dissemination and project administration, respectively.

Links

SmartDataLake project website: https://smartdatalake.eu/

Keywords

Big data, data lake, entity resolution, graph analysis, scalability, data visualization