MODAClouds Multi-Cloud DevOps Toolbox is a set of tools and best practice methods specifically created "for Clouds", for the most demanding Multi-Cloud scenarios. MODAClouds Toolbox is here to help developers and operators to change and improve the way software is created and operated, in a more agile manner.
The toolbox helps lowering existing barriers between Development and Operation Teams and helps embracing DevOps practices within IT teams. This means that MODAClouds software tools can help organizations of any size to Build and Run Cloud Applications driven by business and technical needs and quality requirements.
However, the impact of MODAClouds Project extends outside the toolbox. Various tools have also been develop to help us use our technologies with other relevant third party tools. These tools are also shared with the community, they will help you use Multi-Cloud with various cutting edge solutions for IT automation with Puppet © PuppetLabs; for BigData with Kundera by Impetus, and for Palladio Optimization Tools. Learn more about MODAClouds By-Products here.
The toolbox is comprised by a set of components under the same idea "MODAClouds Multi-Cloud DevOps Toolbox": Creator 4Clouds, an Open Source Integrated Development Environment (IDE) for high-level application design, and Energizer 4Clouds, a Multi-Cloud Run-time Environment energized to provide early prototyping, automatic deployment and execution with built-in capabilities such as self-adaptation of applications with guaranteed Quality of Service (QoS) on compatible Multi-Clouds.
The toolbox also includes a separate tool, Venues 4Clouds, which is a decision support system that helps decision makers identify and select the Best Execution Venue for cloud applications not only considering Technical requirements, but more importantly, including the business perspective into the equation when designing and operating cloud applications.
who need to define Business-Driven Cloud strategies in which cloud applications and services assure performance, quality, cost and other requirement from a provider agnostic perspective.
who are seeking for agnostic and independent cloud frameworks and building tools and at the same time can help them later address automated runtime monitoring and adaptation based on QoS for their apps.
who need to use the right tools to automate operations based on QoS SLAs and guarantee that applications are executing over optimal performance quality thresholds or under cost thresholds with a minimum quality regardless of the provider, and migrate if necessary to other providers .
The Design-Time Platform provided by MODAClouds Toolbox is a piece of software that allows users to design cloud applications, identify components and key functions of applications, perform performance and cost evaluation, get help concerning the mapping of data to scalable databases, define Business and QoS Requirements, plan the initial deployment strategy of our application on a multi-cloud environment, and finally, choose the service provider that best suits all technical and business requirements defined for the application.
Creator4Clouds will result in an architecture inspired by the OMG Model-Driven Architecture (MDA), which is a model-based approach for the development of software systems. The MDA relies on three types of models for three layers of abstractions. The closer to the system a layer is, the more technical the description. These three MDA layers, from the more abstract to the more detailed are:
This environment is comprised by a set of tools orchestrated by an Integrated Development Environment (IDE). From the point of view of the user, this IDE is the main piece of software to be used at design time. It realizes the MDA approach proposed by the MODAClouds project, and its main output is the set of models and artefact's required by the runtime components to deploy, monitor and adapt cloud applications. The IDE provides support for a set of models such as requirements models, service definition and orchestration, data model etc. It also provides transformation, reverse engineering, traceability and document and code generation capabilities. As an Integrated Development Environment it provides a common access point for the following set of tools (its core functionality):
This environment is comprised by the following complementary tools which provide advanced services to application developers and operators.
Energizer 4Clouds Platform includes the following built in capabilities to empower Multi-Cloud DevOps:
The capabilities mentioned above are organized within the following subcomponents:
Venues 4Clouds is designed to support organizations on the task of choosing the most suitable cloud provider, or Best Execution Venue, for their Cloud applications, considering different aspects such as application architecture and more importantly form the business perspective, business and technical risk, quality and cost.
Venues 4Clouds Service Platform includes the following built in capabilities:
Venues 4Clouds is a separated component itself with its own IDE, however, deeply integrated into the whole design workflow proposed by Creator 4Clouds. Its IDE facilitates users with the analysis of the Best Execution Venue for running cloud applications, helping not only technologists, but managers collect:
MODAClouds is not only about technology. What it is important is that all MODAClouds technologies and tools provide also a methodology, a workflow that helps on both the design-time and the run-time of cloud applications. The proposed methodology and framework is also a relevant part of the results of the project. This process is detailed in other technical deliverables, however, it is worth highlighting here that this workflow exploits all the pieces described in the previous section to enable a coherent DevOps process for multi-cloud applications.
The previous picture shows an overview of the MODAClouds workflow. Various actors are involved in this process. The Feasibility Study Engineer is the one in charge of assessing if the development or migration of some application on the cloud is possible within the identified time and cost constraints, taking into account all possible risks. Application Developers are the ones in charge of all technical aspects related to the design, development and validation of the application. He/she interacts with the QoS Engineer who makes sure that the application and its allocation to the available resources fulfill the QoS requirements identified for the application. Then, Cloud App Admins or Operators are the ones in charge of overseeing the deployment and operation of the application. All these actors interact with MODAClouds tools by following the best practices and guidelines we are developing (see D11.4.1). There is not a single entry point for our workflow:
In both cases, there can be various iterations between feasibility study and the functional modeling phase. At any point of these iterations, the QoS Engineer will be involved to analyze the application model and its possible allocation on specific cloud services in order to verify the fulfillment of the application QoS. If needed, the QoS Engineer will offer to the others some alternative solutions that optimize the high priority QoS requirements.
When all these actors will be satisfied of the achieved design solution, the application code will be developed and a deployment model in CloudML will be derived. At this point, the Cloud App Admin will be able to run the deployment model through the CloudML engine and will be able to control the execution of the application and of its self-adaptation thanks to Energizer 4Clouds.
There are two major aspects that make the work flow interesting:
It is important to highlight that MODAClouds has not only created technology for MODAClouds expected goals. The technical team has created various “products” that have been developed during the project in the form of tools generated with MODAClouds technologies, or to somehow assist MODAClouds project. Some of them are built using other tools, and provide stand-alone value, or in conjunction with those other tools, for end-users with specific demands.
DICE Framework is a set of tools and techniques specifically created to help developers of Data-Intensive Applications speed up the design and deployment process while simultaneously reasoning on their product's quality. DICE Framework is here to help developers automate their workstreams and make product delivery pipeline quality-aware and agile.
The Framework consists of Development Tools, which primarily focus on the development stage of a Data-Intensive Application, and Runtime Tools, which collect data during the application quality testing to characterise the efficiency, reliability and correctness of the components. This dual approach helps lowering existing barriers between Development and Operation Teams, facilitating shorter lead-times and embracing DevOps practices within IT teams. This means that DICE Framework can help organisations to Build and Run Data-Intensive Applications driven by quality requirements in a more efficient manner.
Learn more about technologies available
The tools comprising the Framework can be divided into three functional groups:
Model Data-Intensive Applications with one thing in mind: Quality Requirements
DICE Integrated Development Environment (IDE) is the central element of the DICE Framework, where the developer specifies the Data-Intensive Application using a Model-Driven Engineering approach. From the point of view of the user, DICE IDE is the main piece of software to be used at design time.
DICE IDE facilitates creation of an architecture inspired by the Model-Driven Architecture (MDA). MDA is a design approach for the development of software systems. The MDA relies on three types of models for three layers of abstractions. The closer to the system a layer is, the more technical the description. These three MDA layers, from the more abstract to the more detailed are:
Such models can be aligned by DICE model-to-model transformations that are automatically executed within the IDE to reduce the amount of manual work required from the developer.
The DICE Profile is a UML Profile based on MARTE (Modeling and Analysis of Real-Time and Embedded Systems) and DAM (Dependability Modeling and Analysis) profiles. The DICE Profile introduces concepts needed for the modeling of DIA at the different abstraction levels of DICE described above (DPIM, DTSM and DDSM).
A set of tools orchestrated by the DICE IDE allows to assess application's quality properties (efficiency, costs, safety/correctness, etc.).
DICE Deployment Modelling Tool provides a simple visual specification of the DDSM model to produce a fully deployable TOSCA blueprint in a manner of minutes in order to speed up the deployment process.
Continuously deploy and test Data-Intensive Applications in a testbed.
This environment is comprised of the following complementary tools which provide advanced services to application developers and operators.
The workflow and features of the Continuous Delivery & Testing Toolbox are described below:
After the initial prototyping of the application, the developer will request to deploy the current prototype. After an automatic commit of all models and code to the external repository, the continuous deployment tool will retrieve a copy of them from the repository, build the application, and internally store the outputs and their associated artifacts. The delivery tool will then initialize the deployment environment (if not already created), consisting of VMs and software stack, and deploy (or update the existing deployment of) the application.
The quality testing tool will support the generation of test workloads to the application. Such workloads are those that will be used to assess the quality of prototypes. Similarly, the fault injection tool will generate faults and malicious interferences that can help to verify the application resilience. They will also be exploited by the configuration optimization tool to generate an experimental plan automatically given a time budget. The output of this tool is to confirm the optimal configuration of the deployment for an application in its final stages before being pushed to production. Compared to the optimization tool of the design time, configuration optimization will also deal with technology-specific parameters, which are difficult to model in design-time exploration. Moreover, configuration optimization is black-box and solely measurement-driven, whereas design space exploration is primarily model-driven.
Monitor and iteratively improve your application
DICE monitoring platform (DMon) is at the heart of the DICE Feedback and Iterative Enhancement Tool. It collects, stores, indexes and visualizes monitoring data in real-time from applications running on Big Data frameworks. It supports DevOps professionals in iterative quality enhancements of the source code and with the optimisation of the deployment environment. DMon is a fully distributed, highly available and horizontally scalable platform.
The platform is able to monitor both the infrastructure (memory, CPU, disk, network etc.) and multiple Big Data frameworks, currently supported being Apache HDFS, YARN, Spark, Storm and MongoDB. The core components of the platform and the node components running on the monitored cluster are easily controlled thanks to a Web-based user interface. Visualization of collected data is fully customizable and can be structured in multiple dashboards based on developer's needs, or tailored to specific roles in the organization, such as administrator, quality assurance engineer or software architect. DMon is already integrated with the DICE Continuous Integration tool
The Anomaly Detection and Trace Checking Tools query the monitoring platform for relevant metrics and use them to generate analyses concerning anomaly in performance, reliability or operational behavior of the application at a given release version. The Anomaly Detection Tool reasons on the base of black-box and machine-learning models constructed from the monitoring data. Conversely, the trace checking tools are going to analyze the correctness of traces. These analysis results will be manually retrieved by the developer from the IDE plugin. The Enhancement Tool's objective is to close the loop between runtime (monitoring data) and design time (models and tools) by automatically annotating the DICE models stored in the repository with statistics on the inferred and recorded monitoring data, helping the user to inspect the root-causes of performance or reliability anomalies.