User Tools

Site Tools


overview

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
overview [2018/05/09 23:38]
dmitry [Command-line execution]
overview [2019/10/31 12:59] (current)
dmitry [EasyMorph Server]
Line 1: Line 1:
-===== EasyMorph ​Overview =====+=====Overview ​of EasyMorph ====== 
 +{{:​screen-5.png?​nolink&​600|}}
  
-==== Data Import ​==== +==== Data import ​==== 
-In order to start transforming data in EasyMorph, all data subject to manipulations has to be imported into EasyMorph ​as tables. As of version ​3.EasyMorph ​supports ​the following data sources: +In order to start transforming data in EasyMorph, all data subject to manipulations has to be imported into EasyMorph. As of version ​4.EasyMorph ​can import ​the following data sources/formats
-  * Relational databases ​(Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite) +  * Native database drivers ​(Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite) 
-  * ODBC data sources ​(Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite, Amazon Redshift, Vertica, SAP HANA, Apache Hive)+  * ODBC drivers ​(Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite, Amazon Redshift, Vertica, SAP HANA, Apache Hive, Generic ODBC) 
 +  * REST APIs (JSON/XML)
   * Text files with delimiters (e.g. CSV)   * Text files with delimiters (e.g. CSV)
-  * Text files with fixed with columns +  * Text files with fixed width columns 
-  * Excel files (.xlsx, .xlsm, .xls)+  * Excel spreadsheets ​(.xlsx, .xlsm, .xls) 
 +  * XML files 
 +  * Google Sheets 
 +  * SharePoint lists
   * Qlik data files (.qvd)   * Qlik data files (.qvd)
   * SAS files (.sas7bdat)   * SAS files (.sas7bdat)
   * SPSS/PSPP files (.sav)   * SPSS/PSPP files (.sav)
 +  * EasyMorph datasets (.dset)
  
-Once tables are loaded into EasyMorph they can further be modified using transformations that are applied one after another.+Once tables are loaded into EasyMorph they can further be modified using actions (transformationsthat are applied ​sequentially ​one after another.
  
-Import is performed by import actions which are available in the Main menu, by pressing "Add action"​.+Databases can be queried using the visual Query Editor which is intended for users that don't know SQL.
  
-Databases ​can be queries using the Query Editor.+Files can be loaded by simply dragging them into EasyMorph. In many cases, it's possible to load and automatically append multiple files at once.
  
-Files can be loaded by simply dragging them into EasyMorph. 
 ==== Actions ==== ==== Actions ====
-All manipulations with tables are done with the help of various actions. E.g. renaming a column is an action. Filtering, joining, aggregations -- all are actions. ​Actions ​can be one of types:+All manipulations with tables are done with the help of various ​[[[[transformations:​start|actions]]. E.g. renaming a column is an action. Filtering, joining, aggregations -- all are actions. ​As of version 4.0 EasyMorph offers more than 110 various actions that can be divided into categories:
  
-  * Create -- actions that create or generate a table +  * [[transformations:​start#​create|Create]] -- actions that create or generate a table 
-  * Import -- actions that import data into EasyMorph +  * [[transformations:​start#​import|Import]] -- actions that import data into EasyMorph 
-  * Transform -- actions that transform previously loaded data +  * [[transformations:​start#​transform_basic|Transform]] -- actions that transform previously loaded data 
-  * Export -- actions that export data into files and databases +  * [[transformations:​start#​export|Export]] -- actions that export data into filesdatabases, applications and external systems 
-  * Workflow -- actions that help organize project workflow ​and perform various ​automation+  * [[transformations:​start#​workflow_internal|Workflow]] -- actions that organize project workflowperform various ​operations with files and external systems 
 + 
 +{{::​datamorphing.png?​nolink|}}
  
 Actions are executed consequently,​ step-by-step. One action is one step. Every action is applied to the result (output) of previous action. Therefore sequence of actions is important. The same actions in different order would usually produce a different result (although not always). Actions are executed consequently,​ step-by-step. One action is one step. Every action is applied to the result (output) of previous action. Therefore sequence of actions is important. The same actions in different order would usually produce a different result (although not always).
  
-Final state (or result) of a table is the result of all actions of this table applied consequently,​ one after another.+The final state (or result) of a table is the result of all actions of this table applied consequently,​ one after another.
  
 Some actions (e.g. [[transformations:​append|Append Table]]) require another table. These actions have two inputs -- one input is the result of previous action in the same table and the other input is the final state (i.e. the result of last action) of the other table. Some actions (e.g. [[transformations:​append|Append Table]]) require another table. These actions have two inputs -- one input is the result of previous action in the same table and the other input is the final state (i.e. the result of last action) of the other table.
- 
-Here is full list of all [[transformations:​start|actions]]. 
  
 ==== Expressions ==== ==== Expressions ====
-Some actions can use [[syntax:​start|expressions]] in order to calculate new columns or filter rows. These expressions can be composed using arithmetical ​operations, column names and functions. Read more about [[syntax:start|expressions]].+Some actions can use [[syntax:​start|expressions]] in order to calculate new columns or filter rows. These expressions can be composed using [[syntax:​operators|arithmetical ​operators]], column names, [[parameters|parameters]] ​and [[syntax:functions|functions]].
  
 ==== Derived tables ==== ==== Derived tables ====
-Derived tables is a powerful feature that allows transforming data in a non-linear way. When one table is derived from another (source) table it replicates the final result of the latter. So if the source table changes, the derived table changes automatically. In everything else derived tables is just a regular table and can further be modified using various ​transformations.+Derived tables is a powerful feature that allows transforming data in a non-linear way. When one table is derived from another (source) table it replicates the final result of the latter. So if the source table changes, the derived table changes automatically. In everything else derived tables is just a regular table and can further be modified using various ​actions.
  
-Since you can derive any number of tables from a table, and then keep transforming every derived table in its own way, you can create non-linear ​transformation ​chains. For instance, you can produce two derived tables from a table, transform each of them in its own way and then append or merge them back into one table.+Since you can derive any number of tables from a table, and then keep transforming every derived table in its own way, you can create non-linear ​action ​chains. For instance, you can produce two derived tables from a table, transform each of them in its own way and then append or merge them back into one table.
  
-Circular dependencies are not allowed in EasyMorphI.e. you can't append table A to table B if table A is derived from table B directly or indirectly.+Derived tables can be calculated //​conditionally//​ which allows designing conditionally branched workflowsFor instance, ​if a certain data quality check failed, send and email with bad records attached.
  
-==== Data Export ​==== +==== Data export ​==== 
-Exporting data is performed using export ​transformations. They can be inserted at any point of transformation sequence. Export ​transformations ​do not modify tables, they just export them into specified file format. Therefore output of export ​transformation ​is the same as input. I.e. it contains exactly the same number of rows and columns in exactly the same order.+Exporting data is performed using export ​actions. They can be inserted at any point of workflow. Export ​actions ​do not modify tables, they just export them into specified file format ​or external system. Therefore ​the output of an export ​action ​is always exactly ​the same as the input.
  
-Exporting ​data to databases is done using [[transformations:sqlcommand|SQL Command]] transformation that instructs the target database to upload a file previously created in EasyMorph.+As of version 4.0 EasyMorh can export into the following ​data formats and systems:
  
 +  * Native database drivers (Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite)
 +  * ODBC drivers (Oracle, MS SQL Server, MySQL, PostgreSQL, SQLite, Amazon Redshift, Vertica, SAP HANA, Apache Hive, Generic ODBC)
 +  * REST APIs (JSON/XML)
 +  * Text files with delimiters (e.g. CSV)
 +  * Excel spreadsheets (.xlsx)
 +  * XML files (.xml)
 +  * Google Sheets
 +  * Tableau extracts (.tde)
 +  * Qlik data files (.qvd)
 +  * EasyMorph datasets (.dset)
 +
 +==== Automation ====
 +EasyMorph workflows can have following automation:
 +  * File and folder operations (copy/​rename/​delete/​unzip/​zip/​download)
 +  * Running external applications,​ scripts and batch files
 +  * Execution of Windows shell commands
 +  * Embedded PowerShell scripting
 +
 +==== Integration with external systems ====
 +EasyMorph workflows can also integrate and interact with external systems and applications.
 +
 +^ External system ^ Integration ^
 +| IMAP email server ​ | Send / receive emails |
 +| Microsoft Exchange | Send / receive emails |
 +| SFTP/​SCP ​ | Download / upload files  |
 +| REST API | Send HTTP requests and receive responses |
 +| Microsoft SharePoint ​ | Download / upload files  |
 +| Google Drive  | Download / upload files  |
 +| Amazon S3  | Download / upload files  |
 +| Tableau Server / Online ​ | Publish extracts, trigger extract refreshes |
 +| Qlik Sense  | Trigger task reloads |
 +| SSH  | Execute commands ​ |
 ==== Auto-documentation ==== ==== Auto-documentation ====
-EasyMorph can automatically generate documentation ​from current ​transformation ​project. Read more about [[documentation|Auto-documentation]].+EasyMorph can automatically generate ​human-readable ​documentation ​in plain English for current project. Read more about [[documentation|Auto-documentation]]
 + 
 +==== Scheduling ==== 
 +[[https://​easymorph.com/​scheduler.html|EasyMorph Launcher]] is a complementary scheduling/​triggering utility that comes with EasyMorph Desktop. Launcher allows creating tasks with EasyMorph projects that triggered manually when needed, or scheduled on various schedules (Once, Continuous, Daily, Weekly). Launcher resides in the system tray, starts on Windows start and is tightly integrated with Desktop.
  
 ==== Command-line execution ==== ==== Command-line execution ====
 EasyMorph projects can be executed from command line thus allowing integration with third-party applications. Read more about [[command_line|command line parameters]]. EasyMorph projects can be executed from command line thus allowing integration with third-party applications. Read more about [[command_line|command line parameters]].
 +
 +==== EasyMorph Server ====
 +[[https://​easymorph.com/​server.html|EasyMorph Server]] is a Windows service with a web interface that employs the same in-memory engine as EasyMorph Desktop under the hood. Server allows scheduled execution of projects, sharing files and datasets between users, and hosting connector repositories. It also provides a REST API for programmatic integration with external systems.
  
 **See also:** **See also:**
-  * [[https://​easymorph.com/​learn.html|Tutorial]]+  ​* [[https://​www.youtube.com/​watch?​v=1MtXLkJzKuI|10-minute video overview of EasyMorph 3.6]] 
 +  ​* [[https://​easymorph.com/​learn.html|Illustrated tutorial]]
   * [[https://​easymorph.com/​examples.html|Examples]]   * [[https://​easymorph.com/​examples.html|Examples]]
overview.1525909121.txt.gz · Last modified: 2018/05/09 23:38 by dmitry