Version - 2.1.0

• ❇️ New
•   CSV data sources can now use files with any extension as a CSV entity, a filetype filter has been added when creating CSV entities   (ref 5805) 🆕[04/10/2023]
•   Added Primary and Foreign key schema information for Progress   (ref 5660)
• #844   Added write options for the CsvFileWriter   (ref 5137)
•   Progress DB (OpenEdge) read-only data source Community Technology Preview   (ref 5634)
•   Implemented Progress Data Source   (ref 5633) ℹ️
•   Implemented Progress Reader into IDC   (ref 5547) ℹ️
•   Improved error messages when data sources fail to connect   (ref 5334)
•   Added validation for SAPHana connections   (ref 5147) ℹ️
•   Added RRWeb Replayer feature to record and replay the last 10 minutes of user actions   (ref 5332) ℹ️
•   Added a migration for abp upgrade changes   (ref 5381) ℹ️
•   Added Enable/DisableChecksOnModifiedRecords to update using unique columns instead of all values   (ref 4893)
•   Enabled Postgresql in the setup scripts   (ref 5257) ℹ️
•   Upgraded ABP 5.3.5   (ref 5280) ℹ️
•   Added examples of OLAP table behaviour to the manual   (ref 5178)
• #674   Added more AddText transforms and added column functionality to it. AddColText has been deprecated   (ref 5222)
•   OLAP analysis is now toggleable with a checkbox on the Run Script screen for Profiling stages   (ref 5236)
•   Implemented OLAP Analysis along with DuckDb as Reader/Writer   (ref 5165)
•   Added a seperate section in Swagger documentation for available iData Client APIs   (ref 5091)
• #626,#652   Project name, link and filters added to admin job queue page   (ref 5079)
•   Implemented both Log and LogLine transformations   (ref 4915)
• #597   Remove multiple data entities at once   (ref 4984)
• #272   Added copy to clipboard button for Machine Id   (ref 4911)
•   Added entity create feature for creating/copying Data Entities   (ref 5075)
•   Added a analyse tab to the profiling results page   (ref 5077)
•   Exporting a project now includes triggers, the export version increased to 2   (ref 5041)
•   Implemented new Triggers system that replaces the existing Schedules system with expanded functionality   (ref 5046)
•   Added a button to regenerate the client secret while editing client access   (ref 4912)
•   Added size of files in the virtual file system   (ref 4693)
•   Added datasource type licence restrictions   (ref 4858)
•   Added a user interface guide   (ref 4850)
•   Json files now can be imported as part of a script   (ref 4722)
• 552   Added open source licences to the Help page   (ref 4856)
• 🔀 Changed
•   Made trigger stage action parameters fill the card   (ref 5551) 🆕[04/10/2023] ℹ️
• #865   Improved ingest performance when file has a large amount of columns   (ref 5693) 🆕[04/10/2023]
• #853   Optimise updates when the source and destination are the same   (ref 5668) 🆕[04/10/2023]
•   Profiler selections use a list internally instead of a string   (ref 5761) ℹ️
• #876   Front files given unique ID's at each build to prevent using outdated cached versions   (ref 5728)
• #894   Updated user interface document to include an introduction to Quick Fix Actions and a warning advising users not to delete the scripts   (ref 5745)
•   Progress Db data source is now readonly   (ref 5686) ℹ️
• #602   Updated the setup guide with improved connection string information   (ref 5638)
• #854   Allowed the user to reuse deleted trigger names by archiving the old one   (ref 5669)
•   Removed licence file reload from page theme   (ref 5597) ℹ️
• #772   The Check-login API method has been relocated to Identity -> User Information. This method pertains to the iData login status   (ref 5482)
• #740   Improved error messages for using order by in OpenDbQuery   (ref 5426) ℹ️
• #755   Added Log and Logline to the DataStream, and allowed the user to log column values directly using column references   (ref 5455)
• #730   Updated Salesforce OAuth instructions   (ref 5407)
• #756   Improved the load time of the Run History page when there's a large amount of entries   (ref 5459)
• #741   Updated documentation to explain when and why setup scripts prompt users to use update scripts instead   (ref 5433) ℹ️
•   Include validation for no destination selected in column mapping   (ref 5264)
• #748   Exporting a project now includes column mappings and created entities   (ref 5437)
•   Profiler: changed "Analysis" tab to "Warnings", and "Enable OLAP Analysis" setting to "Enable warnings analysis". When the setting is marked as False then the Warnings tab will not appear.   (ref 5277)
•   Upgraded cypress to 12.14   (ref 5363) ℹ️
• #673   Updated the deprecated section of the reference manual and improved IntelliSense   (ref 5223)
•   Set profile Unique Values counters using the distinct count from OLAP   (ref 4665) ℹ️
•   Removed permissions from LicencesInternal   (ref 5253) ℹ️
•   Removed the ability to Clone or Create custom salesforce entities   (ref 5162)
•   Added the ability to hide non-public release notes (like this one!)   (ref 5186) ℹ️
• #644   Setup scripts support existing local/domain user account   (ref 5119)
•   Refactored entity create schema backend   (ref 5176)
• #649   Running scripts from anywhere will bring up View Progress window   (ref 5126)
•   Main scripts for stages are no longer regenerated on import; added new Host function GetJsonFile   (ref 4921)
• 600   Updated the data seed error text for failed data seed generation to clarify that seed-set columns cannot be used for generating seeds   (ref 4987)
•   Updated DataGen scripts to better handle foreign keys   (ref 5090)
• 580   AutoFixAddress verification now replaces responses with a return value of 100, 200, 300 and 400 by default. WithAllowedFlags() can be used to set a custom list of allowed flags.   (ref 4892)
• #567   Display data entity name in column selection screen   (ref 4901)
•   Number of processes and keep alive time for iData Workers can now be edited via the UI. These options have moved inside of appsettings.json (See Getting Started Guide)   (ref 4206)
• #585   Importing projects will now redirect users to update datasources. Stages are prevented from running until all datasources are valid.   (ref 4928)
•   Reduced the load time of Data Entities when creating a stage   (ref 4844)
•   Refactored username lookups for datatables   (ref 4867)
• #420   The transform script output now has specific error text for attempting to use OnNthRow while no unique keys are set   (ref 4865)
• #549   Data sources are now individually licensed. An updated license file will be required.   (ref 4857)
• #417   Updated deprecated RndPatterns method in default transform script   (ref 4864)
• #534,#536   Users with user management permission allowed to enable 2FA   (ref 4810)
• ☑️ Fixed
• #902   Resolved an issue that caused the trigger screen not to render actions and conditions correctly after validation issue   (ref 5757) 🆕[04/10/2023]
• #904   Fix run history missing the report link issue   (ref 5759) 🆕[04/10/2023]
•   Fixed an issue with random person generation   (ref 5835) 🆕[04/10/2023]
• #892   Resolved an issue preventing edited triggers from being saved   (ref 5748) 🆕[04/10/2023]
• #937   Fixed a timeout issue with folder uploads   (ref 5829) 🆕[04/10/2023]
•   Fixed issue when creating entities that incorrectly requested a schema to be selected   (ref 5825) 🆕[04/10/2023]
• #905   Fixed an issue where certain table names would cause their progress bars to behave erratically   (ref 5762) 🆕[04/10/2023]
• #920   Fixed error while generating creating stage scripts   (ref 5800) 🆕[04/10/2023]
• #924   Files that are too large to edit will be opened in read-only mode, displaying only the initial portion in the editor.   (ref 5803) 🆕[04/10/2023]
• #907   Fixed issue with Progress DB Editing Entities   (ref 5766)
• #917   Fixed an issue with the QuickFixActions feature preventing the profiling page from loading   (ref 5797)
• #901,#914   Fixed issue where large file uploads caused timeouts   (ref 5726)
• #910   Fixed issue with ProgressDB not handling table names with dashes   (ref 5768)
• #895   Adjusted the update job progress logic to prevent deadlocks   (ref 5747)
• #893   Fixed issue with profiling list view when paging through results after changing filters   (ref 5746)
• #896   Resolved the issue with profiling reports when filtering on multiple tables   (ref 5751)
• #885   Fixed intermittent error where job script files could not be opened in the script editor   (ref 5739)
• #843   Fixed an issue with SetUniqueKeysFrom when keys were set on the destination   (ref 5639)
• #882   Resolved an issue where the Profiler was logging an authorised exception when running against Progress DB   (ref 5729)
• #883   Added editing for Progress DB DataSources   (ref 5730)
• #881   Added an image for the Progress DB Data Source   (ref 5737)
•   Fixed an issue handling table names containing non-standard characters   (ref 5719)
• #845   Fixed an issue with the New Entity button on empty Data Sources   (ref 5651)
• #863   Fixed an issue with hyphens causing OLAPProfiling to fail   (ref 5691)
• #836   Fixed an issue with profiling displaying null for blank strings or 0 valued ints   (ref 5620)
• #762   Resolved an issue with data generation scripts not setting unique keys correctly   (ref 5470)
•   Fixed an issue updating tables with identity columns   (ref 5667)
• #814   Fixed an issue with dependancies when a table had a foriegn key refering back to itself   (ref 5548)
• #811   Fix database locks preventing the job status screens from being up to date   (ref 5500)
• #765   Fixed issue where CSV Settings aren't being set on stage script generation   (ref 5317) ℹ️
• #773   Fixed import and export CSV file issues, and added a upload CSV file sample into TestScript   (ref 5480)
• #780   Fixed 'quick fix' error in shortest and longest profiler   (ref 5539)
•   Fixed trigger webhook issue when 'additional data' field was used   (ref 5553)
• #808   Fixed a bug with missing files when downloading folders   (ref 5528)
• #630   Fixed an issue with renaming DataSources   (ref 5576)
• #824   Fixed an error uploading profiling results for large data sets   (ref 5584)
• #802   Fixed false negatives in the licence manager   (ref 5524)
• #623   Fixed an internal error when deleting reports   (ref 5559)
• #770   Fixed documentation formatting for Trigger Api example in User Interface guide   (ref 5475) ℹ️
• #786   Updated the profile warnings UI and fixed an issue with spaces in column names   (ref 5494)
• #776   Fixed an issue when creating a Data Entity for an ingested file   (ref 5486)
• #779   Fixed rendering issue with Similar Records column selection   (ref 5489)
• #735   Fixed the upload file over 128MB issues   (ref 5535)
• #758   Custom code in Entity scripts is no longer overwritten when a stage is updated   (ref 5463)
• #809   Fixed an issue when migrating the database   (ref 5533)
•   Fixed issue that caused iData to fail to detect the database hadn't been upgraded successfully   (ref 5534)
• #732   Fixed the issue with an entity name starting with a number that was causing a JavaScript error   (ref 5536)
• #760   Fixed issues with breadcrumbs displaying the wrong path   (ref 5466)
•   Implemented changes to improve performance and prevent deadlocks   (ref 5503) ℹ️
• #801   Fixed an issue preventing the trigger success notification from sending, updated email contents   (ref 5517)
•   Fixed issues with renaming datasources   (ref 5520)
• #783   Fixed an issue where stages would fail to start in some circumstances   (ref 5492)
• #781   Resolved the issue of generating random values for address line 1 to address line 3   (ref 5493)
• #778   Deleting an executed external command trigger no longer causes an internal exception   (ref 5487)
• #780   Fixed QuickFixAction scripts and added transforms to remove character classes   (ref 5488) ℹ️
• #734   Fixed issue with CopyColVal when source column is null   (ref 5420)
• #774   Fixed upload file size limited to 4MB issue   (ref 5484)
• #775   Data Generation should now work out of the box with empty csv files and new sql tables that have not been created yet   (ref 5483)
• #773   Resolved the object reference not set error when using CopyColVal   (ref 5419)
•   Fixed an issue where mapped columns with different names would not be updated   (ref 5423) ℹ️
•   Fixed JavaScript error when changing profiling report filter table   (ref 5444)
•   Fixed issue with empty csv files causing a null reference exception   (ref 5418)
• #744   Fixed incorrect product version in release notes documentation   (ref 5431) ℹ️
• #751   Fix column mapping not refreshing after column name change issues   (ref 5441)
• #747   Threshold and Max count boxes on the profiling warnings screen will now always have enough space to show the value selected   (ref 5436)
• #752   Running a stage should no longer throw a internal error when retrieving stage id   (ref 5454)
• #742   Fixed profiling page "Range Outliers" loading doesn't update issues   (ref 5430)
• #746   Fix profile report showing HTML issue   (ref 5435)
•   Fixed an issue where datasources with no entities do not display the Create Entities button   (ref 5346)
• #704   Fix job progress page "Launched By" column sorting issues   (ref 5294)
• #250,#251   Fixed an issue preventing updates to tables with identity columns   (ref 4158)
•   Added drag and drop to ETL custom column test   (ref 5283) ℹ️
• #667   Improved the script editors ability to resolve import and reference paths   (ref 5213)
•   Fixed an issue renaming Csv Data Sources   (ref 5184)
•   Improvements and fixes for 100-login.cy   (ref 5301) ℹ️
•   Fixed various bugs on the edit entity page, mainly relating to created entities   (ref 5325) ℹ️
• #694   If a job fails any jobs that depend on it will not be run   (ref 5268)
•   Fixed issue where the Run History page would occasionally fail to load   (ref 5322)
• #680   Fixed issue that caused lookup csv files not to be closed correctly   (ref 5248)
• #684   Fixed file history not displaying edit time   (ref 5249)
• #637   Users are now redirected to Update Scripts when attempting to run Setup Scripts after iData has started successfully. Users are also redirected to Step2 of Setup Scripts to prevent password null error when starting from Step8   (ref 5102) ℹ️
•   Fixed various schedule issues from 2-0-5   (ref 5256) ℹ️
• #620   Fixed error in data gen when setUniqueKey is not applied   (ref 5048) ℹ️
• #671   Improved the performance of Salesforce inserts   (ref 5220)
• #656   Fixed issues with documentation   (ref 5132) ℹ️
• #665   Running jobs no longer stop projects from hard deleting   (ref 5210)
• #638   Amend update instructions, added Step 7 and 8   (ref 5104)
• #573   Corrected the reference manual entry for Trans.MaskExcept   (ref 3960)
• #403   Fixed an Authorisation issue preventing users without licencing permissions from viewing the Data Sources page   (ref 4908)
• #582   Updated comments in generated _Stage_jobs.js to display appropriate parameter type and name   (ref 4909)
• #591   Replace spaces in report output with marker   (ref 4955)
• #616   Fixed a namespace issue with SOAP requests   (ref 5055)
• #635   Fixed an issue causing an incorrect Run Status to be set after a failed job   (ref 5098)
• #603   User is now redirected to login page when clicking on New Project button and not logged in   (ref 5088)
•   Fixed trigger runs causing projects to not fully delete   (ref 5211)
•   Fixed the show stack trace param and improved error logging   (ref 4338)
• #623   Screen stays on last page after record deletion   (ref 5072)
• #648   Errors when exporting projects are now shown to the user   (ref 5123)
• #655   An error is now thrown if an undefined value is passed to SetColValue   (ref 5146)
•   API endpoints scripts/runs and scripts/runs-with-usernames now require a ProjectId   (ref 5092)
• #651   Fixed incorrect time filtering on Run History and Job Queue pages   (ref 5133)
•   Fixed various issues with Profiling Results Analyse View involving filters   (ref 5129)
• #633   Fixed the copy to clipboard button for updated stages   (ref 5099)
• #601   Fixed incorrect typescript labelling for methods with similar names   (ref 4986)
• #611,#613,#615   Improved the robustness of the SoapResponseAdapter along with improving the documentation   (ref 5049)
•   Resolved some issue and duplicates in the reference manual   (ref 5069)
• #629   Resolved issue with KeepFirst when used in a Data Generation script   (ref 5086)
• #604   Fixed issue when clicking on "Group On" text on similar records stages caused the incorrect checkbox to be ticked   (ref 5010)
• #617   Fixed typo and updated documentation related to 2FA   (ref 5062)
•   Fixed issue where empty CSV files would cause Validate and Clean steps to throw an error   (ref 5056)
• #627   Fix NewCategoryLabel typo in reference manual   (ref 5080)
• #621   Fixed admins being able to export projects when the export project feature is disabled   (ref 5064)
• 501   Fixed issue where cancelling a script was not cleaning up the process correctly   (ref 5045)
• #609   Removed docs button shown before login   (ref 5050)
• 630   Renaming a Data Source now renames associated Stage files, and fixed a bug that occured when renaming folders   (ref 5087)
• #624   Fixed minimum and maximum values in profiler not appearing for datetimes   (ref 5071)
•   Fixed issue where users were unable to log in for a short while after a server restart   (ref 5022)
• #594   Fixed error when clicking on an "In Progress" report   (ref 4960)
• 496   Fixed SetNullValue logic for CSVs   (ref 4741)
• #574   Add stream.VerfiyEmailFields() to iData reference manual   (ref 4897)
• #576   Fixed iData reference manual typo in TagAsDomain   (ref 4895)
• #575   Updated iData reference manual "Manually tagging" reference   (ref 4896)
• #587   Fixed issue running stages after migration   (ref 4937)
• #565   Fixed issue where the edit entity page was returning a 400 error   (ref 4903)
•   Stages of imported projects can now be executed   (ref 4913)
• #556   Fixed long banner text causes layout issues   (ref 4870)
• #249   Files uploaded to lookup folder are automatically made accessible from scripts   (ref 4854)
• #415   Implemented RndGuid() when primary key type is uniqueidentifier for default transform scripts   (ref 4863)
• #546,#547,#558   Fixed issues in creating non-word stage name   (ref 4829)
•   Data Entity dependencies are now automatically created to prevent errors from missing references   (ref 4845)
• #547   Fixed an issue where stage names with spaces would not generate scripts   (ref 4855)
• #503   iData now handles controlled shutdowns gracefully   (ref 4755)
• #279   Fixed issue where the newly created project is not selected by default   (ref 4798)
• #531   Fixed issue with log download API pointing to the wrong location   (ref 4808)
• #533   Fixed to display role names for users with the Delete Role permission only   (ref 4809)
• #543   Fixed restore user confirm message   (ref 4828)
• #535   Updated roles, permissions, and project permissions to prevent users from getting more access by creating another account   (ref 4811)

Version - 2.0.5

• ❇️ New
•   Implemented OAuth login for Salesforce Data Sources   (ref 4983)
•   On complete triggers can now send webhooks   (ref 4948)
•   Implement OEM salesforce database driver   (ref 4885)
•   Added trigger completion UI for calling a webhook   (ref 3773)
•   Added a user interface guide   (ref 4850)
• #552   Added open source licences to the Help page   (ref 4856)
• #405   Allow rows to be tagged with categories   (ref 4669)
• #385   Added Import and Export feature for Projects   (ref 4487)
• #537   Added view only mode to the Edit Data Entity screen for users without permissions   (ref 4826)
•   Added maintenance mode for orderly updates   (ref 4533)
•   Added SoapClient to allow for iData scripts to interact with SOAP API's   (ref 4705)
•   Added Host.XmlToJson and Host.JsonToXml that allow converstion between Json and XML   (ref 4707)
• #491   Added Host.GetMachineID() to get machine ID in scripts   (ref 4714)
• 🔀 Changed
•   Improved the performance of Salesforce inserts   (ref 5370)
• #577   Update iData Reference Manual regarding TagAsEmail().WithSimpleValidation()   (ref 4894)
•   A valid server certificate is now required for MSSql data sources. To proceed without a valid certificate, the “Trust Server Certificate” flag under a selected data source can be enabled.   (ref 4933)
•   Added further script injection protection   (ref 4950)
• #585   Importing projects will now redirect users to update datasources. Stages are prevented from running until all datasources are valid.   (ref 4928)
•   Reduced the load time of Data Entities when creating a stage   (ref 4844)
•   Refactored username lookups for datatables   (ref 4867)
• #420   The transform script output now has specific error text for attempting to use OnNthRow while no unique keys are set   (ref 4865)
• #549   Data sources are now individually licensed. An updated license file will be required.   (ref 4857)
• #417   Updated deprecated RndPatterns method in default transform script   (ref 4864)
•   Updated SalesForce documentation in the iData Reference Manual   (ref 4843)
• #534,#536   Users with user management permission allowed to enable 2FA   (ref 4810)
•   Clicking "Close" after ingesting a table now takes the user directly to entity creation, letting them create an entity from the ingested table   (ref 4735)
• ☑️ Fixed
• #723   Fixed an issue when comparing values to DBNull   (ref 5388)
• #720   Remove the vertical scroll bar in stage edit page   (ref 5389)
• #721   Fix display confirm button issues in column mapping page   (ref 5390)
• #250,#251   Fixed an issue preventing updates to tables with identity columns   (ref 4158)
• #710   Fixed issue with the Salesforce data source screen stopping the user from using Login With Salesforce option   (ref 5309)
• #710   Fixed an issue that caused Use Sandbox to not work correctly when using Login with Salesforce   (ref 5309)
•   Fixed issue with Excel writer that caused the 'writer must set transform state' error message   (ref 5321)
• #698   Fix 'MaxFileSizeMb' setting ignored for file uploads issue   (ref 5285)
• #692   Fixed issue with renamed files not being downloaded as part of a zip file   (ref 5267)
• #697   Fixed issue when renaming a file to an existing file name   (ref 5271)
•   Fixed an issue with some Day Of Week schedules not running   (ref 5259)
• #687   Resolved an issue with report filters   (ref 5252)
• #689   Fixed an issue that caused schedules to be disabled after editing   (ref 5254)
• #689,#681,#654   Fixes various UI bugs on the edit schedule page   (ref 5258)
• #685   Fixed issue pressing the escape key on the run script making the page unresponsive   (ref 5250)
• #686   Fix issue that stopped tables with a large number of long column names being opened in the profiling report   (ref 5251)
• #659   Fixed a issue that caused test to be filled into the Salesforce DataSource fields   (ref 5201)
• #681   Fixed schedule set multiple days issue   (ref 5245)
• #654   Fix schedule end date being unticked when edited   (ref 5142)
• #663   Resolved issue that could cause stages to fail to generate if the entity dependencies got outdated   (ref 5172)
• #647   Fixed password being shown in connection extra properties   (ref 5128)
• #640   Fixed an error that occured when submitting an invalid server name   (ref 5113)
• #642   The new Stage screen no longer throws an error when a licence is missing   (ref 5105)
• #634   Fixed issue with parameter box on the schedules page not loading correctly   (ref 5097)
• #586   Fixed authorisation error on Job Queue page   (ref 5095)
• #539   Fixed issue with schedules page showing schedules from other projects   (ref 5083)
•   Resolved an issue that stopped Excel files being written in iData 2.0   (ref 4667)
•   Fixed an issue that caused Excel files to become currupt in iData 2.0   (ref 4666)
• #619   Added bool to default MSSqlWhitelist   (ref 5060)
• #602   Added reminder to update connection string (step 5) in error message (step 6).   (ref Story 5008)
• #608   Solved null reference exception when creating Data Gen stage with a Salesforce entity   (ref 5044)
• #594   Fixed error when clicking on an "In Progress" report   (ref 4960)
• #606   Fixed running data generation on salesforce   (ref 5016)
• #576   Fixed iData reference manual typo in TagAsDomain   (ref 4895)
• #575   Updated iData reference manual "Manually tagging" reference   (ref 4896)
• #587   Fixed issue running stages after migration   (ref 4937)
• #565   Fixed issue where the edit entity page was returning a 400 error   (ref 4903)
• #566   Resolved the "Method GetScriptArguments threw ArgumentNullException: Value cannot be null. (Parameter 'g')" error message   (ref 4902)
•   Stages of imported projects can now be executed   (ref 4913)
• #556   Fixed long banner text causes layout issues   (ref 4870)
• #249   Files uploaded to lookup folder are automatically made accessible from scripts   (ref 4854)
• #415   Implemented RndGuid() when primary key type is uniqueidentifier for default transform scripts   (ref 4863)
• #546,#547,#558   Fixed issues in creating non-word stage name   (ref 4829)
•   Data Entity dependencies are now automatically created to prevent errors from missing references   (ref 4845)
• #547   Fixed an issue where stage names with spaces would not generate scripts   (ref 4855)
• #503   iData now handles controlled shutdowns gracefully   (ref 4755)
• #279   Fixed issue where the newly created project is not selected by default   (ref 4798)
• #559   Ensure all database migration steps are recorded to avoid database upgrade warning message   (ref 4879)
• #531   Fixed issue with log download API pointing to the wrong location   (ref 4808)
• #533   Fixed to display role names for users with the Delete Role permission only   (ref 4809)
• #543   Fixed restore user confirm message   (ref 4828)
• #535   Updated roles, permissions, and project permissions to prevent users from getting more access by creating another account   (ref 4811)
• #545   Fixed an issue with Profile Results that prevented List View from displaying when View All Tables was selected in the filter   (ref 4819)
• #357   User account search now returning results   (ref 4432)
• #485   Source Entities no longer fail to load when editing mappings for Single Mapping stages   (ref 4698)
• #512   Fixed file locks when writing to file in scripts   (ref 4779)
• #518   Add permission checking in File browser   (ref 4792)
• #538   Fixed view only permission closing script editor issue   (ref 4813)
• #495   Re-visited users being logged out when marked as inactive   (ref 3999)
•   Similar records no longer skips a row after matching a group   (ref 4458)
• #484   Users are now forced to select a column in Similar Records stage   (ref 4697)
• #492   Apply consistent row ordering in reports   (ref 4737)
• #493   TagAsCompanyEmail tag added to differentiate between different types of emails. Appropriate email now generated according to selected email tag   (ref 4739)
• #495   Fixed issue where inactive users were being sent to the email confirmation screen when they attempt to log in   (ref 4743)
• #523   Allow admin role users delete other users project   (ref 4794)
• #526   Datatables in modals no longer have the same fixed height as datatables on the page   (ref 4804)
• #530   Fixed typo on 2FA setting page   (ref 4807)

Version - 2.0.4

• ❇️ New
•   Added a page before ingesting a CSV file into a database that describes what the action will do  
•   Added a "Create Folder" button on CSV Datasource creation modal. This allows new folders to be used as bases for Datasources  
•   Added a "New Folder" button for Files  
•   Added a "Browse and Upload Files" option for CSV Datasources  
•   Added option to ingest CSV file into a database via the action menu  
•   Files and folders can now be downloaded via the actions menu  
•   Added ability to categories records and view them in the report  
• 🔀 Changed
•   Removed SetColumnToChar due to SQL incompatibility and JSON object issues  
•   #442 update EULA in installer  
•   Removed legacy Host.NewDbCompare  
•   Depreciated SearchGroups for SearchGroup while an issue with multiple runs is resolved  
•   #487 Default data generation scripts for number columns will now generate data relevant to that field type instead of just integers  
•   Moved the "Upload Folder" button for CSV Datasources to the top of the modal  
•   The folder rename dialog in Files no longer requires you to manually remove the backslash and is now titled "Renamed Folder"  
•   Changed how client secrets are stored, this may require creating a new client secret for the worker via the Administration -> Settings -> Worker Settings menu  
•   #479 iData virtual file system storage and performance improvements  
•   Csv datasource re-enabled  
• ☑️ Fixed
•   #490 Fixed issue involving characters in file paths  
•   #469 Fixed issue where execution time was not taking UTC into account  
•   #454 Fixed issue where data type mappings are not applied correctly when comparing a CSV to a table  
•   ETL tooltips now show the names of each row state  
•   Fixed issue with updating progress bars that caused exceptions in the log and the progress bar to not show in the UI  
•   Fixed issue where DBMigrator was not accepting some characters as symbols  
•   #463 Fixed ussue on Schedule page where parameters with checkboxes where not working correctly  
•   #450 Fixed grammar in the limited results text for the report details page  
•   #480 The new CSV Datasource dialog now fits the screen  
•   #464 Permission are now all visible without scrolling  
•   #382 Fixed similar records skipping rows when two groups are close together  
•   #497 Fixed the select data source screen not showing data sources  
•   #506 Fixed edit CSV Datasource no responses issue  
•   #514 Fixed issue where Host.ExitCode was being ignored. Negative values will now cause the script to fail  
•   #527 Fixed new CSV Datasource page upload folder button disappear issue  
•   #529 Fixed stage page and others pages that were not displaying results because the user who created the item had been deleted  

Version - 2.0.3

• 🔀 Changed
•   Disabled csv datasource until #479 resolved  

Version - 2.0.2

• ❇️ New
•   On application launch systeminfo.txt is written to ProgramData\IntelligentDeliverySolutions\iData  
• 🔀 Changed
•   iDataConsole now uses the same encryption as iDataSaaS  
•   #438 Updated 2FA warning message on users page to be more readable  
•   #446 Auto focus username field on login screen  
• ☑️ Fixed
•   No longer shows UpdateProgress exception in script logs when a job ends prematurely  
•   Fixed tooltips on long report values that overflow in Similar Records  
•   A CSV with a duplicate header no longer causes an unhandled exception  
•   #384 Fixed issue where users without CreateProject permission didn't have breadcrumbs  
•   #423 Fixed issue where script path would hide script editor controls  
•   #444 Fixed issue with incorrect filters showing on the report page  
•   #453 Cancelling a job no longer throws a stack trace to the console output  
•   #462 Fixed Typo on Log Settings Screen  
•   #468 Fixed CSV Download issue  
•   #470 Fixed issue allow editing entities with same name on different datasource  
•   #471 Fixed where cancelling a stage generates an error and fails to stop the stage from running  

Version - 2.0.1

• ❇️ New
•   Web user interface  
•   Integrated editor  
•   Datasource and Data entity management through the User interface  
•   Manage and update stages through the user interface (Replacing wizards)  
•   Schedule execution  
•   Interactive reports and drilldown  
•   Projects to organize work and users  
•   Run multiple jobs in parallel  
•   Auto logout user when the user is deleted or has permission change  
• 🔀 Changed
•   Removed support for legacy configuration files  
•   Removed support for legacy wizards. Host.HeadlessWizard, Host.NewWizard, Host.MessageBoxYesNo, Host.MessageBoxWarning have been removed  
•   Host.Exec is deprecated, use javascripts import/export statements. Find out more here: https://javascript.info/import-export  
•   EnableNewColumns() on text writers has been replaced with PreventNewColumns()  
•   Removed support for RemoveDuplicates transform  
•   Updated the UI so only one column can be selected for grouping on SimilarRecords  
• ☑️ Fixed
•   #261 & #231 Fixed sorting by name on project and stage screens  
•   #260 Changes to stages through the UI overwrite manual changes to scripts  
•   #254 Unhandled data types now fails gracefully, however further change will be made in #259  
•   #249 Manually adding a lookup file updates lookups.js to support code completion in the editor  
•   #248 Renaming a lookup file removes contents  
•   #246 Users with execute rights don't have to be in the admin group  
•   #245 Incorrect version number returned from command line  
•   #244 RndIntBetween added for generating inclusive numbers, help for RndIntRange updated to state it generates numbers excluding the end value  
•   #243 Set generate 0 or 1 values for bit fields  
•   #242 ConvertToText now takes requested region and culture into account when formatting dates  
•   #239 Column names with apostrophe's are now handled correctly  
•   #237 #228 #227 #226 #220 Cosmetic layout fixes  
•   #233 Exported results location shown with link  
•   #232 Incorrect number of pages in results  
•   #230 Re-ordered profiling list view filters into logical groups rather than alphabetically  
•   #229 Increased profile results size limits to handle larger tables  
•   #213 Favicon now displayed on login screen  
•   #184 Reports and stages can be deleted to manage database size  
•   #181 Running jobs can be cancelled  
•   #178 Timeout profiling large tables  
•   #168 Entities can be added or ammended after creation  
•   #241 Fixed Inconsistent timezone on Job Progress  
•   #224 Removed references to wizards on some screens  
•   #235 LetsEncrypt certificates can be requested via to the UI  
•   #256 Solved issue with Intermittent failure of scheduled jobs  
•   #255 Action button for schedule missing under some circumstances  
•   #263 MakeRndCardCols returns an error  
•   #253 rename GetDataEntity to GetDataEntityConfig for clearity and implement RndRecordFrom on DBQuery  
•   #259 Creating stage where entities have foreign keys to other ommited entities cases error 500  
•   #262 Error using 'Run again' from job progress or run history  
•   #276 The iData worker is now setup on application start if not already configured  
•   #274/#275 Fixed an issue where the admin role was not being properly checked for when running a script  
•   Resolved a issue when running a job "Issuer name does not match authority"  
•   #269 iData currently does not support MSSQL Spatial data types such as Geography  
•   #269 Skip unreadable data types such as spatial  
•   #300 Fixed link to job in the notification Email from showing an error when clicked  
•   #291 Fixed an issue where CSV entity names containing spaces would generate bad variable names  
•   #288 Resolved an issue when deleting a stage  
•   #297 Solved issue with uploading licence  
•   #343/#344 Fixed various styling and css issues  
•   #326 Solved issue where not all profilers where shown when hitting the 150 column limit  
•   #343/#344/338/339 Fixed various styling and css issues  
•   #323 Fixed AutoMapping CSV to SQL for Wide Table  
•   #355 Solved issue with referential integrity and data generation  
•   #356 Improved register validation to prevent User Enumeration  
•   #317 Set Similar Record modal width to prevent overflow of mapping container  
•   #318 Documents should no longer be incorrectly translated by the browser  
•   #347 Fixed an issue where new user accounts weren't being activated  
•   #309 Removed exec from user help commands  
•   #367 Fixed typo on LetsEncrypt screen along with adding a domain field and some instruction on how to change it  
•   #380 Added Two Factor Authentication  
•   #381 Added "Use Domain in Credentials" to email settings screen to allow for Gmail credentials to authenticate correctly  
•   #386 Fixed Number of Columns Compared counter not reporting anything  
•   #387 Fixed issue running stages with long names  
•   #396 Fixed CSV to CSV data mappings for strings  
•   #397 Fixed 500 error when opening a comparison report with mapped columns  
•   #400 By default we now limit the number of rows reported in each ETL and Comparison report and increased timeout on query to allow for bigger reports to be loaded  
•   #410 Resolved issue where deleting a report that is being processed courses an exception  
•   #412 Fixed 500 Error on the Settings screen  
•   #424 Fixed incorrect reference to step 4 in getting started guide  
•   #426 Updated formating on stage completion emails  
•   #432 Users are now prompted to login after registration. New Users get a welcome email  
•   #433 Fixed issue with apostrophe in CSV column name breaks scripts  
•   #449 Uploading a file no longer creates a folder with the same name  
•   #452 Encoded modal header titles before displaying them to prevent script injection  
•   #457 Fixed issue with default csv options that caused issues reading headers  

Version - 1.8.1

• ❇️ New
•   new data generation method GetRndRecord can be used to select multiple fields from a randomly selected lookup record  
•   values can be set using a 'Get' from another column when using SetText, SetInt etc  
•   RndRange transforms can now accept range parameters using a 'Get' from other columns  
•   values can be assigned using a general 'Set' that uses the data type of the parameter  
•   added RndDateTimeRange which returns a random date and time between the specified values  
•   AddressValidationAPI has an option to refresh the cache once the records are older than a configurable amount of days  
•   added request logging to AddressValidationAPI using LogRequestContaining config setting  
•   AddressValidationAPI cache is now encrypted  
•   MakeRndPersonCols when used for obfuscating guarantees a generated name will be different from original name  
•   Domains for generated email addresses have been updated to avoid clashing with real email addresses  
•   Limited SetMaxUniqueValues(int) to 100,000 to ensure stability when profiling  
•   Added ReplaceControlCharacters to remove control characters.  
•   Added ReplaceMultipleSpaces to replace multiple spaces.  
•   Added ReplaceBlanksWithNulls to replace blanks to Null.  
•   ConvertToText can now support non-numeric format specificer and culture code.  
•   Added data type validation to Csv readers when types are defined by the user, with functions to handle validation errors, OnReadErrorContinue() and OnReadErrorAbort()  
•   Allow scripts to prevent automatic failure of a row after validating an email or address and implement their own handling of the issue including retrieving the error message. See IValidateCondition functions  
• 🔀 Changed
•   #580 AutoFixAddress verification now replaces responses with a return value of 100, 200, 300 and 400 by default. WithAllowedFlags() can be used to set a custom list of allowed flags.  
•   IMPORTANT: Selecting random values based on weighting was found to be inaccurate and has been updated. Random selections with weightings will generate a different selection of results after this change.  
•   when using Get to obtain a reference to a field a default value can supplied if the field isn't found  
•   RndDateRange returns dates with time set to midnight see RndDateTimeRange  
•   Improved the profiling logic for detecting a columns data type  
•   Parameters on Transformations and Conditions now accept column references  
•   Skipping a row now deletes it from the output  
•   Column name references available for more functions, updated list in iData Reference Manual  
•   SetTimeZoneOffset replaced with SetUTCTimeZoneOffset and SetLocalTimeZoneOffset  
•   TestDataSet values set with NextXXX or ForEachXXX are now set before all other transforms take place, rather than after, so their values can be used by other transforms  
•   AutoFixAddress verification now maps addresses containing a Company Name, Property Name, or Locality, to the street/town columns  
•   AutoFixAddress verification only replaces responses with a return value of 100 and 400 by default, WithAllowedFlags() can be used to set a custom list of allowed flags. More information on flags can be found in the docs.  
•   Removed Binary Transforms as they were not useful in their current form  
•   Amended ReplaceNearest to support default value  
•   Added RntIntBetween which generates values including the maximum value  
•   Generated credit cards now give an invalid checksum by default. Added EnableValidCheckDigit() to the tag settings toa allow valid checksums to be generated.  
• ☑️ Fixed
•   #257 Fixed inconsistent report rows order output  
•   Lookups from csv files will use column types defined using SetColumnToXXX  
•   Fixed infinite data generation when no row limit or foreach  
•   Report viewer now correctly reports invalid filenames instead of reporting missing files as invalid encryption keys  
•   Report viewer now correctly handles splitting of files correctly  
•   FormatText now works on fields that contain a NULL value  
•   Fixed error when using SetPatternLiteral  
•   Fixed 'UpdateColumnDefinition' which now correctly increases column sizes when ingesting csv data  
•   Miscapitalised column names used in SetColumnMapping no longer throw an error with counter group '???'  
•   AddressValidationAPI no longer requires external dependencies to run  
•   AddressValidationAPI will now create missing cache  
•   Stopped Abs transform displaying warning for positive numbers  
•   Fixed an issue when generating data that would cause the primary keys to be set to null  
•   Fixed NULL value handling when used with references to other fields  
•   #225 Fixed an issue when generating data that would cause the primary keys to be set to null  
•   CSV entries longer than 8000 characters will no longer throw an error  

Version - 1.8.0

• ❇️ New
•   Reports are compressed and encrypted to a single .idr report file by default  
•   Added SetVirtualFolder to filemanager to direct all folder contents to a single report file  
•   Encryption keys can be created and set through a encryption key pass phrase in the user interface  
•   Installer now deploys Report Viewer to view .idr files  
• 🔀 Changed
•   Profilers are now listed alphabetically  
• ☑️ Fixed
•   Fixed issue when using multiple lookups in one script  
•   Fixed profile reports with group results  
•   Fixed opening reports on a network drive accessed via a UNC path  
•   Fixed AutoFix address output miscapitalising words  
•   Fixed error when address config not set  
•   Fixed regex match error when the db value is null  

Version - 1.7.2

• ❇️ New
•   Added RestClient to allow for iData scripts to interact with Rest endpoints  
•   Added TextFileReader to allow for any text based file to be open and be used in a datastream  
•   Added TextFileWriter allowing transform engine to write to text files  
•   Added RestartLimit and RestartDisabled for Next lookup transforms  
•   Added SetTwoDigitYearMax to set the default century for two digit year date formats  
•   Added SetColumnToGuid to database readers to ensure unique identifiers are compared correctly  
•   Allow csv writers to append or overwrite existing files  
• 🔀 Changed
•   Updated ETL wizard to EnableIdentityInsert when target has identity column.  
•   MakeRndAddressCols() can use existing country or region or filter by country and region with UseCurrentCountry UseCurrentCountryAndRegion options  
•   Removed VaryBy()  
•   Added foreign key references to wizard script  
•   SetColumnToDateTime now also takes optional DateFormat to override the default  
•   CreateTableIfMissing now uses the input column name not a normalised version  
•   AddRecord now uses Upsert when there are no comparisons  
•   Added column mapping to the source stream, a compare is no longer necessary to map columns, compare mapping is being deprecated  
•   AutoFix now capitalises location names, and places houses in address line 1  
•   Dates are now formatted to the local time before being written to reports  
• ☑️ Fixed
•   Fixed data generation resetting identity columns  
•   RepeatRnd() changed to inclusive range to match Repeat() and missing ref bug fixed  
•   Allow updates to unique keys when source or destination have no rows  
•   Ensure error messages are reported for all row states  
•   Fixed error when uploading from excel sheets with numbers for sheet names  
•   Fixed TrackRowUpdates when used with existing data  
•   Fixed Importing csv file with only headers  
•   Fixed wild cards not applying tags to each column  
•   Fixed cardinality max value estimator not tracking past the set limit  

Version - 1.7.1

• ❇️ New
•   DbWriter and SQLWriter now supports creating and updating schema using CreateTableIfMissing, CreateColumnIfMissing and UpdateColumnDefinition.  
•   Added ExcelWriter allowing transform engine to write to excel sheets  
•   Added support for password protected excel files  
•   Added ExtractRegex() transform which allows text matching a given regex pattern to be extracted and optionally replaced with other text  
•   Added ToCol() which allows text extracted from ExtractRegex() to be placed in another column  
•   Added Company, Email, and Phone number to random address generation  
•   Added SetPatternLiteral() to toggle between pattern generation and literal characters in RndPatterns()  
•   Added Single, Double, and Triple word occurance count to the profiling report  
•   Added AddressLine1, AddressLine2, and AddressLine3 for generic columns  
•   Added SetColumnMappingFrom() to allow the use of lookup tables to map compared columns  
•   Added EnableAddressFixLookupService to enable the use of AFD instead of LibPostal to clean addresses, alongside EnableAddressFixMachineLearning, default option is set in AppSettings.ini  
•   Added abort to transforms when too many writes fail in a row can limit can be set with SetWriterMaxWriteFail  
•   MakeCol now accepts Wildcards to select multiple columns at once  
•   MakeCol now accepts multiple columns to be passed at once  
•   Added ExcelReader allowing transform engine to read from excel sheets  
•   ExcelReader stops reading when blank rows found  
•   New Licencing system. Contact Intelligent Delievery Solutions for support.  
•   TrackRowUpdates() now writes comments to SQL, CSV and Excel files  
•   iData Address Validation API is now packaged with the iData install  
•   Added date and time to console logs  
•   Added new conditions to match profiler results. E.G. HasPunctuation  
• 🔀 Changed
•   Lookup matches not support patterns  
•   Merged CardPrefix and CardPrefixRange for card definition in AppSettings  
•   Cards higher than 19 digits can now be validated  
•   AutoFixAddressFields now works on tables with joined number and street fields  
•   Reports no longer show [no value] when one of the rows is entirely missing. e.g. No Src/Dest or Add and Delete  
•   Column values can now be used as parameters in conditions such as Contains  
•   You can no longer edit unique key columns while comparing unless the destination table is empty  
•   Limited the UI log output to 1000 lines  
•   AutoFixAddressFields and VerifyAddressFields now work with group tags  
•   IsValid and NotValid has been added to functions VerifyEmailAddress, VerifyAddressFields, AutoFixAddressFields, CheckForeignKeys, and CheckUniqueKeys  
•   Generate more diverse set of company names  
•   By default tables stop at the first empty row, and the first empty header if headers are set  
• ☑️ Fixed
•   Prevent error when uniqueidentifier is table key and also using Rnd transforms  
•   Fixed system buffer version mismatch  
•   Fixed spelling on some Rnd... transforms from Rng...  
•   Fixed filter buttons sometimes not showing up on ETL reports  
•   Set RndPatterns() to accept punctuation.  
•   Report links with spaces now work  
•   Csv files with trialing spaces in headers now work  
•   Allow duplicate values in data for In and Replace operations  
•   Fixed some formatting issues in the documentation  
•   Data writes take column mappings into account and shows correct mapping in headers in report  
•   Fixed showing mapped columns in report when writing to empty table  
•   Fixed table name validation issue  
•   Speacial characters in excel sheet names now upload okay  
•   Address generation email column not defined and street address default seed  

Version - 1.7.0

• ❇️ New
•   More detailed stack trace for errors  
•   Added NextGuid(), RndGuid() and EachGuid() transforms  
•   Added OnNthRow() transform  
•   Added ConvertToBigInt() which converts the value to a 64 bit integer  
•   More detailed stack trace for errors  
•   Added a UniquePrefixes profiler  
•   Added RndWords() transform which generates a random Lorum Ipsum paragraph  
•   Added ability for tags to be grouped together  
•   Added enhanced Mask() which allows full or partial masking of text with a given mask character  
•   Added enhanced MaskExcept() to mask all except a given range  
•   Added Repeat/RepeatRnd modifier(s) to Each and Next Transformations to repeat the value the requested number of times  
•   Added MakeKeys and ForEachRowFrom/NextRowFrom to setup keys values from a parent table  
•   Added default value option to conversion transformations  
•   Added the ability to set the seed for randomly generated data to an alternative name instead of the column name with UseSeedName()  
•   Added AllUppercase, AllLowercase and MixedCase profilers  
•   Added Flush() to reports to write any buffered output  
•   Added ability to set RequiredRows for a TestSet  
•   Added RepeatRnd() function which allows Each and Next transforms to be randomly reset  
•   Each functions are now known as ForEach. e.g. EachText is now ForEachText  
•   Added RestartRnd() which restarts a ForEach sequence after a random amount of iterations between the given minValue and maxValue  
•   Added SetBinary(), RndBinary(), NextBinary(), EachBinary() and EachFrom versions of these  
•   Added RowLimit() to Each data items to allow for generation of data for smaller sub sets  
•   Added SetDifferenceFilename() option to comparison options  
•   All transforms can now be contained within a TestDataSet()  
•   All TestDataSet() calls must now have an accompanying EndTestDataSet()  
•   Added GetCardType() to check a card number for a matching card type, returns the found type as a string  
•   Added MatchKeysAlphbetically() method to comparison settings to allow for keys to be compared alphanumerically  
•   Added the ability to enable soundex for similar record checker via similar record tags  
•   Added the ability to ignore words for similar record checker via similar record tags  
•   Added the a sort by method the similar record configuration object  
•   Added BlankCol() to blank selected columns  
•   Added UniquePatternLengthIndependent profiler  
•   Added MakeRndCreditCardCols() to generate credit card data on tagged columns. Additionally, SetValidCardTypes() can be used to select card types to generate.  
• 🔀 Changed
•   Temp data is now written to local app data  
•   By default both MakeRndAddressCols() and MakeRndPersonCols() use predefined seeds. This can be changed with the new UseDefaultSeeds()/UseColSeeds() options  
• ☑️ Fixed
•   Prevent error when uniqueidentifier is table key and also using Rnd transforms  
•   Fixed system buffer version mismatch  
•   Fixed spelling on some Rnd... transforms from Rng...  

Version - 1.6.1

• ❇️ New
•   Added the ability to set a timeout when calling Execute() on a DBQuery  
•   Added NextIntAsc(), NextFloatAsc(), NextDecimalAsc(), NextYearAsc(), NextMonthAsc(), NextDayAsc(), NextHourAsc(), NextMinuteAsc(), NextSecondAsc() and NextMillisecondAsc() which increment a value by a given step  
•   Added the RngRegex transformation to randomly generate values that match the specified regular expression  
•   Added ability to generate synthetic data using OpenTableGenerator(). Additionally, use SetTotalRows() and UseOptimalRows to set the amount generated  
•   Added a new DataType profiler that reports back the type of each column profiled  
•   Added a new Median profiler that reports back an estimated median value when the number of unique values has been exceeded. Actual median is reported below this limit  
•   Added Host.NewDateTimeFormat() and Host.SetDefaultDateTimeFormat() which allow for parsing custom date formats  
•   Added a progress bar when the profiler is writing out html reports  
•   Added a new VaryBy transformation that can be used when obfuscating values  
•   Added OnRndRows() and OnRows() to allow for transformations to be applied on a subset of rows  
•   Added grouped profiling results to the profiling reports  
•   Added IsBlank() and NotBlank() conditional transformations  
•   Added Between() and NotBetween() conditional transformations  
•   Added option to paginate sql and csv files via a parameter on Host.WriteSqlScript() and Host.WriteCsvFile()  
•   Added the ability to edit and add new connections from the connection picker when running a wizard  
•   Added Or() conditional transformation  
•   Added SetBigInt() which sets the value to the given int as Int64  
•   Added ConvertToBigInt() which converts the value to a 64 bit integer  
• 🔀 Changed
•   MoreThan(), Equals(), AtLeast() and other comparisons will convert text values to dates before comparing them  
•   NextIntFrom(), NextDoubleFrom(),NextFloatFrom(), NextTextFrom() and NextDateFrom() have been renamed to NextInt(), NextDouble() etc when using a list of values opposed to a lookup  
•   Null values are now included in the unique values profiler results  
•   Total row count is now shown next to the Unique values on the profiling report  
•   Now when the number of unique values passes 1000(by default) an estimation of the number of unique values is provided for the profiling reports  
• ☑️ Fixed
•   Error message is displayed in reports when invalid column mappings are provided  

Version - 1.6.0

• ❇️ New
•   When updating from 1.5.* to 1.6 iData needs to be uninstalled prior to updating  
•   Added SetWriteFailedRecords() which allows failed records to be written to a separate destination  
•   Added new setting to profiler results SetMaxCharLimit() which prevents the profiling of exceedingly large text values  
•   Added NewLookup() to Host that allows the building of lookup tables from within scripts  
•   Added NextIntFrom(), NextDecimalFrom(), NextDateFrom(), NextFloatFrom() and NextTextFrom() which loop through given values replacing each value in turn  
•   Added RndTextFrom(), RndIntFrom(), RndDecimalFrom(), RndFloatFrom(), RndDateTimeFrom() and RndTextFrom() which select a random value from provided values  
•   Added RndTextPattern() which converts a pattern, e.g. 'A9A 9A9', into a random string with the same pattern e.g A7A 3R8  
•   Added SetGenerationSeedCols() to set which column values are used as the seed for generated data  
•   Added RndIntRange(),RndFloatRange(),RndDoubleRange(),RndDecimalRange() and RndDateRange() used to generate random data  
•   Added the ability to sample datastreams with the SetSampleSize() method  
•   Added the ability to generate random addresses with the MakeRndAddressCols() transformation  
•   Added Each transforms which generate data in combination with other Each transforms. Added for Text, Int, Decimal, Float and Dates. Each transforms can be used with LookupTables using Each...From  
• 🔀 Changed
•   Renamed AutoFixPhoneNumber() transformation to FormatPhoneNumber()  

Version - 1.5.1

• ☑️ Fixed
•   Fixed comparison summary not reporting differences when source rows missing from destination  
•   Fixed row count out by 1 when reading from csv files  

Version - 1.5.0

• ❇️ New
•   Added Profiling and Profiling wizard to discover data patterns and potential data quality issues  
•   Added Check for Similar Records and Similar records Wizard to search for records that could be duplicates  
•   Created data transform engine for ETL as well as comparison See NewDataTransform in docs  
•   Added transformations: ConvertToDecimal(), ConvertToText(), ConvertToDateTime(), ReformatDateTimeText()  
•   Added AutoCapitalise() transformation allowing for words to be automatically capitalised  
•   Added transformations: Trim(), TrimEnd(), TrimStart()  
•   Added the ability to tag columns as the following: TagAsPersonTitleName(), TagAsPersonFirstName(),TagAsPersonMiddleName(),TagAsPersonLastName()  
•   Intelligently clean person/name related data: AutoFixPersonFields(), DistributePersonFullName() that have been tagged as person fields  
•   Added EnableDomainCheck() on TagAsEmail to check if the email domain exists  
•   New transformations: CopyCol() and CopyColValue()  
•   New transformation: ReplaceWildcard()  
•   Added for lookup table: EnableWildcards(), EnableRegex()  
•   Added conditions: IsValid(), NotValid() to report if tagged columns such as emails are valid  
•   Added TagAsPhoneNumber() to allow tagging of phone number columns see AutoFixPhoneNumbers  
•   Added AutoFixPhoneNumbers() to convert phone numbers to standarised formatting  
•   Reports are now timestamped so results do not get lost between runs  
•   Added conditions: MatchesDateTimeText(format) and NotMatchDateTimeText(format)  
•   Added CheckForeignKey() to stream to check if all foreign keys are present  
•   Added CheckUniqueKey() to streams to check all key values are unique  
•   Added TagProfilerSetting() to allow for custom profiler settings  
•   An SQL output can be added for both CheckForeignKeys() and CheckUniqueKeys() via SetSqlResultsFilename()  
•   Common values outputted by the Profiler can be save to CSV via the WriteCommonValuesToCSV() command  
•   Added PadLeft() and PadRight() to add spaces to a value  
•   Added DifferenceFrom condition to allow for approximate string matching  
•   Added ConvertToJulianDateTime() and ConvertToUnixDateTime()  
•   The ConvertToDateTime transformation now works with unix timestamps and julian dates.  
•   Added SetLong(), SetBool(), SetDecimal(), SetGuid(), SetShort(), SetByte(), and SetDateTimeOffset()  
•   Added EnableMultipleSelections()/DisableMultipleSelections() to SetControlToListBox() for use with Wizards  
•   Added an Else statement for use with conditions  
•   The name parameter of MakeCol() can now be omitted, MakeCol() will use the column name provided in preceding IfCol() or IfSrcCol() methods  
•   Added ability to enable and disable warnings with via EnableLogWarnings(), DisableLogWarnings(), EnableAbortOnWarnings(), DisableAbortOnWarnings(), RestoreLogWarning()  
•   Added CoalesceCols() to get the first non null value from a list of columns  
•   Added Substring() transform to get part of a text value  
•   Added JoinCols() transform to combine several colums into one value  
•   Added InsertText() and InsertColText() transform to add text at the begining or middle of a value  
•   Added TagAsDomain() to allow tagging of a web domain column  
•   Added NewDBResultsWriter() to the Host object  
•   Added TrackRowUpdates() and StopTrackRowUpdates()  
•   Added SetDNSServer(), ExportDNSCache() and ImportDNSCache()  
•   Added TagAsCreditCard() and validation to check if credit card numbers are valid  
•   Added EnableSoundex() to lookup tables  
•   Added Obfuscate() transform to replace original characters with random characters  
•   Added Mask() transform to replace original characters with fixed characters  
•   Profiling Results can now be grouped by columns  
•   Lookup tables can now be filtered using FilterOn() and Using()  
• 🔀 Changed
•   Database connections and application settings are now read from %LocalAppData%/iData/Settings  
•   Data comparison engine still works but is deprecated, use NewDataTransform instead  
•   NewInputStream still works but is deprecated, use NewStream instead  

Version - 1.4.0

• ❇️ New
•   Upgrade to V8 jscript engine, for better performance, error handling and latest jscript syntax  
•   Doubled performance running scripts in gui with background thread  
•   Allow debugging scripts from VSCode. See Docs  
•   Allow scripts to be stopped from the UI by clicking the run button a second time  
•   NewDbCompare can use SetSrcData or SetDestData instead of SetSrcTable or SetDestTable. SetSrcData can be passed value from OpenCsvFile. This allows control over csv file separators and column settings  
•   NewDbCompare can use SetSrcQuery or SetDestQuery to allow comparison of joined tables or pre-processing of data using sql before iData transformations  
•   Use a lookup table from a query or csv file to replace values using ReplaceValueWithLookup  
•   The database definition of a Null value can now be accessed through Host.DBNull which has a different value from null  
•   IgnoreCase can be applied to make transformations such as Replace case insensitive and MatchCase can turn it case sensitive again  
•   Added SetInt, SetFloat, and SetDateTime to set a column to a specific value of the given type  
•   Created maths operations for Add, Subtract and Remainder  
•   SetMatchDistance added to allow nearest close match to data in lookup table. See Docs  
•   Added SetDateTimeParts to allow part of a dateTime to be changed.  
•   Added SetANSICodePage to allow encoding to be specified when reading CSV files.  
• 🔀 Changed
•   Interactive scripts must have 'Wizard' as part of their filename, for automatic optimisation  
•   ReplaceRegex ignore case parameter deprecated in favor of IgnoreCase command  
•   ReplaceValueWithLookup deprecated and renamed ReplaceLookup  
• ☑️ Fixed
•   Fixed invalid warning when comparing source query to destination csv file  
•   Fixed crash caused by duplicate database connection names  
•   Report any errors generated by wizards  
•   Prevent ConfirmRequiredVersion flagging identical build numbers as different  
•   Prevent missing dates in CSV files generating invalid SQL dateTimes  
•   Fixed cleanup of database connections after DbCompare  

Version - 1.3.6

• ❇️ New
•   Add testConnection wizard. See getting started  
•   Accept jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs  
• 🔀 Changed
•   Database connections details reloaded on demand. No need to re-start gui to refresh  
•   Improved error message for script errors in comparisons  
• ☑️ Fixed
•   Update progress report when skipping rows during comparison  

Version - 1.3.5

• ❇️ New
•   Recored and report statitics for each column  
•   Accept native jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs  
•   Remove duplicates rows before comparison. See RemoveSrcDuplicates in Api Docs  
• 🔀 Changed
•   Limit comparison reports of mismatched rows to avoid hiding other issues  
• ☑️ Fixed
•   Report attempts to map columns incorrectly in comparisons  
•   Prevent installer overwritting connection strings on re-install  

Version - 1.3.4

• ❇️ New
•   Allow comparison to specify IncludeColumns as well as SkipColumns. See Api docs  
•   Allow columns to be SetNull. See api docs  

Version - 1.3.3

• ❇️ New
•   All In to use lookup tables from CSV or queries. See API docs  

Version - 1.3.2

• ❇️ New
•   Additional conditions In, StartsWith etc. See API docs  

Version - 1.3.1

• 🔀 Changed
•   Improved database connection error messages  

Version - 1.3.0

• ❇️ New
•   Support for splitting and joining columns. e.g. AddColText. See Api docs  

Version - 1.2.0

• ❇️ New
•   Database comparison wizard  
•   Import CSV data for comparison  

Version - 1.1.0

• ❇️ New
•   Database comparison  
•   Read/write to Excel files