Version - 2.0.5

• ❇️ New
•   Implemented OAuth login for Salesforce Data Sources   (ref 4983)
•   Implement OEM salesforce database driver   (ref 4885)
•   Added a user interface guide   (ref 4850)
• #552   Added open source licences to the Help page   (ref 4856)
• #405   Allow rows to be tagged with categories   (ref 4669)
• #385   Added Import and Export feature for Projects   (ref 4487)
• #537   Added view only mode to the Edit Data Entity screen for users without permissions   (ref 4826)
•   Added maintenance mode for orderly updates   (ref 4533)
•   Added SoapClient to allow for iData scripts to interact with SOAP API's   (ref 4705)
•   Added Host.XmlToJson and Host.JsonToXml that allow converstion between Json and XML   (ref 4707)
• #491   Added Host.GetMachineID() to get machine ID in scripts   (ref 4714)
• 🔀 Changed
• #577   Update iData Reference Manual regarding TagAsEmail().WithSimpleValidation()   (ref 4894)
•   A valid server certificate is now required for MSSql data sources. To proceed without a valid certificate, the “Trust Server Certificate” flag under a selected data source can be enabled.   (ref 4933)
• #585   Importing projects will now redirect users to update datasources. Stages are prevented from running until all datasources are valid.   (ref 4928)
•   Reduced the load time of Data Entities when creating a stage   (ref 4844)
•   Refactored username lookups for datatables   (ref 4867)
• #420   The transform script output now has specific error text for attempting to use OnNthRow while no unique keys are set   (ref 4865)
• #549   Data sources are now individually licensed. An updated license file will be required.   (ref 4857)
• #417   Updated deprecated RndPatterns method in default transform script   (ref 4864)
•   Updated SalesForce documentation in the iData Reference Manual   (ref 4843)
• #534,#536   Users with user management permission allowed to enable 2FA   (ref 4810)
•   Clicking "Close" after ingesting a table now takes the user directly to entity creation, letting them create an entity from the ingested table   (ref 4735)
• ☑️ Fixed
• #663   Resolved issue that could cause stages to fail to generate if the entity dependencies got outdated   (ref 5172) 🆕[18/04/2023]
• #640   Fixed an error that occured when submitting an invalid server name   (ref 5113)
• #642   The new Stage screen no longer throws an error when a licence is missing   (ref 5105)
• #634   Fixed issue with parameter box on the schedules page not loading correctly   (ref 5097)
• #586   Fixed authorisation error on Job Queue page   (ref 5095)
• #539   Fixed issue with schedules page showing schedules from other projects   (ref 5083)
•   Resolved an issue that stopped Excel files being written in iData 2.0   (ref 4667) 🆕[18/04/2023]
•   Fixed an issue that caused Excel files to become currupt in iData 2.0   (ref 4666)
• #619   Added bool to default MSSqlWhitelist   (ref 5060)
• #602   Added reminder to update connection string (step 5) in error message (step 6).   (ref Story 5008)
• #608   Solved null reference exception when creating Data Gen stage with a Salesforce entity   (ref 5044)
• #594   Fixed error when clicking on an "In Progress" report   (ref 4960)
• #606   Fixed running data generation on salesforce   (ref 5016)
• #576   Fixed iData reference manual typo in TagAsDomain   (ref 4895)
• #575   Updated iData reference manual "Manually tagging" reference   (ref 4896)
• #587   Fixed issue running stages after migration   (ref 4937)
• #565   Fixed issue where the edit entity page was returning a 400 error   (ref 4903)
• #566   Resolved the "Method GetScriptArguments threw ArgumentNullException: Value cannot be null. (Parameter 'g')" error message   (ref 4902)
•   Stages of imported projects can now be executed   (ref 4913)
• #556   Fixed long banner text causes layout issues   (ref 4870)
• #249   Files uploaded to lookup folder are automatically made accessible from scripts   (ref 4854)
• #415   Implemented RndGuid() when primary key type is uniqueidentifier for default transform scripts   (ref 4863)
• #546,#547,#558   Fixed issues in creating non-word stage name   (ref 4829)
•   Data Entity dependencies are now automatically created to prevent errors from missing references   (ref 4845)
• #547   Fixed an issue where stage names with spaces would not generate scripts   (ref 4855)
• #503   iData now handles controlled shutdowns gracefully   (ref 4755)
• #279   Fixed issue where the newly created project is not selected by default   (ref 4798)
• #559   Ensure all database migration steps are recorded to avoid database upgrade warning message   (ref 4879)
• #531   Fixed issue with log download API pointing to the wrong location   (ref 4808)
• #533   Fixed to display role names for users with the Delete Role permission only   (ref 4809)
• #543   Fixed restore user confirm message   (ref 4828)
• #535   Updated roles, permissions, and project permissions to prevent users from getting more access by creating another account   (ref 4811)
• #545   Fixed an issue with Profile Results that prevented List View from displaying when View All Tables was selected in the filter   (ref 4819)
• #357   User account search now returning results   (ref 4432)
• #485   Source Entities no longer fail to load when editing mappings for Single Mapping stages   (ref 4698)
• #512   Fixed file locks when writing to file in scripts   (ref 4779)
• #518   Add permission checking in File browser   (ref 4792)
• #538   Fixed view only permission closing script editor issue   (ref 4813)
• #495   Re-visited users being logged out when marked as inactive   (ref 3999)
•   Similar records no longer skips a row after matching a group   (ref 4458)
• #484   Users are now forced to select a column in Similar Records stage   (ref 4697)
• #492   Apply consistent row ordering in reports   (ref 4737)
• #493   TagAsCompanyEmail tag added to differentiate between different types of emails. Appropriate email now generated according to selected email tag   (ref 4739)
• #495   Fixed issue where inactive users were being sent to the email confirmation screen when they attempt to log in   (ref 4743)
• #523   Allow admin role users delete other users project   (ref 4794)
• #526   Datatables in modals no longer have the same fixed height as datatables on the page   (ref 4804)
• #530   Fixed typo on 2FA setting page   (ref 4807)

Version - 2.0.4

• ❇️ New
•   Added a page before ingesting a CSV file into a database that describes what the action will do  
•   Added a "Create Folder" button on CSV Datasource creation modal. This allows new folders to be used as bases for Datasources  
•   Added a "New Folder" button for Files  
•   Added a "Browse and Upload Files" option for CSV Datasources  
•   Added option to ingest CSV file into a database via the action menu  
•   Files and folders can now be downloaded via the actions menu  
•   Added ability to categories records and view them in the report  
• 🔀 Changed
•   Removed SetColumnToChar due to SQL incompatibility and JSON object issues  
•   #442 update EULA in installer  
•   Removed legacy Host.NewDbCompare  
•   Depreciated SearchGroups for SearchGroup while an issue with multiple runs is resolved  
•   #487 Default data generation scripts for number columns will now generate data relevant to that field type instead of just integers  
•   Moved the "Upload Folder" button for CSV Datasources to the top of the modal  
•   The folder rename dialog in Files no longer requires you to manually remove the backslash and is now titled "Renamed Folder"  
•   Changed how client secrets are stored, this may require creating a new client secret for the worker via the Administration -> Settings -> Worker Settings menu  
•   #479 iData virtual file system storage and performance improvements  
•   Csv datasource re-enabled  
• ☑️ Fixed
•   #490 Fixed issue involving characters in file paths  
•   #469 Fixed issue where execution time was not taking UTC into account  
•   #454 Fixed issue where data type mappings are not applied correctly when comparing a CSV to a table  
•   ETL tooltips now show the names of each row state  
•   Fixed issue with updating progress bars that caused exceptions in the log and the progress bar to not show in the UI  
•   Fixed issue where DBMigrator was not accepting some characters as symbols  
•   #463 Fixed ussue on Schedule page where parameters with checkboxes where not working correctly  
•   #450 Fixed grammar in the limited results text for the report details page  
•   #480 The new CSV Datasource dialog now fits the screen  
•   #464 Permission are now all visible without scrolling  
•   #382 Fixed similar records skipping rows when two groups are close together  
•   #497 Fixed the select data source screen not showing data sources  
•   #506 Fixed edit CSV Datasource no responses issue  
•   #514 Fixed issue where Host.ExitCode was being ignored. Negative values will now cause the script to fail  
•   #527 Fixed new CSV Datasource page upload folder button disappear issue  
•   #529 Fixed stage page and others pages that were not displaying results because the user who created the item had been deleted  

Version - 2.0.3

• 🔀 Changed
•   Disabled csv datasource until #479 resolved  

Version - 2.0.2

• ❇️ New
•   On application launch systeminfo.txt is written to ProgramData\IntelligentDeliverySolutions\iData  
• 🔀 Changed
•   iDataConsole now uses the same encryption as iDataSaaS  
•   #438 Updated 2FA warning message on users page to be more readable  
•   #446 Auto focus username field on login screen  
• ☑️ Fixed
•   No longer shows UpdateProgress exception in script logs when a job ends prematurely  
•   Fixed tooltips on long report values that overflow in Similar Records  
•   A CSV with a duplicate header no longer causes an unhandled exception  
•   #384 Fixed issue where users without CreateProject permission didn't have breadcrumbs  
•   #423 Fixed issue where script path would hide script editor controls  
•   #444 Fixed issue with incorrect filters showing on the report page  
•   #453 Cancelling a job no longer throws a stack trace to the console output  
•   #462 Fixed Typo on Log Settings Screen  
•   #468 Fixed CSV Download issue  
•   #470 Fixed issue allow editing entities with same name on different datasource  
•   #471 Fixed where cancelling a stage generates an error and fails to stop the stage from running  

Version - 2.0.1

• ❇️ New
•   Web user interface  
•   Integrated editor  
•   Datasource and Data entity management through the User interface  
•   Manage and update stages through the user interface (Replacing wizards)  
•   Schedule execution  
•   Interactive reports and drilldown  
•   Projects to organize work and users  
•   Run multiple jobs in parallel  
•   Auto logout user when the user is deleted or has permission change  
• 🔀 Changed
•   Removed support for legacy configuration files  
•   Removed support for legacy wizards. Host.HeadlessWizard, Host.NewWizard, Host.MessageBoxYesNo, Host.MessageBoxWarning have been removed  
•   Host.Exec is deprecated, use javascripts import/export statements. Find out more here: https://javascript.info/import-export  
•   EnableNewColumns() on text writers has been replaced with PreventNewColumns()  
•   Removed support for RemoveDuplicates transform  
•   Updated the UI so only one column can be selected for grouping on SimilarRecords  
• ☑️ Fixed
•   #261 & #231 Fixed sorting by name on project and stage screens  
•   #260 Changes to stages through the UI overwrite manual changes to scripts  
•   #254 Unhandled data types now fails gracefully, however further change will be made in #259  
•   #249 Manually adding a lookup file updates lookups.js to support code completion in the editor  
•   #248 Renaming a lookup file removes contents  
•   #246 Users with execute rights don't have to be in the admin group  
•   #245 Incorrect version number returned from command line  
•   #244 RndIntBetween added for generating inclusive numbers, help for RndIntRange updated to state it generates numbers excluding the end value  
•   #243 Set generate 0 or 1 values for bit fields  
•   #242 ConvertToText now takes requested region and culture into account when formatting dates  
•   #239 Column names with apostrophe's are now handled correctly  
•   #237 #228 #227 #226 #220 Cosmetic layout fixes  
•   #233 Exported results location shown with link  
•   #232 Incorrect number of pages in results  
•   #230 Re-ordered profiling list view filters into logical groups rather than alphabetically  
•   #229 Increased profile results size limits to handle larger tables  
•   #213 Favicon now displayed on login screen  
•   #184 Reports and stages can be deleted to manage database size  
•   #181 Running jobs can be cancelled  
•   #178 Timeout profiling large tables  
•   #168 Entities can be added or ammended after creation  
•   #241 Fixed Inconsistent timezone on Job Progress  
•   #224 Removed references to wizards on some screens  
•   #235 LetsEncrypt certificates can be requested via to the UI  
•   #256 Solved issue with Intermittent failure of scheduled jobs  
•   #255 Action button for schedule missing under some circumstances  
•   #263 MakeRndCardCols returns an error  
•   #253 rename GetDataEntity to GetDataEntityConfig for clearity and implement RndRecordFrom on DBQuery  
•   #259 Creating stage where entities have foreign keys to other ommited entities cases error 500  
•   #262 Error using 'Run again' from job progress or run history  
•   #276 The iData worker is now setup on application start if not already configured  
•   #274/#275 Fixed an issue where the admin role was not being properly checked for when running a script  
•   Resolved a issue when running a job "Issuer name does not match authority"  
•   #269 iData currently does not support MSSQL Spatial data types such as Geography  
•   #269 Skip unreadable data types such as spatial  
•   #300 Fixed link to job in the notification Email from showing an error when clicked  
•   #291 Fixed an issue where CSV entity names containing spaces would generate bad variable names  
•   #288 Resolved an issue when deleting a stage  
•   #297 Solved issue with uploading licence  
•   #343/#344 Fixed various styling and css issues  
•   #326 Solved issue where not all profilers where shown when hitting the 150 column limit  
•   #343/#344/338/339 Fixed various styling and css issues  
•   #323 Fixed AutoMapping CSV to SQL for Wide Table  
•   #355 Solved issue with referential integrity and data generation  
•   #356 Improved register validation to prevent User Enumeration  
•   #317 Set Similar Record modal width to prevent overflow of mapping container  
•   #318 Documents should no longer be incorrectly translated by the browser  
•   #347 Fixed an issue where new user accounts weren't being activated  
•   #309 Removed exec from user help commands  
•   #367 Fixed typo on LetsEncrypt screen along with adding a domain field and some instruction on how to change it  
•   #380 Added Two Factor Authentication  
•   #381 Added "Use Domain in Credentials" to email settings screen to allow for Gmail credentials to authenticate correctly  
•   #386 Fixed Number of Columns Compared counter not reporting anything  
•   #387 Fixed issue running stages with long names  
•   #396 Fixed CSV to CSV data mappings for strings  
•   #397 Fixed 500 error when opening a comparison report with mapped columns  
•   #400 By default we now limit the number of rows reported in each ETL and Comparison report and increased timeout on query to allow for bigger reports to be loaded  
•   #410 Resolved issue where deleting a report that is being processed courses an exception  
•   #412 Fixed 500 Error on the Settings screen  
•   #424 Fixed incorrect reference to step 4 in getting started guide  
•   #426 Updated formating on stage completion emails  
•   #432 Users are now prompted to login after registration. New Users get a welcome email  
•   #433 Fixed issue with apostrophe in CSV column name breaks scripts  
•   #449 Uploading a file no longer creates a folder with the same name  
•   #452 Encoded modal header titles before displaying them to prevent script injection  
•   #457 Fixed issue with default csv options that caused issues reading headers  
• ⚠️ Known Issues
•   #257 Order of source and target rows can swap depending on row status  
•   #252 ForEachRowFrom can't be used multiple times for the same OpenQuery  
•   #250 & #251 Generating data for tables with identity columns  
•   #183 Job stopped during processing  
•   Db Migrator always shows a warning message  
•   Unclear warning message when row size exceeded  
•   user search returns no users  
•   ignore values on similar records causes duplicates to be missed  
•   Similar records tooltip appears in wrong place  
•   Similar records occasionally missing expected results  
•   Restoring a db and ProgramData to a new server causes issues  
•   User with View Only permissions has no breadcrumb menus when viewing a report  
•   #368 Value cannot be null when using OnRndRows method on column that accepts null  
•   #401 script log updates are added to the current page even when its not the latest page  
•   #406 project owner not listed in project permissions  
•   Skipping all records seems to cause an issue  
•   #421 report columns don't line up with headers on tables with a large number of columns  
•   #422 need to scroll to see popup menu  
•   #423 Editing a script with a long path causes save and close buttons to be partially hiden  
•   #426 alert notification email message needs tidying up  
•   #436 Internittent timeout on results screen and error sending progress to source  
•   #439 job status intermittently fails to go to a green tick  
•   short columns with long names causes poor formatting in reports  

Version - 1.8.1

• ❇️ New
•   new data generation method GetRndRecord can be used to select multiple fields from a randomly selected lookup record  
•   values can be set using a 'Get' from another column when using SetText, SetInt etc  
•   RndRange transforms can now accept range parameters using a 'Get' from other columns  
•   values can be assigned using a general 'Set' that uses the data type of the parameter  
•   added RndDateTimeRange which returns a random date and time between the specified values  
•   AddressValidationAPI has an option to refresh the cache once the records are older than a configurable amount of days  
•   added request logging to AddressValidationAPI using LogRequestContaining config setting  
•   AddressValidationAPI cache is now encrypted  
•   MakeRndPersonCols when used for obfuscating guarantees a generated name will be different from original name  
•   Domains for generated email addresses have been updated to avoid clashing with real email addresses  
•   Limited SetMaxUniqueValues(int) to 100,000 to ensure stability when profiling  
•   Added ReplaceControlCharacters to remove control characters.  
•   Added ReplaceMultipleSpaces to replace multiple spaces.  
•   Added ReplaceBlanksWithNulls to replace blanks to Null.  
•   ConvertToText can now support non-numeric format specificer and culture code.  
•   Added data type validation to Csv readers when types are defined by the user, with functions to handle validation errors, OnReadErrorContinue() and OnReadErrorAbort()  
•   Allow scripts to prevent automatic failure of a row after validating an email or address and implement their own handling of the issue including retrieving the error message. See IValidateCondition functions  
• 🔀 Changed
•   IMPORTANT: Selecting random values based on weighting was found to be inaccurate and has been updated. Random selections with weightings will generate a different selection of results after this change.  
•   when using Get to obtain a reference to a field a default value can supplied if the field isn't found  
•   RndDateRange returns dates with time set to midnight see RndDateTimeRange  
•   Improved the profiling logic for detecting a columns data type  
•   Parameters on Transformations and Conditions now accept column references  
•   Skipping a row now deletes it from the output  
•   Column name references available for more functions, updated list in iData Reference Manual  
•   SetTimeZoneOffset replaced with SetUTCTimeZoneOffset and SetLocalTimeZoneOffset  
•   TestDataSet values set with NextXXX or ForEachXXX are now set before all other transforms take place, rather than after, so their values can be used by other transforms  
•   AutoFixAddress verification now maps addresses containing a Company Name, Property Name, or Locality, to the street/town columns  
•   AutoFixAddress verification only replaces responses with a return value of 100 and 400 by default, WithAllowedFlags() can be used to set a custom list of allowed flags. More information on flags can be found in the docs.  
•   Removed Binary Transforms as they were not useful in their current form  
•   Amended ReplaceNearest to support default value  
•   Added RntIntBetween which generates values including the maximum value  
•   Generated credit cards now give an invalid checksum by default. Added EnableValidCheckDigit() to the tag settings toa allow valid checksums to be generated.  
• ☑️ Fixed
•   Lookups from csv files will use column types defined using SetColumnToXXX  
•   Fixed infinite data generation when no row limit or foreach  
•   Report viewer now correctly reports invalid filenames instead of reporting missing files as invalid encryption keys  
•   Report viewer now correctly handles splitting of files correctly  
•   FormatText now works on fields that contain a NULL value  
•   Fixed error when using SetPatternLiteral  
•   Fixed 'UpdateColumnDefinition' which now correctly increases column sizes when ingesting csv data  
•   Miscapitalised column names used in SetColumnMapping no longer throw an error with counter group '???'  
•   AddressValidationAPI no longer requires external dependencies to run  
•   AddressValidationAPI will now create missing cache  
•   Stopped Abs transform displaying warning for positive numbers  
•   Fixed an issue when generating data that would cause the primary keys to be set to null  
•   Fixed NULL value handling when used with references to other fields  
•   #225 Fixed an issue when generating data that would cause the primary keys to be set to null  
•   CSV entries longer than 8000 characters will no longer throw an error  

Version - 1.8.0

• ❇️ New
•   Reports are compressed and encrypted to a single .idr report file by default  
•   Added SetVirtualFolder to filemanager to direct all folder contents to a single report file  
•   Encryption keys can be created and set through a encryption key pass phrase in the user interface  
•   Installer now deploys Report Viewer to view .idr files  
• 🔀 Changed
•   Profilers are now listed alphabetically  
• ☑️ Fixed
•   Fixed issue when using multiple lookups in one script  
•   Fixed profile reports with group results  
•   Fixed opening reports on a network drive accessed via a UNC path  
•   Fixed AutoFix address output miscapitalising words  
•   Fixed error when address config not set  
•   Fixed regex match error when the db value is null  

Version - 1.7.2

• ❇️ New
•   Added RestClient to allow for iData scripts to interact with Rest endpoints  
•   Added TextFileReader to allow for any text based file to be open and be used in a datastream  
•   Added TextFileWriter allowing transform engine to write to text files  
•   Added RestartLimit and RestartDisabled for Next lookup transforms  
•   Added SetTwoDigitYearMax to set the default century for two digit year date formats  
•   Added SetColumnToGuid to database readers to ensure unique identifiers are compared correctly  
•   Allow csv writers to append or overwrite existing files  
• 🔀 Changed
•   Updated ETL wizard to EnableIdentityInsert when target has identity column.  
•   MakeRndAddressCols() can use existing country or region or filter by country and region with UseCurrentCountry UseCurrentCountryAndRegion options  
•   Removed VaryBy()  
•   Added foreign key references to wizard script  
•   SetColumnToDateTime now also takes optional DateFormat to override the default  
•   CreateTableIfMissing now uses the input column name not a normalised version  
•   AddRecord now uses Upsert when there are no comparisons  
•   Added column mapping to the source stream, a compare is no longer necessary to map columns, compare mapping is being deprecated  
•   AutoFix now capitalises location names, and places houses in address line 1  
•   Dates are now formatted to the local time before being written to reports  
• ☑️ Fixed
•   Fixed data generation resetting identity columns  
•   RepeatRnd() changed to inclusive range to match Repeat() and missing ref bug fixed  
•   Allow updates to unique keys when source or destination have no rows  
•   Ensure error messages are reported for all row states  
•   Fixed error when uploading from excel sheets with numbers for sheet names  
•   Fixed TrackRowUpdates when used with existing data  
•   Fixed Importing csv file with only headers  
•   Fixed wild cards not applying tags to each column  
•   Fixed cardinality max value estimator not tracking past the set limit  

Version - 1.7.1

• ❇️ New
•   DbWriter and SQLWriter now supports creating and updating schema using CreateTableIfMissing, CreateColumnIfMissing and UpdateColumnDefinition.  
•   Added ExcelWriter allowing transform engine to write to excel sheets  
•   Added support for password protected excel files  
•   Added ExtractRegex() transform which allows text matching a given regex pattern to be extracted and optionally replaced with other text  
•   Added ToCol() which allows text extracted from ExtractRegex() to be placed in another column  
•   Added Company, Email, and Phone number to random address generation  
•   Added SetPatternLiteral() to toggle between pattern generation and literal characters in RndPatterns()  
•   Added Single, Double, and Triple word occurance count to the profiling report  
•   Added AddressLine1, AddressLine2, and AddressLine3 for generic columns  
•   Added SetColumnMappingFrom() to allow the use of lookup tables to map compared columns  
•   Added EnableAddressFixLookupService to enable the use of AFD instead of LibPostal to clean addresses, alongside EnableAddressFixMachineLearning, default option is set in AppSettings.ini  
•   Added abort to transforms when too many writes fail in a row can limit can be set with SetWriterMaxWriteFail  
•   MakeCol now accepts Wildcards to select multiple columns at once  
•   MakeCol now accepts multiple columns to be passed at once  
•   Added ExcelReader allowing transform engine to read from excel sheets  
•   ExcelReader stops reading when blank rows found  
•   New Licencing system. Contact Intelligent Delievery Solutions for support.  
•   TrackRowUpdates() now writes comments to SQL, CSV and Excel files  
•   iData Address Validation API is now packaged with the iData install  
•   Added date and time to console logs  
•   Added new conditions to match profiler results. E.G. HasPunctuation  
• 🔀 Changed
•   Lookup matches not support patterns  
•   Merged CardPrefix and CardPrefixRange for card definition in AppSettings  
•   Cards higher than 19 digits can now be validated  
•   AutoFixAddressFields now works on tables with joined number and street fields  
•   Reports no longer show [no value] when one of the rows is entirely missing. e.g. No Src/Dest or Add and Delete  
•   Column values can now be used as parameters in conditions such as Contains  
•   You can no longer edit unique key columns while comparing unless the destination table is empty  
•   Limited the UI log output to 1000 lines  
•   AutoFixAddressFields and VerifyAddressFields now work with group tags  
•   IsValid and NotValid has been added to functions VerifyEmailAddress, VerifyAddressFields, AutoFixAddressFields, CheckForeignKeys, and CheckUniqueKeys  
•   Generate more diverse set of company names  
•   By default tables stop at the first empty row, and the first empty header if headers are set  
• ☑️ Fixed
•   Prevent error when uniqueidentifier is table key and also using Rnd transforms  
•   Fixed system buffer version mismatch  
•   Fixed spelling on some Rnd... transforms from Rng...  
•   Fixed filter buttons sometimes not showing up on ETL reports  
•   Set RndPatterns() to accept punctuation.  
•   Report links with spaces now work  
•   Csv files with trialing spaces in headers now work  
•   Allow duplicate values in data for In and Replace operations  
•   Fixed some formatting issues in the documentation  
•   Data writes take column mappings into account and shows correct mapping in headers in report  
•   Fixed showing mapped columns in report when writing to empty table  
•   Fixed table name validation issue  
•   Speacial characters in excel sheet names now upload okay  
•   Address generation email column not defined and street address default seed  

Version - 1.7.0

• ❇️ New
•   More detailed stack trace for errors  
•   Added NextGuid(), RndGuid() and EachGuid() transforms  
•   Added OnNthRow() transform  
•   Added ConvertToBigInt() which converts the value to a 64 bit integer  
•   More detailed stack trace for errors  
•   Added a UniquePrefixes profiler  
•   Added RndWords() transform which generates a random Lorum Ipsum paragraph  
•   Added ability for tags to be grouped together  
•   Added enhanced Mask() which allows full or partial masking of text with a given mask character  
•   Added enhanced MaskExcept() to mask all except a given range  
•   Added Repeat/RepeatRnd modifier(s) to Each and Next Transformations to repeat the value the requested number of times  
•   Added MakeKeys and ForEachRowFrom/NextRowFrom to setup keys values from a parent table  
•   Added default value option to conversion transformations  
•   Added the ability to set the seed for randomly generated data to an alternative name instead of the column name with UseSeedName()  
•   Added AllUppercase, AllLowercase and MixedCase profilers  
•   Added Flush() to reports to write any buffered output  
•   Added ability to set RequiredRows for a TestSet  
•   Added RepeatRnd() function which allows Each and Next transforms to be randomly reset  
•   Each functions are now known as ForEach. e.g. EachText is now ForEachText  
•   Added RestartRnd() which restarts a ForEach sequence after a random amount of iterations between the given minValue and maxValue  
•   Added SetBinary(), RndBinary(), NextBinary(), EachBinary() and EachFrom versions of these  
•   Added RowLimit() to Each data items to allow for generation of data for smaller sub sets  
•   Added SetDifferenceFilename() option to comparison options  
•   All transforms can now be contained within a TestDataSet()  
•   All TestDataSet() calls must now have an accompanying EndTestDataSet()  
•   Added GetCardType() to check a card number for a matching card type, returns the found type as a string  
•   Added MatchKeysAlphbetically() method to comparison settings to allow for keys to be compared alphanumerically  
•   Added the ability to enable soundex for similar record checker via similar record tags  
•   Added the ability to ignore words for similar record checker via similar record tags  
•   Added the a sort by method the similar record configuration object  
•   Added BlankCol() to blank selected columns  
•   Added UniquePatternLengthIndependent profiler  
•   Added MakeRndCreditCardCols() to generate credit card data on tagged columns. Additionally, SetValidCardTypes() can be used to select card types to generate.  
• 🔀 Changed
•   Temp data is now written to local app data  
•   By default both MakeRndAddressCols() and MakeRndPersonCols() use predefined seeds. This can be changed with the new UseDefaultSeeds()/UseColSeeds() options  
• ☑️ Fixed
•   Prevent error when uniqueidentifier is table key and also using Rnd transforms  
•   Fixed system buffer version mismatch  
•   Fixed spelling on some Rnd... transforms from Rng...  

Version - 1.6.1

• ❇️ New
•   Added the ability to set a timeout when calling Execute() on a DBQuery  
•   Added NextIntAsc(), NextFloatAsc(), NextDecimalAsc(), NextYearAsc(), NextMonthAsc(), NextDayAsc(), NextHourAsc(), NextMinuteAsc(), NextSecondAsc() and NextMillisecondAsc() which increment a value by a given step  
•   Added the RngRegex transformation to randomly generate values that match the specified regular expression  
•   Added ability to generate synthetic data using OpenTableGenerator(). Additionally, use SetTotalRows() and UseOptimalRows to set the amount generated  
•   Added a new DataType profiler that reports back the type of each column profiled  
•   Added a new Median profiler that reports back an estimated median value when the number of unique values has been exceeded. Actual median is reported below this limit  
•   Added Host.NewDateTimeFormat() and Host.SetDefaultDateTimeFormat() which allow for parsing custom date formats  
•   Added a progress bar when the profiler is writing out html reports  
•   Added a new VaryBy transformation that can be used when obfuscating values  
•   Added OnRndRows() and OnRows() to allow for transformations to be applied on a subset of rows  
•   Added grouped profiling results to the profiling reports  
•   Added IsBlank() and NotBlank() conditional transformations  
•   Added Between() and NotBetween() conditional transformations  
•   Added option to paginate sql and csv files via a parameter on Host.WriteSqlScript() and Host.WriteCsvFile()  
•   Added the ability to edit and add new connections from the connection picker when running a wizard  
•   Added Or() conditional transformation  
•   Added SetBigInt() which sets the value to the given int as Int64  
•   Added ConvertToBigInt() which converts the value to a 64 bit integer  
• 🔀 Changed
•   MoreThan(), Equals(), AtLeast() and other comparisons will convert text values to dates before comparing them  
•   NextIntFrom(), NextDoubleFrom(),NextFloatFrom(), NextTextFrom() and NextDateFrom() have been renamed to NextInt(), NextDouble() etc when using a list of values opposed to a lookup  
•   Null values are now included in the unique values profiler results  
•   Total row count is now shown next to the Unique values on the profiling report  
•   Now when the number of unique values passes 1000(by default) an estimation of the number of unique values is provided for the profiling reports  
• ☑️ Fixed
•   Error message is displayed in reports when invalid column mappings are provided  

Version - 1.6.0

• ❇️ New
•   When updating from 1.5.* to 1.6 iData needs to be uninstalled prior to updating  
•   Added SetWriteFailedRecords() which allows failed records to be written to a separate destination  
•   Added new setting to profiler results SetMaxCharLimit() which prevents the profiling of exceedingly large text values  
•   Added NewLookup() to Host that allows the building of lookup tables from within scripts  
•   Added NextIntFrom(), NextDecimalFrom(), NextDateFrom(), NextFloatFrom() and NextTextFrom() which loop through given values replacing each value in turn  
•   Added RndTextFrom(), RndIntFrom(), RndDecimalFrom(), RndFloatFrom(), RndDateTimeFrom() and RndTextFrom() which select a random value from provided values  
•   Added RndTextPattern() which converts a pattern, e.g. 'A9A 9A9', into a random string with the same pattern e.g A7A 3R8  
•   Added SetGenerationSeedCols() to set which column values are used as the seed for generated data  
•   Added RndIntRange(),RndFloatRange(),RndDoubleRange(),RndDecimalRange() and RndDateRange() used to generate random data  
•   Added the ability to sample datastreams with the SetSampleSize() method  
•   Added the ability to generate random addresses with the MakeRndAddressCols() transformation  
•   Added Each transforms which generate data in combination with other Each transforms. Added for Text, Int, Decimal, Float and Dates. Each transforms can be used with LookupTables using Each...From  
• 🔀 Changed
•   Renamed AutoFixPhoneNumber() transformation to FormatPhoneNumber()  

Version - 1.5.1

• ☑️ Fixed
•   Fixed comparison summary not reporting differences when source rows missing from destination  
•   Fixed row count out by 1 when reading from csv files  

Version - 1.5.0

• ❇️ New
•   Added Profiling and Profiling wizard to discover data patterns and potential data quality issues  
•   Added Check for Similar Records and Similar records Wizard to search for records that could be duplicates  
•   Created data transform engine for ETL as well as comparison See NewDataTransform in docs  
•   Added transformations: ConvertToDecimal(), ConvertToText(), ConvertToDateTime(), ReformatDateTimeText()  
•   Added AutoCapitalise() transformation allowing for words to be automatically capitalised  
•   Added transformations: Trim(), TrimEnd(), TrimStart()  
•   Added the ability to tag columns as the following: TagAsPersonTitleName(), TagAsPersonFirstName(),TagAsPersonMiddleName(),TagAsPersonLastName()  
•   Intelligently clean person/name related data: AutoFixPersonFields(), DistributePersonFullName() that have been tagged as person fields  
•   Added EnableDomainCheck() on TagAsEmail to check if the email domain exists  
•   New transformations: CopyCol() and CopyColValue()  
•   New transformation: ReplaceWildcard()  
•   Added for lookup table: EnableWildcards(), EnableRegex()  
•   Added conditions: IsValid(), NotValid() to report if tagged columns such as emails are valid  
•   Added TagAsPhoneNumber() to allow tagging of phone number columns see AutoFixPhoneNumbers  
•   Added AutoFixPhoneNumbers() to convert phone numbers to standarised formatting  
•   Reports are now timestamped so results do not get lost between runs  
•   Added conditions: MatchesDateTimeText(format) and NotMatchDateTimeText(format)  
•   Added CheckForeignKey() to stream to check if all foreign keys are present  
•   Added CheckUniqueKey() to streams to check all key values are unique  
•   Added TagProfilerSetting() to allow for custom profiler settings  
•   An SQL output can be added for both CheckForeignKeys() and CheckUniqueKeys() via SetSqlResultsFilename()  
•   Common values outputted by the Profiler can be save to CSV via the WriteCommonValuesToCSV() command  
•   Added PadLeft() and PadRight() to add spaces to a value  
•   Added DifferenceFrom condition to allow for approximate string matching  
•   Added ConvertToJulianDateTime() and ConvertToUnixDateTime()  
•   The ConvertToDateTime transformation now works with unix timestamps and julian dates.  
•   Added SetLong(), SetBool(), SetDecimal(), SetGuid(), SetShort(), SetByte(), and SetDateTimeOffset()  
•   Added EnableMultipleSelections()/DisableMultipleSelections() to SetControlToListBox() for use with Wizards  
•   Added an Else statement for use with conditions  
•   The name parameter of MakeCol() can now be omitted, MakeCol() will use the column name provided in preceding IfCol() or IfSrcCol() methods  
•   Added ability to enable and disable warnings with via EnableLogWarnings(), DisableLogWarnings(), EnableAbortOnWarnings(), DisableAbortOnWarnings(), RestoreLogWarning()  
•   Added CoalesceCols() to get the first non null value from a list of columns  
•   Added Substring() transform to get part of a text value  
•   Added JoinCols() transform to combine several colums into one value  
•   Added InsertText() and InsertColText() transform to add text at the begining or middle of a value  
•   Added TagAsDomain() to allow tagging of a web domain column  
•   Added NewDBResultsWriter() to the Host object  
•   Added TrackRowUpdates() and StopTrackRowUpdates()  
•   Added SetDNSServer(), ExportDNSCache() and ImportDNSCache()  
•   Added TagAsCreditCard() and validation to check if credit card numbers are valid  
•   Added EnableSoundex() to lookup tables  
•   Added Obfuscate() transform to replace original characters with random characters  
•   Added Mask() transform to replace original characters with fixed characters  
•   Profiling Results can now be grouped by columns  
•   Lookup tables can now be filtered using FilterOn() and Using()  
• 🔀 Changed
•   Database connections and application settings are now read from %LocalAppData%/iData/Settings  
•   Data comparison engine still works but is deprecated, use NewDataTransform instead  
•   NewInputStream still works but is deprecated, use NewStream instead  

Version - 1.4.0

• ❇️ New
•   Upgrade to V8 jscript engine, for better performance, error handling and latest jscript syntax  
•   Doubled performance running scripts in gui with background thread  
•   Allow debugging scripts from VSCode. See Docs  
•   Allow scripts to be stopped from the UI by clicking the run button a second time  
•   NewDbCompare can use SetSrcData or SetDestData instead of SetSrcTable or SetDestTable. SetSrcData can be passed value from OpenCsvFile. This allows control over csv file separators and column settings  
•   NewDbCompare can use SetSrcQuery or SetDestQuery to allow comparison of joined tables or pre-processing of data using sql before iData transformations  
•   Use a lookup table from a query or csv file to replace values using ReplaceValueWithLookup  
•   The database definition of a Null value can now be accessed through Host.DBNull which has a different value from null  
•   IgnoreCase can be applied to make transformations such as Replace case insensitive and MatchCase can turn it case sensitive again  
•   Added SetInt, SetFloat, and SetDateTime to set a column to a specific value of the given type  
•   Created maths operations for Add, Subtract and Remainder  
•   SetMatchDistance added to allow nearest close match to data in lookup table. See Docs  
•   Added SetDateTimeParts to allow part of a dateTime to be changed.  
•   Added SetANSICodePage to allow encoding to be specified when reading CSV files.  
• 🔀 Changed
•   Interactive scripts must have 'Wizard' as part of their filename, for automatic optimisation  
•   ReplaceRegex ignore case parameter deprecated in favor of IgnoreCase command  
•   ReplaceValueWithLookup deprecated and renamed ReplaceLookup  
• ☑️ Fixed
•   Fixed invalid warning when comparing source query to destination csv file  
•   Fixed crash caused by duplicate database connection names  
•   Report any errors generated by wizards  
•   Prevent ConfirmRequiredVersion flagging identical build numbers as different  
•   Prevent missing dates in CSV files generating invalid SQL dateTimes  
•   Fixed cleanup of database connections after DbCompare  

Version - 1.3.6

• ❇️ New
•   Add testConnection wizard. See getting started  
•   Accept jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs  
• 🔀 Changed
•   Database connections details reloaded on demand. No need to re-start gui to refresh  
•   Improved error message for script errors in comparisons  
• ☑️ Fixed
•   Update progress report when skipping rows during comparison  

Version - 1.3.5

• ❇️ New
•   Recored and report statitics for each column  
•   Accept native jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs  
•   Remove duplicates rows before comparison. See RemoveSrcDuplicates in Api Docs  
• 🔀 Changed
•   Limit comparison reports of mismatched rows to avoid hiding other issues  
• ☑️ Fixed
•   Report attempts to map columns incorrectly in comparisons  
•   Prevent installer overwritting connection strings on re-install  

Version - 1.3.4

• ❇️ New
•   Allow comparison to specify IncludeColumns as well as SkipColumns. See Api docs  
•   Allow columns to be SetNull. See api docs  

Version - 1.3.3

• ❇️ New
•   All In to use lookup tables from CSV or queries. See API docs  

Version - 1.3.2

• ❇️ New
•   Additional conditions In, StartsWith etc. See API docs  

Version - 1.3.1

• 🔀 Changed
•   Improved database connection error messages  

Version - 1.3.0

• ❇️ New
•   Support for splitting and joining columns. e.g. AddColText. See Api docs  

Version - 1.2.0

• ❇️ New
•   Database comparison wizard  
•   Import CSV data for comparison  

Version - 1.1.0

• ❇️ New
•   Database comparison  
•   Read/write to Excel files