Version - 2.0.4

• ❇️ New
• Added a "New Folder" button for Files
• Added a "Browse and Upload Files" option for CSV Datasources
• Added option to ingest CSV file into a database via the action menu
• Files and folders can now be downloaded via the actions menu
• 🔀 Changed
• #442 update EULA in installer
• Removed legacy Host.NewDbCompare
• Depreciated SearchGroups for SearchGroup while an issue with multiple runs is resolved
• The folder rename dialog in Files no longer requires you to manually remove the backslash and is now titled "Renamed Folder"
• Changed how client secrets are stored, this may require creating a new client secret for the worker via the Administration -> Settings -> Worker Settings menu
• #479 iData virtual file system storage and performance improvements
• Csv datasource re-enabled
• ☑️ Fixed
• #469 Fixed issue where execution time was not taking UTC into account
• #454 Fixed issue where data type mappings are not applied correctly when comparing a CSV to a table
• ETL tooltips now show the names of each row state
• Fixed issue with updating progress bars that caused exceptions in the log and the progress bar to not show in the UI
• Fixed issue where DBMigrator was not accepting some characters as symbols
• #463 Fixed ussue on Schedule page where parameters with checkboxes where not working correctly
• #464 Permission are now all visible without scrolling
• #382 Fixed similar records skipping rows when two groups are close together
• #497 Fixed the select data source screen not showing data sources
• #506 Fixed edit CSV Datasource no responses issue
• #514 Fixed issue where Host.ExitCode was being ignored. Negative values will now cause the script to fail

Version - 2.0.3

• 🔀 Changed
• Disabled csv datasource until #479 resolved

Version - 2.0.2

• ❇️ New
• On application launch systeminfo.txt is written to ProgramData\IntelligentDeliverySolutions\iData
• 🔀 Changed
• iDataConsole now uses the same encryption as iDataSaaS
• #438 Updated 2FA warning message on users page to be more readable
• #446 Auto focus username field on login screen
• ☑️ Fixed
• No longer shows UpdateProgress exception in script logs when a job ends prematurely
• Fixed tooltips on long report values that overflow in Similar Records
• A CSV with a duplicate header no longer causes an unhandled exception
• #384 Fixed issue where users without CreateProject permission didn't have breadcrumbs
• #423 Fixed issue where script path would hide script editor controls
• #444 Fixed issue with incorrect filters showing on the report page
• #453 Cancelling a job no longer throws a stack trace to the console output
• #462 Fixed Typo on Log Settings Screen
• #468 Fixed CSV Download issue
• #470 Fixed issue allow editing entities with same name on different datasource
• #471 Fixed where cancelling a stage generates an error and fails to stop the stage from running

Version - 2.0.1

• ❇️ New
• Web user interface
• Integrated editor
• Datasource and Data entity management through the User interface
• Manage and update stages through the user interface (Replacing wizards)
• Schedule execution
• Interactive reports and drilldown
• Projects to organize work and users
• Run multiple jobs in parallel
• Auto logout user when the user is deleted or has permission change
• 🔀 Changed
• Removed support for legacy configuration files
• Removed support for legacy wizards. Host.HeadlessWizard, Host.NewWizard, Host.MessageBoxYesNo, Host.MessageBoxWarning have been removed
• Host.Exec is deprecated, use javascripts import/export statements. Find out more here: https://javascript.info/import-export
• EnableNewColumns() on text writers has been replaced with PreventNewColumns()
• Removed support for RemoveDuplicates transform
• Updated the UI so only one column can be selected for grouping on SimilarRecords
• ☑️ Fixed
• #261 & #231 Fixed sorting by name on project and stage screens
• #260 Changes to stages through the UI overwrite manual changes to scripts
• #254 Unhandled data types now fails gracefully, however further change will be made in #259
• #249 Manually adding a lookup file updates lookups.js to support code completion in the editor
• #248 Renaming a lookup file removes contents
• #246 Users with execute rights don't have to be in the admin group
• #245 Incorrect version number returned from command line
• #244 RndIntBetween added for generating inclusive numbers, help for RndIntRange updated to state it generates numbers excluding the end value
• #243 Set generate 0 or 1 values for bit fields
• #242 ConvertToText now takes requested region and culture into account when formatting dates
• #239 Column names with apostrophe's are now handled correctly
• #237 #228 #227 #226 #220 Cosmetic layout fixes
• #233 Exported results location shown with link
• #232 Incorrect number of pages in results
• #230 Re-ordered profiling list view filters into logical groups rather than alphabetically
• #229 Increased profile results size limits to handle larger tables
• #213 Favicon now displayed on login screen
• #184 Reports and stages can be deleted to manage database size
• #181 Running jobs can be cancelled
• #178 Timeout profiling large tables
• #168 Entities can be added or ammended after creation
• #241 Fixed Inconsistent timezone on Job Progress
• #224 Removed references to wizards on some screens
• #235 LetsEncrypt certificates can be requested via to the UI
• #256 Solved issue with Intermittent failure of scheduled jobs
• #255 Action button for schedule missing under some circumstances
• #263 MakeRndCardCols returns an error
• #253 rename GetDataEntity to GetDataEntityConfig for clearity and implement RndRecordFrom on DBQuery
• #259 Creating stage where entities have foreign keys to other ommited entities cases error 500
• #262 Error using 'Run again' from job progress or run history
• #276 The iData worker is now setup on application start if not already configured
• #274/#275 Fixed an issue where the admin role was not being properly checked for when running a script
• Resolved a issue when running a job "Issuer name does not match authority"
• #269 iData currently does not support MSSQL Spatial data types such as Geography
• #269 Skip unreadable data types such as spatial
• #300 Fixed link to job in the notification Email from showing an error when clicked
• #291 Fixed an issue where CSV entity names containing spaces would generate bad variable names
• #288 Resolved an issue when deleting a stage
• #297 Solved issue with uploading licence
• #343/#344 Fixed various styling and css issues
• #326 Solved issue where not all profilers where shown when hitting the 150 column limit
• #343/#344/338/339 Fixed various styling and css issues
• #323 Fixed AutoMapping CSV to SQL for Wide Table
• #355 Solved issue with referential integrity and data generation
• #356 Improved register validation to prevent User Enumeration
• #317 Set Similar Record modal width to prevent overflow of mapping container
• #318 Documents should no longer be incorrectly translated by the browser
• #347 Fixed an issue where new user accounts weren't being activated
• #309 Removed exec from user help commands
• #367 Fixed typo on LetsEncrypt screen along with adding a domain field and some instruction on how to change it
• #380 Added Two Factor Authentication
• #381 Added "Use Domain in Credentials" to email settings screen to allow for Gmail credentials to authenticate correctly
• #386 Fixed Number of Columns Compared counter not reporting anything
• #387 Fixed issue running stages with long names
• #396 Fixed CSV to CSV data mappings for strings
• #397 Fixed 500 error when opening a comparison report with mapped columns
• #400 By default we now limit the number of rows reported in each ETL and Comparison report and increased timeout on query to allow for bigger reports to be loaded
• #410 Resolved issue where deleting a report that is being processed courses an exception
• #412 Fixed 500 Error on the Settings screen
• #424 Fixed incorrect reference to step 4 in getting started guide
• #426 Updated formating on stage completion emails
• #432 Users are now prompted to login after registration. New Users get a welcome email
• #433 Fixed issue with apostrophe in CSV column name breaks scripts
• #449 Uploading a file no longer creates a folder with the same name
• #452 Encoded modal header titles before displaying them to prevent script injection
• #457 Fixed issue with default csv options that caused issues reading headers
• ⚠️ Known Issues
• #257 Order of source and target rows can swap depending on row status
• #252 ForEachRowFrom can't be used multiple times for the same OpenQuery
• #250 & #251 Generating data for tables with identity columns
• #183 Job stopped during processing
• Db Migrator always shows a warning message
• Unclear warning message when row size exceeded
• user search returns no users
• ignore values on similar records causes duplicates to be missed
• Similar records tooltip appears in wrong place
• Similar records occasionally missing expected results
• Restoring a db and ProgramData to a new server causes issues
• User with View Only permissions has no breadcrumb menus when viewing a report
• #368 Value cannot be null when using OnRndRows method on column that accepts null
• #401 script log updates are added to the current page even when its not the latest page
• #406 project owner not listed in project permissions
• Skipping all records seems to cause an issue
• #421 report columns don't line up with headers on tables with a large number of columns
• #422 need to scroll to see popup menu
• #423 Editing a script with a long path causes save and close buttons to be partially hiden
• #426 alert notification email message needs tidying up
• #436 Internittent timeout on results screen and error sending progress to source
• #439 job status intermittently fails to go to a green tick
• short columns with long names causes poor formatting in reports

Version - 1.8.1

• ❇️ New
• new data generation method GetRndRecord can be used to select multiple fields from a randomly selected lookup record
• values can be set using a 'Get' from another column when using SetText, SetInt etc
• RndRange transforms can now accept range parameters using a 'Get' from other columns
• values can be assigned using a general 'Set' that uses the data type of the parameter
• added RndDateTimeRange which returns a random date and time between the specified values
• AddressValidationAPI has an option to refresh the cache once the records are older than a configurable amount of days
• added request logging to AddressValidationAPI using LogRequestContaining config setting
• AddressValidationAPI cache is now encrypted
• MakeRndPersonCols when used for obfuscating guarantees a generated name will be different from original name
• Domains for generated email addresses have been updated to avoid clashing with real email addresses
• Limited SetMaxUniqueValues(int) to 100,000 to ensure stability when profiling
• Added ReplaceControlCharacters to remove control characters.
• Added ReplaceMultipleSpaces to replace multiple spaces.
• Added ReplaceBlanksWithNulls to replace blanks to Null.
• ConvertToText can now support non-numeric format specificer and culture code.
• Added data type validation to Csv readers when types are defined by the user, with functions to handle validation errors, OnReadErrorContinue() and OnReadErrorAbort()
• Allow scripts to prevent automatic failure of a row after validating an email or address and implement their own handling of the issue including retrieving the error message. See IValidateCondition functions
• 🔀 Changed
• IMPORTANT: Selecting random values based on weighting was found to be inaccurate and has been updated. Random selections with weightings will generate a different selection of results after this change.
• when using Get to obtain a reference to a field a default value can supplied if the field isn't found
• RndDateRange returns dates with time set to midnight see RndDateTimeRange
• Improved the profiling logic for detecting a columns data type
• Parameters on Transformations and Conditions now accept column references
• Skipping a row now deletes it from the output
• Column name references available for more functions, updated list in iData Reference Manual
• SetTimeZoneOffset replaced with SetUTCTimeZoneOffset and SetLocalTimeZoneOffset
• TestDataSet values set with NextXXX or ForEachXXX are now set before all other transforms take place, rather than after, so their values can be used by other transforms
• AutoFixAddress verification now maps addresses containing a Company Name, Property Name, or Locality, to the street/town columns
• AutoFixAddress verification only replaces responses with a return value of 100 and 400 by default, WithAllowedFlags() can be used to set a custom list of allowed flags. More information on flags can be found in the docs.
• Removed Binary Transforms as they were not useful in their current form
• Amended ReplaceNearest to support default value
• Added RntIntBetween which generates values including the maximum value
• Generated credit cards now give an invalid checksum by default. Added EnableValidCheckDigit() to the tag settings toa allow valid checksums to be generated.
• ☑️ Fixed
• Lookups from csv files will use column types defined using SetColumnToXXX
• Fixed infinite data generation when no row limit or foreach
• Report viewer now correctly reports invalid filenames instead of reporting missing files as invalid encryption keys
• Report viewer now correctly handles splitting of files correctly
• FormatText now works on fields that contain a NULL value
• Fixed error when using SetPatternLiteral
• Fixed 'UpdateColumnDefinition' which now correctly increases column sizes when ingesting csv data
• Miscapitalised column names used in SetColumnMapping no longer throw an error with counter group '???'
• AddressValidationAPI no longer requires external dependencies to run
• AddressValidationAPI will now create missing cache
• Stopped Abs transform displaying warning for positive numbers
• Fixed an issue when generating data that would cause the primary keys to be set to null
• Fixed NULL value handling when used with references to other fields
• #225 Fixed an issue when generating data that would cause the primary keys to be set to null
• CSV entries longer than 8000 characters will no longer throw an error

Version - 1.8.0

• ❇️ New
• Reports are compressed and encrypted to a single .idr report file by default
• Added SetVirtualFolder to filemanager to direct all folder contents to a single report file
• Encryption keys can be created and set through a encryption key pass phrase in the user interface
• Installer now deploys Report Viewer to view .idr files
• 🔀 Changed
• Profilers are now listed alphabetically
• ☑️ Fixed
• Fixed issue when using multiple lookups in one script
• Fixed profile reports with group results
• Fixed opening reports on a network drive accessed via a UNC path
• Fixed AutoFix address output miscapitalising words
• Fixed error when address config not set
• Fixed regex match error when the db value is null

Version - 1.7.2

• ❇️ New
• Added RestClient to allow for iData scripts to interact with Rest endpoints
• Added TextFileReader to allow for any text based file to be open and be used in a datastream
• Added TextFileWriter allowing transform engine to write to text files
• Added RestartLimit and RestartDisabled for Next lookup transforms
• Added SetTwoDigitYearMax to set the default century for two digit year date formats
• Added SetColumnToGuid to database readers to ensure unique identifiers are compared correctly
• Allow csv writers to append or overwrite existing files
• 🔀 Changed
• Updated ETL wizard to EnableIdentityInsert when target has identity column.
• MakeRndAddressCols() can use existing country or region or filter by country and region with UseCurrentCountry UseCurrentCountryAndRegion options
• Removed VaryBy()
• Added foreign key references to wizard script
• SetColumnToDateTime now also takes optional DateFormat to override the default
• CreateTableIfMissing now uses the input column name not a normalised version
• AddRecord now uses Upsert when there are no comparisons
• Added column mapping to the source stream, a compare is no longer necessary to map columns, compare mapping is being deprecated
• AutoFix now capitalises location names, and places houses in address line 1
• Dates are now formatted to the local time before being written to reports
• ☑️ Fixed
• Fixed data generation resetting identity columns
• RepeatRnd() changed to inclusive range to match Repeat() and missing ref bug fixed
• Allow updates to unique keys when source or destination have no rows
• Ensure error messages are reported for all row states
• Fixed error when uploading from excel sheets with numbers for sheet names
• Fixed TrackRowUpdates when used with existing data
• Fixed Importing csv file with only headers
• Fixed wild cards not applying tags to each column
• Fixed cardinality max value estimator not tracking past the set limit

Version - 1.7.1

• ❇️ New
• DbWriter and SQLWriter now supports creating and updating schema using CreateTableIfMissing, CreateColumnIfMissing and UpdateColumnDefinition.
• Added ExcelWriter allowing transform engine to write to excel sheets
• Added support for password protected excel files
• Added ExtractRegex() transform which allows text matching a given regex pattern to be extracted and optionally replaced with other text
• Added ToCol() which allows text extracted from ExtractRegex() to be placed in another column
• Added Company, Email, and Phone number to random address generation
• Added SetPatternLiteral() to toggle between pattern generation and literal characters in RndPatterns()
• Added Single, Double, and Triple word occurance count to the profiling report
• Added AddressLine1, AddressLine2, and AddressLine3 for generic columns
• Added SetColumnMappingFrom() to allow the use of lookup tables to map compared columns
• Added EnableAddressFixLookupService to enable the use of AFD instead of LibPostal to clean addresses, alongside EnableAddressFixMachineLearning, default option is set in AppSettings.ini
• Added abort to transforms when too many writes fail in a row can limit can be set with SetWriterMaxWriteFail
• MakeCol now accepts Wildcards to select multiple columns at once
• MakeCol now accepts multiple columns to be passed at once
• Added ExcelReader allowing transform engine to read from excel sheets
• ExcelReader stops reading when blank rows found
• New Licencing system. Contact Intelligent Delievery Solutions for support.
• TrackRowUpdates() now writes comments to SQL, CSV and Excel files
• iData Address Validation API is now packaged with the iData install
• Added date and time to console logs
• Added new conditions to match profiler results. E.G. HasPunctuation
• 🔀 Changed
• Lookup matches not support patterns
• Merged CardPrefix and CardPrefixRange for card definition in AppSettings
• Cards higher than 19 digits can now be validated
• AutoFixAddressFields now works on tables with joined number and street fields
• Reports no longer show [no value] when one of the rows is entirely missing. e.g. No Src/Dest or Add and Delete
• Column values can now be used as parameters in conditions such as Contains
• You can no longer edit unique key columns while comparing unless the destination table is empty
• Limited the UI log output to 1000 lines
• AutoFixAddressFields and VerifyAddressFields now work with group tags
• IsValid and NotValid has been added to functions VerifyEmailAddress, VerifyAddressFields, AutoFixAddressFields, CheckForeignKeys, and CheckUniqueKeys
• Generate more diverse set of company names
• By default tables stop at the first empty row, and the first empty header if headers are set
• ☑️ Fixed
• Prevent error when uniqueidentifier is table key and also using Rnd transforms
• Fixed system buffer version mismatch
• Fixed spelling on some Rnd... transforms from Rng...
• Fixed filter buttons sometimes not showing up on ETL reports
• Set RndPatterns() to accept punctuation.
• Report links with spaces now work
• Csv files with trialing spaces in headers now work
• Allow duplicate values in data for In and Replace operations
• Fixed some formatting issues in the documentation
• Data writes take column mappings into account and shows correct mapping in headers in report
• Fixed showing mapped columns in report when writing to empty table
• Fixed table name validation issue
• Speacial characters in excel sheet names now upload okay
• Address generation email column not defined and street address default seed

Version - 1.7.0

• ❇️ New
• More detailed stack trace for errors
• Added NextGuid(), RndGuid() and EachGuid() transforms
• Added OnNthRow() transform
• Added ConvertToBigInt() which converts the value to a 64 bit integer
• More detailed stack trace for errors
• Added a UniquePrefixes profiler
• Added RndWords() transform which generates a random Lorum Ipsum paragraph
• Added ability for tags to be grouped together
• Added enhanced Mask() which allows full or partial masking of text with a given mask character
• Added enhanced MaskExcept() to mask all except a given range
• Added Repeat/RepeatRnd modifier(s) to Each and Next Transformations to repeat the value the requested number of times
• Added MakeKeys and ForEachRowFrom/NextRowFrom to setup keys values from a parent table
• Added default value option to conversion transformations
• Added the ability to set the seed for randomly generated data to an alternative name instead of the column name with UseSeedName()
• Added AllUppercase, AllLowercase and MixedCase profilers
• Added Flush() to reports to write any buffered output
• Added ability to set RequiredRows for a TestSet
• Added RepeatRnd() function which allows Each and Next transforms to be randomly reset
• Each functions are now known as ForEach. e.g. EachText is now ForEachText
• Added RestartRnd() which restarts a ForEach sequence after a random amount of iterations between the given minValue and maxValue
• Added SetBinary(), RndBinary(), NextBinary(), EachBinary() and EachFrom versions of these
• Added RowLimit() to Each data items to allow for generation of data for smaller sub sets
• Added SetDifferenceFilename() option to comparison options
• All transforms can now be contained within a TestDataSet()
• All TestDataSet() calls must now have an accompanying EndTestDataSet()
• Added GetCardType() to check a card number for a matching card type, returns the found type as a string
• Added MatchKeysAlphbetically() method to comparison settings to allow for keys to be compared alphanumerically
• Added the ability to enable soundex for similar record checker via similar record tags
• Added the ability to ignore words for similar record checker via similar record tags
• Added the a sort by method the similar record configuration object
• Added BlankCol() to blank selected columns
• Added UniquePatternLengthIndependent profiler
• Added MakeRndCreditCardCols() to generate credit card data on tagged columns. Additionally, SetValidCardTypes() can be used to select card types to generate.
• 🔀 Changed
• Temp data is now written to local app data
• By default both MakeRndAddressCols() and MakeRndPersonCols() use predefined seeds. This can be changed with the new UseDefaultSeeds()/UseColSeeds() options
• ☑️ Fixed
• Prevent error when uniqueidentifier is table key and also using Rnd transforms
• Fixed system buffer version mismatch
• Fixed spelling on some Rnd... transforms from Rng...

Version - 1.6.1

• ❇️ New
• Added the ability to set a timeout when calling Execute() on a DBQuery
• Added NextIntAsc(), NextFloatAsc(), NextDecimalAsc(), NextYearAsc(), NextMonthAsc(), NextDayAsc(), NextHourAsc(), NextMinuteAsc(), NextSecondAsc() and NextMillisecondAsc() which increment a value by a given step
• Added the RngRegex transformation to randomly generate values that match the specified regular expression
• Added ability to generate synthetic data using OpenTableGenerator(). Additionally, use SetTotalRows() and UseOptimalRows to set the amount generated
• Added a new DataType profiler that reports back the type of each column profiled
• Added a new Median profiler that reports back an estimated median value when the number of unique values has been exceeded. Actual median is reported below this limit
• Added Host.NewDateTimeFormat() and Host.SetDefaultDateTimeFormat() which allow for parsing custom date formats
• Added a progress bar when the profiler is writing out html reports
• Added a new VaryBy transformation that can be used when obfuscating values
• Added OnRndRows() and OnRows() to allow for transformations to be applied on a subset of rows
• Added grouped profiling results to the profiling reports
• Added IsBlank() and NotBlank() conditional transformations
• Added Between() and NotBetween() conditional transformations
• Added option to paginate sql and csv files via a parameter on Host.WriteSqlScript() and Host.WriteCsvFile()
• Added the ability to edit and add new connections from the connection picker when running a wizard
• Added Or() conditional transformation
• Added SetBigInt() which sets the value to the given int as Int64
• Added ConvertToBigInt() which converts the value to a 64 bit integer
• 🔀 Changed
• MoreThan(), Equals(), AtLeast() and other comparisons will convert text values to dates before comparing them
• NextIntFrom(), NextDoubleFrom(),NextFloatFrom(), NextTextFrom() and NextDateFrom() have been renamed to NextInt(), NextDouble() etc when using a list of values opposed to a lookup
• Null values are now included in the unique values profiler results
• Total row count is now shown next to the Unique values on the profiling report
• Now when the number of unique values passes 1000(by default) an estimation of the number of unique values is provided for the profiling reports
• ☑️ Fixed
• Error message is displayed in reports when invalid column mappings are provided

Version - 1.6.0

• ❇️ New
• When updating from 1.5.* to 1.6 iData needs to be uninstalled prior to updating
• Added SetWriteFailedRecords() which allows failed records to be written to a separate destination
• Added new setting to profiler results SetMaxCharLimit() which prevents the profiling of exceedingly large text values
• Added NewLookup() to Host that allows the building of lookup tables from within scripts
• Added NextIntFrom(), NextDecimalFrom(), NextDateFrom(), NextFloatFrom() and NextTextFrom() which loop through given values replacing each value in turn
• Added RndTextFrom(), RndIntFrom(), RndDecimalFrom(), RndFloatFrom(), RndDateTimeFrom() and RndTextFrom() which select a random value from provided values
• Added RndTextPattern() which converts a pattern, e.g. 'A9A 9A9', into a random string with the same pattern e.g A7A 3R8
• Added SetGenerationSeedCols() to set which column values are used as the seed for generated data
• Added RndIntRange(),RndFloatRange(),RndDoubleRange(),RndDecimalRange() and RndDateRange() used to generate random data
• Added the ability to sample datastreams with the SetSampleSize() method
• Added the ability to generate random addresses with the MakeRndAddressCols() transformation
• Added Each transforms which generate data in combination with other Each transforms. Added for Text, Int, Decimal, Float and Dates. Each transforms can be used with LookupTables using Each...From
• 🔀 Changed
• Renamed AutoFixPhoneNumber() transformation to FormatPhoneNumber()

Version - 1.5.1

• ☑️ Fixed
• Fixed comparison summary not reporting differences when source rows missing from destination
• Fixed row count out by 1 when reading from csv files

Version - 1.5.0

• ❇️ New
• Added Profiling and Profiling wizard to discover data patterns and potential data quality issues
• Added Check for Similar Records and Similar records Wizard to search for records that could be duplicates
• Created data transform engine for ETL as well as comparison See NewDataTransform in docs
• Added transformations: ConvertToDecimal(), ConvertToText(), ConvertToDateTime(), ReformatDateTimeText()
• Added AutoCapitalise() transformation allowing for words to be automatically capitalised
• Added transformations: Trim(), TrimEnd(), TrimStart()
• Added the ability to tag columns as the following: TagAsPersonTitleName(), TagAsPersonFirstName(),TagAsPersonMiddleName(),TagAsPersonLastName()
• Intelligently clean person/name related data: AutoFixPersonFields(), DistributePersonFullName() that have been tagged as person fields
• Added EnableDomainCheck() on TagAsEmail to check if the email domain exists
• New transformations: CopyCol() and CopyColValue()
• New transformation: ReplaceWildcard()
• Added for lookup table: EnableWildcards(), EnableRegex()
• Added conditions: IsValid(), NotValid() to report if tagged columns such as emails are valid
• Added TagAsPhoneNumber() to allow tagging of phone number columns see AutoFixPhoneNumbers
• Added AutoFixPhoneNumbers() to convert phone numbers to standarised formatting
• Reports are now timestamped so results do not get lost between runs
• Added conditions: MatchesDateTimeText(format) and NotMatchDateTimeText(format)
• Added CheckForeignKey() to stream to check if all foreign keys are present
• Added CheckUniqueKey() to streams to check all key values are unique
• Added TagProfilerSetting() to allow for custom profiler settings
• An SQL output can be added for both CheckForeignKeys() and CheckUniqueKeys() via SetSqlResultsFilename()
• Common values outputted by the Profiler can be save to CSV via the WriteCommonValuesToCSV() command
• Added PadLeft() and PadRight() to add spaces to a value
• Added DifferenceFrom condition to allow for approximate string matching
• Added ConvertToJulianDateTime() and ConvertToUnixDateTime()
• The ConvertToDateTime transformation now works with unix timestamps and julian dates.
• Added SetLong(), SetBool(), SetDecimal(), SetGuid(), SetShort(), SetByte(), and SetDateTimeOffset()
• Added EnableMultipleSelections()/DisableMultipleSelections() to SetControlToListBox() for use with Wizards
• Added an Else statement for use with conditions
• The name parameter of MakeCol() can now be omitted, MakeCol() will use the column name provided in preceding IfCol() or IfSrcCol() methods
• Added ability to enable and disable warnings with via EnableLogWarnings(), DisableLogWarnings(), EnableAbortOnWarnings(), DisableAbortOnWarnings(), RestoreLogWarning()
• Added CoalesceCols() to get the first non null value from a list of columns
• Added Substring() transform to get part of a text value
• Added JoinCols() transform to combine several colums into one value
• Added InsertText() and InsertColText() transform to add text at the begining or middle of a value
• Added TagAsDomain() to allow tagging of a web domain column
• Added NewDBResultsWriter() to the Host object
• Added TrackRowUpdates() and StopTrackRowUpdates()
• Added SetDNSServer(), ExportDNSCache() and ImportDNSCache()
• Added TagAsCreditCard() and validation to check if credit card numbers are valid
• Added EnableSoundex() to lookup tables
• Added Obfuscate() transform to replace original characters with random characters
• Added Mask() transform to replace original characters with fixed characters
• Profiling Results can now be grouped by columns
• Lookup tables can now be filtered using FilterOn() and Using()
• 🔀 Changed
• Database connections and application settings are now read from %LocalAppData%/iData/Settings
• Data comparison engine still works but is deprecated, use NewDataTransform instead
• NewInputStream still works but is deprecated, use NewStream instead

Version - 1.4.0

• ❇️ New
• Upgrade to V8 jscript engine, for better performance, error handling and latest jscript syntax
• Doubled performance running scripts in gui with background thread
• Allow debugging scripts from VSCode. See Docs
• Allow scripts to be stopped from the UI by clicking the run button a second time
• NewDbCompare can use SetSrcData or SetDestData instead of SetSrcTable or SetDestTable. SetSrcData can be passed value from OpenCsvFile. This allows control over csv file separators and column settings
• NewDbCompare can use SetSrcQuery or SetDestQuery to allow comparison of joined tables or pre-processing of data using sql before iData transformations
• Use a lookup table from a query or csv file to replace values using ReplaceValueWithLookup
• The database definition of a Null value can now be accessed through Host.DBNull which has a different value from null
• IgnoreCase can be applied to make transformations such as Replace case insensitive and MatchCase can turn it case sensitive again
• Added SetInt, SetFloat, and SetDateTime to set a column to a specific value of the given type
• Created maths operations for Add, Subtract and Remainder
• SetMatchDistance added to allow nearest close match to data in lookup table. See Docs
• Added SetDateTimeParts to allow part of a dateTime to be changed.
• Added SetANSICodePage to allow encoding to be specified when reading CSV files.
• 🔀 Changed
• Interactive scripts must have 'Wizard' as part of their filename, for automatic optimisation
• ReplaceRegex ignore case parameter deprecated in favor of IgnoreCase command
• ReplaceValueWithLookup deprecated and renamed ReplaceLookup
• ☑️ Fixed
• Fixed invalid warning when comparing source query to destination csv file
• Fixed crash caused by duplicate database connection names
• Report any errors generated by wizards
• Prevent ConfirmRequiredVersion flagging identical build numbers as different
• Prevent missing dates in CSV files generating invalid SQL dateTimes
• Fixed cleanup of database connections after DbCompare

Version - 1.3.6

• ❇️ New
• Add testConnection wizard. See getting started
• Accept jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs
• 🔀 Changed
• Database connections details reloaded on demand. No need to re-start gui to refresh
• Improved error message for script errors in comparisons
• ☑️ Fixed
• Update progress report when skipping rows during comparison

Version - 1.3.5

• ❇️ New
• Recored and report statitics for each column
• Accept native jscript regex expressions in MatchRegex and NotMatchRegex. See Api docs
• Remove duplicates rows before comparison. See RemoveSrcDuplicates in Api Docs
• 🔀 Changed
• Limit comparison reports of mismatched rows to avoid hiding other issues
• ☑️ Fixed
• Report attempts to map columns incorrectly in comparisons
• Prevent installer overwritting connection strings on re-install

Version - 1.3.4

• ❇️ New
• Allow comparison to specify IncludeColumns as well as SkipColumns. See Api docs
• Allow columns to be SetNull. See api docs

Version - 1.3.3

• ❇️ New
• All In to use lookup tables from CSV or queries. See API docs

Version - 1.3.2

• ❇️ New
• Additional conditions In, StartsWith etc. See API docs

Version - 1.3.1

• 🔀 Changed
• Improved database connection error messages

Version - 1.3.0

• ❇️ New
• Support for splitting and joining columns. e.g. AddColText. See Api docs

Version - 1.2.0

• ❇️ New
• Database comparison wizard
• Import CSV data for comparison

Version - 1.1.0

• ❇️ New
• Database comparison
• Read/write to Excel files